# Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias

**Authors:** Andres Algaba, Carmen Mazijn, Vincent Holst, Floriano Tori, Sylvia Wenmackers, Vincent Ginis  
**arXiv:** [2405.15739](https://arxiv.org/abs/2405.15739)  
**Date:** 2024-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2405.15739v2/

## Key points

- Authors studied the impact of Large Language Models (LLMs) like GPT-4 on citation practices in scientific knowledge dissemination.
- Citation practices play a crucial role in shaping the structure of scientific knowledge.
- These practices are influenced by contemporary norms and biases.
- LLMs like GPT-4 introduce a new dynamic to citation practices, recommending references based on parametric knowledge rather than search or retrieval-augmented generation.
- Experiment used a dataset of 166 papers from prestigious conferences published after GPT-4's cut-off date, with a total of 3,066 references.
- GPT-4 was tasked with suggesting scholarly references for anonymized in-text citations within these papers.
- Similarity between human and LLM citation patterns observed.
- More pronounced high citation bias in GPT-4 compared to human patterns, even after controlling for variables like publication year and venue.
- Model internalized citation patterns to a considerable extent.
- References recommended by GPT-4 were embedded within relevant citation contexts, indicating deeper conceptual internalization of citation networks.
- LLMs have potential to aid in citation generation but can also amplify existing biases and introduce new ones that may skew scientific knowledge dissemination.
- Importance of identifying and addressing biases within LLMs emphasized.
- Need for developing balanced methods to interact effectively with these models highlighted.

## Summary

In their study titled "Large Language Models Reflect Human Citation Patterns with a Heightened Citation Bias," authors Andres Algaba, Carmen Mazijn, Vincent Holst, Floriano Tori, Sylvia Wenmackers, and Vincent Ginis delve into the impact of Large Language Models (LLMs) like GPT-4 on citation practices in scientific knowledge dissemination. The researchers highlight the crucial role citation practices play in shaping the structure of scientific knowledge while acknowledging that these practices are often influenced by contemporary norms and biases. The introduction of LLMs such as GPT-4 introduces a new dynamic to citation practices. This is particularly evident in the context of recommending references based solely on parametric knowledge rather than search or retrieval-augmented generation. Despite this shift, the characteristics and potential biases of references recommended by LLMs remain largely unexplored. To address this gap, the authors conducted an experiment using a dataset comprising 166 papers from prestigious conferences such as AAAI, NeurIPS, ICML, and ICLR. These papers were published after GPT-4's knowledge cut-off date and collectively included 3,066 references. In the experiment, GPT-4 was tasked with suggesting scholarly references for anonymized in-text citations within these papers. The findings of the study revealed a striking similarity between human and LLM citation patterns. However, it also uncovered a more pronounced high citation bias in GPT-4 compared to human patterns. This bias persisted even after controlling for variables such as publication year, title length, number of authors, and venue. Additionally, there was a significant consistency between the characteristics of existing references in GPT-4's database and those that were generated but did not exist previously. This suggests that the model has internalized citation patterns to a considerable extent. By analyzing citation graphs, the researchers demonstrated that the references recommended by GPT-4 were embedded within relevant citation contexts. This finding indicates a deeper conceptual internalization of citation networks by the model. While LLMs have the potential to aid in citation generation, they also have the capacity to amplify existing biases and introduce new ones that could skew scientific knowledge dissemination. Overall, this study underscores the importance of identifying and addressing biases within LLMs and emphasizes the need for developing balanced methods to interact with these models effectively. The research sheds light on how advancements in language models can impact scholarly communication and calls for continued scrutiny and refinement in utilizing LLMs within academic contexts.