Benchmarking Large Language Models in Retrieval-Augmented Generation
Auteurs : Jiawei Chen, Hongyu Lin, Xianpei Han, Le Sun
Résumé scientifique
Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness. To this end, we establish Retrieval-Augmented Generation Benchmark (RGB), a new corpus for RAG evaluation in both English and Chinese. RGB divides the instances within the benchmark into 4 separate testbeds based on the aforementioned fundamental abilities required to resolve the case. Then we evaluate 6 representative LLMs on RGB to diagnose the challenges of current LLMs when applying RAG. Evaluation reveals that while LLMs exhibit a certain degree of noise robustness, they still struggle significantly in terms of negative rejection, information integration, and dealing with false information. The aforementioned assessment outcomes indicate that there is still a considerable journey ahead to effectively apply RAG to LLMs.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
Evaluation of Large Language Models Enhanced with Retrieval-Augmented Generation: A literature reviewDobrovodsky Patrik, Balázsné Dr. Kail Eszter, Tafferner-Gulyás Viktória et al. · 2026 · International Symposium on Applied Machine Intelligence and Informatics · cité 1 fois · Pas encore résumé
Résumé scientifique
Large Language Models (LLMs) demonstrate remarkable capabilities in text generation but face persistent challenges with hallucination and factual accuracy. Retrieval-Augmented Generation (RAG) addresses these limitations by combining generative models with external knowledge retrieval. However, standardized evaluation frameworks remain lacking. This literature review analyzes eleven evaluation frameworks for LLMs and RAG systems published between 2023 and 2025, examining their methodologies, metrics, and approaches to hallucination detection. The frameworks span diverse domains and employ varying automation levels, from manual annotation to LLM-as-a-Judge paradigms.Our analysis reveals a par…
-
CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language ModelsYuanjie Lyu, Zhi-Yu Li, Si-Min Niu et al. · 2024 · ACM Trans. Inf. Syst. · cité 153 fois · Pas encore résumé
Résumé scientifique
Retrieval-augmented generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate “hallucinated” content. However, evaluating RAG systems is a challenge. Most benchmarks focus primarily on question-answering applications, neglecting other potential scenarios where RAG could be beneficial. Accordingly, in the experiments, these benchmarks often assess only the LLM components of the RAG pipeline or the retriever in knowledge-intensive scenarios, overlooking the impact of external knowledge base construc…
-
Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive SurveyAo-Ran Gan, Hao Yu, Kai Zhang et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 58 fois · Pas encore résumé
Résumé scientifique
Recent advancements in Retrieval-Augmented Generation (RAG) have revolutionized natural language processing by integrating Large Language Models (LLMs) with external information retrieval, enabling accurate, up-to-date, and verifiable text generation across diverse applications. However, evaluating RAG systems presents unique challenges due to their hybrid architecture that combines retrieval and generation components, as well as their dependence on dynamic knowledge sources in the LLM era. In response, this paper provides a comprehensive survey of RAG evaluation methods and frameworks, systematically reviewing traditional and emerging evaluation approaches, for system performance, factual a…
-
Retrieval-Augmented Generation for Large Language Models: A SurveyYunfan Gao, Yun Xiong, Xinyu Gao et al. · 2023 · Computation and Language · preprint, non relu par les pairs · cité 4 438 fois · Résumé IA disponible
Cette synthèse présente les usages, avantages et limites de la génération augmentée par récupération, tandis que l’article évalue systématiquement ses effets sur plusieurs LLM.
Résumé scientifique
Large language models (LLMs) demonstrate powerful capabilities, but they still face challenges in practical applications, such as hallucinations, slow knowledge updates, and lack of transparency in answers. Retrieval-Augmented Generation (RAG) refers to the retrieval of relevant information from external knowledge bases before answering questions with LLMs. RAG has been demonstrated to significantly enhance answer accuracy, reduce model hallucination, particularly for knowledge-intensive tasks. By citing sources, users can verify the accuracy of answers and increase trust in model outputs. It also facilitates knowledge updates and the introduction of domain-specific knowledge. RAG effectivel…
-
A Comprehensive Evaluation of Large Language Models for Retrieval-Augmented Generation under Noisy ConditionsJosué Caldas, E. D. de Souza · 2025 · Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settin · cité 1 fois · Pas encore résumé
Résumé scientifique
Retrieval-Augmented Generation (RAG) has emerged as an effective strategy to ground Large Language Models (LLMs) with reliable, real-time information. This paper investigates the trade-off between cost and performance by evaluating 13 LLMs within a RAG pipeline for the Question Answering (Q&A) task under noisy retrieval conditions. We assess four ex-tractive and nine generative models—spanning both open-and closed-source ones of varying sizes—on a journalistic benchmark specifically designed for RAG. By systematically varying the level of noise injected into the retrieved context, we analyze not only which models perform best, but also their robustness to noisy input. Results show that large…
-
PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented GenerationZhehao Tan, Yi-Han Jiao, Dan Yang et al. · 2025 · AAAI Conference on Artificial Intelligence · cité 3 fois · Pas encore résumé
Résumé scientifique
Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, where the LLM's ability to generate responses based on the combination of a given query and retrieved documents is crucial. However, most benchmarks focus on overall RAG system performance, rarely assessing LLM-specific capabilities. Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization. To this end, we introduce Placeholder-RAG-Benchmark, a multi-level fine-grained benchmark, emphasizing the following progressive dimensions: (1) multi-level filtering abilities, (2) combination abilities, a…
-
When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMsYongjie Wang, Yue Yu, Kai-Song Song et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 2 fois · Pas encore résumé
Résumé scientifique
Large Language Models (LLMs) have enabled a wide range of applications through their powerful capabilities in language understanding and generation. However, as LLMs are trained on static corpora, they face difficulties in addressing rapidly evolving information or domain-specific queries. Retrieval-Augmented Generation (RAG) was developed to overcome this limitation by integrating LLMs with external retrieval mechanisms, allowing them to access up-to-date and contextually relevant knowledge. However, as LLMs themselves continue to advance in scale and capability, the relative advantages of traditional RAG frameworks have become less pronounced and necessary. Here, we present a comprehensive…
-
Retrieval-Augmented Generation and Hallucination in Large Language Models: A Scholarly OverviewSagar Gupta · 2025 · Scholars Journal of Engineering and Technology · cité 8 fois · Pas encore résumé
Résumé scientifique
Large Language Models (LLMs) have revolutionized natural language processing tasks, yet they often suffer from "hallucination” the confident generation of factually incorrect information. Retrieval-Augmented Generation (RAG) has emerged as a promising technique to mitigate hallucinations by grounding model responses in external documents. This article explores the underlying causes of hallucinations in LLMs, the mechanisms and architectures of RAG systems, their effectiveness in reducing hallucinations, and ongoing challenges. We conclude with a discussion of future directions for integrating retrieval mechanisms more seamlessly into generative architecture.
Les travaux antérieurs sur lesquels il s’appuie
-
Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval AugmentationRuiyang Ren, Yu-Hao Wang, Yingqi Qu et al. · 2023 · International Conference on Computational Linguistics · cité 215 fois · Résumé IA disponible
Résumé scientifique
Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly under retrieval augmentation settings. In this study, we present the first analysis on the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain question answering (QA), with a bunch of important findings. Specifically, we focus on three research questions and analyze them by examining QA, priori judgement and posteriori judgement capabilities of LLMs. We show evidence that LLMs possess unwavering confidence in their knowledge a…
-
REPLUG: Retrieval-Augmented Black-Box Language ModelsWeijia Shi, Sewon Min, Michihiro Yasunaga et al. · 2023 · North American Chapter of the Association for Computational Linguistics · cité 1 172 fois · Pas encore résumé
Résumé scientifique
We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improv…
-
A Survey on Evaluation of Large Language ModelsYu-Peng Chang, Xu Wang, Jin-Dong Wang et al. · 2023 · ACM Transactions on Intelligent Systems and Technology · cité 4 011 fois · Résumé IA disponible
Résumé scientifique
Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the …
-
Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated FeedbackBaolin Peng, Michel Galley, Pengcheng He et al. · 2023 · arXiv.org · preprint, non relu par les pairs · cité 563 fois · Résumé IA disponible
Résumé scientifique
Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. However, applying LLMs to real-world, mission-critical applications remains challenging mainly due to their tendency to generate hallucinations and their inability to use external knowledge. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules. Our system makes the LLM generate responses grounded in external knowledge, e.g., stored in task-specific databases. It also iteratively revises LLM prompts to improve model responses using feedback generated by utility fu…
-
Rethinking with Retrieval: Faithful Large Language Model InferenceHangfeng He, Hong-Ming Zhang, Dan Roth · 2022 · arXiv.org · preprint, non relu par les pairs · cité 211 fois · Pas encore résumé
Résumé scientifique
Despite the success of large language models (LLMs) in various natural language processing (NLP) tasks, the stored knowledge in these models may inevitably be incomplete, out-of-date, or incorrect. This motivates the need to utilize external knowledge to assist LLMs. Unfortunately, current methods for incorporating external knowledge often require additional training or fine-tuning, which can be costly and may not be feasible for LLMs. To address this issue, we propose a novel post-processing approach, rethinking with retrieval (RR), which retrieves relevant external knowledge based on the decomposed reasoning steps obtained from the chain-of-thought (CoT) prompting. This lightweight approac…
Les articles qui le citent
-
Graph Retrieval-Augmented Generation for Large Language Models: A SurveyTyler Procko, Omar Ochoa · 2024 · Conference on AI, Science, Engineering, and Technology (AIxSET) · cité 108 fois · Pas encore résumé
Résumé scientifique
Large Language Models (LLMs) demonstrate general knowledge, but they suffer when specifically needed knowledge is not present in their training set. Two approaches to ameliorating this, without retraining, are 1) prompt engineering and 2) Retrieval-Augmented Generation (RAG). RAG is a form of prompt engineering, insofar as relevant lexical snippets retrieved from RAG corpora are vectorized and aggregated with prompts. However, RAG documents are often noisy, i.e., while relevant to a given prompt, they can contain much other information that obfuscates the desired snippet. If the purpose of pretraining a LLM on massive and general corpora is to engender a generally applicable model, RAG is no…
-
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A ReviewWan Zhang, Jing Zhang · 2025 · Mathematics · cité 178 fois · Pas encore résumé
Résumé scientifique
Retrieval-augmented generation (RAG) leverages the strengths of information retrieval and generative models to enhance the handling of real-time and domain-specific knowledge. Despite its advantages, limitations within RAG components may cause hallucinations, or more precisely termed confabulations in generated outputs, driving extensive research to address these limitations and mitigate hallucinations. This review focuses on hallucination in retrieval-augmented large language models (LLMs). We first examine the causes of hallucinations from different sub-tasks in the retrieval and generation phases. Then, we provide a comprehensive overview of corresponding hallucination mitigation techniqu…
-
Evaluating Retrieval Augmented Generation: A Comprehensive Review of Evaluation Dimensions, Question Types, and ApplicationSimon Knollmeyer, Oğuz Caymazer, Leonid Koval et al. · 2026 · SN Computer Science · cité 4 fois · Pas encore résumé
Résumé scientifique
This study addresses limitations of traditional benchmarking methods for Retrieval-Augmented Generation (RAG) systems by proposing an evaluation framework for RAG-enhanced Large Language Models (LLMs). The framework structures evaluation dimensions and metrics, identifies suitable datasets and question types, and provides guidance for applying the framework in practice. A systematic literature review (SLR) was conducted, synthesizing evidence from 12 studies focused on the evaluation of RAG systems. The review employs a concept matrix to classify evaluative approaches and maps metrics to dimensions, evaluator types, and pipeline stages. In addition, we systematize dataset and question-type r…
-
Improving negative rejection ability in language models: A review of fine-tuned LLMs, RAG, and RAFTBo-Wen Li, Zhuoqing Zhong, Hengyu Zhang et al. · 2025 · Journal of King Saud University: Computer and Information Sciences · cité 4 fois · Pas encore résumé
Résumé scientifique
Large Language Models (LLMs) excel in text understanding and generation but struggle to reject irrelevant, ambiguous, or misleading queries, termed negative rejection, impacting reliability in high-stakes contexts. This paper reviews negative rejection, analyzing three approaches: fine-tuned LLMs, Retrieval-Augmented Generation (RAG), and Retrieval-Augmented Fine-Tuning (RAFT). Through a literature review, we assess their rejection accuracy, hallucination rates, computational costs, and domain adaptability. We introduce a four-dimensional framework—architecture, metrics, domain application, and dataset diversity—to organize existing studies and benchmarks for comparative analysis (Cao, 2024)…
-
RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language ModelsAnge Maiztegi, J. Ayerdi, Miren Illarramendi et al. · 2026 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
Résumé scientifique
Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatib…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 4 novembre 2026.