Benchmarking Large Language Models in Retrieval-Augmented Generation

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Review Not relevant

   Evaluation of Large Language Models Enhanced with Retrieval-Augmented Generation: A literature review

   Dobrovodsky Patrik, Balázsné Dr. Kail Eszter, Tafferner-Gulyás Viktória et al. · 2026 · International Symposium on Applied Machine Intelligence and Informatics · cited 1 time · Not summarized yet

   The literature review compares evaluation frameworks for LLMs and RAG, while this paper benchmarks LLM capabilities across four RAG testbeds.

   Abstract

   Large Language Models (LLMs) demonstrate remarkable capabilities in text generation but face persistent challenges with hallucination and factual accuracy. Retrieval-Augmented Generation (RAG) addresses these limitations by combining generative models with external knowledge retrieval. However, standardized evaluation frameworks remain lacking. This literature review analyzes eleven evaluation frameworks for LLMs and RAG systems published between 2023 and 2025, examining their methodologies, metrics, and approaches to hallucination detection. The frameworks span diverse domains and employ varying automation levels, from manual annotation to LLM-as-a-Judge paradigms.Our analysis reveals a par…

2. Very close Builds on this paper Cites this paper Not relevant

   CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models

   Yuanjie Lyu, Zhi-Yu Li, Si-Min Niu et al. · 2024 · ACM Trans. Inf. Syst. · cited 153 times · Not summarized yet

   CRUD-RAG cites this paper and introduces a broader Chinese RAG benchmark spanning scenarios beyond question answering.

   Abstract

   Retrieval-augmented generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This method addresses common LLM limitations, including outdated information and the tendency to produce inaccurate “hallucinated” content. However, evaluating RAG systems is a challenge. Most benchmarks focus primarily on question-answering applications, neglecting other potential scenarios where RAG could be beneficial. Accordingly, in the experiments, these benchmarks often assess only the LLM components of the RAG pipeline or the retriever in knowledge-intensive scenarios, overlooking the impact of external knowledge base construc…

3. Very close Review Not relevant

   Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

   Ao-Ran Gan, Hao Yu, Kai Zhang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 58 times · Not summarized yet

   The survey organizes methods and frameworks for evaluating RAG systems, whereas this paper evaluates six LLMs on four capabilities.

   Abstract

   Recent advancements in Retrieval-Augmented Generation (RAG) have revolutionized natural language processing by integrating Large Language Models (LLMs) with external information retrieval, enabling accurate, up-to-date, and verifiable text generation across diverse applications. However, evaluating RAG systems presents unique challenges due to their hybrid architecture that combines retrieval and generation components, as well as their dependence on dynamic knowledge sources in the LLM era. In response, this paper provides a comprehensive survey of RAG evaluation methods and frameworks, systematically reviewing traditional and emerging evaluation approaches, for system performance, factual a…

4. Very close Review Not relevant

   Retrieval-Augmented Generation for Large Language Models: A Survey

   Yunfan Gao, Yun Xiong, Xinyu Gao et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 4,438 times · AI summary available

   The survey reviews RAG methods and applications, while this paper measures how LLMs perform when using retrieved information.

   Abstract

   Large language models (LLMs) demonstrate powerful capabilities, but they still face challenges in practical applications, such as hallucinations, slow knowledge updates, and lack of transparency in answers. Retrieval-Augmented Generation (RAG) refers to the retrieval of relevant information from external knowledge bases before answering questions with LLMs. RAG has been demonstrated to significantly enhance answer accuracy, reduce model hallucination, particularly for knowledge-intensive tasks. By citing sources, users can verify the accuracy of answers and increase trust in model outputs. It also facilitates knowledge updates and the introduction of domain-specific knowledge. RAG effectivel…

5. Very close Another approach to the problem Not relevant

   A Comprehensive Evaluation of Large Language Models for Retrieval-Augmented Generation under Noisy Conditions

   Josué Caldas, E. D. de Souza · 2025 · Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settin · cited 1 time · Not summarized yet

   This study tests 13 LLMs on question answering under varied retrieval noise, whereas this paper evaluates four RAG abilities across English and Chinese.

   Abstract

   Retrieval-Augmented Generation (RAG) has emerged as an effective strategy to ground Large Language Models (LLMs) with reliable, real-time information. This paper investigates the trade-off between cost and performance by evaluating 13 LLMs within a RAG pipeline for the Question Answering (Q&A) task under noisy retrieval conditions. We assess four ex-tractive and nine generative models—spanning both open-and closed-source ones of varying sizes—on a journalistic benchmark specifically designed for RAG. By systematically varying the level of noise injected into the retrieved context, we analyze not only which models perform best, but also their robustness to noisy input. Results show that large…

6. Very close Another approach to the problem Not relevant

   PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation

   Zhehao Tan, Yi-Han Jiao, Dan Yang et al. · 2025 · AAAI Conference on Artificial Intelligence · cited 3 times · Not summarized yet

   PRGB introduces a fine-grained benchmark for document filtering and combination, complementing this paper’s tests of four fundamental RAG abilities.

   Abstract

   Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by integrating external knowledge, where the LLM's ability to generate responses based on the combination of a given query and retrieved documents is crucial. However, most benchmarks focus on overall RAG system performance, rarely assessing LLM-specific capabilities. Current benchmarks emphasize broad aspects such as noise robustness, but lack a systematic and granular evaluation framework on document utilization. To this end, we introduce Placeholder-RAG-Benchmark, a multi-level fine-grained benchmark, emphasizing the following progressive dimensions: (1) multi-level filtering abilities, (2) combination abilities, a…

7. Very close Builds on this paper Cites this paper Not relevant

   When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs

   Yongjie Wang, Yue Yu, Kai-Song Song et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 2 times · Not summarized yet

   When Retrieval Succeeds and Fails cites this paper while reconsidering when traditional RAG benefits LLMs.

   Abstract

   Large Language Models (LLMs) have enabled a wide range of applications through their powerful capabilities in language understanding and generation. However, as LLMs are trained on static corpora, they face difficulties in addressing rapidly evolving information or domain-specific queries. Retrieval-Augmented Generation (RAG) was developed to overcome this limitation by integrating LLMs with external retrieval mechanisms, allowing them to access up-to-date and contextually relevant knowledge. However, as LLMs themselves continue to advance in scale and capability, the relative advantages of traditional RAG frameworks have become less pronounced and necessary. Here, we present a comprehensive…

8. Very close Broader context Not relevant

   Retrieval-Augmented Generation and Hallucination in Large Language Models: A Scholarly Overview

   Sagar Gupta · 2025 · Scholars Journal of Engineering and Technology · cited 8 times · Not summarized yet

   The overview discusses how RAG can mitigate hallucinations and the challenges that remain, a broader concern motivating this paper’s benchmark.

   Abstract

   Large Language Models (LLMs) have revolutionized natural language processing tasks, yet they often suffer from "hallucination” the confident generation of factually incorrect information. Retrieval-Augmented Generation (RAG) has emerged as a promising technique to mitigate hallucinations by grounding model responses in external documents. This article explores the underlying causes of hallucinations in LLMs, the mechanisms and architectures of RAG systems, their effectiveness in reducing hallucinations, and ongoing challenges. We conclude with a discussion of future directions for integrating retrieval mechanisms more seamlessly into generative architecture.

Prior work it builds on

1. Close Groundwork for this paper A method it uses Not relevant

   Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

   Ruiyang Ren, Yu-Hao Wang, Yingqi Qu et al. · 2023 · International Conference on Computational Linguistics · cited 215 times · AI summary available

   This study analyzes LLM factual knowledge boundaries under retrieval augmentation and is cited by this paper.

   Abstract

   Large language models (LLMs) have shown impressive prowess in solving a wide range of tasks with world knowledge. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly under retrieval augmentation settings. In this study, we present the first analysis on the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain question answering (QA), with a bunch of important findings. Specifically, we focus on three research questions and analyze them by examining QA, priori judgement and posteriori judgement capabilities of LLMs. We show evidence that LLMs possess unwavering confidence in their knowledge a…

2. Close Groundwork for this paper A method it uses Not relevant

   REPLUG: Retrieval-Augmented Black-Box Language Models

   Weijia Shi, Sewon Min, Michihiro Yasunaga et al. · 2023 · North American Chapter of the Association for Computational Linguistics · cited 1,172 times · Not summarized yet

   REPLUG presents a retrieval-augmented framework that prepends retrieved documents to a frozen black-box language model, an approach relevant to this paper’s evaluation.

   Abstract

   We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improv…

3. Close Groundwork for this paper Not relevant

   A Survey on Evaluation of Large Language Models

   Yu-Peng Chang, Xu Wang, Jin-Dong Wang et al. · 2023 · ACM Transactions on Intelligent Systems and Technology · cited 4,011 times · AI summary available

   The survey reviews evaluation methods for LLMs across tasks and settings and is cited by this paper.

   Abstract

   Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the …

4. Close Groundwork for this paper Not relevant

   Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback

   Baolin Peng, Michel Galley, Pengcheng He et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 563 times · AI summary available

   LLM-Augmenter grounds black-box LLM responses in external knowledge and revises prompts using feedback, an approach cited by this paper.

   Abstract

   Large language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. However, applying LLMs to real-world, mission-critical applications remains challenging mainly due to their tendency to generate hallucinations and their inability to use external knowledge. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules. Our system makes the LLM generate responses grounded in external knowledge, e.g., stored in task-specific databases. It also iteratively revises LLM prompts to improve model responses using feedback generated by utility fu…

5. Close Groundwork for this paper Not relevant

   Rethinking with Retrieval: Faithful Large Language Model Inference

   Hangfeng He, Hong-Ming Zhang, Dan Roth · 2022 · arXiv.org · preprint, not peer reviewed · cited 211 times · Not summarized yet

   Rethinking with Retrieval adds external knowledge at reasoning steps without retraining and is cited by this paper.

   Abstract

   Despite the success of large language models (LLMs) in various natural language processing (NLP) tasks, the stored knowledge in these models may inevitably be incomplete, out-of-date, or incorrect. This motivates the need to utilize external knowledge to assist LLMs. Unfortunately, current methods for incorporating external knowledge often require additional training or fine-tuning, which can be costly and may not be feasible for LLMs. To address this issue, we propose a novel post-processing approach, rethinking with retrieval (RR), which retrieves relevant external knowledge based on the decomposed reasoning steps obtained from the chain-of-thought (CoT) prompting. This lightweight approac…

Papers that cite it

1. Very close Groundwork for this paper Not relevant

   Graph Retrieval-Augmented Generation for Large Language Models: A Survey

   Tyler Procko, Omar Ochoa · 2024 · Conference on AI, Science, Engineering, and Technology (AIxSET) · cited 108 times · Not summarized yet

   The survey discusses noise in retrieved documents and is cited by this paper, which tests LLM noise robustness in RAG.

   Abstract

   Large Language Models (LLMs) demonstrate general knowledge, but they suffer when specifically needed knowledge is not present in their training set. Two approaches to ameliorating this, without retraining, are 1) prompt engineering and 2) Retrieval-Augmented Generation (RAG). RAG is a form of prompt engineering, insofar as relevant lexical snippets retrieved from RAG corpora are vectorized and aggregated with prompts. However, RAG documents are often noisy, i.e., while relevant to a given prompt, they can contain much other information that obfuscates the desired snippet. If the purpose of pretraining a LLM on massive and general corpora is to engender a generally applicable model, RAG is no…

2. Very close Review Not relevant

   Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review

   Wan Zhang, Jing Zhang · 2025 · Mathematics · cited 178 times · Not summarized yet

   The review categorizes hallucination causes and mitigation techniques in retrieval-augmented LLMs, while this paper evaluates RAG-related capabilities.

   Abstract

   Retrieval-augmented generation (RAG) leverages the strengths of information retrieval and generative models to enhance the handling of real-time and domain-specific knowledge. Despite its advantages, limitations within RAG components may cause hallucinations, or more precisely termed confabulations in generated outputs, driving extensive research to address these limitations and mitigate hallucinations. This review focuses on hallucination in retrieval-augmented large language models (LLMs). We first examine the causes of hallucinations from different sub-tasks in the retrieval and generation phases. Then, we provide a comprehensive overview of corresponding hallucination mitigation techniqu…

3. Very close Review Not relevant

   Evaluating Retrieval Augmented Generation: A Comprehensive Review of Evaluation Dimensions, Question Types, and Application

   Simon Knollmeyer, Oğuz Caymazer, Leonid Koval et al. · 2026 · SN Computer Science · cited 4 times · Not summarized yet

   The review synthesizes RAG evaluation dimensions, metrics, datasets, and question types, while this paper establishes a benchmark of four capabilities.

   Abstract

   This study addresses limitations of traditional benchmarking methods for Retrieval-Augmented Generation (RAG) systems by proposing an evaluation framework for RAG-enhanced Large Language Models (LLMs). The framework structures evaluation dimensions and metrics, identifies suitable datasets and question types, and provides guidance for applying the framework in practice. A systematic literature review (SLR) was conducted, synthesizing evidence from 12 studies focused on the evaluation of RAG systems. The review employs a concept matrix to classify evaluative approaches and maps metrics to dimensions, evaluator types, and pipeline stages. In addition, we systematize dataset and question-type r…

4. Very close Review Uses this paper in depth Not relevant

   Improving negative rejection ability in language models: A review of fine-tuned LLMs, RAG, and RAFT

   Bo-Wen Li, Zhuoqing Zhong, Hengyu Zhang et al. · 2025 · Journal of King Saud University: Computer and Information Sciences · cited 4 times · Not summarized yet

   The review focuses on negative rejection in LLMs, RAG, and RAFT, one of the four capabilities tested in this paper.

   Abstract

   Large Language Models (LLMs) excel in text understanding and generation but struggle to reject irrelevant, ambiguous, or misleading queries, termed negative rejection, impacting reliability in high-stakes contexts. This paper reviews negative rejection, analyzing three approaches: fine-tuned LLMs, Retrieval-Augmented Generation (RAG), and Retrieval-Augmented Fine-Tuning (RAFT). Through a literature review, we assess their rejection accuracy, hallucination rates, computational costs, and domain adaptability. We introduce a four-dimensional framework—architecture, metrics, domain application, and dataset diversity—to organize existing studies and benchmarks for comparative analysis (Cao, 2024)…

5. Very close Another approach to the problem Not relevant

   RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models

   Ange Maiztegi, J. Ayerdi, Miren Illarramendi et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   RAG-TESTER automates end-to-end tests using generated cases and an LLM judge, while this paper benchmarks LLMs on four RAG abilities.

   Abstract

   Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to use external and domain-specific knowledge, but its reliability depends on the interaction between the generative model, embedding model, retrieval mechanism, and prompt construction strategy. We present RagTester, an automated end-to-end testing approach for RAG systems. RagTester generates retrieval documents, test inputs, and expected outputs; executes the tests; and evaluates the resulting answers using an LLM as a judge. Its test-generation strategy targets complex passages, unsupported queries, and document-coverage criteria. We evaluate RagTester using eight LLMs and six embedding models, yielding 24 compatib…