RAGAS: Automated Evaluation of Retrieval Augmented Generation

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Review Not relevant

   Assessing RAG: A Comprehensive Review of Evaluation Frameworks

   Inaya Imtiyaz, Mahdi Parvaz, Shafaq Mandha et al. · 2025 · 2025 International Conference on Computational Intelligence and Knowledge Economy (ICCIKE) · cited 3 times · Not summarized yet

   Assessing RAG surveys evaluation frameworks and compares metrics such as precision and faithfulness for RAG pipelines.

   Abstract

   Retrieval-Augmented Generation (RAG) pipelines, which combine retrieval and generative AI components, are becoming crucial for accurate context-aware text generation in a variety of applications. However, this remains challenging for RAG systems to evaluate due to their complexity and the need to evaluate retrieval and generation components. This survey is conducted to fill the gap in systematic evaluation methodologies by examining existing RAG evaluation frameworks and providing practical advice to improve performance. We compare and evaluate according to metrics such as precision, faithfulness, and adaptability. Our analysis shows that open-source tools focus on transparency and customiza…

2. Very close Another approach to the problem Not relevant

   Universal End-to-End Automated Evaluation Framework for Retrieval-Augmented Generation System

   Shen Li, Haochen Zhang, Yue-Jiao Wang et al. · 2025 · 2025 IEEE 8th International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · Not summarized yet

   The universal framework automates end-to-end evaluation of RAG systems, addressing evaluation of the same retrieval-and-generation pipeline as RAGAs.

   Abstract

   Retrieval-Augmented Generation (RAG) has become one of the key functionalities in current large language model (LLM) applications. It effectively mitigates hallucination issues in LLMs and provides users with the latest domain-specific information by accessing knowledge bases, serving as a costefficient method for LLM capability migration. However, users primarily interact with LLMs through application systems like OpenWebUI, where RAG functionality is typically embedded as a “black box.” This makes large-scale automated evaluation challenging for existing RAG assessment systems. To address this, this paper proposes an universal end-to-end automated evaluation framework. This framework can e…

3. Very close Another approach to the problem Not relevant

   Evaluating Precision and Recall at Retrieval Time in Retrieval-Augmented Generation (RAG) Systems

   Gopichand Agnihotram, Joydeep Sarkar · 2025 · American Journal of Computer Science and Technology · cited 4 times · Not summarized yet

   The paper analyzes precision and recall for RAG retrieval, whereas RAGAs evaluates retrieval alongside generation with a suite of metrics.

   Abstract

   Retrieval-Augmented Generation (RAG) systems signify a pivotal advancement in natural language processing, merging information retrieval with large language models (LLMs) to ground responses in external knowledge. This hybrid approach enhances the factual accuracy and currency of generated content, mitigating common issues like hallucination. The efficacy of a RAG system, however, is fundamentally dependent on the performance of its retrieval component. This paper provides a detailed analysis of precision and recall as critical metrics for evaluating and optimizing this retrieval step. We explore the distinct roles and inherent trade-offs of these metrics within a RAG pipeline, demonstrating…

4. Very close Review Cites this paper Not relevant

   Evaluation of Retrieval-Augmented Generation: A Survey

   Hao Yu, Ao-Ran Gan, Kai Zhang et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 355 times · Not summarized yet

   Evaluation of Retrieval-Augmented Generation surveys evaluation methods and benchmarks for RAG retrieval and generation.

   Abstract

   Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct A Unified Evaluation Process of RAG (Auepora) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accurac…

5. Very close Builds on this paper Not relevant

   RAG-based LLM Evaluation

   André Ribeiro, Rúben Garrido, V. Ramos et al. · 2026 · 2026 International Conference Automatics, Robotics and Artificial Intelligence (ICARAI) · Not summarized yet

   RAG-based LLM Evaluation combines RAGAS metrics with classical retrieval measures to assess a RAG pipeline.

   Abstract

   Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLM) by grounding responses in retrieved context, addressing hallucination concerns in knowledge-intensive applications. However, evaluating RAG-based LLMs remains challenging due to complex interactions between retrieval and generation components. This work presents a comprehensive evaluation framework combining RAGAS LLM-based metrics with classical information retrieval measures. We evaluate a RAG pipeline using Qwen 3 14B on the Neural Bridge RAG Dataset 12000, comparing RAG and non-RAG grounded response generation. To address same-model evaluation bias, we introduce cross-model evaluation using Gemma 3 12B, demo…

6. Very close Another approach to the problem Not relevant

   RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

   Dongyu Ru, Lin Qiu, Xiangkun Hu et al. · 2024 · Computation and Language · cited 115 times · Not summarized yet

   RAGChecker proposes fine-grained diagnostic metrics for retrieval and generation and tests their correlation with human judgments.

   Abstract

   Despite Retrieval-Augmented Generation (RAG) has shown promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patte…

7. Very close Review Not relevant

   Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

   Ao-Ran Gan, Hao Yu, Kai Zhang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 58 times · Not summarized yet

   The survey reviews traditional and emerging methods and frameworks for evaluating RAG systems.

   Abstract

   Recent advancements in Retrieval-Augmented Generation (RAG) have revolutionized natural language processing by integrating Large Language Models (LLMs) with external information retrieval, enabling accurate, up-to-date, and verifiable text generation across diverse applications. However, evaluating RAG systems presents unique challenges due to their hybrid architecture that combines retrieval and generation components, as well as their dependence on dynamic knowledge sources in the LLM era. In response, this paper provides a comprehensive survey of RAG evaluation methods and frameworks, systematically reviewing traditional and emerging evaluation approaches, for system performance, factual a…

8. Very close Another approach to the problem Not relevant

   Automated Testing Framework for Retrieval-Augmented Generation Systems: Evaluation Metrics and Optimization Strategies

   Jun Zhang, Rui He, Xiao-Dong Zheng et al. · 2025 · 2025 5th International Symposium on Artificial Intelligence and Big Data (AIBDF) · Not summarized yet

   The testing framework evaluates retrieval and generation with quantifiable metrics, reporting gains from combining context recall and answer correctness.

   Abstract

   The rapid development of Retrieval-Augmented Generation (RAG) systems has addressed critical limitations in large language models (LLMs), including factual hallucinations, knowledge obsolescence, and domain-specific incompetence. However, the complexity of RAG architectures introduces new challenges in performance evaluation. This paper proposes a comprehensive automated testing framework that systematically assesses both retrieval and generation components through quantifiable metrics. Our experimental results demonstrate that the weighted integration of context recall (CR) and answer correctness (AC) yields 17.1% higher discriminative power than single metrics, while semantic-aware documen…

Prior work it builds on

1. Close Broader context A method it uses Not relevant

   In-Context Retrieval-Augmented Language Models

   Ori Ram, Yoav Levine, Itay Dalmedigos et al. · 2023 · Transactions of the Association for Computational Linguistics · cited 1,191 times · AI summary available

   In-Context RALM grounds language-model generation by prepending retrieved documents, describing a RAG approach rather than its evaluation.

   Abstract

   Abstract Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, witho…

2. Related Groundwork for this paper Used in depth by this paper A method it uses Not relevant

   BARTScore: Evaluating Generated Text as Text Generation

   Weizhe Yuan, Graham Neubig, Peng-Fei Liu · 2021 · Neural Information Processing Systems · cited 1,186 times · AI summary available

   BARTScore introduces a pretrained-model metric for generated-text quality, a generation-evaluation approach cited by RAGAs.

   Abstract

   A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with …

3. Related Broader context A method it uses Not relevant

   Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

   O. Khattab, Keshav Santhanam, Xiang Lisa Li et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 417 times · AI summary available

   Demonstrate-Search-Predict composes retrieval and language models for grounded predictions, addressing RAG system construction rather than evaluation.

   Abstract

   Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple"retrieve-then-read"pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematicall…

4. Related Broader context Not relevant

   REPLUG: Retrieval-Augmented Black-Box Language Models

   Weijia Shi, Sewon Min, Michihiro Yasunaga et al. · 2023 · North American Chapter of the Association for Computational Linguistics · cited 1,167 times · Not summarized yet

   REPLUG augments a black-box language model with retrieved documents, presenting a retrieval-augmented modeling approach rather than evaluation.

   Abstract

   We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improv…

5. Related Groundwork for this paper Not relevant

   Is ChatGPT a Good NLG Evaluator? A Preliminary Study

   Jiaan Wang, Yun-Long Liang, Fan-Dong Meng et al. · 2023 · NEWSUM · cited 670 times · Not summarized yet

   The study tests ChatGPT as an automatic natural-language-generation evaluator, an evaluation approach cited by RAGAs.

   Abstract

   Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regar…

Papers that cite it

1. Very close Broader context Not relevant

   Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance

   Binita Saha, Utsha Saha, Muhammad Zubair Malik · 2024 · IEEE Access · cited 40 times · Not summarized yet

   The inverted-question-matching architecture targets better RAG question-answering, while RAGAs evaluates RAG pipeline quality.

   Abstract

   This work presents a novel architecture for building Retrieval-Augmented Generation (RAG) systems to improve Question Answering (QA) tasks from a target corpus. Large Language Models (LLMs) have revolutionized the analyzing and generation of human-like text. These models rely on pre-trained data and lack real-time updates unless integrated with live data tools. RAG enhances LLMs by integrating online resources and databases to generate contextually appropriate responses. However, traditional RAG still encounters challenges like information dilution and hallucinations when handling vast amounts of data. Our approach addresses these challenges by converting corpora into a domain-specific datas…

2. Very close Review Not relevant

   Retrieval-Augmented Generation (RAG): Architectures, Evaluation Metrics, and Security Challenges in Large Language Models

   Asker Shariff Subahan, S. Khan, Sourav Sarkar et al. · 2026 · 2026 International Conference on Secure Information Systems and Technologies (ICSIST) · Not summarized yet

   The paper surveys RAG architecture, evaluation processes, and security challenges.

   Abstract

   Retrieval-Augmented Generation (RAG) is a sophisticated approach employed to enhance the efficiency of Large Language Models (LLMs). Most of the existing LLMs primarily rely on pre-trained data, which can be either obsolete or insufficient at times, hence providing suboptimal outcomes. To mitigate this problem, RAG begins by searching for relevant content and employs that content to produce a response. In this way, it becomes possible to yield better outputs. This paper provides insight into the concept of RAG through its architecture, working mechanism, and evaluation processes. It outlines various challenges associated with the technique, including sourcing quality data, delayed output, an…

3. Very close Review Not relevant

   Rethinking Knowledge Retrieval for Generation: A Survey on RAG Architectures and Applications

   Meghana Sunil, V. Shravya, Shravan Venkatraman et al. · 2026 · Not summarized yet

   The survey examines RAG architectures and applications, including how retrieval grounds generation in external information.

   Abstract

   Large Language Models (LLMs) have demonstrated remarkable fluency and versatility across natural language tasks but remain fundamentally limited by their static knowledge and susceptibility to hallucinations, especially in domains requiring up to date or attribute grounded information. Retrieval Augmented Generation (RAG) addresses these challenges by integrating external retrieval mechanisms with generative models, enabling dynamic, context aware generation grounded in verifiable data sources. This survey presents a comprehensive examination of RAG as a modular and evolving paradigm that enhances factual reliability, adaptability, and task alignment in LLM based systems. We formalize the RA…

4. Very close Another approach to the problem Not relevant

   RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis

   N. Hart, Nicolas Weeger · 2026 · Not summarized yet

   RAG-NIMBF evaluates RAG configurations through controlled ablations and combines answer-quality, retrieval-correctness, and runtime measures.

   Abstract

   Retrieval-Augmented Generation (RAG) has become the dominant paradigm for grounding the outputs of large language models (LLM) in external knowledge. However, practitioners are faced with a vast configuration space comprising chunking, embedding, retrieval, reranking, prompting and generation models, and there is limited guidance on how these components interact. We present RAG-NIMBF, a modular benchmarking framework that attaches to existing RAG pipelines as a non-invasive evaluation layer. It evaluates the configuration space through controlled ablation with end-to-end experiment tracking. The evaluation combines LLM-based answer quality scores, retrieval correctness measures, and runtime …

5. Very close Shared data, tool or method Not relevant

   MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

   Yannis Katsis, Sara Rosenthal, Kshitij Fadnis et al. · 2025 · Qeios preprint · preprint, not peer reviewed · cited 9 times · Not summarized yet

   mtRAG supplies a human-generated multi-turn benchmark for evaluating the full RAG pipeline, complementing RAGAs' evaluation metrics.

   Abstract

   Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on _multi-turn_ RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present mtRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. mtRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judg…