RAGAS: Automated Evaluation of Retrieval Augmented Generation

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   Assessing RAG: A Comprehensive Review of Evaluation Frameworks

   Inaya Imtiyaz, Mahdi Parvaz, Shafaq Mandha et al. · 2025 · 2025 International Conference on Computational Intelligence and Knowledge Economy (ICCIKE) · cité 3 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) pipelines, which combine retrieval and generative AI components, are becoming crucial for accurate context-aware text generation in a variety of applications. However, this remains challenging for RAG systems to evaluate due to their complexity and the need to evaluate retrieval and generation components. This survey is conducted to fill the gap in systematic evaluation methodologies by examining existing RAG evaluation frameworks and providing practical advice to improve performance. We compare and evaluate according to metrics such as precision, faithfulness, and adaptability. Our analysis shows that open-source tools focus on transparency and customiza…

2. Très proche Pas pertinent

   Universal End-to-End Automated Evaluation Framework for Retrieval-Augmented Generation System

   Shen Li, Haochen Zhang, Yue-Jiao Wang et al. · 2025 · 2025 IEEE 8th International Conference on Pattern Recognition and Artificial Intelligence (PRAI) · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) has become one of the key functionalities in current large language model (LLM) applications. It effectively mitigates hallucination issues in LLMs and provides users with the latest domain-specific information by accessing knowledge bases, serving as a costefficient method for LLM capability migration. However, users primarily interact with LLMs through application systems like OpenWebUI, where RAG functionality is typically embedded as a “black box.” This makes large-scale automated evaluation challenging for existing RAG assessment systems. To address this, this paper proposes an universal end-to-end automated evaluation framework. This framework can e…

3. Très proche Pas pertinent

   Evaluating Precision and Recall at Retrieval Time in Retrieval-Augmented Generation (RAG) Systems

   Gopichand Agnihotram, Joydeep Sarkar · 2025 · American Journal of Computer Science and Technology · cité 4 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) systems signify a pivotal advancement in natural language processing, merging information retrieval with large language models (LLMs) to ground responses in external knowledge. This hybrid approach enhances the factual accuracy and currency of generated content, mitigating common issues like hallucination. The efficacy of a RAG system, however, is fundamentally dependent on the performance of its retrieval component. This paper provides a detailed analysis of precision and recall as critical metrics for evaluating and optimizing this retrieval step. We explore the distinct roles and inherent trade-offs of these metrics within a RAG pipeline, demonstrating…

4. Très proche Cite cet article Pas pertinent

   Evaluation of Retrieval-Augmented Generation: A Survey

   Hao Yu, Ao-Ran Gan, Kai Zhang et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 355 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) has recently gained traction in natural language processing. Numerous studies and real-world applications are leveraging its ability to enhance generative models through external information retrieval. Evaluating these RAG systems, however, poses unique challenges due to their hybrid structure and reliance on dynamic knowledge sources. To better understand these challenges, we conduct A Unified Evaluation Process of RAG (Auepora) and aim to provide a comprehensive overview of the evaluation and benchmarks of RAG systems. Specifically, we examine and compare several quantifiable metrics of the Retrieval and Generation components, such as relevance, accurac…

5. Très proche Pas pertinent

   RAG-based LLM Evaluation

   André Ribeiro, Rúben Garrido, V. Ramos et al. · 2026 · 2026 International Conference Automatics, Robotics and Artificial Intelligence (ICARAI) · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) systems enhance Large Language Models (LLM) by grounding responses in retrieved context, addressing hallucination concerns in knowledge-intensive applications. However, evaluating RAG-based LLMs remains challenging due to complex interactions between retrieval and generation components. This work presents a comprehensive evaluation framework combining RAGAS LLM-based metrics with classical information retrieval measures. We evaluate a RAG pipeline using Qwen 3 14B on the Neural Bridge RAG Dataset 12000, comparing RAG and non-RAG grounded response generation. To address same-model evaluation bias, we introduce cross-model evaluation using Gemma 3 12B, demo…

6. Très proche Autre approche du problème Pas pertinent

   RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

   Dongyu Ru, Lin Qiu, Xiangkun Hu et al. · 2024 · Computation and Language · cité 115 fois · Pas encore résumé

   RAGChecker évalue finement les modules de récupération et de génération avec des métriques diagnostiques, tandis que RAGAS propose une évaluation sans annotations de référence.

   Résumé scientifique

   Despite Retrieval-Augmented Generation (RAG) has shown promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patte…

7. Très proche Pas pertinent

   Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey

   Ao-Ran Gan, Hao Yu, Kai Zhang et al. · 2025 · arXiv.org · preprint, non relu par les pairs · cité 58 fois · Pas encore résumé

   Résumé scientifique

   Recent advancements in Retrieval-Augmented Generation (RAG) have revolutionized natural language processing by integrating Large Language Models (LLMs) with external information retrieval, enabling accurate, up-to-date, and verifiable text generation across diverse applications. However, evaluating RAG systems presents unique challenges due to their hybrid architecture that combines retrieval and generation components, as well as their dependence on dynamic knowledge sources in the LLM era. In response, this paper provides a comprehensive survey of RAG evaluation methods and frameworks, systematically reviewing traditional and emerging evaluation approaches, for system performance, factual a…

8. Très proche Pas pertinent

   Automated Testing Framework for Retrieval-Augmented Generation Systems: Evaluation Metrics and Optimization Strategies

   Jun Zhang, Rui He, Xiao-Dong Zheng et al. · 2025 · 2025 5th International Symposium on Artificial Intelligence and Big Data (AIBDF) · Pas encore résumé

   Résumé scientifique

   The rapid development of Retrieval-Augmented Generation (RAG) systems has addressed critical limitations in large language models (LLMs), including factual hallucinations, knowledge obsolescence, and domain-specific incompetence. However, the complexity of RAG architectures introduces new challenges in performance evaluation. This paper proposes a comprehensive automated testing framework that systematically assesses both retrieval and generation components through quantifiable metrics. Our experimental results demonstrate that the weighted integration of context recall (CR) and answer correctness (AC) yields 17.1% higher discriminative power than single metrics, while semantic-aware documen…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Une méthode qu’il utilise Pas pertinent

   In-Context Retrieval-Augmented Language Models

   Ori Ram, Yoav Levine, Itay Dalmedigos et al. · 2023 · Transactions of the Association for Computational Linguistics · cité 1 191 fois · Résumé IA disponible

   Résumé scientifique

   Abstract Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, witho…

2. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   BARTScore: Evaluating Generated Text as Text Generation

   Weizhe Yuan, Graham Neubig, Peng-Fei Liu · 2021 · Neural Information Processing Systems · cité 1 186 fois · Résumé IA disponible

   Résumé scientifique

   A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with …

3. Apparenté Une méthode qu’il utilise Pas pertinent

   Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

   O. Khattab, Keshav Santhanam, Xiang Lisa Li et al. · 2022 · arXiv.org · preprint, non relu par les pairs · cité 417 fois · Résumé IA disponible

   Résumé scientifique

   Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple"retrieve-then-read"pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematicall…

4. Apparenté Pas pertinent

   REPLUG: Retrieval-Augmented Black-Box Language Models

   Weijia Shi, Sewon Min, Michihiro Yasunaga et al. · 2023 · North American Chapter of the Association for Computational Linguistics · cité 1 167 fois · Pas encore résumé

   Résumé scientifique

   We introduce REPLUG, a retrieval-augmented language modeling framework that treats the language model (LM) as a black box and augments it with a tuneable retrieval model. Unlike prior retrieval-augmented LMs that train language models with special cross-attention mechanisms to encode the retrieved text, REPLUG simply prepends retrieved documents to the input for the frozen black-box LM. This simple design can be easily applied to any existing language models. Furthermore, we show that the LM can be used to supervise the retrieval model, which can then find documents that help the LM make better predictions. Our experiments demonstrate that REPLUG with the tuned retriever significantly improv…

5. Apparenté Pas pertinent

   Is ChatGPT a Good NLG Evaluator? A Preliminary Study

   Jiaan Wang, Yun-Long Liang, Fan-Dong Meng et al. · 2023 · NEWSUM · cité 670 fois · Pas encore résumé

   Résumé scientifique

   Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community. Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics. However, the ability of ChatGPT to serve as an evaluation metric is still underexplored. Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether ChatGPT is a good NLG evaluation metric. In this report, we provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. In detail, we regar…

Les articles qui le citent

1. Très proche Pas pertinent

   Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance

   Binita Saha, Utsha Saha, Muhammad Zubair Malik · 2024 · IEEE Access · cité 40 fois · Pas encore résumé

   Résumé scientifique

   This work presents a novel architecture for building Retrieval-Augmented Generation (RAG) systems to improve Question Answering (QA) tasks from a target corpus. Large Language Models (LLMs) have revolutionized the analyzing and generation of human-like text. These models rely on pre-trained data and lack real-time updates unless integrated with live data tools. RAG enhances LLMs by integrating online resources and databases to generate contextually appropriate responses. However, traditional RAG still encounters challenges like information dilution and hallucinations when handling vast amounts of data. Our approach addresses these challenges by converting corpora into a domain-specific datas…

2. Très proche Pas pertinent

   Retrieval-Augmented Generation (RAG): Architectures, Evaluation Metrics, and Security Challenges in Large Language Models

   Asker Shariff Subahan, S. Khan, Sourav Sarkar et al. · 2026 · 2026 International Conference on Secure Information Systems and Technologies (ICSIST) · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) is a sophisticated approach employed to enhance the efficiency of Large Language Models (LLMs). Most of the existing LLMs primarily rely on pre-trained data, which can be either obsolete or insufficient at times, hence providing suboptimal outcomes. To mitigate this problem, RAG begins by searching for relevant content and employs that content to produce a response. In this way, it becomes possible to yield better outputs. This paper provides insight into the concept of RAG through its architecture, working mechanism, and evaluation processes. It outlines various challenges associated with the technique, including sourcing quality data, delayed output, an…

3. Très proche Pas pertinent

   Rethinking Knowledge Retrieval for Generation: A Survey on RAG Architectures and Applications

   Meghana Sunil, V. Shravya, Shravan Venkatraman et al. · 2026 · Pas encore résumé

   Résumé scientifique

   Large Language Models (LLMs) have demonstrated remarkable fluency and versatility across natural language tasks but remain fundamentally limited by their static knowledge and susceptibility to hallucinations, especially in domains requiring up to date or attribute grounded information. Retrieval Augmented Generation (RAG) addresses these challenges by integrating external retrieval mechanisms with generative models, enabling dynamic, context aware generation grounded in verifiable data sources. This survey presents a comprehensive examination of RAG as a modular and evolving paradigm that enhances factual reliability, adaptability, and task alignment in LLM based systems. We formalize the RA…

4. Très proche Pas pertinent

   RAG-NIMBF: A Non-Invasive, Modular Benchmarking Framework for Automated RAG Configuration Analysis

   N. Hart, Nicolas Weeger · 2026 · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) has become the dominant paradigm for grounding the outputs of large language models (LLM) in external knowledge. However, practitioners are faced with a vast configuration space comprising chunking, embedding, retrieval, reranking, prompting and generation models, and there is limited guidance on how these components interact. We present RAG-NIMBF, a modular benchmarking framework that attaches to existing RAG pipelines as a non-invasive evaluation layer. It evaluates the configuration space through controlled ablation with end-to-end experiment tracking. The evaluation combines LLM-based answer quality scores, retrieval correctness measures, and runtime …

5. Très proche Pas pertinent

   MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems

   Yannis Katsis, Sara Rosenthal, Kshitij Fadnis et al. · 2025 · Qeios preprint · preprint, non relu par les pairs · cité 9 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-augmented generation (RAG) has recently become a very popular task for Large Language Models (LLMs). Evaluating them on _multi-turn_ RAG conversations, where the system is asked to generate a response to a question in the context of a preceding conversation is an important and often overlooked task with several additional challenges. We present mtRAG: an end-to-end human-generated multi-turn RAG benchmark that reflects several real-world properties across diverse dimensions for evaluating the full RAG pipeline. mtRAG contains 110 conversations averaging 7.7 turns each across four domains for a total of 842 tasks. We also explore automation paths via synthetic data and LLM-as-a-Judg…