# RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework

**Authors:** Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, Maosong Sun  
**arXiv:** [2408.01262](https://arxiv.org/abs/2408.01262)  
**Date:** 2024-08-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.01262v1/

## Key points

- The paper introduces the "RAGEval" framework for evaluating Large Language Models (LLMs) in various scenarios.
- RAGEval automates the generation of evaluation datasets and assesses LLMs' performance across different domains.
- Existing benchmarks for Retrieval-Augmented Generation (RAG) systems focus on general knowledge questions, lacking evaluation in diverse vertical domains.
- RAGEval addresses this limitation with innovative metrics: Completeness, Hallucination, and Irrelevance.
- The framework summarizes schemas from seed documents, applies configurations to produce varied documents, and constructs question-answering pairs based on articles and configurations.
- By benchmarking RAG models across vertical domains using these metrics, RAGEval enhances the evaluation of LLMs' knowledge usage abilities.
- This research significantly advances evaluation methodologies for LLMs by providing a comprehensive framework for assessing their performance in different scenarios and domains.

## Summary

, , , , 

The paper titled "RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework" by Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, and Maosong Sun introduces a novel framework for evaluating the knowledge usage ability of Large Language Models (LLMs) in different scenarios. The proposed  framework automates the generation of evaluation datasets and offers a comprehensive approach to assessing LLMs' performance across various domains. Retrieval-Augmented Generation () systems have proven effective in addressing the hallucination issue of LLMs. However, existing RAG benchmarks primarily focus on general knowledge questions and lack the capability to evaluate how well these systems handle data from diverse vertical domains. This limitation is addressed by , which provides a more nuanced understanding of LLMs' performance through its three innovative metrics - Completeness, Hallucination, and Irrelevance. The authors achieve this by summarizing schemas from seed documents and applying configurations to produce varied documents. They then construct question-answering pairs based on both articles and configurations. By benchmarking RAG models across vertical domains using the proposed metrics,  enhances the evaluation of LLMs' knowledge usage abilities and eliminates confusion about where knowledge comes from when answering questions in existing QA datasets. This research contributes significantly to advancing the evaluation methodologies for LLMs by providing a comprehensive framework that can assess their performance across different scenarios and domains. It offers a valuable tool for researchers and developers working with LLMs to enhance their capabilities and address challenges related to domain-specific information retrieval and utilization.