# Benchmarking Large Language Models in Retrieval-Augmented Generation

**Authors:** Jiawei Chen, Hongyu Lin, Xianpei Han, Le Sun  
**arXiv:** [2309.01431](https://arxiv.org/abs/2309.01431)  
**Date:** 2023-09-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.01431v1/

## Key points

- Authors address the issue of hallucination in large language models (LLMs)
- Propose the use of Retrieval-Augmented Generation (RAG) to mitigate this problem
- Introduce a new corpus called Retrieval-Augmented Generation Benchmark (RGB)
- RGB corpus divided into four separate testbeds based on fundamental abilities required
- Evaluate six representative LLMs on RGB to diagnose challenges faced by current models
- LLMs struggle with negative rejection, information integration, and dealing with false information
- More work needed before effectively applying RAG to LLMs
- Paper provides insights into limitations and challenges of using RAG for LLMs
- Establish benchmark and evaluate multiple LLMs across various abilities required for RAG

## Summary

In their paper titled "Benchmarking Large Language Models in Retrieval-Augmented Generation," authors Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun address the issue of hallucination in large language models (LLMs) and propose the use of Retrieval-Augmented Generation (RAG) as a promising approach to mitigate this problem. To address the gap in existing research on the impact of retrieval-augmented generation on different LLMs, they systematically investigate its impact and introduce a new corpus called Retrieval-Augmented Generation Benchmark (RGB), which includes instances in both English and Chinese. The RGB corpus is divided into four separate testbeds based on fundamental abilities required to resolve each case. The authors then evaluate six representative LLMs on RGB to diagnose the challenges faced by current models when applying RAG. The evaluation reveals that while LLMs demonstrate a certain degree of noise robustness, they still struggle significantly with negative rejection, information integration, and dealing with false information. These findings indicate that there is still much work to be done before effectively applying RAG to LLMs. Overall, this paper provides valuable insights into the limitations and challenges associated with using retrieval-augmented generation for large language models. By establishing a benchmark and evaluating multiple LLMs across various abilities required for RAG, the authors shed light on areas that require further improvement in order to enhance the performance and capabilities of these models.