# Enabling Large Language Models to Generate Text with Citations

**Authors:** Tianyu Gao, Howard Yen, Jiatong Yu, Danqi Chen  
**arXiv:** [2305.14627](https://arxiv.org/abs/2305.14627)  
**Date:** 2023-05-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14627v2/

## Key points

- Large language models (LLMs) are popular tools for information seeking
- LLMs often generate outputs with false or misleading information
- Authors aim to enable LLMs to generate text with citations to improve accuracy and verifiability
- Existing approaches rely on commercial search engines and human evaluation, making reproducibility and comparison difficult
- Authors propose ALCE (Automatic LLMs' Citation Evaluation) as a benchmark for evaluating LLMs' citation generation capabilities
- ALCE includes diverse questions and retrieval corpora, requiring end-to-end systems for evidence retrieval and citation generation
- Automatic metrics are introduced to evaluate fluency, correctness, and citation quality, showing correlation with human judgments
- Experiments reveal current systems have room for improvement, even the best models lack complete citation support 50% of the time on ELI5 dataset
- Promising directions for future research include enhancing retrievers, advancing long-context LLMs, and improving synthesis of information from multiple sources
- Paper presents comprehensive approach to addressing hallucination in LLM-generated text by enabling them to generate text with citations
- Proposed benchmark ALCE provides standardized evaluation framework for comparing different modeling approaches in terms of fluency, correctness, and citation quality

## Summary

Large language models (LLMs) have become popular tools for information seeking. However, they often generate outputs that contain false or misleading information. To address this issue, the authors of this paper aim to enable LLMs to generate text with citations. This would improve the accuracy and verifiability of their generated content. Existing approaches in this area rely heavily on commercial search engines and human evaluation. This makes it difficult to reproduce and compare different modeling techniques. To overcome these challenges, the authors propose ALCE (Automatic LLMs' Citation Evaluation). It is the first benchmark for evaluating LLMs' citation generation capabilities. ALCE includes a diverse set of questions and retrieval corpora. This requires the development of end-to-end systems that can retrieve supporting evidence and generate answers with citations. The authors also introduce automatic metrics to evaluate fluency, correctness, and citation quality. These demonstrate a strong correlation with human judgments. The experiments conducted using state-of-the-art LLMs and novel prompting strategies reveal that current systems have significant room for improvement. For instance, even the best models lack complete citation support 50% of the time on the ELI5 dataset. The authors' analysis suggests several promising directions for future research including enhancing retrievers, advancing long-context LLMs, and improving the synthesis of information from multiple sources. Overall, this paper presents a comprehensive approach to addressing hallucination in LLM-generated text by enabling them to generate text with citations. The proposed benchmark ALCE provides a standardized evaluation framework for comparing different modeling approaches in terms of fluency, correctness, and citation quality. The findings highlight areas where current systems fall short and offer insights into potential avenues for further improvement in generating accurate and verifiable information using LLMs.