# Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

**Authors:** Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, Haifeng Wang  
**arXiv:** [2307.11019](https://arxiv.org/abs/2307.11019)  
**Date:** 2023-07-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.11019v1/

## Key points

- Study title: "Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation"
- Researchers: Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, Haifeng Wang
- Focus on knowledge-intensive tasks like open-domain question answering (QA) that require external information support
- Large language models (LLMs) like ChatGPT show remarkable ability in handling tasks relying on world knowledge
- Ambiguity around LLMs' discernment of factual knowledge boundaries and adaptation with retrieval augmentation
- Analysis of QA performance to understand LLMs' awareness of their capabilities pre and post feedback
- Findings show LLMs exhibit confidence in answering questions accurately but benefit from retrieval augmentation for improved judgemental abilities
- LLMs rely on retrieved information for formulating responses, influenced by the quality of results
- Importance of retrieval augmentation in enhancing LLMs' performance in complex tasks requiring factual knowledge

## Summary

In the study titled "Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation," conducted by Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang, the researchers delve into the realm of knowledge-intensive tasks such as open-domain question answering (QA). These tasks necessitate a significant amount of factual knowledge and often require external information for support. The advent of large language models (LLMs) like ChatGPT has showcased their remarkable ability to tackle a diverse array of tasks that rely on world knowledge, including those that are knowledge-intensive. However, a critical aspect that remains ambiguous is how well LLMs can discern their factual knowledge boundaries and how they adapt when incorporating retrieval augmentation. The researchers present an initial analysis focusing on three primary research questions to shed light on this matter. By evaluating QA performance and examining both priori judgement (before receiving feedback) and posteriori judgement (after receiving feedback) of LLMs , they aim to understand the extent of these models' awareness of their own capabilities. The findings reveal that LLMs exhibit unwavering confidence in their capacity to answer questions accurately. Moreover, the study demonstrates that retrieval augmentation serves as an effective strategy in enhancing LLMs' understanding of their  and subsequently improving their judgemental abilities. Additionally, it is observed that LLMs tend to rely on retrieved information when formulating responses , with the quality of these results significantly influencing their reliance. Overall, this research contributes valuable insights into how large language models navigate complex tasks requiring substantial factual knowledge and underscores the importance of retrieval augmentation in enhancing their performance . The code for replicating this study is accessible at https://github.com/RUCAIBox/LLM-Knowledge-Boundary.