# WikiChat: A Few-Shot LLM-Based Chatbot Grounded with Wikipedia

**Authors:** Sina J. Semnani, Violet Z. Yao, Heidi C. Zhang, Monica S. Lam  
**arXiv:** [2305.14292](https://arxiv.org/abs/2305.14292)  
**Date:** 2023-05-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.14292v1/

## Key points

- WikiChat is a few-shot Large Language Model (LLM)-based chatbot that addresses limitations of existing LLMs in providing accurate and reliable information
- LLMs struggle to speak accurately about events that occurred after their training and are prone to hallucination when discussing less popular topics
- WikiChat leverages live information from Wikipedia to ground its responses
- The development of WikiChat involved multiple iterations of experimentation and resulted in a pipeline based on information retrieval with three main components: suggesting facts using LLMs, verifying facts against Wikipedia for accuracy, and retrieving additional up-to-date information for coherent and engaging responses
- The authors propose a hybrid human-and-LLM evaluation methodology to assess the factuality and conversationality of LLM-based chatbots, specifically focusing on recent and less popular topics
- In evaluations against strong baselines, WikiChat outperforms all baselines in terms of factual accuracy by significant margins (up to 12.1%, 28.3%, and 32.7% on head, recent, and tail topics respectively)
- WikiChat maintains naturalness, relevance, non-repetitiveness, and informational quality comparable to GPT-3
- Overall, this paper presents WikiChat as an effective solution for enhancing the reliability and accuracy of LLM-based chatbots by grounding them with live information from Wikipedia

## Summary

, , , , 

This paper introduces WikiChat, a few-shot Large Language Model (LLM)-based chatbot that addresses the limitations of existing LLMs in providing accurate and reliable information. Despite recent advancements in LLMs, users still cannot fully trust the responses generated by these models. LLMs struggle to speak accurately about events that occurred after their training, which are often of great interest to users. Additionally, they are prone to hallucination when discussing less popular topics. To overcome these challenges, WikiChat leverages live information from Wikipedia to ground its responses. The development of WikiChat involved multiple iterations of experimentation and resulted in a pipeline based on information retrieval. This pipeline has three main components: 

1. Using LLMs to suggest interesting and relevant facts. 2. Individually verifying these facts against Wikipedia for accuracy. 3. Retrieving additional up-to-date information and composing coherent and engaging time-aware responses. The authors propose a novel hybrid human-and-LLM evaluation methodology to assess the factuality and conversationality of LLM-based chatbots. They specifically focus on evaluating the ability of chatbots to converse about recent and less popular topics, which have been previously neglected. In evaluations against strong fine-tuned and LLM-based baselines across diverse conversation topics, WikiChat outperforms all baselines in terms of factual accuracy by significant margins. It achieves improvements of up to 12.1%, 28.3%, and 32.7% on head, recent, and tail topics respectively while maintaining naturalness, relevance, non-repetitiveness, and informational quality comparable to GPT-3. Overall, this paper presents WikiChat as an effective solution for enhancing the reliability and accuracy of LLM-based chatbots by grounding them with live information from Wikipedia. The proposed evaluation methodology sheds light on important aspects such as conversing about recent events and less popular topics that were previously overlooked.