# Evaluating Hallucinations in Chinese Large Language Models

**Authors:** Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, Xipeng Qiu  
**arXiv:** [2310.03368](https://arxiv.org/abs/2310.03368)  
**Date:** 2023-10-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.03368v4/

## Key points

- Authors introduce HalluQA benchmark to assess hallucinations in Chinese large language models
- Two main types of hallucinations defined: imitative falsehoods and factual errors
- Adversarial samples created using GLM-130B and ChatGPT
- Automated evaluation method developed using GPT-4 to detect model hallucinations
- Study focuses on 450 adversarial questions covering various domains and Chinese culture
- Only 18 out of 24 models achieve non-hallucination rates lower than 50%
- Analysis of primary types of hallucinations observed in different models and their causes
- Discussion on prioritizing types of hallucinations based on model characteristics
- HalluQA benchmark provides valuable tool for evaluating prevalence of hallucinations in Chinese large language models

## Summary

, , , , 

In the paper titled "Evaluating Hallucinations in Chinese Large Language Models," authors Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu introduce a benchmark called HalluQA (Chinese Hallucination Question-Answering) to assess the phenomenon of hallucinations in Chinese large language models. This comprehensive evaluation sheds light on the prevalence of hallucinations in Chinese large language models and provides valuable insights for improving their performance and reliability. The authors first define two main types of hallucinations: imitative falsehoods and factual errors. To create adversarial samples for evaluation purposes, they leverage GLM-130B and ChatGPT. They also develop an automated evaluation method using GPT-4 to determine whether a model's output is hallucinated. The study focuses on 450 carefully crafted adversarial questions that cover various domains while considering Chinese historical culture, customs, and social phenomena. The research team conducts extensive experiments involving 24 large language models such as ERNIE-Bot, Baichuan2, ChatGLM, Qwen, SparkDesk among others. Surprisingly, only 18 out of the 24 models achieve non-hallucination rates lower than 50%, indicating the high level of challenge presented by HalluQA. Furthermore,the authors analyze the primary types of hallucinations observed in different models and delve into their underlying causes. They also discuss which types of hallucinations should be given priority based on the specific characteristics of each model. In conclusion, the HalluQA benchmark provides a valuable tool for evaluating the prevalence of hallucinations in Chinese large language models. The results of this study highlight the need for further improvements and developments in these models to ensure their reliability and accuracy.