# ChatGPT as a Factual Inconsistency Evaluator for Abstractive Text Summarization

**Authors:** Zheheng Luo, Qianqian Xie, Sophia Ananiadou  
**arXiv:** [2303.15621](https://arxiv.org/abs/2303.15621)  
**Date:** 2023-03-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.15621v1/

## Key points

- Performance of abstractive text summarization has improved with pre-trained language models
- Major challenge: factual inconsistency in generated summaries
- Efforts to develop factuality evaluation metrics have limitations (e.g., high computational complexity, reliance on annotated data)
- Study evaluates ChatGPT's ability to assess factual inconsistency under zero-shot setting
- Tasks include binary natural language inference (NLI), summary ranking, and consistency rating at different levels
- ChatGPT outperforms previous metrics on 6 out of 9 datasets for assessing factual inconsistency
- Limitations: evaluation bias, wrong reasoning, hallucination in ChatGPT's outputs
- Existing factuality evaluation metrics categorized into unsupervised and semi-supervised methods (e.g., IE, NLI-based methods, QA-based methods)
- Natural language inference explored for entity-level factuality consistency in generated summaries
- Further research needed to address limitations and enhance accuracy in evaluating factual consistency

## Summary

, , , , 

In recent years, the performance of abstractive text summarization has significantly improved with the emergence of pre-trained language models. However, a major challenge faced by existing methods is factual inconsistency in generated summaries. Efforts have been made to develop effective factuality evaluation metrics based on natural language inference and question answering, but they have limitations such as high computational complexity and reliance on annotated data. This study focuses on evaluating ChatGPT's ability to assess factual inconsistency under the zero-shot setting. The tasks include binary natural language inference (NLI), summary ranking, and consistency rating at coarse-grained and fine-grained levels. Results show that ChatGPT outperforms previous state-of-the-art metrics on 6 out of 9 datasets, highlighting its potential for effectively assessing factual inconsistency in a zero-shot setting. However, there are still limitations to be addressed, such as evaluation bias, wrong reasoning, and hallucination in ChatGPT's outputs. Existing factuality evaluation metrics can be categorized into unsupervised and semi-supervised methods, including information extraction (IE), NLI-based methods, and question answering (QA) based methods. Natural language inference has also been explored for factuality evaluation by assessing entity-level factuality consistency in generated summaries. While advancements in pre-trained language models like ChatGPT show promise for improving factuality evaluation in abstractive summarization, further research is needed to address existing limitations and enhance accuracy in evaluating factual consistency.