# G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment

**Authors:** Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, Chenguang Zhu  
**arXiv:** [2303.16634](https://arxiv.org/abs/2303.16634)  
**Date:** 2023-03-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.16634v3/

## Key points

- The paper addresses the challenge of automatically measuring the quality of texts generated by NLG systems.
- Conventional reference-based metrics like BLEU and ROUGE have low correlation with human judgments for tasks requiring creativity and diversity.
- Recent studies propose using large language models (LLMs) as reference-free metrics for NLG evaluation.
- LLM-based evaluators still exhibit lower human correspondence compared to medium-size neural evaluators.
- G-Eval is a framework that utilizes large language models with chain-of-thoughts (CoT) and a form-filling paradigm to assess NLG outputs.
- G-Eval achieves a Spearman correlation of 0.514 with human judgments in text summarization when leveraging GPT-4 as the backbone model, outperforming previous methods significantly.
- LLM-based evaluators may exhibit bias towards LLM-generated texts, highlighting the importance of diverse evaluation methods.
- The code for G-Eval is available on GitHub at https://github.com/nlpyang/geval.

## Summary

The paper titled "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" addresses the challenge of automatically measuring the quality of texts generated by natural language generation (NLG) systems. Conventional reference-based metrics like BLEU and ROUGE have shown low correlation with human judgments, particularly for tasks that require creativity and diversity. To overcome this limitation, recent studies propose using large language models (LLMs) as reference-free metrics for NLG evaluation, which can be applied to new tasks lacking human references. However, LLM-based evaluators still exhibit lower human correspondence compared to medium-size neural evaluators. In response to this issue, the authors present G-Eval, a framework that utilizes large language models with chain-of-thoughts (CoT) and a form-filling paradigm to assess the quality of NLG outputs. The framework is tested on two generation tasks: text summarization and dialogue generation. The results demonstrate that G-Eval achieves a Spearman correlation of 0.514 with human judgments in the text summarization task when leveraging GPT-4 as the backbone model. This outperforms previous methods by a significant margin. Furthermore, the authors conduct preliminary analysis on the behavior of LLM-based evaluators and highlight a potential issue where these evaluators may exhibit bias towards LLM-generated texts. This highlights the importance of using diverse evaluation methods in order to accurately measure the quality of NLG outputs. Overall, this work presents an improved approach for evaluating NLG outputs using large language models and provides insights into their performance compared to other evaluation methods. The code for G-Eval is available on GitHub at https://github.com/nlpyang/geval.