# Large Language Models Are State-of-the-Art Evaluators of Translation Quality

**Authors:** Tom Kocmi, Christian Federmann  
**arXiv:** [2302.14520](https://arxiv.org/abs/2302.14520)  
**Date:** 2023-02-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.14520v1/

## Key points

- Introducing GEMBA, a metric for assessing translation quality using GPT-based models
- Evaluation of the metric through zero-shot prompting with four prompt variants and two modes (with and without reference translation)
- Effective performance of the method with GPT 3.5 and larger models
- State-of-the-art accuracy compared to MQM-based human labels in English-German, English-Russian, and Chinese-English translation pairs
- Potential of pre-trained large language models for assessing translation quality
- Release of code, prompt templates, and scoring results for external validation and reproducibility
- Future research plans include exploring few-shot learning, model fine-tuning, error-based evaluation or post-editing efforts as areas for improvement
- Possibility of using GPT-enhanced metrics for document-level evaluation due to their ability to handle larger context windows

## Summary

The paper introduces GEMBA, a metric for assessing translation quality using GPT-based models. The authors evaluate the metric through zero-shot prompting, comparing four prompt variants in two modes: with and without a reference translation. They investigate seven versions of GPT models, including ChatGPT, and find that their method only works effectively with GPT 3.5 and larger models. Comparing their results to the WMT22 Metrics shared task, they achieve state-of-the-art accuracy in both modes when compared to MQM-based human labels for three language pairs: English into German, English into Russian, and Chinese into English. This study highlights the potential of pre-trained large language models for assessing translation quality. The authors release their code, prompt templates, and scoring results for external validation and reproducibility. In conclusion, the authors plan to continue researching the application of GPT models for quality assessment by exploring few-shot learning and model fine-tuning to improve accuracy. They also suggest modifying prompts to support error-based evaluation or post-editing efforts as potential areas for improvement. Additionally, they mention the possibility of using GPT-enhanced metrics for document-level evaluation due to their ability to handle larger context windows which is an area that currently lacks sufficient research.