# Towards Explainable Evaluation Metrics for Machine Translation

**Authors:** Christoph Leiter, Piyawat Lertvittayakumjorn, Marina Fomicheva, Wei Zhao, Yang Gao, Steffen Eger  
**arXiv:** [2306.13041](https://arxiv.org/abs/2306.13041)  
**Date:** 2023-06-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.13041v1/

## Key points

- Shift from traditional lexical overlap metrics like BLEU to newer black-box models such as COMET and BERTScore
- Preference for classical metrics due to transparent decision-making processes
- Importance of explainability in promoting adoption of high-quality metrics
- Taxonomy of previous efforts in explainable MT evaluation showcasing various methods and techniques used in the field
- Discussion on underexplored research directions and potential future paths for improving MT evaluation through explainability
- Utilization of large language models like ChatGPT and GPT4 in the context of explainable MT metrics
- Guidance provided for researchers and metric developers to enhance understanding and explanation of MT metrics
- Envisioned benefits including improved MT metrics, translation selection, and semi-automatic labeling
- Background information on machine translation evaluation metrics explaining key dimensions such as input type, granularity, quality aspect, learning objective, etc.
- Comprehensive guide offering insights into current trends, future possibilities, and potential advancements in the field

## Summary

This paper delves into the realm of explainable machine translation (MT) metrics, highlighting the shift from traditional lexical overlap metrics like BLEU to newer black-box models such as COMET and BERTScore. While these newer models show strong correlations with human judgments, there is still a preference for classical metrics due to their transparent decision-making processes. To address this disparity and promote the adoption of high-quality metrics, the concept of explainability becomes paramount. The paper outlines a taxonomy of previous efforts in explainable MT evaluation, showcasing various methods and techniques used in the field. Additionally, it discusses underexplored research directions and potential future paths for improving MT evaluation through explainability. The utilization of large language models like ChatGPT and GPT4 is also explored in this context. Furthermore, the authors aim to solidify the field of explainable MT metrics by providing guidance for researchers and metric developers looking to enhance their understanding and explanation of MT metrics. They envision that this work will not only lead to improved MT metrics but also benefit other applications such as translation selection and semi-automatic labeling. The paper also provides background information on machine translation evaluation metrics, explaining key dimensions such as input type, granularity, quality aspect, learning objective, etc. This foundational knowledge sets the stage for a comprehensive understanding of explainable MT metrics. Overall, this paper serves as a comprehensive guide to explainable machine translation metrics , offering insights into current trends , future possibilities , and potential advancements in the field. By emphasizing transparency and clarity in evaluating machine translations the authors hope to contribute to better  MT systems in the future.