# Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains

**Authors:** Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, Brian Thompson  
**arXiv:** [2402.18747](https://arxiv.org/abs/2402.18747)  
**Date:** 2024-02-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.18747v1/

## Key points

- Authors: Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, Brian Thompson
- Introduce a comprehensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain
- Primary objective: Assess robustness of fine-tuned machine translation (MT) metrics in face of domain shifts between training and inference
- Findings: Fine-tuned metrics show significant decline in performance in unseen domains compared to surface form or pre-trained metrics
- Challenges: Adapting MT metrics to new domains highlighted; importance of developing more resilient evaluation methods emphasized
- Complexity: Optimizing MT metrics for diverse domains discussed; need for continued research and development stressed

## Summary

In their study titled "Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains," authors Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, and Brian Thompson introduce a comprehensive multidimensional quality metrics (MQM) annotated dataset that covers 11 language pairs within the biomedical domain. The primary objective of their research is to assess the robustness of machine translation (MT) metrics that have been fine-tuned based on human-generated MT quality judgments when faced with domain shifts between training and inference. Through their analysis, the researchers discovered that fine-tuned metrics experienced a significant decline in performance when operating in unseen domains compared to metrics that rely solely on surface form or pre-trained metrics that do not undergo fine-tuning based on MT quality judgments. This finding underscores the challenges associated with adapting MT metrics to new domains and highlights the importance of developing more resilient evaluation methods for machine translation systems. The study sheds light on the complexities involved in optimizing MT metrics for diverse domains and emphasizes the need for continued research and development in this area to enhance the overall effectiveness and reliability of machine translation technologies across various linguistic contexts.