# Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains

**Authors:** Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, Brian Thompson  
**arXiv:** [2402.18747](https://arxiv.org/abs/2402.18747)  
**Date:** 2024-02-28  
**Summary page:** https://www.summarizepaper.com/fr/paper/2402.18747v1/

## Abstract

We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to metrics that rely on the surface form, as well as pre-trained metrics which are not fine-tuned on MT quality judgments.