# Predicting Perfect Quality Segments in MT Output with Fine-Tuned OpenAI LLM: Is it possible to capture editing distance patterns from historical data?

**Authors:** Serge Gladkoff, Gleb Erofeev, Lifeng Han, Goran Nenadic  
**arXiv:** [2308.00158](https://arxiv.org/abs/2308.00158)  
**Date:** 2023-07-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.00158v1/

## Key points

- Study investigates potential of fine-tuning large language models (LLMs) for Translation Quality Estimation (TQE)
- TQE is crucial for assessing quality of machine translation (MT) and human translation (HT)
- Approach TQE as a binary classification task using English-Italian and English-German training data
- Fine-tuned ChatGPT achieves relatively high score in predicting translation quality, but room for improvement in accuracy
- Methodology involves collecting historical post-editing data and using OpenAI's API access to fine-tune ChatGPT model
- Contributes to understanding how LLMs can be utilized for TQE and highlights areas for further enhancement in predicting perfect quality segments in MT output

## Summary

This study investigates the potential of fine-tuning large language models (LLMs), specifically ChatGPT, for the task of Translation Quality Estimation (TQE). TQE is a crucial step in assessing the quality of machine translation (MT) and human translation (HT) without relying on reference translations. The researchers approach TQE as a binary classification task and use English-Italian and English-German training data to experiment with fine-tuned ChatGPT via its API. The results show that ChatGPT achieves a relatively high score in predicting translation quality, indicating whether a translation needs editing. However, there is still room for improvement in accuracy. The authors provide an overview of their methodology, which involves collecting historical post-editing data from past projects on ERP content translation in multiple languages. They then use OpenAI's API access to fine-tune the ChatGPT model for the classification task. Overall, this work contributes to understanding how LLMs can be utilized for TQE and highlights areas for further enhancement in predicting perfect quality segments in MT output.