Predicting Perfect Quality Segments in MT Output with Fine-Tuned OpenAI LLM: Is it possible to capture editing distance patterns from historical data?
AI-generated Key Points
- Study investigates potential of fine-tuning large language models (LLMs) for Translation Quality Estimation (TQE)
- TQE is crucial for assessing quality of machine translation (MT) and human translation (HT)
- Approach TQE as a binary classification task using English-Italian and English-German training data
- Fine-tuned ChatGPT achieves relatively high score in predicting translation quality, but room for improvement in accuracy
- Methodology involves collecting historical post-editing data and using OpenAI's API access to fine-tune ChatGPT model
- Contributes to understanding how LLMs can be utilized for TQE and highlights areas for further enhancement in predicting perfect quality segments in MT output
Authors: Serge Gladkoff, Gleb Erofeev, Lifeng Han, Goran Nenadic
Abstract: Translation Quality Estimation (TQE) is an important step before deploying the output translation into usage. TQE is also critical in assessing machine translation (MT) and human translation (HT) quality without seeing the reference translations. In this work, we examine if the state-of-the-art large language models (LLMs) can be fine-tuned for the TQE task and their capability. We take ChatGPT as one example and approach TQE as a binary classification task. Using English-Italian and English-German training corpus, our experimental results show that fine-tuned ChatGPT via its API can achieve a relatively high score on predicting translation quality, i.e. if the translation needs to be edited, but there is definitely space to improve the accuracy. English-Italiano bilingual Abstract is available in the paper.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
Welcome to our AI assistant! Here are some important things to keep in mind:
- The assistant will only answer questions related to this specific paper.
- Please note that this is not a bot for casual chatting.
- If you want to keep the history of your questions/answers you should create an account.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Similar papers summarized with our AI tools
Navigate through even more similar papers through atree representation
Look for similar papers (in beta version)