# Large language models effectively leverage document-level context for literary translation, but critical errors persist

**Authors:** Marzena Karpinska, Mohit Iyyer  
**arXiv:** [2304.03245](https://arxiv.org/abs/2304.03245)  
**Date:** 2023-04-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.03245v1/

## Key points

- Large language models (LLMs) are effective in translating paragraphs and documents
- Evaluating LLMs' performance on larger units of text is challenging due to cost and difficulty
- Authors conducted a human evaluation of Gpt-3.5 LLM's ability to translate literary paragraphs across 18 diverse languages
- Results show that discourse-level LLM translators commit fewer errors than sentence-level approaches
- Critical errors still exist, including occasional content omissions that require human intervention
- The evaluation took approximately 350 hours of effort for annotation and analysis
- The authors publicly release their dataset and error annotations to spur future research on evaluation of document-level literary translation

## Summary

In their paper titled "Large language models effectively leverage document-level context for literary translation, but critical errors persist," authors Marzena Karpinska and Mohit Iyyer explore the ability of large language models (LLMs) to translate paragraphs and documents. While LLMs have shown competitive performance on sentence-level translation datasets, evaluating their performance on larger units of text such as paragraphs and documents has been challenging due to the cost and difficulty involved in conducting such evaluations. The authors conduct a rigorous human evaluation of Gpt-3.5 (text-davinci-003) LLM's ability to translate an entire literary paragraph from novels across 18 linguistically-diverse language pairs including Japanese, Polish, and English. The evaluation involves hiring translators fluent in both the source and target languages to provide span-level error annotations as well as preference judgments of which system's translations are better. The results show that asking the LLM to translate an entire literary paragraph at once results in higher-quality translations than standard sentence-by-sentence translation approaches. Discourse-level LLM translators commit fewer mistranslations, grammar errors, and stylistic inconsistencies than sentence-level approaches. However, critical errors still abound, including occasional content omissions that require human translator intervention to ensure that the author's voice remains intact. The evaluation took approximately 350 hours of effort for annotation and analysis, and the authors publicly release their dataset and error annotations to spur future research on evaluation of document-level literary translation. Overall, this study highlights the potential of LLMs in translating larger units of text while also emphasizing the need for continued improvement in machine translation technology to minimize critical errors.