# Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis

**Authors:** Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, Shujian Huang  
**arXiv:** [2304.04675](https://arxiv.org/abs/2304.04675)  
**Date:** 2023-04-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.04675v1/

## Key points

- Authors Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang focus on large language models (LLMs) in multilingual machine translation (MMT)
- Study addresses two key questions: 1) Effectiveness of LLMs in translating various languages; 2) Factors influencing LLM performance in translation tasks
- Evaluation of popular LLMs like XGLM, OPT, BLOOMZ, and ChatGPT across 102 languages reveals ChatGPT falls short compared to NLLB in 83.33% of translation directions
- Strong performance with contextual exemplars even with unconventional prompts
- Cross-lingual exemplars offer better guidance for low-resource translation tasks
- Overestimation of BLOOMZ's performance on dataset Flores-101 highlights risks of relying solely on public datasets for evaluation
- Research provides insights into how LLMs operate in MMT frameworks and identifies key factors influencing their performance

## Summary

In their paper titled "Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis," authors Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang delve into the realm of large language models (LLMs) and their application in multilingual machine translation (MMT). The study aims to explore the strengths and challenges of utilizing LLMs for MMT by addressing two key questions: 1) How effective are LLMs in translating a wide array of languages? 2) What factors influence the performance of LLMs in translation tasks? The researchers conduct a comprehensive evaluation of popular LLMs such as XGLM, OPT, BLOOMZ, and ChatGPT across 102 different languages. Surprisingly, despite the advancements in LLM technology, even the top-performing model ChatGPT falls short when compared to the supervised baseline NLLB in approximately 83.33% of translation directions. This discrepancy underscores the complexity and nuances involved in achieving high-quality multilingual translations using LLMs. Through meticulous analysis, the study uncovers novel operational patterns exhibited by LLMs when applied to MMT scenarios. Firstly, it is observed that prompt semantics can be disregarded to some extent when contextual exemplars are provided, leading to strong performance even with unconventional prompts. Secondly, cross-lingual exemplars prove to offer superior task guidance for low-resource translation tasks compared to exemplars within the same language pairs. Lastly, a notable finding highlights the overestimation of BLOOMZ's performance on dataset Flores-101, emphasizing potential risks associated with relying solely on public datasets for evaluation purposes. Overall, this research sheds light on both the promise and limitations of leveraging LLMs for multilingual machine translation. By unraveling new insights into how these models operate within MMT frameworks and identifying key influencing factors on their performance, this study contributes valuable knowledge towards enhancing the efficacy of multilingual translation systems powered by large language models.