# mT5: A massively multilingual pre-trained text-to-text transformer

**Authors:** Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel  
**arXiv:** [2010.11934](https://arxiv.org/abs/2010.11934)  
**Date:** 2020-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.11934v3/

## Key points

- mT5 is introduced as a multilingual variant of the T5 model, pre-trained on a Common Crawl-based dataset covering 101 languages.
- The authors detail the design and modified training process of mT5 in their study.
- mT5 demonstrates exceptional performance on various multilingual benchmarks, establishing itself as a cutting-edge model.
- The paper addresses the issue of "accidental translation" in the zero-shot setting and proposes an effective technique to prevent such errors.
- All code and model checkpoints used in the research are publicly available for transparency and reproducibility.

## Summary

, , , , 

In their paper titled "mT5: A massively multilingual pre-trained text-to-text transformer," authors Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel introduce mT5 as a multilingual variant of the "Text-to-Text Transfer Transformer" (T5). The T5 model had previously achieved state-of-the-art results on various English-language NLP tasks by leveraging a unified text-to-text format and scale. In contrast, mT5 was pre-trained on a new Common Crawl-based dataset that covers an impressive 101 languages. The authors delve into the design and modified training process of mT5 in detail within their study. They showcase the exceptional performance of mT5 on numerous multilingual benchmarks, solidifying its position as a cutting-edge model in the field. Additionally, the paper addresses a crucial issue known as "accidental translation" in the zero-shot setting. This phenomenon occurs when a generative model mistakenly translates its prediction into an unintended language partially. The authors propose a simple yet effective technique to prevent such errors from occurring. Furthermore, it is highlighted that all code and model checkpoints utilized in this research are made publicly available for transparency and reproducibility purposes.