# How multilingual is Multilingual BERT?

**Authors:** Telmo Pires, Eva Schlinger, Dan Garrette  
**arXiv:** [1906.01502](https://arxiv.org/abs/1906.01502)  
**Date:** 2019-06-04  
**Summary page:** https://www.summarizepaper.com/en/paper/1906.01502v1/

## Key points

- Multilingual BERT (M-BERT) is a language model pre-trained in 104 languages
- M-BERT performs well in zero-shot cross-lingual model transfer
- Probing experiments reveal M-BERT's ability to transfer knowledge across languages with different scripts
- Transfer works best between typologically similar languages
- Monolingual corpora can train models for code-switching
- M-BERT can identify translation pairs
- Systematic deficiencies affect certain language pairs in M-BERT's multilingual representations
- The study provides insights into the strengths and limitations of M-BERT's multilingual capabilities
- The research highlights areas for further improvement in multilingual representation learning.

## Summary

In their paper titled "How multilingual is Multilingual BERT? ", Telmo Pires, Eva Schlinger, and Dan Garrette explore the capabilities of Multilingual BERT (M-BERT), a language model pre-trained from monolingual corpora in 104 languages. They find that M-BERT performs remarkably well in zero-shot cross-lingual model transfer, where task-specific annotations in one language are used to fine-tune the model for evaluation in another language. To understand the reasons behind this success, the authors conduct numerous probing experiments. Their findings reveal that M-BERT can effectively transfer knowledge even to languages with different scripts. Additionally, they observe that transfer works best between typologically similar languages and demonstrate that monolingual corpora can train models for code-switching. Furthermore, M-BERT is capable of identifying translation pairs. Based on these results, the authors conclude that M-BERT indeed creates multilingual representations; however, they also identify systematic deficiencies that affect certain language pairs. This research sheds light on the strengths and limitations of M-BERT's multilingual capabilities and provides valuable insights into how it can be utilized for cross-lingual tasks. Overall, this study highlights areas for further improvement in multilingual representation learning.