# Identifying Necessary Elements for BERT's Multilinguality

**Authors:** Philipp Dufter, Hinrich Schütze  
**arXiv:** [2005.00396](https://arxiv.org/abs/2005.00396)  
**Date:** 2020-05-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2005.00396v3/

## Key points

- Authors explore multilingual BERT (mBERT) and its ability to generate high-quality multilingual representations without crosslingual signal during training
- Aim to identify architectural properties of BERT and linguistic properties of languages essential for enabling multilinguality in BERT
- Proposed setup using small BERT models trained on a combination of synthetic and natural data
- Four architectural elements and two linguistic elements influencing the multilinguality of BERT discovered
- Experimented with modified masking strategy using VecMap in a multilingual pretraining setup
- Experiments on XNLI with three languages conducted to evaluate findings
- Results show identified elements transfer from small-scale setup to larger-scale settings
- Study provides insights into how mBERT generates high-quality multilingual representations and enables effective zero-shot transfer
- Research contributes to advancing understanding of how BERT becomes multilingual.

## Summary

In their paper titled "Identifying Necessary Elements for BERT's Multilinguality," authors Philipp Dufter and Hinrich Schütze explore the phenomenon of multilingual BERT (mBERT) and its ability to generate high-quality multilingual representations without using any crosslingual signal during training. Previous research has investigated this surprising characteristic; however, the reasons behind mBERT's multilinguality remain unclear. To understand this topic better, the authors aim to identify the architectural properties of BERT and linguistic properties of languages that are essential for enabling multilinguality in BERT. To facilitate efficient experimentation, they propose a setup using small BERT models trained on a combination of synthetic and natural data. Through their investigation, four architectural elements and two linguistic elements that influence the multilinguality of BERT are discovered. Building upon these insights, they experiment with a modified masking strategy using VecMap - an unsupervised embedding alignment technique - in a multilingual pretraining setup. To evaluate the effectiveness of their findings, experiments on XNLI with three languages are conducted. The results show that their identified elements for achieving multilinguality transfer from their small-scale setup to larger-scale settings. Overall, this study provides valuable insights into understanding how mBERT is able to generate high-quality multilingual representations and enable effective zero-shot transfer. By identifying key architectural and linguistic elements that contribute to mBERT's ability to become multilingual, this research contributes to advancing our understanding of how BERT becomes multilingual.