# XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

**Authors:** Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, Madian Khabsa  
**arXiv:** [2301.10472](https://arxiv.org/abs/2301.10472)  
**Date:** 2023-01-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.10472v2/

## Key points

- Large multilingual language models, such as XLM-R, have a single vocabulary shared across more than 100 languages.
- The vocabulary size has not kept up with the growth in model size and complexity, creating a "vocabulary bottleneck."
- The authors propose a new approach called XLM-V to overcome this limitation by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity for each individual language.
- XLM-V is a multilingual language model with a one million token vocabulary that outperforms XLM-R on various tasks including natural language inference, question answering, and named entity recognition.
- XLM-V performs exceptionally well on low-resource language tasks and shows significant improvements compared to XLM-R.
- The paper introduces the concept of average log probability (ALP) to evaluate the ability of a vocabulary to represent a particular language.
- A greedy algorithm is proposed to determine the desired vocabulary capacity for individual languages based on ALP.
- The authors train individual monolingual sentencepiece models for each language using the Unigram Language Model algorithm and cluster them using K-Means clustering to construct multilingual vocabularies.
- Vocabulary capacities are assigned to each cluster based on ALP, resulting in per-cluster vocabularies.
- This research presents an innovative approach for scaling multilingual vocabularies and demonstrates improved performance compared to existing models like XLM-R.
- The proposed methodology for vocabulary allocation provides a systematic way to optimize vocabulary capacity for individual languages.

## Summary

Large multilingual language models, such as XLM-R, typically rely on a single vocabulary shared across more than 100 languages. However, as these models have grown in size and complexity, the vocabulary size has remained largely unchanged. This creates a "vocabulary bottleneck" that limits the representational capabilities of these models. In this paper, the authors propose a new approach to overcome this limitation by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to ensure sufficient coverage for each individual language. The authors introduce XLM-V, a multilingual language model with a one million token vocabulary. They demonstrate that XLM-V outperforms XLM-R on various tasks including natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and named entity recognition (WikiAnn). Notably, XLM-V performs exceptionally well on low-resource language tasks and achieves an absolute improvement of 11.2% and 5.8% on MasakhaNER and Americas NLI respectively compared to XLM-R. The paper also discusses the issue of vocabulary allocation and introduces the concept of average log probability (ALP) to evaluate the ability of a vocabulary to represent a particular language. The authors propose a greedy algorithm to determine the desired vocabulary capacity for individual languages in the multilingual vocabulary based on ALP. To construct the multilingual vocabularies, the authors train individual monolingual sentencepiece models for each language using the Unigram Language Model algorithm. They then use per-language vocabularies to construct lexical representation vectors and cluster them using K-Means clustering. Vocabulary capacities are assigned to each cluster based on ALP, resulting in per-cluster vocabularies. Overall, this paper presents an innovative approach for scaling multilingual vocabularies and demonstrates its effectiveness through improved performance on various tasks compared to existing models like XLM-R. The proposed methodology for vocabulary allocation provides a systematic way to optimize vocabulary capacity for individual languages. This research has significant implications for improving the representational capabilities of multilingual language models and enhancing their performance on diverse linguistic tasks.