# MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer

**Authors:** Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, Sebastian Ruder  
**arXiv:** [2005.00052](https://arxiv.org/abs/2005.00052)  
**Date:** 2020-04-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2005.00052v1/

## Key points

- State-of-the-art pretrained multilingual models aim to facilitate NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer
- These models have limited capacity, resulting in weak transfer performance for low-resource and unseen languages during pretraining
- MAD-X is an adapter-based framework that enables high portability and parameter efficient transfer to any task and language by learning modular language and task representations
- MAD-X introduces an innovative invertible adapter architecture and a strong baseline method for adapting pretrained multilingual models to new languages
- In experiments, MAD-X outperforms existing methods in cross-lingual transfer for named entity recognition across diverse typologically varied languages
- MAD-X also achieves competitive results for question answering tasks
- The proposed framework significantly improves the ability of pretrained multilingual models to handle cross-lingual transfer tasks effectively
- MAD-X offers a promising solution for enhancing the performance of pretrained multilingual models on low resource and previously unseen languages during pretraining
- By enabling efficient transfer to arbitrary tasks and languages, MAD-X has the potential to advance NLP applications in various linguistic contexts.

## Summary

The main goal of state-of-the-art pretrained multilingual models like multilingual BERT and XLM-R is to facilitate NLP applications in low-resource languages through zero-shot or few-shot cross-lingual transfer. However, these models have limited capacity, resulting in weak transfer performance for low-resource languages and unseen languages during pretraining. To address this issue, the authors propose MAD-X, an adapter-based framework that enables high portability and parameter efficient transfer to any task and language by learning modular language and task representations. The framework also introduces an innovative invertible adapter architecture and a strong baseline method for adapting pretrained multilingual models to new languages. In their experiments, the authors demonstrate that MAD-X outperforms existing methods in cross-lingual transfer across a diverse set of typologically varied languages for named entity recognition. Additionally, MAD-X achieves competitive results for question answering tasks. This indicates that the proposed framework significantly improves the ability of pretrained multilingual models to handle cross-lingual transfer tasks effectively. Overall, MAD-X offers a promising solution for enhancing the performance of pretrained multilingual models on low resource languages and previously unseen languages during pretraining. By enabling efficient transfer to arbitrary tasks and languages, MAD-X has the potential to advance NLP applications in various linguistic contexts.