# LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding

**Authors:** Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei  
**arXiv:** [2104.08836](https://arxiv.org/abs/2104.08836)  
**Date:** 2021-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.08836v1/

## Key points

- Multimodal pre-training has advanced visually-rich document understanding (VrDU)
- Joint learning across different modalities has potential for achieving state-of-the-art performance on VrDU tasks
- Microsoft Research Asia and Microsoft Azure AI have introduced LayoutXLM, a multilingual pre-trained model for VrDU in different languages
- LayoutXLM is evaluated using XFUN, a new multilingual form understanding benchmark dataset with key-value labeled forms in seven languages
- Experiment results show that LayoutXLM outperforms existing cross-lingual pre-trained models on the XFUN dataset
- The pre-trained LayoutXLM model and XFUN dataset are publicly available to advance research in document understanding
- Other related works discussed include mBART and mT5 based on contrastive learning and multilingual variants of T5 respectively
- Future research plans include enlarging the multilingual training data to cover more languages as well as more document layouts and templates, and investigating how to leverage the co-occurrence of business documents with similar content but in different languages to improve accuracy and performance.

## Summary

The field of visually-rich document understanding (VrDU) has seen significant advancements in recent times with the use of multimodal pre-training that incorporates text, layout, and image information. This joint learning across different modalities has demonstrated great potential for achieving state-of-the-art (SOTA) performance on VrDU tasks. To further bridge the language barriers for VrDU, Microsoft Research Asia and Microsoft Azure AI have introduced LayoutXLM, a multilingual pre-trained model that aims to accurately understand visually-rich documents in different languages. The LayoutXLM model is evaluated using a new multilingual form understanding benchmark dataset named XFUN, which includes key-value labeled forms in seven languages (Chinese, Japanese, Spanish, French, Italian, German, Portuguese). Experiment results show that LayoutXLM significantly outperforms existing SOTA cross-lingual pre-trained models on the XFUN dataset. The pre-trained LayoutXLM model and XFUN dataset are publicly available to advance research in document understanding. The paper also discusses other related works such as mBART and mT5 that are based on contrastive learning and multilingual variants of T5 respectively. For future research, the authors plan to enlarge the multilingual training data to cover more languages as well as more document layouts and templates. They also aim to investigate how to leverage the co-occurrence of business documents with similar content but in different languages in order to improve accuracy and performance.