# Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

**Authors:** Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, Maosong Sun  
**arXiv:** [2308.12038](https://arxiv.org/abs/2308.12038)  
**Date:** 2023-08-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.12038v1/

## Key points

- Significant surge in multimodal learning in image-to-text and text-to-image generation
- Limitations of progress to English language due to lack of large-scale image-text data in other languages
- Introduction of MPM training paradigm for training large multimodal models in low-resource languages
- Leveraging multilingual language models for zero-shot multimodal learning across languages
- Success of MPM demonstrated through VisCPM models achieving state-of-the-art performance in Chinese
- Contributions of team members crucial in designing, collecting data, and implementing training codebases for MPM and VisCPM
- Importance of initiatives like MPM in bridging the gap between English-centric advancements and non-English languages

## Summary

, , , , 

In recent years, there has been a significant surge in multimodal learning, particularly in the realms of image-to-text and text-to-image generation. This progress has predominantly been limited to the English language, leaving other languages lagging behind due to the lack of large-scale, high-quality image-text data. To address this challenge, a team of researchers proposed MPM - an innovative training paradigm designed to facilitate the training of large multimodal models in low-resource languages. MPM leverages the power of multilingual language models to pivot zero-shot multimodal learning across languages. By utilizing a strong multilingual large language model as a foundation, multimodal models pretrained on English-only image-text data can effectively generalize to other languages in a zero-shot manner for both image-to-text and text-to-image generation. Surprisingly, these models even surpass those trained on native language image-text data. To demonstrate the effectiveness of MPM, the researchers focused on Chinese as a case study and developed large multimodal models known as VisCPM for image-to-text and text-to-image generation. These models achieved state-of-the-art performance in Chinese and have been made available as open-source resources for future research. The contributions of various team members were instrumental throughout the project. From designing the model architecture to collecting extensive multimodal datasets for pretraining and implementing training codebases, each member played a crucial role in ensuring the success of MPM and VisCPM. Additionally, efforts were made towards evaluating the models' performance through both automatic and human evaluations. Overall, with advancements in powerful multimodal models like GPT-4 and Stable Diffusion reshaping the landscape of AI towards achieving Artificial General Intelligence (AGI), initiatives like MPM are vital in bridging the gap between English-centric advancements and non-English languages. By enabling zero-shot transfer learning across languages for multimodal tasks, MPM opens up new possibilities for advancing AI capabilities globally.