# PolyLM: An Open Source Polyglot Large Language Model

**Authors:** Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Binyuan Hui, Bowen Yu, Dayiheng Liu, Baosong Yang, Fei Huang, Jun Xie  
**arXiv:** [2307.06018](https://arxiv.org/abs/2307.06018)  
**Date:** 2023-07-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.06018v1/

## Key points

- PolyLM is a multilingual large language model (LLM) developed to address limitations of existing LLMs focused on high-resource languages like English.
- It has been trained on a massive dataset of 640 billion tokens and is available in two sizes: 1.7B and 13B.
- PolyLM integrates bilingual data into its training process and uses curriculum learning to gradually increase non-English data during pre-training.
- It introduces a novel multilingual self-instruct method that generates 132.7K multilingual instructions for fine-tuning the model.
- PolyLM outperforms other open-source models like LLaMA and BLOOM in multilingual tasks while maintaining comparable performance in English.
- Access to the PolyLM models, instruction data, and a multilingual benchmark can be found at [https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation](https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation).
- The research team includes Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu , Shangjie Li , Binyuan Hui , Bowen Yu Dayiheng Liu Baosong Yang Fei Huang Jun Xie.

## Summary

PolyLM is a multilingual large language model (LLM) that has been developed to address the limitations of existing LLMs, which are primarily focused on high-resource languages like English. This model has been trained on a massive dataset of 640 billion tokens and is available in two sizes: 1.7B and 13B. To improve its multilingual capabilities, PolyLM integrates bilingual data into its training process and adopts a curriculum learning strategy that gradually increases the proportion of non-English data during pre-training. Additionally, PolyLM introduces a novel multilingual self-instruct method that automatically generates 132.7K multilingual instructions for fine-tuning the model. To evaluate the performance of PolyLM, several existing multilingual tasks including understanding, question answering, generation, and translation were collected. Extensive experiments demonstrate that PolyLM outperforms other open-source models such as LLaMA and BLOOM in multilingual tasks while maintaining comparable performance in English. The authors provide access to the PolyLM models, along with the instruction data and a multilingual benchmark, at the following URL: [https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation](https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation). The team behind this research includes Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu , Shangjie Li , Binyuan Hui , Bowen Yu Dayiheng Liu Baosong Yang Fei Huang Jun Xie .