# Yi: Open Foundation Models by 01.AI

**Authors:** 01. AI,  :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie  
**arXiv:** [2403.04652](https://arxiv.org/abs/2403.04652)  
**Date:** 2024-03-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.04652v1/

## Key points

- The Yi base models consist of a series of language and multimodal models with impressive multi-dimensional capabilities.
- These models are built upon 6B and 34B pretrained language models and have been extended to include various specialized models such as chat models, long context models, depth-upscaled models, and vision-language models.
- The performance of these base models has been exceptional on benchmarks like MMLU, with finetuned chat models achieving high human preference rates on evaluation platforms like AlpacaEval and Chatbot Arena.
- High-quality data is attributed to the success of these models, achieved through constructing a massive corpus of 3.1 trillion tokens in English and Chinese for pretraining purposes using cascaded data deduplication and quality filtering pipeline.
- Continual pretraining techniques have been utilized to extend context length to 200K, align visual representations with the semantic space of the language model, and enhance performance by extending the depth of pretrained checkpoints.
- Scaling up model parameters using continual pretraining will lead to even stronger frontier models in the future.
- Post-training methods are being explored to further improve capabilities in long-context modeling, visual understanding, and depth upscaling for Yi base models.
- A meticulous cascaded data-processing pipeline was designed to ensure high-quality bilingual pretraining data for Yi models by removing low-quality text content from web documents sourced from Common Crawl.

## Summary

The , consisting of a series of language and multimodal models, has demonstrated impressive multi-dimensional capabilities. These models are built upon 6B and 34B pretrained language models and have been extended to include chat models, 200K long context models, depth-upscaled models, and vision-language models. The performance of these base models has been exceptional on various benchmarks such as MMLU. In fact, the finetuned chat models have achieved high human preference rates on evaluation platforms like AlpacaEval and Chatbot Arena. This success can be attributed to the high-quality data that is a result of extensive . To construct this massive corpus of 3.1 trillion tokens in English and Chinese for pretraining purposes, a cascaded data deduplication and quality filtering pipeline was used. During finetuning, a small-scale instruction dataset was meticulously polished by machine learning engineers through multiple iterations. In terms of , the chat language model was combined with a vision transformer encoder to align visual representations with the semantic space of the language model. Additionally, through lightweight continual pretraining, the context length was extended to 200K resulting in strong needle-in-a-haystack retrieval performance. Continual pretraining also proved beneficial in enhancing performance by extending the depth of pretrained checkpoints. Moving forward, it is evident that scaling up model parameters using  will lead to even stronger frontier models. To further improve the capabilities of Yi base models in long-context modeling, visual understanding, and depth upscaling post-training methods are being explored. To ensure high-quality bilingual pretraining data for Yi models, a cascaded data-processing pipeline was meticulously designed. This involved heuristic rule filters to remove low-quality text content from web documents sourced from Common Crawl. This process guarantees that only relevant and diverse data is included in the training corpus, ensuring the best possible results for Yi models. Overall, our research showcases the potential for further advancements in language and multimodal modeling by leveraging scalable infrastructure and optimized data processing techniques.