# Qwen2.5-1M Technical Report

**Authors:** An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu  
**arXiv:** [2501.15383](https://arxiv.org/abs/2501.15383)  
**Date:** 2025-01-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2501.15383v1/

## Key points

- Introduction of Qwen2.5-1M series, a significant advancement in long-context modeling
- Utilization of innovative techniques such as long data synthesis, progressive pre-training, and multi-stage supervised fine-tuning to enhance model capabilities
- Open-sourcing of inference framework with length extrapolation method for expanding context lengths without additional training
- Implementation of sparse attention methods and chunked prefill optimization for reduced inference costs while maintaining precision
- Optimizations in the inference engine including kernel optimization, pipeline parallelism, and scheduling optimization leading to improved performance
- Achieving 3x to 7x prefill speedup with the Qwen2.5-1M models in scenarios with 1 million tokens of context
- Outperformance of GPT-4o-mini by Qwen2.5-14B-Instruct-1M model in long-context tasks supporting contexts eight times longer

## Summary

We are proud to introduce the Qwen2.5-1M series - a groundbreaking advancement in long-context modeling. These models have extended context length to an impressive 1 million tokens and include Qwen2.5-7B-Instruct-1M, Qwen2.5-14B-Instruct-1M, and the API-accessed model Qwen2.5-Turbo. Our team has utilized innovative techniques such as long data synthesis, progressive pre-training, and multi-stage supervised fine-tuning to significantly enhance the long-context capabilities of these models. To promote wider adoption of long-context models, we have open-sourced our inference framework which includes a length extrapolation method that expands model context lengths by at least four times without additional training. In addition to this, we have implemented sparse attention methods and chunked prefill optimization for deployment scenarios - reducing inference costs while maintaining precision. Our optimizations in the inference engine - including kernel optimization, pipeline parallelism, and scheduling optimization - have resulted in significant improvements in overall inference performance. Leveraging our framework has enabled the Qwen2.5-1M models to achieve remarkable 3x to 7x prefill speedup in scenarios with 1 million tokens of context. Evaluations have shown that these models excel in long-context tasks without sacrificing performance in short-context scenarios. Notably, the Qwen2.5-14B-Instruct-1M model outperforms GPT-4o-mini in long-context tasks and supports contexts eight times longer. The collaborative efforts of An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu,
Fei Huang,
Haoyan Huang,
Jiandong Jiang,
Jianhong Tu,
Jianwei Zhang,
Jingren Zhou,
Junyang Lin,
Kai Dang,
Kexin Yang,
Le Yu,
Mei Li,
Minmin Sun,
Qin Zhu,
Rui Men,
Tao He
and Weijia Xu have led to the development of these cutting-edge models that redefine what is possible in long-context processing using open-source technology.