# LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

**Authors:** Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, Mao Yang  
**arXiv:** [2402.13753](https://arxiv.org/abs/2402.13753)  
**Date:** 2024-02-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.13753v1/

## Key points

- LongRoPE is a groundbreaking advancement in large language models (LLMs) that extends the context window to an impressive 2048k tokens.
- The extension is achieved with minimal fine-tuning steps, only up to 1k within 256k training lengths, while maintaining performance levels comparable to shorter context windows.
- Three key innovations of LongRoPE include identifying and leveraging non-uniformities in positional interpolation, introducing a progressive extension strategy, and readjusting on an 8k length to restore performance levels associated with shorter context windows.
- Extensive experiments on LLaMA2 and Mistral show the effectiveness of LongRoPE in enhancing language understanding and model capabilities.

## Summary

LongRoPE is a groundbreaking advancement in the realm of large language models (LLMs), aiming to address the limitations imposed by current extended context windows. The significance of a large context window in LLMs cannot be overstated, as it allows for a more comprehensive understanding of language and improves model performance. However, existing models are constrained by factors such as high fine-tuning costs, limited availability of long texts, and issues arising from new token positions. In this paper authored by Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang,<br>
LongRoPE introduces a novel approach that pushes the boundaries of context window extension. Unlike previous models capped at around 128k tokens,<br>
LongRoPE extends the context window to an impressive 2048k tokens. This extension is achieved with minimal fine-tuning steps—only up to 1k within 256k training lengths—while still maintaining performance levels comparable to those achieved with shorter context windows. The success of LongRoPE hinges on three key innovations. Firstly,<br>
the authors identify and leverage two forms of non-uniformities in positional interpolation through an efficient search process.<br>
This not only provides a better initialization for fine-tuning but also enables an eightfold extension in scenarios where fine-tuning is not required.<br>
Secondly,<br>
they introduce a progressive extension strategy wherein a 256k length LLM is first fine-tuned before undergoing a second positional interpolation to reach the desired 2048k context window.<br>
Lastly,<br>
they readjust LongRoPE on an 8k length to restore performance levels associated with shorter context windows. Extensive experiments conducted on LLaMA2 and Mistral across various tasks showcase the effectiveness of LongRoPE. Models extended using this approach retain their original architecture with minor adjustments made to positional embedding while being able to capitalize on pre-existing optimizations. In conclusion, LongRoPE represents a significant leap forward in extending LLM context windows beyond previous limitations. By enabling models to process information from up to 2 million tokens while maintaining performance standards and minimizing fine-tuning requirements, this innovative approach opens up new possibilities for enhancing language understanding and model capabilities in diverse applications.