# YaRN: Efficient Context Window Extension of Large Language Models

**Authors:** Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole  
**arXiv:** [2309.00071](https://arxiv.org/abs/2309.00071)  
**Date:** 2023-08-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.00071v2/

## Key points

- YaRN introduced as a novel method for extending the context window of transformer-based language models
- Efficiency of YaRN highlighted by achieving extension with fewer tokens and training steps compared to existing techniques
- Study evaluates YaRN at different scale factors (s = 16, 32) and compares its performance against other open-source models fine-tuned from Llama-2
- Results show successful extension of effective context size up to 128k, surpassing previous methods
- YaRN (s = 32) models exhibit decreasing perplexity even at 128k despite being fine-tuned on a smaller dataset, demonstrating robust generalization capabilities
- Evaluations on untruncated GovReport documents confirm strong performance on long sequences without dynamic scaling

## Summary

In their paper "YaRN: Efficient Context Window Extension of Large Language Models," the authors introduce YaRN as a novel method for extending the context window of transformer-based language models. This allows these models to effectively extrapolate to longer context lengths and outperform previous state-of-the-art methods. The efficiency of YaRN is highlighted by its ability to achieve this extension with significantly fewer tokens and training steps compared to existing techniques. The study evaluates YaRN at different scale factors (s = 16, 32) and compares its performance against other open-source models fine-tuned from Llama-2 and extended to over 32k context windows. Results show that YaRN interpolation successfully extends the effective context size of Llama-2 up to 128k, surpassing previous methods. Notably, the YaRN (s = 32) models exhibit decreasing perplexity even at 128k despite being fine-tuned on a smaller dataset of only 64k tokens, demonstrating robust generalization capabilities. Additionally, evaluations on untruncated GovReport documents confirm that fine-tuning with YaRN yields strong performance on long sequences without dynamic scaling. Overall, these results showcase the effectiveness and efficiency of YaRN in extending the context window of large language models for improved performance on tasks requiring longer contextual information.