# YaRN: Efficient Context Window Extension of Large Language Models

**Authors:** Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole  
**arXiv:** [2309.00071](https://arxiv.org/abs/2309.00071)  
**Date:** 2023-08-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.00071v1/

## Key points

- YaRN is a compute-efficient approach to extend the context window of transformer-based language models.
- Large Language Models (LLMs) have strong performance in in-context learning tasks but struggle to generalize beyond their trained sequence length.
- YaRN allows LLMs to effectively utilize and extrapolate to longer context lengths than their original pre-training would allow.
- YaRN surpasses state-of-the-art methods for context window extension, using 10x fewer tokens and 2.5x fewer training steps.
- The authors demonstrate the effectiveness of YaRN by fine-tuning Llama 2 7B/13B models with extended context windows of 64k and 128k using YaRN.
- Checkpoints for these models are provided on GitHub.
- YaRN can extrapolate beyond the limited context of a fine-tuning dataset, enhancing its generalization abilities.
- It enables leveraging longer-context data for superior performance in various NLP tasks such as summarization.
- Overall, YaRN presents an efficient solution for extending the context window of LLMs.

## Summary

The paper introduces YaRN (Yet another RoPE extensioN method), a compute-efficient approach to extend the context window of transformer-based language models. These models, known as Large Language Models (LLMs), have shown strong performance in in-context learning (ICL) tasks. However, their ability to generalize beyond the sequence length they were trained on is limited. YaRN addresses this limitation by allowing LLMs to effectively utilize and extrapolate to longer context lengths than their original pre-training would allow. It achieves this while also surpassing the state-of-the-art methods for context window extension, using 10x fewer tokens and 2.5x fewer training steps than previous approaches. The authors demonstrate the effectiveness of YaRN by fine-tuning Llama 2 7B/13B models with extended context windows of 64k and 128k using YaRN and provide the checkpoints for these models on GitHub. In addition, the paper highlights that YaRN exhibits the capability to extrapolate beyond the limited context of a fine-tuning dataset, further enhancing its generalization abilities and enabling it to leverage longer-context data for superior performance in various NLP tasks such as summarization. Overall, YaRN presents an efficient solution for extending the context window of LLMs.