# LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

**Authors:** Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, Xia Hu  
**arXiv:** [2401.01325](https://arxiv.org/abs/2401.01325)  
**Date:** 2024-01-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.01325v1/

## Key points

- Several methods have been developed to extend the context window size of pretrained Large Language Models (LLMs)
- These methods either require fine-tuning on extensive texts or aim for extension without or with minimal fine-tuning
- Some approaches may be resource-intensive and time-consuming
- LLMs are believed to have the ability to handle long contexts, but struggle with predicting important tokens related to long context comprehension
- The proposed method, called Self-Extend, stimulates LLMs' long context handling potential without any fine-tuning
- Self-Extend constructs bi-level attention information using group level and neighbor level attention computed through self-attention in the original model
- With just four lines of code modification, Self-Extend extends existing LLMs' context window effortlessly
- Comprehensive experiments show that Self-Extend significantly extends existing LLMs' context window length and improves performance on real-world long context tasks
- Leveraging the inherent capabilities of LLMs for handling long contexts is more effective than simply extending the content window size

## Summary

In recent years, several methods have been developed to extend the context window size of pretrained Large Language Models (LLMs). These methods either require fine-tuning on extensive texts or aim to achieve extension without or with minimal fine-tuning. However, these approaches may be resource-intensive and time-consuming. They also assume that LLMs lack the ability to handle long content. On the other hand, some fine-tuning-free methods rely on local information in the sequence but may not effectively expand the context window capacity of LLMs. In this paper, we propose a different approach by leveraging the inherent capabilities of LLMs for handling long contexts. This belief is based on the fact that as human beings, we are taught how to read and write using relatively short texts. Yet we can effectively understand longer texts. Therefore, we argue that poor performance of LLMs on long text tasks is not due to a lack of understanding long contexts but rather a challenge in predicting important tokens related to long context comprehension. To address this challenge, we introduce <b>Self-Extend</b>, a method that stimulates LLMs' long context handling potential without any fine-tuning. <b>Self-Extend</b> constructs bi-level attention information by utilizing group level and neighbor level attention computed through self-attention in the original model. With just four lines of code modification,<b>Self-Extend</b> extends existing LLMs' context window effortlessly. We conducted comprehensive experiments to evaluate the effectiveness of <b>Self-Extend</b>. The results show that our proposed method significantly extends existing LLMs' context window length. Additionally, we evaluated its performance on real-world long context tasks using benchmarks such as Longbench and L-Eval. The results demonstrate significant performance improvements when <b>Self-Extend</b> is applied. Overall, our findings suggest that instead of extending the content window size for LLMs, their inherent capabilities for handling long contexts should be leveraged. Our proposed method offers an efficient solution for fully utilizing this inherent ability without the need for extensive fine-tuning.