# LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

**Authors:** Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, Jiaya Jia  
**arXiv:** [2309.12307](https://arxiv.org/abs/2309.12307)  
**Date:** 2023-09-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.12307v1/

## Key points

- LongLoRA is an efficient fine-tuning approach for extending the context sizes of pre-trained large language models (LLMs) with limited computation cost.
- Training LLMs with long context sizes is computationally expensive and requires extensive training hours and GPU resources.
- LongLoRA proposes two aspects to speed up the context extension of LLMs:
- Fine-tuning the model can be effectively and efficiently done by sparse local attention, resulting in significant computation savings with similar performance to fine-tuning with vanilla attention.
- The parameter-efficient fine-tuning regime for context expansion works well under the premise of trainable embedding and normalization.
- LongLoRA demonstrates strong empirical results on various tasks using LLaMA2 models from 7B/13B to 70B, while retaining their original architectures and being compatible with most existing techniques like FlashAttention-2.
- To make LongLoRA practical, the authors provide a dataset called LongQA for supervised fine-tuning, which contains more than 3k long-context question-answer pairs.
- The authors provide code, models, dataset, and a demo for LongLoRA on GitHub at https://github.com/dvlab-research/LongLoRA.

## Summary

LongLoRA is an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs) with limited computation cost. Training LLMs with long context sizes is typically computationally expensive and requires extensive training hours and GPU resources. For example, training on a context length of 8192 needs 16x computational costs in self-attention layers compared to that of 2048. This paper proposes two aspects to speed up the context extension of LLMs. Firstly, while dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shift short attention enables context extension, resulting in significant computation savings with similar performance to fine-tuning with vanilla attention. This approach can be implemented with just two lines of code in training and is optional in inference. Secondly, the paper revisits the parameter-efficient fine-tuning regime for context expansion. It finds that LoRA for context extension works well under the premise of trainable embedding and normalization. LongLoRA demonstrates strong empirical results on various tasks using LLaMA2 models from 7B/13B to 70B. It extends models' context while retaining their original architectures and is compatible with most existing techniques like FlashAttention-2. To make LongLoRA practical, the authors collect a dataset called LongQA for supervised fine-tuning. This dataset contains more than 3k long-context question-answer pairs. The authors provide code, models, dataset, and a demo for LongLoRA on GitHub at https://github.com/dvlab-research/LongLoRA . In summary, LongLoRA presents an efficient approach for extending the context sizes of large language models without significant computational costs. It achieves this through sparse local attention during fine-tuning and parameter-efficient regime for context expansion. The approach demonstrates strong empirical results across various tasks and is compatible with existing techniques. The authors also provide a dataset for supervised fine-tuning to make LongLoRA practical.