# LongAlign: A Recipe for Long Context Alignment of Large Language Models

**Authors:** Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, Juanzi Li  
**arXiv:** [2401.18058](https://arxiv.org/abs/2401.18058)  
**Date:** 2024-01-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.18058v1/

## Key points

- LongAlign introduced as a solution for handling long contexts with large language models
- Comprehensive approach covering instruction data creation, training strategies, and evaluation methods tailored for long context alignment tasks
- Creation of diverse long instruction-following dataset using Self-Instruct to cover various tasks from different long context inputs
- Implementation of packing and sorted batching strategies to expedite supervised fine-tuning on datasets with varying length distributions
- Introduction of loss weighting method to ensure balanced contributions to the loss function across different sequences during training with packed data
- Development of LongBench-Chat benchmark for evaluating instruction-following capabilities on queries ranging from 10k to 100k in length
- Demonstrated superior performance of LongAlign in handling long context tasks compared to existing recipes for large language models by up to 30%
- Proficiency maintained in shorter generic tasks while enhancing performance in extended contextual information handling
- The codebase, dataset, and models developed using LongAlign are openly accessible at https://github.com/THUDM/LongAlign.

## Summary

, , , , 

In their paper titled "LongAlign: A Recipe for Long Context Alignment of Large Language Models," authors Yushi Bai, Xin Lv, Jiajie Zhang, Yuze He, Ji Qi, Lei Hou, Jie Tang, Yuxiao Dong, and Juanzi Li introduce LongAlign as a solution to effectively handle long contexts with large language models. The key challenge addressed is the need for instruction fine-tuning on input sequences of similar length to extend the capabilities of these models. To tackle this issue, the authors present LongAlign as a comprehensive approach encompassing instruction data creation, training strategies, and evaluation methods tailored specifically for long context alignment tasks. They start by creating a diverse long instruction-following dataset using Self-Instruct that covers a wide range of tasks sourced from various long context inputs. In order to expedite supervised fine-tuning on datasets with varying length distributions, the authors implement packing and sorted batching strategies. Additionally, they devise a loss weighting method to ensure balanced contributions to the loss function across different sequences during training with packed data. Furthermore, the authors introduce the LongBench-Chat benchmark for evaluating instruction-following capabilities on queries ranging from 10k to 100k in length. Through experiments conducted using LongAlign, it is demonstrated that this approach outperforms existing recipes for large language models in handling long context tasks by up to 30%, while still maintaining proficiency in shorter generic tasks. The codebase, dataset, and models developed using LongAlign are made openly accessible at https://github.com/THUDM/LongAlign. This detailed summary highlights the innovative contributions of LongAlign in enhancing the performance of large language models when dealing with extended contextual information.