# Effective Long-Context Scaling of Foundation Models

**Authors:** Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang  
**arXiv:** [2309.16039](https://arxiv.org/abs/2309.16039)  
**Date:** 2023-09-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.16039v1/

## Key points

- Long-context Large Language Models (LLMs) with effective context windows of up to 32,768 tokens
- Continual pretraining from Llama 2 with longer training sequences and upsampling of long texts
- Consistent improvements on most regular tasks and significant improvements on long-context tasks compared to Llama 2
- Surpassing the overall performance of gpt-3.5-turbo-16k on long-context tasks using a cost-effective instruction tuning procedure
- In-depth analysis of limitations in modeling long dependencies and impact of design choices in pretraining process
- Evaluation and measurements of other existing open-sourced long-context models for more context
- Importance of maintaining strong performance on standard short-context tasks while demonstrating effectiveness in real-world scenarios
- Evaluation of fine-tuned LLMs on safety benchmarks and red teaming exercises for safety in long-context understanding
- Acknowledgment of limitations such as limited functionality for certain applications and challenges related to tokenizer efficiency

## Summary

In this paper, the authors present a series of long-context Large Language Models (LLMs) that have effective context windows of up to 32,768 tokens. These models are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. The authors extensively evaluate these models on language modeling, synthetic context probing tasks, and various research benchmarks. The results show that the models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks compared to Llama 2. Notably, the 70B variant of the model surpasses the overall performance of gpt-3.5-turbo-16k on a suite of long-context tasks using a cost-effective instruction tuning procedure that does not require human-annotated long instruction data. The paper also includes an in-depth analysis of the individual components of their method. They discuss the limitations of Llama's position encodings in modeling long dependencies and examine the impact of various design choices in the pretraining process. Additionally, the authors provide more context by discussing evaluations and measurements of other existing open-sourced long-context models. They highlight the importance of maintaining strong performance on standard short-context tasks while demonstrating effectiveness in diverse real-world scenarios. Furthermore, they evaluate fine-tuned LLMs on safety benchmarks and perform red teaming exercises to ensure safety in long-context understanding. Although there is currently no open sourced safety benchmark designed for long context understanding, their internal red teaming did not observe significant risks compared to LLAMA 2 CHAT. The paper acknowledges some limitations such as limited functionality for certain applications that require long form outputs and challenges related to tokenizer efficiency. In conclusion, this paper presents a series of long context LLMs that outperform previous models on both regular and long context tasks. The authors provide detailed analyses and evaluations to support their findings while addressing potential limitations and future research directions.