# Efficient Streaming Language Models with Attention Sinks

**Authors:** Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis  
**arXiv:** [2309.17453](https://arxiv.org/abs/2309.17453)  
**Date:** 2023-09-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.17453v2/

## Key points

- Challenges of deploying Large Language Models (LLMs) in streaming applications:
- Extensive memory consumption during decoding stage
- Inability of popular LLMs to generalize to longer texts than training sequence length
- Addresses challenges and enables LLMs to generalize to infinite sequence lengths without fine-tuning
- Leverages attention sink by keeping Key and Value states (KV) of initial tokens to improve performance
- StreamingLLM enables stable and efficient language modeling with up to 4 million tokens or more for LLMs like Llama-2, MPT, Falcon, and Pythia
- Addition of a placeholder token as dedicated attention sink during pre-training improves streaming deployment
- Related work in three main areas: Length Extrapolation, Context Window Extension, and Improving LLMs' Utilization of Long Text
- StreamingLLM outperforms baseline methods like sliding window recomputation with up to 22.2x speedup in streaming settings
- Code and datasets for implementation provided on GitHub.

## Summary

The paper discusses the challenges of deploying Large Language Models (LLMs) in streaming applications, particularly in multi-round dialogue where long interactions are expected. The two main challenges identified are the extensive memory consumption during the decoding stage and the inability of popular LLMs to generalize to longer texts than their training sequence length. The authors propose a framework called StreamingLLM that addresses these challenges and enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. The emergence of a phenomenon called "attention sink" is observed, where initial tokens receive strong attention scores even if they are not semantically important. This observation leads to the introduction of StreamingLLM, which leverages attention sink by keeping the Key and Value states (KV) of initial tokens to improve performance. The authors demonstrate that StreamingLLM enables LLMs like Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens or more. Additionally, the authors propose adding a placeholder token as a dedicated attention sink during pre-training, which further improves streaming deployment. They show that pre-training language models with this single sink token preserves their performance in streaming cases without requiring multiple initial tokens as attention sinks. The paper also discusses related work in three main areas: Length Extrapolation, Context Window Extension, and Improving LLMs' Utilization of Long Text. While progress has been made in these areas individually, none of them achieve infinite length extrapolation or fit for streaming applications like StreamingLLM does. Overall, the proposed StreamingLLM framework addresses the challenges of deploying LLMs in streaming applications by leveraging attention sink and introducing a dedicated attention sink token. It outperforms baseline methods like sliding window recomputation by achieving up to 22.2x speedup in streaming settings. Code and datasets for implementation are provided on GitHub.