# Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

**Authors:** Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal  
**arXiv:** [2404.07143](https://arxiv.org/abs/2404.07143)  
**Date:** 2024-04-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.07143v1/

## Key points

- Paper introduces a novel method for scaling Transformer-based Large Language Models (LLMs) to handle infinitely long inputs
- Key innovation is the development of a new attention mechanism called Infini-attention
- Infini-attention incorporates compressive memory, masked local attention, and long-term linear attention mechanisms within a single Transformer block
- Demonstrated effectiveness through experiments on various tasks such as long-context language modeling benchmarks, passkey context block retrieval, and book summarization tasks
- Method introduces minimal bounded memory parameters, enabling fast streaming inference for LLMs
- Significant advancement in scaling LLMs to handle infinitely long inputs efficiently

## Summary

The paper "Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" by Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal introduces a novel method for scaling Transformer-based Large Language Models (LLMs) to handle infinitely long inputs while maintaining bounded memory and computation requirements. The key innovation in their approach is the development of a new attention mechanism called Infini-attention. This attention technique incorporates compressive memory within the traditional attention mechanism and integrates both masked local attention and long-term linear attention mechanisms within a single Transformer block. The authors demonstrate the effectiveness of their approach through experiments on various tasks such as long-context language modeling benchmarks, 1M sequence length passkey context block retrieval, and 500K length book summarization tasks using LLMs with 1B and 8B parameters. Their method introduces minimal bounded memory parameters, enabling fast streaming inference for LLMs. Overall, the paper presents a significant advancement in scaling LLMs to handle infinitely long inputs efficiently, showcasing the potential for improved performance in tasks requiring extensive contextual information processing.