Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing

   Xiaoju Ye, Zhichun Wang, Jingyuan Wang · 2025 · Computation and Language · preprint, not peer reviewed · cited 9 times · AI summary available

   Abstract

   Limited by the context window size of Large Language Models(LLMs), handling various tasks with input tokens exceeding the upper limit has been challenging, whether it is a simple direct retrieval task or a complex multi-hop reasoning task. Although various methods have been proposed to enhance the long-context processing capabilities of LLMs, they either incur substantial post-training costs, or require additional tool modules(e.g.,RAG), or have not shown significant improvement in realistic tasks. Our work observes the correlation between the attention distribution and generated answers across each layer, and establishes the attention allocation aligns with retrieval-augmented capabilities …

2. Very close Cites this paper Uses this paper in depth Not relevant

   EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices

   Jiyu Chen, Shuang Peng, Daxiong Luo et al. · 2025 · Computation and Language · cited 7 times · AI summary available

   Abstract

   Transformer-based large language models (LLMs) encounter challenges in processing long sequences on edge devices due to the quadratic complexity of attention mechanisms and growing memory demands from Key-Value (KV) cache. Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks, while alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructure. We present EdgeInfinite, a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module. This approach maintains full compatibility with standard Transformer architectures,…

3. Very close Not relevant

   ReAttention: Training-Free Infinite Context with Finite Attention Scope

   Xiaoran Liu, Rui-Xiao Li, Zhigeng Liu et al. · 2024 · International Conference on Learning Representations · cited 15 times · Not summarized yet

   Abstract

   The long-context capability of the Large Language Models (LLM) has made significant breakthroughs, but the maximum supported context length in length extrapolation remains a critical bottleneck limiting their practical applications. The constraint of context length in LLMs arises from the self-attention mechanism, which cannot effectively and efficiently capture the semantic relationships within infinitely long contexts via the limited pre-trained positional information and attention scope. In this work, we propose ReAttention, a training-free approach enabling LLM based on the self-attention mechanism to support an infinite context with a finite attention scope under sufficient memory resou…

4. Very close Cited by this paper A method it uses Not relevant

   Landmark Attention: Random-Access Infinite Context Length for Transformers

   Amirkeivan Mohtashami, Martin Jaggi · 2023 · Computation and Language · cited 256 times · AI summary available

   Abstract

   While transformers have shown remarkable success in natural language processing, their attention mechanism's large memory requirements have limited their ability to handle longer contexts. Prior approaches, such as recurrent memory or retrieval-based augmentation, have either compromised the random-access flexibility of attention (i.e., the capability to select any token in the entire context) or relied on separate mechanisms for relevant context retrieval, which may not be compatible with the model's attention. In this paper, we present a novel approach that allows access to the complete context while retaining random-access flexibility, closely resembling running attention on the entire co…

5. Very close Cites this paper Uses this paper in depth Not relevant

   Probing the Limits of Compressive Memory: A Study of Infini-Attention in Small-Scale Pretraining

   Rui-Zhe Huang, Kexuan Zhang, Yihao Fang et al. · 2025 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   This study investigates small-scale pretraining for Small Language Models (SLMs) to enable efficient use of limited data and compute, improve accessibility in low-resource settings and reduce costs. To enhance long-context extrapolation in compact models, we focus on Infini-attention, which builds a compressed memory from past segments while preserving local attention. In our work, we conduct an empirical study using 300M-parameter LLaMA models pretrained with Infini-attention. The model demonstrates training stability and outperforms the baseline in long-context retrieval. We identify the balance factor as a key part of the model performance, and we found that retrieval accuracy drops with …

6. Very close Not relevant

   Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache

   Bin Lin, Chen Zhang, Tao Peng et al. · 2024 · Distributed, Parallel, and Cluster Computing · preprint, not peer reviewed · cited 114 times · AI summary available

   Abstract

   Large Language Models (LLMs) demonstrate substantial potential across a diverse array of domains via request serving. However, as trends continue to push for expanding context sizes, the autoregressive nature of LLMs results in highly dynamic behavior of the attention layers, showcasing significant differences in computational characteristics and memory requirements from the non-attention layers. This presents substantial challenges for resource management and performance optimization in service systems. Existing static model parallelism and resource allocation strategies fall short when dealing with this dynamicity. To address the issue, we propose Infinite-LLM, a novel LLM serving system d…

7. Very close Cites this paper Not relevant

   InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory

   Chao-Jun Xiao, Pengle Zhang, Xu Han et al. · 2024 · Neural Information Processing Systems · cited 248 times · Not summarized yet

   Abstract

   Large language models (LLMs) have emerged as a cornerstone in real-world applications with lengthy streaming inputs (e.g., LLM-driven agents). However, existing LLMs, pre-trained on sequences with a restricted maximum length, cannot process longer sequences due to the out-of-domain and distraction issues. Common solutions often involve continual pre-training on longer sequences, which will introduce expensive computational overhead and uncontrollable change in model capabilities. In this paper, we unveil the intrinsic capacity of LLMs for understanding extremely long sequences without any fine-tuning. To this end, we introduce a training-free memory-based method, InfLLM. Specifically, InfLLM…

8. Very close Cites this paper Not relevant

   Towards Infinite Length Extrapolation: A Unified Approach

   Nitin Vetcha · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Large language models (LLMs) have revolutionized natural language processing, but their ability to process long sequences is fundamentally limited by the context window size during training. Existing length extrapolation methods often suffer from performance degradation or computational inefficiencies. We thereby use a unified framework that reinterprets positional encoding methods as a decomposition of the attention score into a multiplicative transformation and an additive bias. This perspective not only subsumes popular approaches such as relative position embeddings and attention-bias moderated approaches but also exposes their inherent limitations in handling long-range dependencies. To…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   In-context Autoencoder for Context Compression in a Large Language Model

   Tao Ge, Jing Hu, Xun Wang et al. · 2023 · Computation and Language · cited 242 times · AI summary available

   Abstract

   We propose the In-context Autoencoder (ICAE) for context compression in a large language model (LLM). The ICAE has two modules: a learnable encoder adapted with LoRA from an LLM for compressing a long context into a limited number of memory slots, and a fixed decoder which is the target LLM that can condition on the memory slots for various purposes. We first pretrain the ICAE using both autoencoding and language modeling objectives on massive text data, enabling it to generate memory slots that accurately and comprehensively represent the original context. Then, we fine-tune the pretrained ICAE on a small amount of instruct data to enhance its interaction with various prompts for producing …

2. Close Used in depth by this paper A method it uses Not relevant

   Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

   Zihang Dai, Zhilin Yang, Yiming Yang et al. · 2019 · Machine Learning · cited 4,640 times · AI summary available

   Abstract

   Transformer networks have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. As a solution, we propose a novel neural architecture, \textit{Transformer-XL}, that enables Transformer to learn dependency beyond a fixed length without disrupting temporal coherence. Concretely, it consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the problem of context fragmentation. As a result, Transformer-XL learns dependency that is about 80\% longer than RNNs and 450\% longer than vanilla Transformers, achieves bett…

3. Close Used in depth by this paper A method it uses Not relevant

   Adapting Language Models to Compress Contexts

   A. Chevalier, Alexander Wettig, Anirudh Ajith et al. · 2023 · Conference on Empirical Methods in Natural Language Processing · cited 465 times · Not summarized yet

   Abstract

   Transformer-based language models (LMs) are powerful and widely-applicable tools, but their usefulness is constrained by a finite context window and the expensive computational cost of processing long text documents. We propose to adapt pre-trained LMs into AutoCompressors. These models are capable of compressing long contexts into compact summary vectors, which are then accessible to the model as soft prompts. Summary vectors are trained with an unsupervised objective, whereby long documents are processed in segments and summary vectors from all previous segments are used in language modeling. We fine-tune OPT models on sequences of up to 30,720 tokens and show that AutoCompressors can util…

4. Close A method it uses Not relevant

   LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models

   Yukang Chen, Shengju Qian, Haotian Tang et al. · 2023 · Computation and Language · cited 310 times · AI summary available

   Abstract

   We present LongLoRA, an efficient fine-tuning approach that extends the context sizes of pre-trained large language models (LLMs), with limited computation cost. Typically, training LLMs with long context sizes is computationally expensive, requiring extensive training hours and GPU resources. For example, training on the context length of 8192 needs 16x computational costs in self-attention layers as that of 2048. In this paper, we speed up the context extension of LLMs in two aspects. On the one hand, although dense global attention is needed during inference, fine-tuning the model can be effectively and efficiently done by sparse local attention. The proposed shift short attention effecti…

5. Close Used in depth by this paper A method it uses Not relevant

   Compressive Transformers for Long-Range Sequence Modelling

   Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar et al. · 2019 · International Conference on Learning Representations · cited 1,042 times · Not summarized yet

   Abstract

   We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97 bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Recurrent Context Compression: Efficiently Expanding the Context Window of LLM

   Chensen Huang, Gui-Bo Zhu, Xuepeng Wang et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet

   Abstract

   To extend the context length of Transformer-based large language models (LLMs) and improve comprehension capabilities, we often face limitations due to computational resources and bounded memory storage capacity. This work introduces a method called Recurrent Context Compression (RCC), designed to efficiently expand the context window length of LLMs within constrained storage space. We also investigate the issue of poor model responses when both instructions and context are compressed in downstream tasks, and propose an instruction reconstruction method to mitigate this problem. We validated the effectiveness of our approach on multiple tasks, achieving a compression rate of up to 32x on tex…

2. Very close Not relevant

   Star Attention: Efficient LLM Inference over Long Sequences

   Shantanu Acharya, Fei Jia, Boris Ginsburg · 2024 · International Conference on Machine Learning · cited 37 times · Not summarized yet

   Abstract

   Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory re…

3. Close Not relevant

   MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

   Huiqiang Jiang, Yucheng Li, Chengruidong Zhang et al. · 2024 · Neural Information Processing Systems · cited 507 times · Not summarized yet

   Abstract

   The computational challenges of Large Language Model (LLM) inference remain a significant barrier to their widespread deployment, especially as prompt lengths continue to increase. Due to the quadratic complexity of the attention computation, it takes 30 minutes for an 8B LLM to process a prompt of 1M tokens (i.e., the pre-filling stage) on a single A100 GPU. Existing methods for speeding up prefilling often fail to maintain acceptable accuracy or efficiency when applied to long-context LLMs. To address this gap, we introduce MInference (Milliontokens Inference), a sparse calculation method designed to accelerate pre-filling of long-sequence processing. Specifically, we identify three unique…

4. Close Uses this paper in depth Not relevant

   Artificial Hippocampus Networks for Efficient Long-Context Modeling

   Yun-Hao Fang, Wei-Hao Yu, Shu Zhong et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet

   Abstract

   Long-sequence modeling faces a fundamental trade-off between the efficiency of compressive fixed-size memory in RNN-like models and the fidelity of lossless growing memory in attention-based Transformers. Inspired by the Multi-Store Model in cognitive science, we introduce a memory framework of artificial neural networks. Our method maintains a sliding window of the Transformer's KV cache as lossless short-term memory, while a learnable module termed Artificial Hippocampus Network (AHN) recurrently compresses out-of-window information into a fixed-size compact long-term memory. To validate this framework, we instantiate AHNs using modern RNN-like architectures, including Mamba2, DeltaNet, an…

5. Close Not relevant

   Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modeling

   Xiang Hu, Zhihao Teng, Jun Zhao et al. · 2024 · International Conference on Machine Learning · cited 9 times · Not summarized yet

   Abstract

   Despite the success of Transformers, handling long contexts remains challenging due to the limited length generalization and quadratic complexity of self-attention. Thus Transformers often require post-training with a larger attention window, significantly increasing computational and memory costs. In this paper, we propose a novel attention mechanism based on dynamic context, Grouped Cross Attention (GCA), which can generalize to 1000 times the pre-training context length while maintaining the ability to access distant information with a constant attention window size. For a given input sequence, we split it into chunks and use each chunk to retrieve top-k relevant past chunks for subsequen…