LongNet: Scaling Transformers to 1,000,000,000 Tokens

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Broader context Not relevant

   BitNet: Scaling 1-bit Transformers for Large Language Models

   Hongyu Wang, Shuming Ma, Li Dong et al. · 2023 · Computation and Language · AI summary available

   BitNet replaces linear layers with 1-bit weights to reduce memory and energy, addressing Transformer scaling through parameter efficiency rather than sequence length.

   Abstract

   The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scal…

2. Very close Another approach to the problem Not relevant

   Longformer: The Long-Document Transformer

   Iz Beltagy, Matthew E. Peters, Arman Cohan · 2020 · Computation and Language · AI summary available

   Longformer uses local windowed and task-motivated global attention for linear-complexity long-document processing, while LongNet uses dilated attention to scale sequence length.

   Abstract

   Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contra…

3. Close Another approach to the problem Not relevant

   Scaling Transformer to 1M tokens and beyond with RMT

   Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev · 2023 · Computation and Language · AI summary available

   The RMT report extends context length with recurrent memory across input segments, whereas LongNet uses dilated attention to model extremely long sequences.

   Abstract

   This technical report presents the application of a recurrent memory to extend the context length of BERT, one of the most effective Transformer-based models in natural language processing. By leveraging the Recurrent Memory Transformer architecture, we have successfully increased the model's effective context length to an unprecedented two million tokens, while maintaining high memory retrieval accuracy. Our method allows for the storage and processing of both local and global information and enables information flow between segments of the input sequence through the use of recurrence. Our experiments demonstrate the effectiveness of our approach, which holds significant potential to enhanc…

4. Close Another approach to the problem Not relevant

   Unlimiformer: Long-Range Transformers with Unlimited Length Input

   Amanda Bertsch, Uri Alon, Graham Neubig et al. · 2023 · Computation and Language · AI summary available

   Unlimiformer retrieves top-k keys through a k-nearest-neighbor index for long inputs, rather than LongNet’s dilated attention mechanism.

   Abstract

   Transformer-based models typically have a predefined bound to their input length, because of their need to potentially attend to every token in the input. In this work, we propose Unlimiformer: a general approach that can wrap any existing pretrained encoder-decoder transformer, and offload the attention computation across all layers to a single $k$-nearest-neighbor index; this index can be kept on either the GPU or CPU memory and queried in sub-linear time. This way, we can index extremely long input sequences, while every attention head in every decoder layer retrieves its top-$k$ keys, instead of attending to every key. We demonstrate Unlimiformers's efficacy on several long-document and …

5. Close Another approach to the problem Not relevant

   Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

   Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal · 2024 · Computation and Language · AI summary available

   Infini-attention combines compressive memory with local and long-term linear attention for extended inputs, while LongNet uses dilated attention.

   Abstract

   This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minima…

6. Close Broader context Not relevant

   Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing

   Xiaoju Ye, Zhichun Wang, Jingyuan Wang · 2025 · Computation and Language · AI summary available

   Infinite Retrieval studies long-context retrieval and reasoning in LLMs, focusing on attention allocation rather than LongNet’s sequence-length scaling architecture.

   Abstract

   Limited by the context window size of Large Language Models(LLMs), handling various tasks with input tokens exceeding the upper limit has been challenging, whether it is a simple direct retrieval task or a complex multi-hop reasoning task. Although various methods have been proposed to enhance the long-context processing capabilities of LLMs, they either incur substantial post-training costs, or require additional tool modules(e.g.,RAG), or have not shown significant improvement in realistic tasks. Our work observes the correlation between the attention distribution and generated answers across each layer, and establishes the attention allocation aligns with retrieval-augmented capabilities …

7. Close Broader context Not relevant

   Retentive Network: A Successor to Transformer for Large Language Models

   Yutao Sun, Li Dong, Shaohan Huang et al. · 2023 · Computation and Language · AI summary available

   RetNet proposes retention for parallel training and efficient recurrent inference, addressing Transformer sequence modeling rather than scaling input length with dilated attention.

   Abstract

   In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation …

8. Close Groundwork for this paper Not relevant

   Attention Is All You Need

   Ashish Vaswani, Noam Shazeer, Niki Parmar et al. · 2017 · Computation and Language · AI summary available

   Attention Is All You Need introduces the Transformer architecture whose attention mechanism LongNet modifies to handle much longer sequences.

   Abstract

   The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ense…