LongNet: Scaling Transformers to 1,000,000,000 Tokens

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   BitNet: Scaling 1-bit Transformers for Large Language Models

   Hongyu Wang, Shuming Ma, Li Dong et al. · 2023 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   The increasing size of large language models has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. In this work, we introduce BitNet, a scalable and stable 1-bit Transformer architecture designed for large language models. Specifically, we introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to train 1-bit weights from scratch. Experimental results on language modeling show that BitNet achieves competitive performance while substantially reducing memory footprint and energy consumption, compared to state-of-the-art 8-bit quantization methods and FP16 Transformer baselines. Furthermore, BitNet exhibits a scal…

2. Très proche Pas pertinent

   Longformer: The Long-Document Transformer

   Iz Beltagy, Matthew E. Peters, Arman Cohan · 2020 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contra…

3. Proche Pas pertinent

   Scaling Transformer to 1M tokens and beyond with RMT

   Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev · 2023 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   This technical report presents the application of a recurrent memory to extend the context length of BERT, one of the most effective Transformer-based models in natural language processing. By leveraging the Recurrent Memory Transformer architecture, we have successfully increased the model's effective context length to an unprecedented two million tokens, while maintaining high memory retrieval accuracy. Our method allows for the storage and processing of both local and global information and enables information flow between segments of the input sequence through the use of recurrence. Our experiments demonstrate the effectiveness of our approach, which holds significant potential to enhanc…

4. Proche Pas pertinent

   Unlimiformer: Long-Range Transformers with Unlimited Length Input

   Amanda Bertsch, Uri Alon, Graham Neubig et al. · 2023 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   Transformer-based models typically have a predefined bound to their input length, because of their need to potentially attend to every token in the input. In this work, we propose Unlimiformer: a general approach that can wrap any existing pretrained encoder-decoder transformer, and offload the attention computation across all layers to a single $k$-nearest-neighbor index; this index can be kept on either the GPU or CPU memory and queried in sub-linear time. This way, we can index extremely long input sequences, while every attention head in every decoder layer retrieves its top-$k$ keys, instead of attending to every key. We demonstrate Unlimiformers's efficacy on several long-document and …

5. Proche Pas pertinent

   Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

   Tsendsuren Munkhdalai, Manaal Faruqui, Siddharth Gopal · 2024 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minima…

6. Proche Pas pertinent

   Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing

   Xiaoju Ye, Zhichun Wang, Jingyuan Wang · 2025 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   Limited by the context window size of Large Language Models(LLMs), handling various tasks with input tokens exceeding the upper limit has been challenging, whether it is a simple direct retrieval task or a complex multi-hop reasoning task. Although various methods have been proposed to enhance the long-context processing capabilities of LLMs, they either incur substantial post-training costs, or require additional tool modules(e.g.,RAG), or have not shown significant improvement in realistic tasks. Our work observes the correlation between the attention distribution and generated answers across each layer, and establishes the attention allocation aligns with retrieval-augmented capabilities …

7. Proche Pas pertinent

   Retentive Network: A Successor to Transformer for Large Language Models

   Yutao Sun, Li Dong, Shaohan Huang et al. · 2023 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation …

8. Proche Pas pertinent

   Attention Is All You Need

   Ashish Vaswani, Noam Shazeer, Niki Parmar et al. · 2017 · Calcul et langage · Résumé IA disponible

   Résumé scientifique

   The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ense…