Packing: Towards 2x NLP BERT Acceleration

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Related Not relevant

   SpanBERT: Improving Pre-training by Representing and Predicting Spans

   Mandar Joshi, Danqi Chen, Yinhan Liu et al. · 2019 · Computation and Language · AI summary available

   Abstract

   We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT-large, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0, respectively. We als…

2. Related Not relevant

   LongNet: Scaling Transformers to 1,000,000,000 Tokens

   Jiayu Ding, Shuming Ma, Li Dong et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Scaling sequence length has become a critical demand in the era of large language models. However, existing methods struggle with either computational complexity or model expressivity, rendering the maximum sequence length restricted. To address this issue, we introduce LongNet, a Transformer variant that can scale sequence length to more than 1 billion tokens, without sacrificing the performance on shorter sequences. Specifically, we propose dilated attention, which expands the attentive field exponentially as the distance grows. LongNet has significant advantages: 1) it has a linear computation complexity and a logarithm dependency between any two tokens in a sequence; 2) it can be served …

3. Related Not relevant

   PaLM: Scaling Language Modeling with Pathways

   Aakanksha Chowdhery, Sharan Narang, Jacob Devlin et al. · 2022 · Computation and Language · AI summary available

   Abstract

   Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning resu…

4. Related Not relevant

   Megatron-LM: Training Multi-Billion Parameter Language Models Using GPU Model Parallelism

   Mohammad Shoeybi, Mostofa Patwary, Raul Puri et al. · 2019 · Computation and Language · Not summarized yet

   Abstract

   Recent work in unsupervised language modeling demonstrates that training large neural language models advances the state of the art in Natural Language Processing applications. However, for very large models, memory constraints limit the size of models that can be practically trained. Model parallelism allows us to train larger models, because the parameters can be split across multiple processors. In this work, we implement a simple, efficient intra-layer model parallel approach that enables training state of the art transformer language models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model para…

5. Related Not relevant

   D4: Improving LLM Pretraining via Document De-Duplication and Diversification

   Kushal Tirumala, Daniel Simig, Armen Aghajanyan et al. · 2023 · Computation and Language · Not summarized yet

   Abstract

   Over recent years, an increasing amount of compute and data has been poured into training large language models (LLMs), usually by doing one-pass learning on as many tokens as possible randomly selected from large-scale web corpora. While training on ever-larger portions of the internet leads to consistent performance improvements, the size of these improvements diminishes with scale, and there has been little work exploring the effect of data selection on pre-training and downstream performance beyond simple de-duplication methods such as MinHash. Here, we show that careful data selection (on top of de-duplicated data) via pre-trained model embeddings can speed up training (20% efficiency g…

6. Related Not relevant

   Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review

   Neha Prakriya, Jui-Nan Yen, Cho-Jui Hsieh et al. · 2024 · Computation and Language · Not summarized yet

   Abstract

   Large Language Model (LLM) pretraining traditionally relies on autoregressive language modeling on randomly sampled data blocks from web-scale datasets. We take inspiration from human learning techniques like spaced repetition to hypothesize that random data sampling for LLMs leads to high training cost and low quality models which tend to forget data. In order to effectively commit web-scale information to long-term memory, we propose the LFR (Learn, Focus, and Review) pedagogy, a new dynamic training paradigm which focuses and repeatedly reviews complex data blocks at systematic intervals based on the model's learning pace and progress. LFR records the model perplexities for different data…

7. Related Not relevant

   Scaling Transformer to 1M tokens and beyond with RMT

   Aydar Bulatov, Yuri Kuratov, Mikhail S. Burtsev · 2023 · Computation and Language · AI summary available

   Abstract

   This technical report presents the application of a recurrent memory to extend the context length of BERT, one of the most effective Transformer-based models in natural language processing. By leveraging the Recurrent Memory Transformer architecture, we have successfully increased the model's effective context length to an unprecedented two million tokens, while maintaining high memory retrieval accuracy. Our method allows for the storage and processing of both local and global information and enables information flow between segments of the input sequence through the use of recurrence. Our experiments demonstrate the effectiveness of our approach, which holds significant potential to enhanc…

8. Related Not relevant

   RoBERTa: A Robustly Optimized BERT Pretraining Approach

   Yinhan Liu, Myle Ott, Naman Goyal et al. · 2019 · Computation and Language · Not summarized yet

   Abstract

   Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight …