Inference with Reference: Lossless Acceleration of Large Language Models

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Close Not relevant

   MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

   Jian Chen, Vashisth Tiwari, Ranajoy Sadhukhan et al. · 2024 · Computation and Language · Not summarized yet

   Abstract

   Large Language Models (LLMs) have become more prevalent in long-context applications such as interactive chatbots, document analysis, and agent workflows, but it is challenging to serve long-context requests with low latency and high throughput. Speculative decoding (SD) is a widely used technique to reduce latency without sacrificing performance but the conventional wisdom suggests that its efficacy is limited to small batch sizes. In MagicDec, we show that surprisingly SD can achieve speedup even for a high throughput inference regime for moderate to long sequences. More interestingly, an intelligent drafting strategy can achieve better speedup with increasing batch size based on our rigor…

2. Close Not relevant

   APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

   Mingdao Liu, Aohan Zeng, Bowen Wang et al. · 2024 · Computation and Language · AI summary available

   Abstract

   The massive adoption of large language models (LLMs) demands efficient deployment strategies. However, the auto-regressive decoding process, which is fundamental to how most LLMs generate text, poses challenges to achieve efficient serving. In this work, we introduce a parallel auto-regressive generation method. By instruct-tuning on general domain data that contains hierarchical structures, we enable LLMs to independently plan their generation process and perform auto-parallel auto-regressive (APAR) generation, significantly reducing the number of generation steps. APAR alone can achieve up to 2x speed-up, and when combined with speculative decoding, the speed-up can reach up to 4x. In addi…

3. Close Not relevant

   Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding

   Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich et al. · 2024 · Computation and Language · AI summary available

   Abstract

   We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory foot…

4. Close Not relevant

   LLM in a flash: Efficient Large Language Model Inference with Limited Memory

   Keivan Alizadeh, Iman Mirzadeh, Dmitry Belenko et al. · 2023 · Computation and Language · AI summary available

   Abstract

   Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their intensive computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters on flash memory but bringing them on demand to DRAM. Our method involves constructing an inference cost model that harmonizes with the flash memory behavior, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous c…

5. Close Not relevant

   Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding

   Xuefei Ning, Zinan Lin, Zixuan Zhou et al. · 2023 · Computation and Language · AI summary available

   Abstract

   This work aims at decreasing the end-to-end generation latency of large language models (LLMs). One of the major causes of the high generation latency is the sequential decoding approach adopted by almost all state-of-the-art LLMs. In this work, motivated by the thinking and writing process of humans, we propose Skeleton-of-Thought (SoT), which first guides LLMs to generate the skeleton of the answer, and then conducts parallel API calls or batched decoding to complete the contents of each skeleton point in parallel. Not only does SoT provide considerable speed-ups across 12 LLMs, but it can also potentially improve the answer quality on several question categories. SoT is an initial attempt…

6. Close Not relevant

   SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator

   Guoxuan Chen, Han Shi, Jiawei Li et al. · 2024 · Computation and Language · AI summary available

   Abstract

   Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and inference speed, due to their quadratic complexity. In this work, we have identified a key pattern: certain seemingly meaningless special tokens (i.e., separators) contribute disproportionately to attention scores compared to semantically meaningful tokens. This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss. Guided by this insi…

7. Close Not relevant

   Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing

   Xiaoju Ye, Zhichun Wang, Jingyuan Wang · 2025 · Computation and Language · AI summary available

   Abstract

   Limited by the context window size of Large Language Models(LLMs), handling various tasks with input tokens exceeding the upper limit has been challenging, whether it is a simple direct retrieval task or a complex multi-hop reasoning task. Although various methods have been proposed to enhance the long-context processing capabilities of LLMs, they either incur substantial post-training costs, or require additional tool modules(e.g.,RAG), or have not shown significant improvement in realistic tasks. Our work observes the correlation between the attention distribution and generated answers across each layer, and establishes the attention allocation aligns with retrieval-augmented capabilities …

8. Close Not relevant

   LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement

   Nicholas Lee, Thanakul Wattanawong, Sehoon Kim et al. · 2024 · Computation and Language · AI summary available

   Abstract

   Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. While many real-world applications still require fine-tuning to reach satisfactory levels of performance, many of them are in the low-data regime, making fine-tuning challenging. To address this, we propose LLM2LLM, a targeted and iterative data augmentation strategy that uses a teacher LLM to enhance a small seed dataset by augmenting additional data that can be used for fine-tuning on a specific task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data, (2) evaluates and extracts data points that the model gets wrong, and (3) uses a t…