Layer Skip: Enabling Early Exit Inference and Self-Speculative Decoding

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

   Walaa Amer, Udayan Das, Fadi J. Kurdahi · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Self-speculative decoding is an inference technique for large language models designed to speed up generation without sacrificing output quality. It combines fast, approximate decoding using a compact version of the model as a draft model with selective re-evaluation by the full target model. Some existing methods form the draft model by dynamically learning which layers to skip during inference, effectively creating a smaller subnetwork to speed up computation. However, using heuristic-based approaches to select layers to skip can often be simpler and more effective. In this paper, we propose ConfLayers, a dynamic plug-and-play approach to forming the draft model in self-speculative decodin…

2. Very close Cites this paper Uses this paper in depth Not relevant

   AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism

   Zhepei Wei, Wei-Lin Chen, Xin-Yu Zhu et al. · 2025 · International Conference on Machine Learning · cited 7 times · Not summarized yet

   Abstract

   Large language models (LLMs) are increasingly used for long-content generation (e.g., long Chain-of-Thought reasoning) where decoding efficiency becomes a critical bottleneck: Autoregressive decoding is inherently limited by its sequential token generation process, where each token must be generated before the next can be processed. This sequential dependency restricts the ability to fully leverage modern hardware's parallel processing capabilities. Existing methods like speculative decoding and layer skipping offer potential speedups but have notable drawbacks: speculative decoding relies on an auxiliary"drafter"model, which can be challenging to acquire and increases memory overhead, while…

3. Very close Cites this paper Not relevant

   CLaSp: In-Context Layer Skip for Self-Speculative Decoding

   Long-Ze Chen, Renke Shan, Huiming Wang et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 14 times · Not summarized yet

   Abstract

   Speculative decoding (SD) is a promising method for accelerating the decoding process of Large Language Models (LLMs). The efficiency of SD primarily hinges on the consistency between the draft model and the verify model. However, existing drafting approaches typically require additional modules to be trained, which can be challenging to implement and ensure compatibility across various LLMs. In this paper, we propose CLaSp, an in-context layer-skipping strategy for self-speculative decoding. Unlike prior methods, CLaSp does not require additional drafting modules or extra training. Instead, it employs a plug-and-play mechanism by skipping intermediate layers of the verify model to construct…

4. Very close Cites this paper Uses this paper in depth Not relevant

   DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decoding

   H. E. Zarch, Lei Gao, Chao-Yi Jiang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet

   Abstract

   Speculative Decoding (SD) is a widely used approach to accelerate the inference of large language models (LLMs) without reducing generation quality. It operates by first using a compact model to draft multiple tokens efficiently, followed by parallel verification using the target LLM. This approach leads to faster inference compared to auto-regressive decoding. While there are multiple approaches to create a draft model, one promising approach is to use early-exit methods. These methods draft candidate tokens by using a subset of layers of the primary model and applying the remaining layers for verification, allowing a single model to handle both drafting and verification. While this techniq…

5. Very close Cites this paper Not relevant

   Adaptive Layer-skipping in Pre-trained LLMs

   Xuan Luo, Weizhi Wang, Xifeng Yan · 2025 · arXiv.org · preprint, not peer reviewed · cited 33 times · Not summarized yet

   Abstract

   Various layer-skipping methods have been proposed to accelerate token generation in large language models (LLMs). However, limited attention has been paid to a fundamental question: How do computational demands vary across the generation of different tokens? In this work, we introduce FlexiDepth, a method that dynamically adjusts the number of Transformer layers used in text generation. By incorporating a plug-in router and adapter, FlexiDepth enables adaptive computation in LLMs without modifying their original parameters. Applied to Llama-3-8B, it skips 8 out of 32 layers while maintaining full benchmark performance. Our experiments reveal that computational demands in LLMs significantly v…

6. Very close Cites this paper Not relevant

   vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains

   Dao-Min Wei, Yavuz Ferhatosmanoglu, E. Kalyvianaki · 2026 · Not summarized yet

   Abstract

   Dynamic layer skipping reduces LLM computation by allowing each token to execute only a subset of the model's layers. However, existing skippers rely on specialized generation loops and do not integrate with modern serving engines. As a result, fewer executed layers do not necessarily translate into lower serving latency: FlexiDepth skips 8 of Llama-3-8B's 32 layers on average, yet its standard generation loop decodes 14.6--21.0% more slowly than the base model. We present vSkipper, a virtualization layer that makes dynamic layer skippers pluggable in serving engines while preserving continuous batching, fixed-shape batches, paged KV caching, and captured decode graphs. At each routed layer,…

7. Very close Cited by this paper Not relevant

   SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference

   Luciano Del Corro, Allison Del Giorno, Sahaj Agarwal et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 110 times · Not summarized yet

   Abstract

   Autoregressive large language models (LLMs) have made remarkable progress in various natural language generation tasks. However, they incur high computation cost and latency resulting from the autoregressive token-by-token generation. To address this issue, several approaches have been proposed to reduce computational cost using early-exit strategies. These strategies enable faster text generation using reduced computation without applying the full computation graph to each token. While existing token-level early exit methods show promising results for online inference, they cannot be readily applied for batch inferencing and Key-Value caching. This is because they have to wait until the las…

8. Very close Cites this paper Uses this paper in depth Not relevant

   The Diminishing Returns of Early-Exit Decoding in Modern LLMs

   Rui Wei, R. Du, Han-Fei Yu et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Abstract

   In Large Language Model (LLM) inference, early-exit refers to stopping computation at an intermediate layer once the prediction is sufficiently confident, thereby reducing latency and cost. However, recent LLMs adopt improved pretraining recipes and architectures that reduce layer redundancy, potentially limiting early-exit opportunities. We re-evaluate layer-wise early-exit in modern LLMs and analyze how intermediate representations evolve during training. We introduce a metric to quantify a model's intrinsic suitability for early-exit and propose a benchmark for researchers to explore the potential early-exit benefits on different models and workloads. Our results show a diminishing trend …

Prior work it builds on

1. Very close Not relevant

   SPEED: Speculative Pipelined Execution for Efficient Decoding

   C. Hooper, Sehoon Kim, Hiva Mohammadzadeh et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 58 times · Not summarized yet

   Abstract

   Generative Large Language Models (LLMs) based on the Transformer architecture have recently emerged as a dominant foundation model for a wide range of Natural Language Processing tasks. Nevertheless, their application in real-time scenarios has been highly restricted due to the significant inference latency associated with these models. This is particularly pronounced due to the autoregressive nature of generative LLM inference, where tokens are generated sequentially since each token depends on all previous output tokens. It is therefore challenging to achieve any token-level parallelism, making inference extremely memory-bound. In this work, we propose SPEED, which improves inference effic…

2. Close Not relevant

   Accelerating Large Language Model Decoding with Speculative Sampling

   Charlie Chen, Sebastian Borgeaud, Geoffrey Irving et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 1,258 times · Not summarized yet

   Abstract

   We present speculative sampling, an algorithm for accelerating transformer decoding by enabling the generation of multiple tokens from each transformer call. Our algorithm relies on the observation that the latency of parallel scoring of short continuations, generated by a faster but less powerful draft model, is comparable to that of sampling a single token from the larger target model. This is combined with a novel modified rejection sampling scheme which preserves the distribution of the target model within hardware numerics. We benchmark speculative sampling with Chinchilla, a 70 billion parameter language model, achieving a 2-2.5x decoding speedup in a distributed setup, without comprom…

3. Close Not relevant

   Fast Inference from Transformers via Speculative Decoding

   Yaniv Leviathan, Matan Kalman, Yossi Matias · 2022 · Machine Learning · cited 2,306 times · AI summary available

   Abstract

   Inference from large autoregressive models like Transformers is slow - decoding K tokens takes K serial runs of the model. In this work we introduce speculative decoding - an algorithm to sample from autoregressive models faster without any changes to the outputs, by computing several tokens in parallel. At the heart of our approach lie the observations that (1) hard language-modeling tasks often include easier subtasks that can be approximated well by more efficient models, and (2) using speculative execution and a novel sampling method, we can make exact decoding from the large models faster, by running them in parallel on the outputs of the approximation models, potentially generating sev…

4. Close Not relevant

   Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping

   Minjia Zhang, Yu-Xiong He · 2020 · Neural Information Processing Systems · cited 126 times · Not summarized yet

   Abstract

   Recently, Transformer-based language models have demonstrated remarkable performance across many NLP domains. However, the unsupervised pre-training step of these models suffers from unbearable overall computational expenses. Current methods for accelerating the pre-training either rely on massive parallelism with advanced hardware or are not applicable to language modeling. In this work, we propose a method based on progressive layer dropping that speeds the training of Transformer-based language models, not at the cost of excessive hardware resources but from model architecture change and training technique boosted efficiency. Extensive experiments on BERT show that the proposed method ach…

5. Close Not relevant

   Confident Adaptive Language Modeling

   Tal Schuster, Adam Fisch, Jai Gupta et al. · 2022 · Neural Information Processing Systems · cited 425 times · Not summarized yet

   Abstract

   Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models' full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit d…

Papers that cite it

1. Very close Uses this paper in depth Not relevant

   SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

   Han-Kun Lin, Patrick Pynadath, Ruqi Zhang · 2026 · Not summarized yet

   Abstract

   Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft. Early-exit methods produce drafts cheaply by terminating computation at intermediate layers, but forgo the deeper representations that later layers provide and thus suffer in draft quality. Multi-token prediction preserves draft quality by emitting from the model's final hidden states, but pays for a full forward pass to produce those states at every drafting step. We propose self-speculative encoder-decoder (SEED), a self-speculative method that obtains high-quality drafts cheaply by reusing the deep…

2. Close Uses this paper in depth Not relevant

   Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

   Mostafa Elhoushi, Alexander D. Pretko, Nolan Dey et al. · 2026 · Not summarized yet

   Abstract

   Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from large language models (LLMs) pre-training recipes. While some prior work has reported that dropout can degrade accuracy, no comprehensive study has quantified, let alone mitigated, this effect. In this study, we show that layer dropout should be used in state-of-the-art LLM training, establishing best practices and scaling analysis for both training and post-training benefits. Concretely, with optimal…

3. Close Uses this paper in depth Not relevant

   ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decoding

   Zi-Yang Ma, Zi-Hong Zhang, Zu-Chao Li et al. · 2026 · cited 1 time · Not summarized yet

   Abstract

   While draft-model-free speculative decoding offers a promising path to efficient LLM inference, it is frequently constrained by stale draft candidates and the high computational cost of the verification. To address these challenges, we propose ECHO, a hierarchical dual-loop framework that exploits the functional asymmetry between LLM layers. Leveraging the high discriminative efficiency of early layers and the authoritative distribution of final layers, ECHO bifurcates inference into a high-frequency inner loop and a low-frequency outer loop. Within the inner loop, early-layer bonus logits drive rapid, multi-step draft-tree exploration at a minimal cost. Simultaneously, the outer loop perfor…

4. Close Uses this paper in depth Not relevant

   Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning

   Yan-Yu Zhu, Hoilam Pao, Niu Hu et al. · 2026 · IEEE International Conference on Acoustics, Speech, and Signal Processing · Not summarized yet

   Abstract

   Large Language Models suffer from slow autoregressive inference. While self-speculative decoding accelerates this process, its efficiency is hampered by static configurations like fixed exit layers and speculation lengths. We reframe this optimization as a \textbf{Markov Decision Process} and propose \textbf{LEDE}, a framework that uses offline reinforcement learning. LEDE learns a policy to dynamically select the optimal exit layer and speculation length based on the local context of the generated sequence at each step, balancing computational cost and draft quality. Comprehensive evaluations on Llama-2 and Llama-3 models show LEDE achieves up to a $2.0\times$$\sim$$2.7\times$ speedup over …

5. Close Uses this paper in depth Not relevant

   Deeper is Not Always Better: Mitigating the Alignment Tax via Confident Layer Decoding

   Xuanming Zhang, Sining Zhoubian, Yuxuan Chen et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Autoregressive generation in large language models (LLMs) conventionally decodes from the final layer, assuming that deeper representations yield more reliable next-token predictions. We revisit this assumption by revealing a recurring Guess-Refine-Perturb dynamic: early layers form coarse guesses, intermediate layers refine reasoning-relevant semantics, and final layers can perturb these refined predictions toward generic or alignment-preferred tokens. We introduce Confident Decoding, a training-free decoding strategy that dynamically selects the most reliable near-final layer through entropy-guided conservative backward search. We further provide a theoretical formulation of layer selectio…