# Fast Inference from Transformers via Speculative Decoding

**Authors:** Yaniv Leviathan, Matan Kalman, Yossi Matias  
**arXiv:** [2211.17192](https://arxiv.org/abs/2211.17192)  
**Date:** 2022-11-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.17192v2/

## Key points

- Efficient inference from large autoregressive models like Transformers is a key focus.
- Various techniques have been developed to speed up inference, including distillation, sparcification, quantization, and architecture modifications.
- Adaptive computation methods adjust computation based on task difficulty.
- The Wisdom of Committees method leverages smaller models but may not guarantee identical outputs due to heuristic decision-making.
- A novel speculative decoding algorithm is introduced to sample from autoregressive models faster without altering outputs.
- Speculative execution and innovative sampling techniques are utilized to accelerate exact decoding by recognizing simpler subtasks within complex language-modeling tasks.
- This approach enhances existing off-the-shelf models without requiring retraining or modifications.
- The technique showcases 2X-3X acceleration on T5-XXL compared to standard implementations while maintaining accuracy and efficiency.

## Summary

In the realm of efficient inference from large autoregressive models like Transformers, the issue of slow decoding has been extensively explored. Various techniques have been developed to speed up inference, with a focus on improving efficiency for all tokens. These approaches include distillation, sparcification, quantization, and architecture modifications. Adaptive computation methods have also been investigated, which adjust the amount of computation based on task difficulty. One notable approach is the Wisdom of Committees method, which leverages smaller models but relies on heuristic decision-making and may not guarantee identical outputs. In this work, a novel speculative decoding algorithm is introduced to sample from autoregressive models faster without altering outputs. By recognizing that complex language-modeling tasks often contain simpler subtasks that can be approximated efficiently by other models, the proposed method utilizes speculative execution and innovative sampling techniques to accelerate exact decoding from large models. This involves running parallel computations on outputs from approximation models to potentially generate multiple tokens simultaneously while maintaining distribution integrity. Unlike adaptive computation methods that typically require architectural changes or custom model training, this approach can enhance existing off-the-shelf models without retraining or modifications. The effectiveness of this technique is demonstrated on T5-XXL, showcasing a 2X-3X acceleration compared to standard implementations without any changes in output results. Overall, speculative decoding offers a promising solution for speeding up inference from large autoregressive models while preserving accuracy and efficiency.