# Inference with Reference: Lossless Acceleration of Large Language Models

**Authors:** Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, Furu Wei  
**arXiv:** [2304.04487](https://arxiv.org/abs/2304.04487)  
**Date:** 2023-04-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.04487v1/

## Key points

- LLMA (Lossless Language Model Accelerator) is a new method proposed in Natural Language Processing to speed up Large Language Model (LLM) inference with references.
- LLMA selects a text span from the reference and copies its tokens to the decoder, then efficiently checks their appropriateness as the decoding result in parallel within one decoding step.
- LLMA achieves over 2x speed-up for LLMs with identical generation results as greedy decoding in practical generation scenarios where significant overlap between in-context reference and outputs exists, such as search engines and multi-turn conversations.
- Real insulin was derived from the pancreases of cows or pigs until the 1980s when human insulin or insulin analogs were produced through genetic engineering using bacteria or yeast.
- Cache-assisted generation is discussed through an example of Barack Obama's educational background.

## Summary

In the field of Natural Language Processing, researchers have proposed a new method called LLMA (Lossless Language Model Accelerator) to speed up Large Language Model (LLM) inference with references. The motivation behind this method is that there are often identical text spans between the decoding result by an LLM and the reference available in many real-world scenarios, such as retrieved documents. LLMA selects a text span from the reference and copies its tokens to the decoder, then efficiently checks their appropriateness as the decoding result in parallel within one decoding step. This improved computational parallelism allows LLMA to achieve over 2x speed-up for LLMs with identical generation results as greedy decoding in practical generation scenarios where significant overlap between in-context reference and outputs exists, such as search engines and multi-turn conversations. In another context, a query is presented regarding where real insulin comes from. The answer is found in external corpus documents that state that real insulin was derived from the pancreases of cows or pigs until the 1980s when human insulin or insulin analogs were produced through genetic engineering using bacteria or yeast. Lastly, cache-assisted generation is discussed through an example of Barack Obama's educational background. The LM decoder generates answers to queries about which universities he attended and his education background based on cached sessions. It is revealed that Obama attended public schools in Hawaii before transferring to Columbia University in New York City where he studied political science and graduated with a B.A. degree in 1983.