# KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

**Authors:** Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami  
**arXiv:** [2401.18079](https://arxiv.org/abs/2401.18079)  
**Date:** 2024-01-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.18079v3/

## Key points

- Large Language Models (LLMs) are widely used in Natural Language Processing for tasks like document analysis and summarization
- LLMs are beneficial for tasks requiring a wide context window, but large context windows lead to increased memory consumption during inference due to activation of KV cache
- Researchers have turned to quantization to compress KV cache activations, but existing solutions struggle with accurately representing activations in ultra-low precisions like sub-4-bit
- A novel approach called KVQuant has been introduced, incorporating innovative methods for quantizing cached KV activations
- Applying the KVQuant method to popular LLM models achieved perplexity degradation of less than 0.1 with 3-bit quantization, surpassing existing approaches
- This breakthrough enables serving the LLaMA-7B model with an impressive context length of up to 1 million on a single A100-80GB GPU or up to 10 million on an 8-GPU system
- The advancement enhances efficiency and performance in LLM inference tasks and allows handling extremely large context windows with minimal memory overhead

## Summary

In the field of Natural Language Processing, Large Language Models (LLMs) have gained significant traction for various applications such as document analysis and summarization. These models are particularly useful for tasks that require a wide context window. However, the use of large context windows in these applications has led to a notable increase in memory consumption during inference. This is primarily due to the activation of KV cache, which has become a major contributor to this issue. To address this challenge, researchers have turned to quantization as a means of compressing KV cache activations. While quantization shows promise in reducing memory usage, existing solutions fall short when it comes to accurately representing activations in ultra-low precisions, specifically sub-4-bit. In response to this limitation, a novel approach called KVQuant has been introduced in this study. KVQuant incorporates innovative methods for quantizing cached KV activations including Per-Channel Key Quantization, Pre-RoPE Key Quantization, Non-Uniform KV Cache Quantization, Per-Vector Dense-and-Sparse Quantization and Q-Norm. By applying the KVQuant method to popular LLM models such as LLaMA, LLaMA-2 and Mistral on datasets like Wikitext-2 and C4, researchers were able to achieve perplexity degradation of less than 0.1 with 3-bit quantization. This surpasses existing approaches and enables serving the LLaMA-7B model with an impressive context length of up to 1 million on a single A100-80GB GPU or up to 10 million on an 8-GPU system. This breakthrough not only enhances efficiency and performance in LLM inference tasks but also opens up new possibilities for handling extremely large context windows with minimal memory overhead.