# KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache

**Authors:** Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia Hu  
**arXiv:** [2402.02750](https://arxiv.org/abs/2402.02750)  
**Date:** 2024-02-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.02750v2/

## Key points

- Efficiently serving large language models (LLMs) requires batching of many requests to reduce cost per request
- Increasing batch sizes and context lengths can bottleneck the key-value (KV) cache in terms of speed and memory usage
- Quantization can reduce the size of the KV cache by decreasing total bytes
- Developed KIVI, a tuning-free 2bit KV cache quantization algorithm based on analyzing element distribution in popular LLMs' KV caches
- Implementing hardware-friendly techniques allows models like Llama, Falcon, and Mistral to use significantly less peak memory while maintaining quality
- Reduction in memory usage allows for larger batch sizes and improves throughput on real LLM inference workloads
- Applied KIVI to various models with minimal impact on accuracy across challenging long context generation tasks in LongBench results
- NIAH results show that KIVI maintains retrieval ability even with 2bit KV Cache
- Benchmarking KIVI on GSM8K in ablation studies to analyze hyperparameters group size G and residual length R on model performance
- Group size significantly impacts KV cache compression effectiveness under long inputs; reasonably large residual length is crucial for performance boosts on difficult tasks like GSM8K
- Efficiency comparison shows that KIVI is an effective method for compressing KV caches without sacrificing model quality across various LLMs and tasks

## Summary

Efficiently serving large language models (LLMs) requires batching of many requests to reduce the cost per request. However, as batch sizes and context lengths increase, the key-value (KV) cache becomes a bottleneck in terms of speed and memory usage. To address this issue, quantization can reduce the size of the KV cache by decreasing its total bytes. Our study focused on analyzing element distribution in popular LLMs' KV caches to optimize quantization. We developed KIVI, a tuning-free 2bit KV cache quantization algorithm based on our findings. By implementing hardware-friendly techniques, models like Llama, Falcon, and Mistral can maintain quality while using significantly less peak memory. This reduction allows for larger batch sizes and improves throughput on real LLM inference workloads. In LongBench results, we applied KIVI to various models with minimal impact on accuracy across challenging long context generation tasks. Additionally, our NIAH results show that KIVI maintains retrieval ability even with 2bit KV Cache. In ablation studies, we benchmarked KIVI on GSM8K to analyze the effect of hyperparameters group size G and residual length R on model performance. The choice of group size significantly impacts KV cache compression effectiveness under long inputs. While there is no consistent pattern between residual lengths and model accuracy, having a reasonably large residual length is crucial for performance boosts on difficult tasks like GSM8K. Overall, our efficiency comparison shows that KIVI is an effective method for compressing KV caches without sacrificing model quality across various LLMs and tasks. The source code for KIVI is available at https://github.com/jy-yuan/KIVI for further exploration and implementation.