# No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

**Authors:** June Yong Yang, Byeongwook Kim, Jeongin Bae, Beomseok Kwon, Gunho Park, Eunho Yang, Se Jung Kwon, Dongsoo Lee  
**arXiv:** [2402.18096](https://arxiv.org/abs/2402.18096)  
**Date:** 2024-02-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.18096v1/

## Key points

- Key-Value (KV) Caching is crucial for accelerating the inference speed and throughput of generative Large Language Models (LLMs).
- Memory footprint of the KV cache poses a significant challenge in LLM deployment.
- Recent methods focus on selecting and evicting unimportant KV pairs to reduce memory consumption.
- Eviction of KV pairs can have detrimental impacts on generative processes, leading to safety breaches, hallucinations, and loss of context.
- Preserving even a small amount of information from evicted KV pairs through reduced precision quantization can mitigate degradation.
- Mixed-precision KV cache (MiKV) method proposed by June Yong Yang et al. involves compressing caches while retaining context details for important KV pairs at higher precision levels.
- MiKV offers a balance between compression ratio and performance, addressing memory consumption challenges in large language models while maintaining high-quality generative outputs.

## Summary

In the realm of accelerating the inference speed and throughput of generative Large Language Models (LLMs), Key-Value (KV) Caching has emerged as a crucial technique. However, a significant challenge arises in LLM deployment due to the memory footprint of the KV cache, which acts as a critical bottleneck. Recent methods have been introduced to address this issue by selecting and evicting unimportant KV pairs from the cache to reduce memory consumption. However, the potential consequences of eviction on the generative process have not been extensively explored. In their study titled "No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization," June Yong Yang et al. conducted an in-depth analysis on the detrimental impact of cache eviction. The research revealed that unforeseen risks surface when information contained in evicted KV pairs is completely discarded. This leads to safety breaches, hallucinations during generation processes, and loss of context. Interestingly, it was discovered that preserving even a small amount of information from evicted KV pairs through reduced precision quantization significantly mitigates degradation. Conversely, important KV pairs must be maintained at higher precision levels to uphold generation quality standards. Motivated by these findings, the researchers proposed a novel method called Mixed-precision KV cache (MiKV). This approach involves compressing caches while simultaneously retaining context details by storing evicted KV pairs in low precision and ensuring generation quality by keeping important KV pairs in high precision. Experimental evaluations conducted on various benchmarks and LLM backbones demonstrated that MiKV offers an exceptional trade-off between compression ratio and performance compared to existing baselines. This innovative solution showcases promising advancements in addressing memory consumption challenges associated with KV caching in large language models while maintaining high-quality generative outputs.