# RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

**Authors:** Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, Alexey Tumanov  
**arXiv:** [2502.14051](https://arxiv.org/abs/2502.14051)  
**Date:** 2025-02-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.14051v1/

## Key points

- Transformer-based Large Language Models heavily rely on KV cache for extended context handling during decoding
- RocketKV introduced as a training-free KV cache compression strategy
- Consists of coarse-grain KV cache eviction using SnapKV++ and fine-grain top-k sparse attention through hybrid attention method
- RocketKV reduces memory bandwidth and storage demands during decoding while maintaining accuracy comparable to full KV cache attention
- Up to 3x end-to-end speedup and up to 31% peak memory reduction on NVIDIA H100 GPU compared to full KV cache baseline
- Negligible accuracy loss across various long-context tasks with RocketKV
- Varying kernel sizes under RULER with different sequence lengths lead to linear improvements in speedup and peak memory savings
- Ablation studies confirm efficacy of RocketKV, highlighting superiority of SnapKV with grouped-query attention enhancement over original SnapKV for performance with low token budgets

## Summary

, , , , 

Transformer-based Large Language Models heavily rely on KV cache to efficiently handle extended contexts during the decoding phase. However, the size of the KV cache grows with the input length, causing memory bandwidth and capacity issues as decoding progresses. To tackle this challenge, RocketKV is introduced as a training-free KV cache compression strategy. It consists of two stages: coarse-grain KV cache eviction using SnapKV++ in the first stage and fine-grain top-k sparse attention through a hybrid attention method in the second stage. RocketKV effectively reduces memory bandwidth and storage demands of the KV cache during decoding while maintaining accuracy comparable to full KV cache attention. The approach delivers up to 3x end-to-end speedup and up to 31% peak memory reduction on an NVIDIA H100 GPU compared to the full KV cache baseline. <break>

Notably, RocketKV shows negligible accuracy loss across various long-context tasks. Furthermore, experiments demonstrate that varying kernel sizes under RULER with different sequence lengths lead to linear improvements in both speedup and peak memory savings. Ablation studies confirm the efficacy of RocketKV, particularly highlighting the superiority of SnapKV with grouped-query attention enhancement over the original SnapKV in terms of performance with low token budgets. In conclusion, <keyword>RocketKV presents a novel solution for accelerating Long-Context LLM Inference by efficiently compressing KV cache without sacrificing accuracy,</keyword> showcasing significant improvements in speedup and memory efficiency compared to traditional methods.