# PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs

**Authors:** Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, Ping Luo  
**arXiv:** [2410.05265](https://arxiv.org/abs/2410.05265)  
**Date:** 2024-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.05265v1/

## Key points

- Quantization in Large Language Models (LLMs) is crucial for enhancing memory efficiency and inference speed.
- Existing activation quantization methods often focus on channel-wise outliers, neglecting token-wise outliers and relying on costly per-token dynamic quantization.
- PrefixQuant is a novel technique that isolates outlier tokens offline without re-training, preventing the generation of outlier tokens during inference by prefixing them in the Key-Value (KV) cache.
- PrefixQuant enables efficient per-tensor static quantization, surpassing expensive per-token dynamic quantization techniques.
- When applied to W4A4KV4 Llama-3-8B models, PrefixQuant coupled with per-tensor static quantization achieves impressive results with improved perplexity and accuracy compared to previous methods like QuaRot.
- W4A4 quantized models using PrefixQuant are faster than FP16 models, showing notable performance gains in terms of speed compared to QuaRot models.
- The authors have made their code available for further exploration and implementation at https://github.com/ChenMnZ/PrefixQuant.

## Summary

In the realm of Large Language Models (LLMs), the process of quantization plays a crucial role in enhancing memory efficiency and inference speed. However, existing methods for activation quantization often focus on addressing channel-wise outliers while neglecting token-wise outliers. This leads to a reliance on costly per-token dynamic quantization. To combat this limitation, a novel technique called PrefixQuant has been introduced. This innovative approach isolates outlier tokens offline without the need for re-training. By identifying high-frequency outlier tokens and prefixing them in the Key-Value (KV) cache, PrefixQuant prevents the generation of outlier tokens during inference. This simplifies the quantization process and stands out as the first method to enable efficient per-tensor static quantization that surpasses expensive per-token dynamic quantization techniques. For instance, in the context of W4A4KV4 Llama-3-8B models (comprising 4-bit weight, 4-bit activation, and 4-bit KV cache), PrefixQuant coupled with per-tensor static quantization achieves impressive results. It delivers a perplexity of 7.43 on WikiText2 and an average accuracy of 71.08% across five common-sense reasoning tasks. This outperforms previous per-token dynamic quantization methods like QuaRot by improving perplexity by 0.98 points and accuracy by +5.98 points. Moreover, when considering inference speed, W4A4 quantized models utilizing PrefixQuant exhibit notable performance gains compared to FP16 models. They are shown to be 1.60x to 2.81x faster than their FP16 counterparts and outperform QuaRot models by 1.2x to 1.3x in terms of speed. The authors behind this groundbreaking work - Mengzhao Chen, Yi Liu, Jiahao Wang, Yi Bin, Wenqi Shao, and Ping Luo - have made their code available at https://github.com/ChenMnZ/PrefixQuant for further exploration and implementation. have also delved into addressing channel-wise outliers through mixed-precision approaches to enhance activation quantization performance further. Overall, PrefixQuant emerges as a game-changing technique that not only streamlines the quantization process but also significantly boosts model performance in terms of perplexity scores, accuracy rates, and inference speeds when applied to Large Language Models like Llama-3-8B models with W4A4KV4 configurations.