# AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

**Authors:** Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Song Han  
**arXiv:** [2306.00978](https://arxiv.org/abs/2306.00978)  
**Date:** 2023-06-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.00978v1/

## Key points

- Authors address challenges of large model size in language models (LLMs) for memory requirements and token generation speed
- Introduce Activation-aware Weight Quantization (AWQ) for low-bit weight-only quantization to make LLMs more hardware-friendly
- AWQ focuses on protecting only 1% of salient weights through optimal per-channel scaling based on activation observations
- Method reduces quantization error without relying on backpropagation or reconstruction processes, preserving generalization ability across domains and modalities
- AWQ outperforms existing techniques in various language modeling tasks, question answering scenarios, and domain-specific benchmarks
- Efficient tensor core kernels with reorder-free online dequantization implemented to accelerate AWQ, resulting in notable speedup compared to other implementations
- AWQ compresses LLMs to 3/4 bits while maintaining high performance levels suitable for efficient deployment scenarios
- Code availability provided at https://github.com/mit-han-lab/llm-awq for further research and application

## Summary

In their paper titled "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration," authors Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han address the challenges posed by the large model size of language models (LLMs) in terms of memory requirements for serving and token generation speed. They introduce a novel approach called Activation-aware Weight Quantization (AWQ) that focuses on low-bit weight-only quantization to make LLMs more hardware-friendly. The key insight behind AWQ is the recognition that not all weights in LLMs are equally important. By identifying and protecting only 1% of salient weights through optimal per-channel scaling based on activation observations rather than weights themselves, AWQ significantly reduces quantization error. This method does not rely on backpropagation or reconstruction processes, ensuring that the generalization ability of LLMs across different domains and modalities is preserved without overfitting to calibration sets. Additionally, AWQ maintains hardware efficiency by avoiding data layout reordering. Through extensive experimentation, the authors demonstrate that AWQ outperforms existing techniques across various language modeling tasks, common sense question answering scenarios, and domain-specific benchmarks. The method's superior generalization capabilities enable exceptional quantization performance for instruction-tuned LMs and even multi-modal LMs for the first time. To further enhance efficiency, efficient tensor core kernels with reorder-free online dequantization are implemented to accelerate AWQ. This optimization results in a notable 1.45x speedup compared to GPTQ and a remarkable 1.85x improvement over cuBLAS FP16 implementation. Overall, AWQ offers a comprehensive solution for compressing LLMs to 3/4 bits while maintaining high performance levels suitable for efficient deployment scenarios. The authors provide code availability for their work at https://github.com/mit-han-lab/llm-awq, facilitating further research and application of their proposed method in real-world settings.