# Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases

**Authors:** Xiaoxia Wu, Cheng Li, Reza Yazdani Aminabadi, Zhewei Yao, Yuxiong He  
**arXiv:** [2301.12017](https://arxiv.org/abs/2301.12017)  
**Date:** 2023-01-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.12017v1/

## Key points

- Authors address the challenge of improving deployment efficiency of transformer-based language models
- INT8 quantization reduces memory cost and latency while preserving model accuracy
- Feasibility of using INT4 quantization for language models is investigated
- INT4 quantization introduces no or negligible accuracy degradation for encoder-only and encoder-decoder models, but significant drop for decoder-only models
- Highly optimized end-to-end INT4 encoder inference pipeline developed to support different quantization strategies
- INT4 pipeline is 8.5 times faster for latency-oriented scenarios and up to 3 times faster for throughput-oriented scenarios compared to FP16 inference
- State-of-the-art BERT INT8 performance improved by up to 1.7 times compared to FasterTransformer
- Insights provided on why applying INT4 quantization fails in some cases for decoder-only models
- Compatibility with other compression techniques like pruning and layer reduction explored
- Potential benefits of using INT4 quantization include improved latency without significant accuracy degradation

## Summary

In this study, the authors address the challenge of improving the deployment efficiency of transformer-based language models, which are known for their high computation and memory cost. While INT8 quantization has been effective in reducing memory cost and latency while preserving model accuracy, it remains unclear whether INT4 quantization can further improve latency by leveraging its ability to double peak hardware throughput. To investigate the feasibility of using INT4 quantization for language models, the authors conducted a comprehensive analysis. They found that using INT4 quantization introduces no or negligible accuracy degradation for encoder-only and encoder-decoder models. However, it does cause a significant accuracy drop for decoder-only models. To realize the performance gain using INT4 quantization, the authors developed a highly-optimized end-to-end INT4 encoder inference pipeline that supports different quantization strategies. Their INT4 pipeline demonstrated impressive results, being 8.5 times faster for latency-oriented scenarios and up to 3 times faster for throughput-oriented scenarios compared to FP16 inference. Additionally, it improved the state-of-the-art BERT INT8 performance from FasterTransformer by up to 1.7 times. The study also provides insights into why applying INT4 quantization to decoder-only models fails in some cases and explores its compatibility with other compression techniques like pruning and layer reduction. Overall, this work highlights the potential benefits of using INT4 quantization for transformer-based language models in terms of improved latency without significant accuracy degradation. The development of an optimized end-to-end pipeline further enhances these benefits and opens avenues for future research on combining INT4 quantization with other compression techniques.