# GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

**Authors:** Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh  
**arXiv:** [2210.17323](https://arxiv.org/abs/2210.17323)  
**Date:** 2022-10-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.17323v2/

## Key points

- Generative Pre-trained Transformer models (GPT or OPT) are exceptional in complex language modeling tasks.
- The large size and high computational and storage costs of GPT models limit their usability.
- The authors propose a new one-shot weight quantization method called GPTQ based on approximate second-order information that is highly accurate and efficient.
- GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours while reducing the bitwidth down to 3 or 4 bits per weight with negligible accuracy degradation relative to the uncompressed baseline.
- This more than doubles the compression gains relative to previously-proposed one-shot quantization methods while preserving accuracy.
- GPTQ can still provide reasonable accuracy even in extreme quantization regimes where weights are quantized to 2-bit or even ternary levels.
- The implementation of GPTQ is available at https://github.com/IST-DASLab/gptq along with acknowledgements for funding from the European Research Council under the European Union’s Horizon 2020 program as well as experimental support from Eldar Kurtic and from IST Austria IT department.
- Experimentally, improvements can be leveraged for end-to-end inference speedups over FP16 of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000).
- While our study focused on "leading accuracy" metrics such as perplexity which is standard in literature, we believe a thorough study of the impact of compression upon secondary measures such as bias effects is warranted.
- In conclusion, GPTQ offers a highly accurate and efficient method for compressing large language models via quantization with little-to-no accuracy loss resulting in end-to-end inference speedups over FP16.

## Summary

Generative Pre-trained Transformer models (GPT or OPT) have revolutionized natural language processing with their exceptional performance in complex language modeling tasks. However, their large size and high computational and storage costs limit their usability. To address this issue, the authors propose a new one-shot weight quantization method called GPTQ based on approximate second-order information that is both highly accurate and efficient. This method can quantize GPT models with 175 billion parameters in approximately four GPU hours while reducing the bitwidth down to 3 or 4 bits per weight with negligible accuracy degradation relative to the uncompressed baseline. This more than doubles the compression gains relative to previously-proposed one-shot quantization methods while preserving accuracy. The authors demonstrate that their method can still provide reasonable accuracy even in extreme quantization regimes where weights are quantized to 2-bit or even ternary levels. They show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16 of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000). The implementation of GPTQ is available at https://github.com/IST-DASLab/gptq along with acknowledgements for funding from the European Research Council under the European Union’s Horizon 2020 program as well as experimental support from Eldar Kurtic and from IST Austria IT department. While our study focused on "leading accuracy" metrics such as perplexity which is standard in literature, we believe a thorough study of the impact of compression upon secondary measures such as bias effects is warranted. Our work makes inference on extremely large language models more accessible, for better or for worse; thus making it necessary to understand its power and limitations even more stringent over time as tools become easier to use and deploy. In conclusion, GPTQ offers a highly accurate and efficient method for compressing large language models via quantization with little-to-no accuracy loss resulting in end-to-end inference speedups over FP16.