# GPTVQ: The Blessing of Dimensionality for LLM Quantization

**Authors:** Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, Paul Whatmough  
**arXiv:** [2402.15319](https://arxiv.org/abs/2402.15319)  
**Date:** 2024-02-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.15319v1/

## Key points

- A new post-training vector quantization technique designed for large language models.
- Increasing quantization dimensionality can improve the balance between size and accuracy in neural network quantization.
- GPTVQ method involves interleaving specific column quantizations with weight updates to enhance performance.
- GPTVQ method is tailored for large language models, demonstrating efficiency in processing complex models.
- GPTVQ improves trade-offs between size and accuracy while enhancing efficiency on mobile CPUs.

## Summary

, , , , 

In the study "GPTVQ: The Blessing of Dimensionality for LLM Quantization," the authors explore how increasing the quantization dimensionality can significantly enhance the trade-off between size and accuracy in neural network quantization. They introduce the GPTVQ method, a rapid post-training vector quantization (VQ) technique tailored for Large Language Models (LLMs). This method involves interleaving quantization of specific columns with updates to unquantized weights, leveraging information from the Hessian of per-layer output reconstruction Mean Squared Error (MSE). The initialization of quantization codebooks is performed using an efficient data-aware version of the Expectation-Maximization (EM) algorithm. Subsequently, these codebooks are updated and further compressed through integer quantization and Singular Value Decomposition (SVD)-based compression techniques. <kw>GPTVQ:</kw> In their study, the authors propose a new post-training vector quantization technique specifically designed for large language models. <kw>Dimensionality:</kw> By increasing the dimensionality of quantization, significant improvements can be made in balancing size and accuracy in neural network quantization. <kw>Quantization:</kw> The GPTVQ method involves interleaving specific column quantizations with weight updates to improve overall performance. <kw>Large Language Models:</kw> The GPTVQ method is tailored specifically for large language models, showcasing its efficiency in processing these complex models. <kw>Trade-Off:</kw> Through comprehensive experimentation and analysis, GPTVQ emerges as a state-of-the-art approach that improves trade-offs between size and accuracy while also enhancing efficiency on mobile CPUs.