# ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization

**Authors:** Zechun Liu, Changsheng Zhao, Hanxian Huang, Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, Lin Xiao, Yuandong Tian, Bilge Soran, Raghuraman Krishnamoorthi, Tijmen Blankevoort, Vikas Chandra  
**arXiv:** [2502.02631](https://arxiv.org/abs/2502.02631)  
**Date:** 2025-02-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.02631v2/

## Key points

- Debate over optimal bit-width for quantized model optimization:
- Some advocate for 4-bit quantization
- Others argue that 1.58-bit yields superior results
- Enables comparisons across bit-widths from 1-bit to 4-bit
- Reveals significant learning transition between 2 and 3 bits
- Surpasses previous methods tailored to specific bit widths
- Ternary model with 600 million parameters outperformed state-of-the-art ternary model with 3 billion parameters while using fewer parameters
- Ternary, 2-bit, and 3-bit quantization consistently deliver comparable performance in size-accuracy trade-off
- Outperform both 4-bit and binary quantization approaches
- Promise of 2-bit quantization in terms of hardware limitations:
- Significant promise in memory reduction and speed enhancement

## Summary

In the realm of quantized model optimization, the debate over the optimal bit-width for balancing model size and accuracy has persisted. While some advocate for 4-bit quantization, others argue that 1.58-bit yields superior results. However, the absence of a comprehensive framework for various bit-widths has rendered such conclusions uncertain. Enter ParetoQ, a groundbreaking unified framework that enables rigorous comparisons across a spectrum of quantization settings ranging from 1-bit to 4-bit. Through meticulous experimentation, it has been revealed that there exists a significant learning transition between 2 and 3 bits: Models fine-tuned with 3-bits or more tend to maintain proximity to their original pre-trained distributions, whereas those trained with 2 bits or fewer undergo drastic representation changes. By honing training methodologies and refining quantization functions, ParetoQ surpasses all previous methods tailored to specific bit widths. Notably, the ParetoQ ternary model boasting 600 million parameters has even outperformed the previous state-of-the-art ternary model with 3 billion parameters in terms of accuracy while utilizing only one-fifth of the parameters. Extensive experimentation further demonstrates that ternary, 2-bit, and 3-bit quantization consistently deliver comparable performance in the size-accuracy trade-off and generally outperform both 4-bit and binary quantization approaches. Considering hardware limitations, it is highlighted that 2-bit quantization holds significant promise in terms of memory reduction and speed enhancement. The findings presented by ParetoQ not only shed light on the intricate nuances of low-bit LLM quantization but also pave the way for more efficient and effective model optimization strategies in the field of machine learning.