# SqueezeLLM: Dense-and-Sparse Quantization

**Authors:** Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, Kurt Keutzer  
**arXiv:** [2306.07629](https://arxiv.org/abs/2306.07629)  
**Date:** 2023-06-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.07629v4/

## Key points

- Deployment of Generative Large Language Models (LLMs) for inference is challenging due to high resource requirements
- SqueezeLLM, a post-training quantization framework, addresses the bottleneck for generative inference with LLMs by focusing on memory bandwidth
- Sensitivity-based non-uniform quantization optimizes bit precision assignment based on second-order information
- Dense-and-Sparse decomposition stores outliers and sensitive weight values efficiently
- Achieves lossless compression to ultra-low precisions of up to 3-bit while maintaining high quantization performance under the same memory constraint
- Reduces perplexity gap from FP16 baseline by up to 2.1x compared to state-of-the-art methods with similar memory requirements when applied to LLaMA models
- A6000 GPU deployment achieves up to a 2.3x speedup compared to the baseline
- Presented at ICML 2024, offering promising advancements in optimizing large language models for more efficient deployment and improved performance outcomes

## Summary

In the realm of Generative Large Language Models (LLMs), the deployment for inference has posed a significant challenge due to their high resource requirements. Existing frameworks have resorted to multi-GPU inference pipelines or smaller, less performant models to tackle this issue. However, a breakthrough comes in the form of SqueezeLLM, a post-training quantization framework introduced by Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. addresses the bottleneck for generative inference with LLMs by focusing on memory bandwidth rather than compute power, especially for single batch inference scenarios. The framework enables lossless compression to ultra-low precisions of up to 3-bit while achieving higher quantization performance under the same memory constraint. This is made possible through two innovative concepts incorporated into 

1. Sensitivity-based non-uniform quantization: This feature searches for the optimal bit precision assignment based on second-order information. 2. Dense-and-Sparse decomposition: By storing outliers and sensitive weight values in an efficient sparse format, this method enhances efficiency. When applied to LLaMA models, 3-bit quantization significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x compared to state-of-the-art methods with similar memory requirements. Moreover, when deployed on an A6000 GPU achieves up to a 2.3x speedup compared to the baseline. This groundbreaking work was presented at ICML 2024 and offers promising advancements in optimizing large language models for more efficient deployment and improved performance outcomes. The code for  is openly available at https://github.com/SqueezeAILab/SqueezeLLM for further exploration and implementation in related research endeavors.