# OneBit: Towards Extremely Low-bit Large Language Models

**Authors:** Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, Wanxiang Che  
**arXiv:** [2402.11295](https://arxiv.org/abs/2402.11295)  
**Date:** 2024-02-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.11295v1/

## Key points

- The paper introduces a novel approach to model quantification by quantizing weight matrices of Large Language Models (LLMs) to 1-bit
- Aims to reduce storage and computational overheads in deploying LLMs
- OneBit includes a unique 1-bit parameter representation method and effective parameter initialization technique based on matrix decomposition
- Experimental results show that OneBit achieves impressive performance, with at least 83% of the non-quantized performance even with 1-bit weight matrices
- Evaluation experiment demonstrates effectiveness through perplexity and zero-shot accuracy metrics on datasets like WikiText2 and C4
- Lower perplexity values indicate better preservation of the output distribution, while high accuracies in zero-shot tasks highlight robustness of compressed models
- Outperforms existing quantization techniques for extremely low bit-width deployment of LLMs
- Presents a promising solution for deploying highly anticipated LLMs with reduced bit-width values without compromising performance

## Summary

The paper "OneBit: Towards Extremely Low-bit Large Language Models" introduces a novel approach to model quantification by boldly quantizing the weight matrices of Large Language Models (LLMs) to 1-bit. This groundbreaking method aims to significantly reduce storage and computational overheads in deploying LLMs. The proposed 1-bit quantization-aware training (QAT) framework named OneBit includes a unique 1-bit parameter representation method and an effective parameter initialization technique based on matrix decomposition. Experimental results demonstrate that OneBit achieves impressive performance, with at least 83% of the non-quantized performance, even when using only 1-bit weight matrices. The evaluation experiment showcases the effectiveness of OneBit by reporting perplexity and zero-shot accuracy metrics on datasets like WikiText2 and C4. Lower perplexity values indicate better preservation of the output distribution of the original model, while high accuracies in zero-shot tasks such as Winograde, HellaSwag, PIQA, and BoolQ highlight the robustness of the compressed models. Furthermore, the study analyzes OneBit's ability to transfer knowledge from the original models and compares its performance with other methods. The results demonstrate that OneBit outperforms existing quantization techniques when it comes to extremely low bit-width deployment of LLMs. In conclusion, OneBit presents a promising solution for deploying highly anticipated LLMs with significantly reduced bit-width values without compromising performance. The innovative approach and robust experimental results make it a valuable contribution to the field of model quantification and optimization.