# 1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs

**Authors:** Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei  
**arXiv:** [2410.16144](https://arxiv.org/abs/2410.16144)  
**Date:** 2024-10-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.16144v1/

## Key points

- Authors Wang, Zhou, Song, Mao, Ma, Wang, Xia, and Wei focus on advancements in 1-bit Large Language Models (LLMs), particularly BitNet and BitNet b1.58
- Developments aim to enhance LLM efficiency by improving speed and reducing energy consumption for local deployment on various devices
- Introduction of bitnet.cpp software stack tailored to optimize performance of 1-bit LLMs
- Series of kernels within bitnet.cpp enable fast and lossless inference of ternary BitNet b1.58 LLMs on CPUs
- Extensive experimentation shows significant speedups with speedups ranging from 2.37x to 6.17x on x86 CPUs and 1.37x to 5.07x on ARM CPUs
- Code available at https://github.com/microsoft/BitNet for further exploration and implementation
- Emphasizes the potential benefits of utilizing 1-bit Large Language Models like BitNet b1.58 for efficient inference tasks on CPUs

## Summary

In their paper titled "1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs," authors Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, and Furu Wei delve into the recent advancements in 1-bit Large Language Models (LLMs), specifically focusing on BitNet and BitNet b1.58. These developments offer a promising avenue for improving the efficiency of LLMs by enhancing speed and reducing energy consumption while also enabling local deployment across a wide array of devices. To harness the full potential of 1-bit LLMs, the authors introduce bitnet.cpp—a tailored software stack designed to optimize the performance of these models. They have developed a series of kernels within this software stack to facilitate fast and lossless inference of ternary BitNet b1.58 LLMs specifically on CPUs. Through extensive experimentation, Wang et al. demonstrate that bitnet.cpp yields significant speedups across various model sizes. On x86 CPUs, the speedups range from 2.37x to 6.17x, while on ARM CPUs, they range from 1.37x to 5.07x. This substantial improvement in performance underscores the efficacy of their approach in enhancing the efficiency of LLMs. For those interested in exploring further or implementing this technology, the authors have made their code available at https://github.com/microsoft/BitNet. This resource provides a valuable tool for researchers and practitioners seeking to leverage 1-bit LLMs for enhanced computational capabilities with reduced energy consumption on CPU architectures. Overall, this work sheds light on the potential benefits of utilizing 1-bit Large Language Models like BitNet b1.58 and underscores the importance of tailored software solutions like bitnet.cpp in unlocking their full capabilities for efficient inference tasks on CPUs.