# DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables

**Authors:** Darshan C. Ganji, Saad Ashfaq, Ehsan Saboori, Sudhakar Sah, Saptarshi Mitra, MohammadHossein AskariHemmat, Alexander Hoffman, Ahmed Hassanien, Mathieu Léonardon  
**arXiv:** [2304.09049](https://arxiv.org/abs/2304.09049)  
**Date:** 2023-04-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.09049v1/

## Key points

- DeepGEMM is a lookup table based approach to accelerate ultra low-precision inference on CPU architectures
- Recent progress in ultra low-bit quantization has shown significant improvements in latency, memory footprint, and energy consumption on edge devices
- Commodity SIMD hardware typically supports no less than 8-bit precision, which limits the execution of ultra low-precision convolutional neural networks
- DeepGEMM precomputes all possible products of weights and activations and stores them in a lookup table for efficient access at inference time
- DeepGEMM enables the execution of ultra low-precision convolutional neural networks on SIMD hardware with improved performance compared to corresponding 8-bit integer kernels in the QNNPACK framework up to 1.74x on x86 platforms for 2-bit implementation
- Quantization methods such as Learned Step Size Quantization can achieve model accuracy comparable to full-precision floating-point baselines even with sub-byte quantization
- DeepGEMM promises significant improvements in latency, memory footprint, and energy consumption while maintaining model accuracy for ultra low-bit quantized models deployed on mainstream CPU devices

## Summary

DeepGEMM is a lookup table based approach proposed by a team of researchers to accelerate ultra low-precision inference on CPU architectures. Recent progress in ultra low-bit quantization has shown significant improvements in latency, memory footprint, and energy consumption on edge devices. To address the limitation of commodity SIMD hardware that typically supports no less than 8-bit precision, DeepGEMM precomputes all possible products of weights and activations and stores them in a lookup table for efficient access at inference time. This method enables the execution of ultra low-precision convolutional neural networks on SIMD hardware with improved performance compared to corresponding 8-bit integer kernels in the QNNPACK framework up to 1.74x on x86 platforms for 2-bit implementation. Moreover, quantization methods such as Learned Step Size Quantization can achieve model accuracy comparable to full-precision floating-point baselines even with sub-byte quantization. Therefore, DeepGEMM promises significant improvements in latency, memory footprint, and energy consumption while maintaining model accuracy for ultra low-bit quantized models deployed on mainstream CPU devices.