# LUT-NN: Towards Unified Neural Network Inference by Table Lookup

**Authors:** Xiaohu Tang, Yang Wang, Ting Cao, Li Lyna Zhang, Qi Chen, Deng Cai, Yunxin Liu, Mao Yang  
**arXiv:** [2302.03213](https://arxiv.org/abs/2302.03213)  
**Date:** 2023-02-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.03213v1/

## Key points

- Deep Neural Network (DNN) inference is computationally intensive and costly
- LUT-NN is a technique for DNN inference by table lookup
- LUT-NN learns typical features, called centroids, of each layer from training data
- Centroids are precomputed with model weights and saved in tables for future input
- LUT-NN achieves comparable accuracy (<5% difference) with original models on real complex datasets such as CIFAR, ImageNet, and GLUE
- LUT-NN simplifies computing operators to only two: closest centroid search and table lookup
- LUT-NN has been implemented for Intel and ARM CPUs reducing model size up to 3.5x for CNN models and 7x for BERT while achieving real speedup up to 7x for BERT and 2x for ResNet latency-wise.
- Current hardware design limitations result in lower speedup than theoretical results.
- The authors expect first-class table lookup support in future hardware designs to unleash the full potential of LUT-NN.
- The proposed approach simplifies DNN inference while maintaining high accuracy levels without requiring extensive system development or resource costs.

## Summary

Deep Neural Network (DNN) inference is a computationally intensive task that requires significant system development and resource cost. To address this challenge, the authors propose LUT-NN, a technique for DNN inference by table lookup. LUT-NN eliminates the need for diverse computation kernels and reduces running costs by learning typical features, called centroids, of each layer from training data. These centroids are precomputed with model weights and saved in tables. For future input, the results of the closest centroids with the input features can be directly read from the table as an approximation of layer output. The authors introduce a novel centroid learning technique for DNN that enables centroid learning through backpropagation and adapts three levels of approximation to minimize model loss. By this technique, LUT-NN achieves comparable accuracy (<5% difference) with original models on real complex datasets such as CIFAR, ImageNet, and GLUE. LUT-NN simplifies computing operators to only two: closest centroid search and table lookup. LUT-NN has been implemented for Intel and ARM CPUs reducing model size up to 3.5x for CNN models and 7x for BERT while achieving real speedup up to 7x for BERT and 2x for ResNet latency-wise. However, current hardware design limitations result in lower speedup than theoretical results. The authors expect first-class table lookup support in future hardware designs to unleash the full potential of LUT-NN. The proposed approach has significant implications as it simplifies DNN inference while maintaining high accuracy levels without requiring extensive system development or resource costs.