# Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework

**Authors:** Sung-En Chang, Yanyu Li, Mengshu Sun, Runbin Shi, Hayden K. -H. So, Xuehai Qian, Yanzhi Wang, Xue Lin  
**arXiv:** [2012.04240](https://arxiv.org/abs/2012.04240)  
**Date:** 2020-12-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2012.04240v1/

## Key points

- The paper proposes a novel FPGA-centric DNN quantization framework for efficient DNN inference engine on FPGA devices.
- Different quantization schemes are applied for different rows of the weight matrix to achieve better utilization of heterogeneous FPGA hardware resources.
- A hardware-friendly quantization scheme named sum-of-power-of-2 (SP2) is proposed for Gaussian-like weight distribution, while fixed-point quantization is suitable for Uniform-like weight distribution.
- An intra-layer multi-scheme quantization framework with an ensemble of SP2 and fixed-point schemes is proposed to fully explore the FPGA resources and maintain or even increase accuracy due to better matching with weight distributions.
- The authors evaluate their framework across multiple application domains with various DNNs such as CNN and RNN, achieving performance improvement of 2.1×−4.1× compared to solely exploiting DSPs for all multiplication operations.
- This research contributes to addressing the critical step of model compression required to deploy DNN models on edge devices while maintaining or even improving accuracy.
- The proposed MSQ approach offers a hardware-friendly solution that enables efficient implementation of DNN inference on edge computing platforms such as ASICs, FPGAs, and embedded systems.

## Summary

This paper focuses on the development of a novel FPGA-centric deep neural network (DNN) quantization framework that enables efficient DNN inference engine on FPGA devices through DNN quantization. The proposed solution applies different quantization schemes for different rows of the weight matrix, which is motivated by the fact that the distribution of weights in different rows is not uniform and there is potential to achieve better utilization of heterogeneous FPGA hardware resources. Unlike existing methods that use the same quantization scheme for all weights, this paper proposes a hardware-friendly quantization scheme named sum-of-power-of-2 (SP2), suitable for Gaussian-like weight distribution. The multiplication arithmetic can be replaced with logic shifter and adder, thereby enabling highly efficient implementations with the FPGA LUT resources. In contrast, fixed-point quantization is suitable for Uniform-like weight distribution and can be implemented efficiently by DSP. To fully explore the FPGA resources, an intra-layer multi-scheme quantization framework with an ensemble of SP2 and fixed-point schemes is proposed. This mixed scheme quantization (MSQ) approach can maintain or even increase accuracy due to better matching with weight distributions. The authors evaluate their FPGA-centric quantization framework across multiple application domains with various DNNs such as convolutional neural networks (CNN) and recurrent neural networks (RNN). With optimal SP2/fixed-point ratios on two FPGA devices, i.e., Zynq XC7Z020 and XC7Z045, they achieve performance improvement of 2.1×−4.1× compared to solely exploiting DSPs for all multiplication operations. This research contributes to addressing the critical step of model compression required to deploy DNN models on edge devices due to their huge model size and computation amount. The proposed MSQ approach offers a hardware-friendly solution that enables efficient implementation of DNN inference on edge computing platforms such as ASICs, FPGAs, and embedded systems while maintaining or even improving accuracy. This work is partly supported by the National Science Foundation CCF-1901378, CCF-1919117, CCF-1919289, CNS 1909172 and DARPA HR00112090055 grants.