# FPGA Implementation of Convolutional Neural Network for Real-Time Handwriting Recognition

**Authors:** Shichen Qiao, Haining Qiu, Lingkai Zhao, Qikun Liu, Eric J. Hoffman  
**arXiv:** [2306.13557](https://arxiv.org/abs/2306.13557)  
**Date:** 2023-06-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.13557v2/

## Key points

- Machine Learning (ML) is a rapidly growing field in Computer Science
- Hardware implementations of popular ML architectures are needed to optimize performance, reliability, and resource usage
- The authors designed a highly-configurable, real-time device using an Altera DE1 FPGA Kit for recognizing handwritten letters and digits
- Various engineering standards such as IEEE-754 32-bit Floating-Point Standard, VGA display protocol, UART protocol, and I2C protocols were followed to improve compatibility, reusability, and simplicity in verifications
- A 32-bit floating-point instruction set architecture (ISA) and a 5-stage RISC processor were developed in System Verilog to manage image processing, matrix multiplications, ML classifications, and user interfaces
- Three different ML architectures were implemented: Linear Classification (LC), fully connected neural network (NN), and LeNet-like Convolutional Neural Network (CNN)
- Training processes were done in Python scripts with resulting kernels and weights stored in hex files loaded into the FPGA's SRAM units
- Firmware guided convolution, pooling, data management, and other ML features using custom assembly language
- The implemented handwriting recognition system demonstrated high accuracy, efficiency, and flexibility with various ML models
- The authors suggest further optimizing the IEEE-754 compliant RISC processor architecture for ML acceleration by enhancing the ISA to cater better to ML requirements
- Exploring superscalar architecture could lead to significant performance gains due to parallelizability of ML computations
- Implementing some of the ML layers at the firmware level allows for future enhancements without extensive modifications to hardware modules; this enables edge detection of individual characters as well as synchronous recognition of multiple characters while also allowing for classification of more complex alphabets for higher accuracy and broader capability.
- -The trend of edge computing using ASIC devices brings computation and data storage closer to where the data is generated; their FPGA design could potentially be implemented as ASIC devices for commercial use, reducing overheads, relaxing memory limitations, improving power consumption while maintaining high accuracy, efficiency, and flexibility with various ML models.
- Exploring fixed point arithmetic in FPGA ML hardware applications may provide potential performance boosts
- The project provides a comprehensive platform for future developments of similar ML applications on FPGAs and discusses potential improvements at both the processor architecture and firmware levels

## Summary

Machine Learning (ML) has become a rapidly growing field in Computer Science, and there is a need for hardware implementations of popular ML architectures to optimize their performance, reliability, and resource usage. In this project, the authors designed a highly-configurable, real-time device using an Altera DE1 FPGA Kit for recognizing handwritten letters and digits. To achieve their goals, the authors followed various engineering standards such as IEEE-754 32-bit Floating-Point Standard, Video Graphics Array (VGA) display protocol, Universal Asynchronous Receiver-Transmitter (UART) protocol, and Inter-Integrated Circuit (I2C) protocols. These standards improved the design's compatibility, reusability, and simplicity in verifications. The authors developed a 32-bit floating-point instruction set architecture (ISA) and a 5-stage RISC processor in System Verilog to manage image processing, matrix multiplications, ML classifications, and user interfaces. They implemented three different ML architectures: Linear Classification (LC), a fully connected neural network (NN), and a LeNet-like Convolutional Neural Network (CNN). The training processes were done in Python scripts with the resulting kernels and weights stored in hex files that were loaded into the FPGA's SRAM units. Firmware guided convolution, pooling, data management, and other ML features using custom assembly language. The implemented handwriting recognition system demonstrated high accuracy, efficiency, and flexibility with various ML models. The authors believe that their IEEE-754 compliant RISC processor architecture could be further optimized for ML acceleration by enhancing the ISA to cater better to ML requirements. They also suggest exploring superscalar architecture for significant performance gains due to parallelizability of ML computations. At the firmware level implementing some of the ML layers allows for future enhancements without extensive modifications to hardware modules; this approach enables edge detection of individual characters as well as synchronous recognition of multiple characters while also allowing for classification of more complex alphabets for higher accuracy and broader capability. The authors highlight the trend of edge computing using ASIC devices which brings computation and data storage closer to where the data is generated; their FPGA design could potentially be implemented as ASIC devices for commercial use reducing overheads relaxing memory limitations improving power consumption while still maintaining high accuracy efficiency flexibility with various ML models. While current implementation is based on floating point arithmetic authors suggest exploring fixed point arithmetic potential performance boosts in FPGA ML hardware applications . Overall this project provides comprehensive platform future developments similar ML applications on FPGAs discussing potential improvements both processor architecture firmware levels potential impact work edge computing using ASIC devices .