# HARFLOW3D: A Latency-Oriented 3D-CNN Accelerator Toolflow for HAR on FPGA Devices

**Authors:** Petros Toupas, Alexander Montgomerie-Corcoran, Christos-Savvas Bouganis, Dimitrios Tzovaras  
**arXiv:** [2303.17218](https://arxiv.org/abs/2303.17218)  
**Date:** 2023-03-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.17218v1/

## Key points

- 3D Convolutional Neural Networks (CNNs) are effective for Human Action Recognition (HAR) tasks
- However, they require more computational and memory resources than 2D CNNs due to the additional temporal dimension
- HARFLOW3D is a novel toolflow that maps 3D CNN models onto Field Programmable Gate Array (FPGA) devices while considering the model's inherent characteristics and the features of the targeted FPGA device
- The toolflow takes as input a 3D CNN in ONNX format and a description of the FPGA characteristics, generating a design that minimizes computation latency
- It comprises several parts, including a 3D CNN parser, performance and resource model, scheduling algorithm for executing 3D models on generated hardware, resource-aware optimization engine tailored for 3D models, and automated mapping to synthesizable code for FPGAs
- Experiments showed that HARFLOW3D can support a broad range of models and devices with high-performing results compared to existing hand-tuned approaches
- HARFLOW3D was able to achieve up to five times better performance compared to some existing hand-tuned approaches
- There is an increasing need for efficient tools like HARFLOW3D that can accelerate computations involving volumetric data using FPGAs in video-related applications such as video surveillance, autonomous driving and patient monitoring.

## Summary

The use of 3D Convolutional Neural Networks (CNNs) has become increasingly popular for Human Action Recognition (HAR) tasks, as they have proven to be highly effective in achieving state-of-the-art results. However, the computational and memory requirements of 3D CNNs are often larger than those of 2D CNNs due to the additional temporal dimension. To address this issue, a novel toolflow called HARFLOW3D has been introduced in this study. The toolflow is based on a streaming architecture that maps 3D CNN models onto Field Programmable Gate Array (FPGA) devices while considering the model's inherent characteristics and the features of the targeted FPGA device. The HARFLOW3D toolflow takes as input a 3D CNN in ONNX format and a description of the FPGA characteristics, generating a design that minimizes the latency of computation. It comprises several parts, including a 3D CNN parser, performance and resource model, scheduling algorithm for executing 3D models on generated hardware, resource-aware optimization engine tailored for 3D models, and automated mapping to synthesizable code for FPGAs. Experiments were conducted on various pairs of 3D CNN models and FPGA systems to demonstrate the ability of HARFLOW3D to support a broad range of models and devices. The toolflow produced high-performing results for some previously unmapped 3D CNN models to FPGAs, demonstrating the potential of FPGA-based systems in this space. In particular, HARFLOW3D was able to achieve up to five times better performance compared to some existing hand-tuned approaches. The growing focus on video-related applications such as video surveillance, autonomous driving and patient monitoring has necessitated algorithms that integrate and take into account the temporal domain. As such there is an increasing need for efficient tools like HARFLOW3D that can accelerate computations involving volumetric data using FPGAs. Overall HARFLOW3Ds has demonstrated its ability to deliver competitive latency and resource utilization compared to a range of state-of-the-art hand tuned approaches.