# Fused Depthwise Tiling for Memory Optimization in TinyML Deep Neural Network Inference

**Authors:** Rafael Stahl, Daniel Mueller-Gritschneder, Ulf Schlichtmann  
**arXiv:** [2303.17878](https://arxiv.org/abs/2303.17878)  
**Date:** 2023-03-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.17878v1/

## Key points

- Memory optimization for deep neural network (DNN) inference is crucial in TinyML.
- TinyML involves deploying DNN inference tasks on tiny, low-power microcontrollers with limited memory.
- Fused Depthwise Tiling (FDT) method has been proposed for the memory optimization of DNNs.
- FDT reduces memory usage without inducing any run time overhead and applies to a larger variety of network layers than existing tiling methods that focus on convolutions.
- An end-to-end flow with a new path discovery method has been proposed to identify the best tiling configuration for a given model.
- FDT achieved significant memory reduction for two models by 76.2% and 18.1% where existing tiling methods could not be applied.
- Five models from a wide range benefit from fused tiling: Keyword Spotting (KWS), Text Sentiment Analysis (TXT), Magic Wand (MW), PoseNet (POS), and MobileNet V2 SSDLite.
- The proposed FDT method provides an effective solution to optimize memory usage in DNNs for TinyML applications.

## Summary

Memory optimization for deep neural network (DNN) inference is crucial in the emerging field of TinyML, which involves deploying DNN inference tasks on tiny, low-power microcontrollers. However, these devices have limited memory and are heavily constrained by the large intermediate run-time buffers required to store activations and other intermediate data during DNN inference. This leads to high memory usage, making it challenging to deploy applications such as audio keyword detection or radar-based gesture recognition on such devices. To address this challenge, a new Fused Depthwise Tiling (FDT) method has been proposed for the memory optimization of DNNs. Compared to existing tiling methods, FDT reduces memory usage without inducing any run time overhead and applies to a larger variety of network layers than existing tiling methods that focus on convolutions. It improves TinyML memory optimization significantly by reducing memory of models where this was not possible before and additionally providing alternative design points for models that show high run time overhead with existing methods. To identify the best tiling configuration for a given model, an end-to-end flow with a new path discovery method has been proposed. This approach applies FDT and existing tiling methods in a fully automated way, including scheduling operations and planning buffer layout in memory. Out of seven evaluated models, FDT achieved significant memory reduction for two models by 76.2% and 18.1% where existing tiling methods could not be applied. Two other models showed significant run time overhead with existing methods, but FDT provided alternative design points with no overhead but reduced memory savings. The study identified five models from a wide range that benefit from fused tiling: Keyword Spotting (KWS), Text Sentiment Analysis (TXT), Magic Wand (MW), PoseNet (POS), and MobileNet V2 SSDLite. All models were quantized to 8 bits, and performance was comparable across different architectures. In conclusion, the proposed FDT method provides an effective solution to optimize memory usage in DNNs for TinyML applications. The end-to-end flow with a new path discovery method enables fully automated tiling configuration and buffer layout planning, making it easier to deploy DNN inference tasks on tiny, low-power microcontrollers.