# Depthwise Spatio-Temporal STFT Convolutional Neural Networks for Human Action Recognition

**Authors:** Sudhakar Kumawat, Manisha Verma, Yuta Nakashima, Shanmuganathan Raman  
**arXiv:** [2007.11365](https://arxiv.org/abs/2007.11365)  
**Date:** 2020-07-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2007.11365v1/

## Key points

- The authors propose a new approach called spatio-temporal short term Fourier transform (STFT) blocks for 3D convolutional neural networks (CNNs)
- STFT blocks overcome challenges of conventional CNNs such as high computational cost, memory requirements, overfitting, and limited feature learning capabilities
- STFT blocks consist of non-trainable convolution layers capturing spatially and/or temporally local Fourier information using an STFT kernel at multiple low-frequency points
- Trainable linear weights are used in STFT blocks for learning channel correlations
- Incorporating STFT blocks into 3D CNNs reduces space-time complexity while enhancing feature learning capabilities
- Compared to state-of-the-art methods, STFT blocks require fewer parameters and have lower computational costs (3.5 to 4.5 times fewer parameters, 1.5 to 1.8 times lower computational costs)
- Extensive experiments on seven action recognition datasets demonstrate that STFT block based 3D CNNs achieve comparable or better performance compared to state-of-the-art methods
- The proposed depthwise spatio-temporal STFT CNNs offer a promising solution for improving action recognition tasks by addressing limitations of conventional approaches while reducing computational complexity and leading to competitive performance on benchmark datasets.

## Summary

The authors propose a new approach called spatio-temporal short term Fourier transform (STFT) blocks to overcome the challenges of conventional 3D convolutional neural networks (CNNs), such as high computational cost, memory requirements, overfitting and limited feature learning capabilities. An STFT block is composed of non-trainable convolution layers that capture spatially and/or temporally local Fourier information using an STFT kernel at multiple low-frequency points followed by a set of trainable linear weights for learning channel correlations. By incorporating STFT blocks into 3D CNNs, the authors significantly reduce the space-time complexity of the network while enhancing feature learning capabilities. Compared to state-of-the-art methods, the STFT blocks require 3.5 to 4.5 times fewer parameters and have 1.5 to 1.8 times lower computational costs. To evaluate their approach, extensive experiments were conducted on seven action recognition datasets including Something-something v1 and v2, Jester, Diving-48, Kinetics-400, UCF 101 and HMDB 51 which demonstrate that STFT block based 3D CNNs achieve comparable or even better performance compared to state-of-the art methods. In conclusion, the proposed depthwise spatio-temporal STFT convolutional neural networks offer a promising solution for improving action recognition tasks by addressing the limitations of conventional approaches while reducing computational complexity and leading to competitive performance on various benchmark datasets.