# HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

**Authors:** Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, Shlomo Dubnov  
**arXiv:** [2202.00874](https://arxiv.org/abs/2202.00874)  
**Date:** 2022-02-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.00874v1/

## Key points

- Audio classification involves mapping audio samples to their corresponding labels.
- Transformer models with self-attention mechanisms have been adopted in this field, but they require large GPU memories and long training time.
- HTS-AT is an audio transformer with a hierarchical structure that reduces model size and training time. It is combined with a token-semantic module for audio event detection.
- Researchers compared HTS-AT with different benchmark models and achieved a new state-of-the-art mAP of 0.471 in a single model, outperforming previous models.
- Ensembling six HTS-ATs achieved an mAP of 0.487, further outperforming previous models.
- HTS-AT achieved new state-of-the-art results on AudioSet and ESC-50 while equaling the state-of-the art on Speech Command V2.
- HTS AT performed better than previous CNN based models in event localization while requiring only 35% model parameters and 15% training time of the previous audio transformer.

## Summary

Audio classification is a crucial task that involves mapping audio samples to their corresponding labels. The transformer model with self-attention mechanisms has recently been adopted in this field, but existing audio transformers require large GPU memories and long training time, limiting their scalability in audio tasks. To address these issues, researchers have introduced HTS-AT: an audio transformer with a hierarchical structure that reduces the model size and training time. It is combined with a token-semantic module to map final outputs into class featuremaps, enabling the model for audio event detection (i.e., localization in time). In Table 1 of their study, the researchers compared HTS-AT with different benchmark models and three self-ablated variations: (1) H: only hierarchical structure; (2) HC: with hierarchical structure and token-semantic module; and (3) HCP: (2) with pretrained vision model (the full setting). Their best setting achieved a new state-of-the-art mAP of 0.471 in a single model as a significant improvement from 0.459 by AST. They also ensembled six HTS-ATs with different training random seeds in the same settings to achieve an mAP of 0.487, outperforming AST's 0.475 and 0.485. The researchers evaluated HTS-AT on three datasets of audio classification where it achieved new state-of-the-art results on AudioSet and ESC-50 while equaling the state-of-the art on Speech Command V2. Additionally, HTS AT performed better than previous CNN based models in event localization while requiring only 35% model parameters and 15% training time of the previous audio transformer. Overall, these results demonstrate the high performance and efficiency of HTS AT as a promising solution for scalable audio classification tasks.