# AST: Audio Spectrogram Transformer

**Authors:** Yuan Gong, Yu-An Chung, James Glass  
**arXiv:** [2104.01778](https://arxiv.org/abs/2104.01778)  
**Date:** 2021-04-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.01778v1/

## Key points

- The Audio Spectrogram Transformer (AST) is a convolution-free and attention-based model for audio classification.
- AST captures long-range global context directly from audio spectrograms, even in the lowest layers.
- Knowledge transfer from the Vision Transformer (ViT) pretrained on ImageNet enhances AST's performance.
- AST outperforms other models on various audio classification tasks and datasets such as AudioSet, ESC-50, and Speech Commands.
- AST can handle variable-length inputs without requiring architectural changes, making it adaptable to different tasks.
- The architecture of AST involves splitting the 2D audio spectrogram into patches and projecting them linearly into 1-D embeddings with positional encodings.
- A classification token at the beginning of the sequence feeds into a Transformer for processing, with the output used for final classification through a linear layer.
- AST is a compelling choice for audio tasks due to its exceptional performance and versatility in handling diverse input lengths and tasks without compromising accuracy or efficiency.

## Summary

The Audio Spectrogram Transformer (AST) is a state-of-the-art convolution-free and purely attention-based model designed for audio classification. It aims to capture long-range global context directly from audio spectrograms, even in the lowest layers. The model utilizes knowledge transfer from the Vision Transformer (ViT) pretrained on ImageNet to significantly enhance its performance. AST offers several advantages over existing systems. Firstly, it outperforms other models on various audio classification tasks and datasets such as AudioSet, ESC-50, and Speech Commands. Secondly, AST can seamlessly handle variable-length inputs without requiring architectural changes, making it adaptable to different tasks. The architecture of AST involves splitting the 2D audio spectrogram into a sequence of patches and projecting them linearly into 1-D embeddings with added positional encodings. A classification token is included at the beginning of the sequence which feeds into a Transformer for processing. The output of this token is used for final classification through a linear layer. In summary, AST presents a compelling choice for audio tasks due to its exceptional performance and versatility in handling diverse input lengths and tasks without compromising accuracy or efficiency.