# Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....

**Authors:** Prateek Verma  
**arXiv:** [2110.03183](https://arxiv.org/abs/2110.03183)  
**Date:** 2021-10-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2110.03183v5/

## Key points

- Deep learning has revolutionized audio signal processing
- Convolutional architectures have achieved state-of-the-art results
- End-to-end Transformer architectures have shown impressive performance in natural language processing tasks
- The proposed method for large-scale audio understanding does not rely on traditional neural architectures or Transformers
- The approach is based on a Bag-of-Words model and utilizes micro and macro level clustered vanilla embeddings
- Feed-forward encoder-decoder models are used to extract bottlenecks of spectral envelops, spectral patches and slices as well as multi-resolution spectra
- A classification head is trained on a learned representation using simple codes learned on latent representations
- This approach does not involve convolutions, recurrence, attention mechanisms or other approaches such as BERT.
- The authors demonstrate that their approach surpasses traditional convolutional neural network architectures and comes strikingly close to outperforming powerful Transformer architectures.
- This work presents a novel way of doing large scale audio understanding that challenges the dominance of traditional convolutional and recurrent neural networks as well as end-to-end Transformer architectures.

## Summary

In recent years, deep learning has revolutionized the field of audio signal processing, with convolutional architectures achieving state-of-the-art results. However, there has been a shift towards end-to-end Transformer architectures which have shown impressive performance in various natural language processing tasks. In this paper, the authors present an alternative approach to large-scale audio understanding that does not rely on traditional neural architectures or Transformers. The proposed method is based on a Bag-of-Words model and utilizes micro and macro level clustered vanilla embeddings. The authors use feed-forward encoder-decoder models to extract bottlenecks of spectral envelops, spectral patches and slices as well as multi-resolution spectra. A classification head (a feed forward layer) is trained on a learned representation using simple codes learned on latent representations. This approach does not involve convolutions, recurrence, attention mechanisms or other approaches such as BERT. The authors demonstrate that their approach surpasses traditional convolutional neural network architectures and comes strikingly close to outperforming powerful Transformer architectures. This work opens up exciting possibilities for advancements in the field of representation learning without relying on massive end-to-end neural architectures. Overall, this paper presents a novel way of doing large scale audio understanding that challenges the dominance of traditional convolutional and recurrent neural networks as well as end to end Transformer architectures. By utilizing Bag of Words model with micro and macro level clustered vanilla embeddings along with feed forward encoder decoder models for feature extraction and classification head training using simple codes learned on latent representations, the authors show promising results in surpassing traditional convolutional neural network architectures while coming close to outperforming powerful Transformer architectures.