# BEATs: Audio Pre-Training with Acoustic Tokenizers

**Authors:** Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Furu Wei  
**arXiv:** [2212.09058](https://arxiv.org/abs/2212.09058)  
**Date:** 2022-12-18  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.09058v1/

## Key points

- Self-supervised learning (SSL) has seen a significant increase in usage across various domains such as language, vision, speech, and audio.
- Implementing semantic-rich acoustic tokenizers for general audio pre-training faces challenges due to the continuous nature of audio data and the lack of phoneme sequences like speech.
- A novel iterative audio pre-training framework called BEATs has been introduced to address these challenges by optimizing an acoustic tokenizer and an audio SSL model through iterative processes.
- BEATs aims to encourage SSL models to abstract high-level audio semantics while discarding redundant details similar to human perception.
- The authors of this work include Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei.
- In the initial iteration of BEATs, random projection is used as the acoustic tokenizer to train an audio SSL model using a mask and label prediction approach.
- Subsequent iterations involve training an acoustic tokenizer by distilling semantic knowledge from either pre-trained or fine-tuned audio SSL models with the expectation of mutual enhancement between the tokenizer and the model.
- Experimental results have shown that BEATs' acoustic tokenizers can generate discrete labels with rich audio semantics surpassing previous approaches that required larger amounts of training data and model parameters significantly.
- BEATs has achieved a state-of-the-art mean average precision (mAP) of 50.6% on AudioSet-2M without external data sources and an accuracy rate of 98.1% on ESC-50 classification tasks.
- Code implementations and pre-trained models for BEATs are accessible at https://aka.ms/beats for further exploration or utilization.

## Summary

In recent years, there has been a significant surge in the use of self-supervised learning (SSL) across various domains such as language, vision, speech, and audio. One of the main challenges faced in implementing semantic-rich acoustic tokenizers for general audio pre-training lies in the continuous nature of audio data and the unavailability of phoneme sequences like speech. To address this challenge head-on, a novel iterative audio pre-training framework called BEATs has been proposed. This framework aims to learn Bidirectional Encoder representations from Audio Transformers by optimizing an acoustic tokenizer and an audio SSL model through iterative processes. has shown to be more effective in encouraging SSL models to abstract high-level audio semantics while discarding redundant details akin to human perception. The authors behind this groundbreaking work include Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu,
Daniel Tompkins,
Zhuo Chen,
and Furu Wei. In the initial iteration of BEATs, random projection is employed as the acoustic tokenizer to train an audio SSL model using a mask and label prediction approach. Subsequently, an acoustic tokenizer is trained for the next iteration by distilling semantic knowledge from either the pre-trained or fine-tuned audio SSL model. This iterative process continues with the expectation of mutual enhancement between the acoustic tokenizer and the audio SSL model. Experimental results have demonstrated that BEATs' acoustic tokenizers are capable of generating discrete labels with rich audio semantics. Furthermore,has been achieved by , surpassing previous approaches that utilized larger amounts of training data and model parameters significantly. Notably, has set a new benchmark with a state-of-the-art mean average precision (mAP) of 50.6% on AudioSet-2M without requiring any external data sources. Additionally, it has achieved an accuracy rate of 98.1% on ESC-50 classification tasks. For those interested in exploring further or utilizing these advancements, both code implementations and pre-trained models are readily accessible at https://aka.ms/beats.