# W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training

**Authors:** Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, Yonghui Wu  
**arXiv:** [2108.06209](https://arxiv.org/abs/2108.06209)  
**Date:** 2021-08-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2108.06209v1/

## Key points

- The paper introduces a framework called w2v-BERT for self-supervised speech pre-training.
- w2v-BERT combines contrastive learning and masked language modeling (MLM).
- It aims to learn contextualized speech representations by solving a masked prediction task using discretized speech tokens obtained through contrastive learning.
- Unlike existing MLM-based speech pre-training frameworks, w2v-BERT can be optimized in an end-to-end fashion.
- Experiments on LibriSpeech benchmarks using the Libri-Light 60k corpus show that w2v-BERT achieves competitive results compared to current state-of-the-art models.
- w2v-BERT demonstrates a relative word error rate (WER) reduction of 5% to 10% compared to conformer-based wav2vec 2.0 and HuBERT on test clean and test other subsets.
- When applied to Google's Voice Search traffic dataset, w2v BERT outperforms internal conformer based wav2vec 2.0 by more than 30% relatively.
- These findings highlight the effectiveness of w2v BERT in self-supervised speech representation learning.

## Summary

The paper introduces a framework called w2v-BERT, which combines contrastive learning and masked language modeling (MLM) for self-supervised speech pre-training. The approach aims to learn contextualized speech representations by solving a masked prediction task using discretized speech tokens obtained through contrastive learning. Unlike existing MLM-based speech pre-training frameworks, such as HuBERT and vq-wav2vec, w2v-BERT can be optimized in an end-to-end fashion by simultaneously solving the contrastive task and MLM. The experiments conducted on the LibriSpeech benchmarks using the Libri-Light 60k corpus as unsupervised data show that w2v-BERT achieves competitive results compared to current state-of-the-art pre-trained models. Specifically, when compared to conformer-based wav2vec 2.0 and HuBERT, w2v-BERT demonstrates a relative word error rate (WER) reduction of 5% to 10% on the test clean and test other subsets. Moreover, when applied to Google's Voice Search traffic dataset, w2v BERT outperforms internal conformer based wav2vec 2.0 by more than 30% relatively. Overall, these findings highlight the effectiveness of w2v BERT in self supervised speech representation learning.