# Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language

**Authors:** Alexei Baevski, Arun Babu, Wei-Ning Hsu, Michael Auli  
**arXiv:** [2212.07525](https://arxiv.org/abs/2212.07525)  
**Date:** 2022-12-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.07525v1/

## Key points

- The paper proposes an approach to increase the training efficiency of data2vec 2.0, a learning objective that generalizes across several modalities.
- The proposed method does not encode masked tokens and uses a fast convolutional decoder while amortizing the effort to build teacher representations.
- The authors leverage rich contextualized target representations introduced in data2vec to enable a fast self-supervised learner.
- Experiments show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time on ImageNet-1K image classification.
- On Librispeech speech recognition tasks it performs as well as wav2vec 2.0 in 10.6x less time and matches a retrained RoBERTa model in half the time on GLUE natural language understanding tasks.
- Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8% with a ViT-L model trained for 150 epochs using data2vec 2.0.
- Overall, this paper presents an efficient self-supervised learning approach that can be applied across multiple modalities without requiring significant computational resources or encoding masked tokens while achieving comparable or better performance than existing methods in significantly less pre-training time.

## Summary

The paper titled "Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language" by Alexei Baevski, Arun Babu, Wei-Ning Hsu and Michael Auli addresses the challenges of current self-supervised learning algorithms that are often modality-specific and require large amounts of computational resources. To overcome these issues, the authors propose an approach to increase the training efficiency of data2vec 2.0, a learning objective that generalizes across several modalities. The proposed method does not encode masked tokens and uses a fast convolutional decoder while amortizing the effort to build teacher representations. The authors leverage rich contextualized target representations introduced in data2vec to enable a fast self-supervised learner. Experiments on ImageNet-1K image classification demonstrate that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time. Additionally, on Librispeech speech recognition tasks it performs as well as wav2vec 2.0 in 10.6x less time and matches a retrained RoBERTa model in half the time on GLUE natural language understanding tasks. The paper also highlights that trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8% with a ViT-L model trained for 150 epochs using data2vec 2.0. Overall, this paper presents an efficient self-supervised learning approach that can be applied across multiple modalities without requiring significant computational resources or encoding masked tokens while achieving comparable or better performance than existing methods in significantly less pre-training time.