# An Empirical Study of Training Self-Supervised Visual Transformers

**Authors:** Xinlei Chen, Saining Xie, Kaiming He  
**arXiv:** [2104.02057](https://arxiv.org/abs/2104.02057)  
**Date:** 2021-04-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.02057v1/

## Key points

- Authors Xinlei Chen, Saining Xie, and Kaiming He focus on self-supervised learning for Visual Transformers (ViT) in computer vision.
- Traditional convolutional networks have established training recipes, but ViT lacks mature guidelines, especially in self-supervised scenarios.
- Study investigates fundamental components affecting training of self-supervised ViT models, highlighting the issue of instability impacting accuracy.
- Addressing instability leads to improved model performance; comparisons are made against MoCo v3 and other frameworks with insights from various ablations.
- Supervised pre-training on larger datasets like ImageNet-21k or JFT-300M shows ViT models outperform self-supervised counterparts when transferring to smaller datasets.
- Future research directions include self-supervised pre-training for large ViT models on extensive datasets similar to advancements in NLP unsupervised pre-training.
- Study offers comparisons between ViT and standard convolutional networks, supervised vs. self-supervised approaches, contrastive learning vs. masked auto-encoding for valuable insights.
- Positive findings alongside challenges, open questions, and opportunities for further exploration aim to bridge the gap between pre-training methodologies in vision-based tasks and language processing domains.

## Summary

In their paper titled "An Empirical Study of Training Self-Supervised Visual Transformers," Xinlei Chen, Saining Xie, and Kaiming He delve into the realm of self-supervised learning for Visual Transformers (ViT) in computer vision. Traditional convolutional networks have well-established training recipes; however, ViT poses a new challenge due to the lack of mature guidelines, especially in self-supervised scenarios. The authors conduct a thorough investigation into fundamental components affecting the training of self-supervised ViT models. One key finding highlighted in the study is the issue of instability which significantly impacts accuracy and can mask subpar results as seemingly successful outcomes. By addressing this instability and making training more stable, the authors demonstrate improvements in model performance. They benchmark ViT results against MoCo v3 and other self-supervised frameworks and provide insights through various ablations. Furthermore, the authors discuss the implications of supervised pre-training on larger datasets such as ImageNet-21k or JFT-300M where ViT models exhibit superior performance compared to their self-supervised counterparts when transferring to smaller datasets. They suggest future research directions involving self-supervised pre-training for large ViT models on extensive datasets akin to advancements seen in unsupervised pre-training in NLP. In conclusion, this empirical study sheds light on training ViT models within self-supervised frameworks in computer vision. The comparisons made between ViT and standard convolutional networks, supervised versus self-supervised approaches, and contrastive learning versus masked auto-encoding offer valuable insights for researchers. The authors present positive findings alongside challenges, open questions, and opportunities for further exploration. Ultimately, they aim to contribute valuable data points and experiences to bridge the gap between pre-training methodologies in vision-based tasks and language processing domains.