# Emerging Properties in Self-Supervised Vision Transformers

**Authors:** Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin  
**arXiv:** [2104.14294](https://arxiv.org/abs/2104.14294)  
**Date:** 2021-04-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2104.14294v1/

## Key points

- Self-supervised learning may provide new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets).
- Self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs or convnets.
- These features are excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT.
- The study highlights the importance of momentum encoder, multi-crop training and the use of small patches with ViTs.
- The authors have implemented their findings into a simple self-supervised method called DINO, which achieves 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.
- Self-supervised pretraining of standard ViT models can achieve performance comparable to convnets specifically designed for this setting.
- Two properties that can be leveraged in future applications are: the quality of features in k-NN classification has potential for image retrieval where ViTs are already showing promising results; and information about scene layout in features can benefit weakly supervised image segmentation.
- The main result of this paper is that there is evidence to suggest that self-supervised learning could be key to developing a BERT-like model based on ViT.
- In future work, the authors plan to explore whether pretraining a large ViT model with DINO on random uncurated images could push the limits of visual features.
- Overall, this study underscores the potential benefits of using self-supervised learning methods with Vision Transformers and offers insights into how these models can be optimized for specific tasks such as image retrieval and weakly supervised image segmentation.

## Summary

In their paper titled "Emerging Properties in Self-Supervised Vision Transformers," Mathilde Caron and her colleagues question whether self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). The authors observe that self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs or convnets. Additionally, these features are excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. The study also highlights the importance of momentum encoder, multi-crop training and the use of small patches with ViTs. To implement their findings into a simple self-supervised method called DINO, the authors interpret it as a form of self-distillation with no labels and show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base. The authors note that self-supervised pretraining of standard ViT models can achieve performance comparable to convnets specifically designed for this setting. They also highlight two properties that can be leveraged in future applications: the quality of features in k-NN classification has potential for image retrieval where ViTs are already showing promising results; and information about scene layout in features can benefit weakly supervised image segmentation. The main result of this paper is that there is evidence to suggest that self-supervised learning could be key to developing a BERT-like model based on ViT. In future work, the authors plan to explore whether pretraining a large ViT model with DINO on random uncurated images could push the limits of visual features. Overall, this study underscores the potential benefits of using self-supervised learning methods with Vision Transformers and offers insights into how these models can be optimized for specific tasks such as image retrieval and weakly supervised image segmentation. The authors' findings could have significant implications for the development of more advanced computer vision systems in the future.