# DINOv2: Learning Robust Visual Features without Supervision

**Authors:** Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve  
**arXiv:** [2304.07193](https://arxiv.org/abs/2304.07193)  
**Date:** 2023-04-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.07193v2/

## Key points

- Recent advancements in natural language processing have led to the development of foundation models that can be pretrained on large datasets in computer vision.
- These models generate all-purpose visual features effective across different image distributions and tasks without needing fine-tuning.
- Existing pretraining methods, especially self-supervised techniques, can produce versatile features when trained on diverse curated data sources.
- Researchers combined various approaches to enhance the scalability of pretraining in terms of data and model size, focusing on accelerating and stabilizing training processes at scale.
- They introduced an automated pipeline for constructing a meticulously curated image dataset and trained a Vision Transformer (ViT) model with 1 billion parameters before distilling it into smaller models.
- The refined models outperformed existing state-of-the-art all-purpose features like OpenCLIP on benchmarks at both image and pixel levels.

## Summary

, , , , 

Recent advancements in natural language processing have paved the way for the development of foundation models that can be pretrained on large datasets in the field of computer vision. These models have the potential to simplify the use of images in various systems by generating all-purpose visual features that are effective across different image distributions and tasks without requiring fine-tuning. This study demonstrates that existing pretraining methods, particularly self-supervised techniques, can indeed produce such versatile features when trained on a diverse set of curated data sources. To further enhance the scalability of pretraining in terms of both data and model size, the researchers combined various approaches and techniques. The focus of their technical contributions was primarily on accelerating and stabilizing training processes at scale. In terms of data preparation, they introduced an automated pipeline for constructing a meticulously curated image dataset, as opposed to using uncurated data commonly found in self-supervised literature. In terms of model development, the team trained a Vision Transformer (ViT) model with an impressive 1 billion parameters before distilling it into a series of smaller models. These refined models outperformed existing state-of-the-art all-purpose features like OpenCLIP on numerous benchmarks at both image and pixel levels. The study titled "DINOv2: Learning Robust Visual Features without Supervision" was authored by Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang ,Shang-Wen Li, Ishan Misra, Micheal Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut ,Armand Joulin,and Piotr Bojanowski. Their work showcases significant progress in leveraging pretraining methods to develop robust visual features without supervision.