# Patch-level Representation Learning for Self-supervised Vision Transformers

**Authors:** Sukmin Yun, Hankook Lee, Jaehyung Kim, Jinwoo Shin  
**arXiv:** [2206.07990](https://arxiv.org/abs/2206.07990)  
**Date:** 2022-06-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.07990v1/

## Key points

- Recent self-supervised learning (SSL) methods have made significant progress in learning visual representations from unlabeled images.
- Leveraging the architectural advantages of the underlying neural network can further improve the performance of SSL methods.
- Vision Transformers (ViTs) are considered a better architectural choice for SSL, outperforming convolutional networks in various visual tasks.
- ViTs process images as a sequence of disjoint patches and internally process patch-level representations.
- The authors propose a visual pretext task called SelfPatch to learn better patch-level representations using ViTs.
- In SelfPatch, each patch is enforced to be invariant against itself and its neighboring patches, treating similar neighboring patches as positive samples.
- Training ViTs with SelfPatch improves semantically meaningful relations among patches without human-annotated labels.
- The proposed method achieves significant improvements in the performance of existing SSL methods for object detection, instance segmentation, and semantic segmentation tasks.
- When applied to the self-supervised ViT model called DINO, SelfPatch achieves impressive results including +1.3 AP on COCO object detection, +1.2 AP on COCO instance segmentation, and +2.9 mIoU on ADE20K semantic segmentation.
- Considering architectural advantages when designing visual pretext tasks for SSL leads to enhanced patch-level representations and improved performance on downstream visual tasks.

## Summary

Recent self-supervised learning (SSL) methods have made significant progress in learning visual representations from unlabeled images. However, the performance of these methods can be further improved by leveraging the architectural advantages of the underlying neural network. The current state-of-the-art visual pretext tasks for SSL are architecture-agnostic, which means they do not fully exploit the potential benefits of specific network architectures. In this paper titled "Patch-level Representation Learning for Self-supervised Vision Transformers," authors Sukmin Yun, Hankook Lee, Jaehyung Kim, and Jinwoo Shin focus on Vision Transformers (ViTs) as a better architectural choice for SSL. ViTs have gained attention recently for their ability to outperform convolutional networks in various visual tasks. Unlike traditional convolutional networks that process images as a whole, ViTs take a sequence of disjoint patches from an image and internally process patch-level representations. Inspired by the unique characteristics of ViTs, the authors propose a simple yet effective visual pretext task called SelfPatch to learn better patch-level representations. In this task, each patch is enforced to be invariant against itself and its neighboring patches. This means that each patch treats similar neighboring patches as positive samples. By training ViTs with SelfPatch, more semantically meaningful relations among patches can be learned without using human-annotated labels. The proposed method demonstrates significant improvements in the performance of existing SSL methods for various visual tasks such as object detection and semantic segmentation. Specifically, when applied to the self-supervised ViT model called DINO, SelfPatch achieves impressive results including +1.3 AP on COCO object detection, +1.2 AP on COCO instance segmentation, and +2.9 mIoU on ADE20K semantic segmentation. Overall, this paper highlights the importance of considering architectural advantages when designing visual pretext tasks for SSL. The use of ViTs and the SelfPatch task leads to enhanced patch-level representations and improved performance on downstream visual tasks.