# Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation

**Authors:** Dahyun Kang, Piotr Koniusz, Minsu Cho, Naila Murray  
**arXiv:** [2307.03407](https://arxiv.org/abs/2307.03407)  
**Date:** 2023-07-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.03407v1/

## Key points

- Paper title: "Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation"
- Addresses the challenge of weakly-supervised few-shot image classification and segmentation
- Proposes a method that leverages a Vision Transformer (ViT) pretrained with self-supervision
- Uses token representations from the self-supervised ViT and exploits their correlations through self-attention
- Generates separate task heads for classification and segmentation predictions
- Uses attention maps created from tokens as pixel-level pseudo-labels instead of relying on pixel-level labels
- Explores a practical setup called "mixed" supervision, where only a small number of training images have ground truth pixel level labels
- Introduces a pseudo label enhancer to improve pseudo labels in the mixed setup, trained using available ground truth pixel level labels
- Experimental results on Pascal 5i and COCO 20i datasets show significant performance gains across various supervision settings, especially with limited or no pixel level labels available
- Shows promise in addressing weakly supervised few shot image classification and segmentation tasks by leveraging self supervision in Vision Transformers.

## Summary

The paper titled "Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation" addresses the challenge of weakly-supervised few-shot image classification and segmentation. The authors propose a method that leverages a Vision Transformer (ViT) pretrained with self-supervision to perform these tasks effectively even in the absence of pixel-level labels during training. The proposed approach utilizes token representations from the self-supervised ViT and exploits their correlations through self-attention. This enables the model to generate separate task heads for classification and segmentation predictions. Instead of relying on pixel-level labels, the model uses attention maps created from tokens generated by the self-supervised ViT backbone as pixel-level pseudo-labels. In addition, the authors explore a practical setup called "mixed" supervision, where only a small number of training images have ground truth pixel level labels while the rest have image level labels only. To improve the pseudo labels in this mixed setup, they introduce a pseudo label enhancer that is trained using the available ground truth pixel level labels. Experimental results on Pascal 5i and COCO 20i datasets demonstrate significant performance gains across various supervision settings, particularly when there are limited or no pixel level labels available. The proposed method shows promise in addressing weakly supervised few shot image classification and segmentation tasks by leveraging self supervision in Vision Transformers.