# Big Self-Supervised Models are Strong Semi-Supervised Learners

**Authors:** Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton  
**arXiv:** [2006.10029](https://arxiv.org/abs/2006.10029)  
**Date:** 2020-06-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2006.10029v1/

## Key points

- Authors introduce a paradigm for learning from limited labeled examples while leveraging a substantial amount of unlabeled data
- Approach involves unsupervised pretraining followed by supervised fine-tuning using a large network during both stages
- Utilization of a big network during pretraining and fine-tuning processes is key insight
- Employing a larger network is more beneficial with fewer labeled examples to leverage unlabeled data effectively
- Semi-supervised learning algorithm consists of unsupervised pretraining, supervised fine-tuning, and distillation with unlabeled data
- Achieves 73.9% ImageNet top-1 accuracy with only 1% of labels (≤13 labeled images per class) using ResNet-50—a tenfold improvement in label efficiency compared to previous methods
- ResNet-50 trained using their method achieves 77.5% top-1 accuracy on ImageNet with 10% of labels, outperforming standard supervised training
- Code and pretrained models provided for replication at https://github.com/google-research/simclr
- Efficacy of employing big self-supervised models as strong semi-supervised learners in computer vision tasks highlighted

## Summary

In their paper titled "Big Self-Supervised Models are Strong Semi-Supervised Learners," authors Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton introduce a paradigm for learning from limited labeled examples while leveraging a substantial amount of unlabeled data. This approach involves unsupervised pretraining followed by supervised fine-tuning, utilizing a large (deep and wide) network during both stages. Contrary to traditional semi-supervised learning methods in computer vision, which typically rely on task-specific use of unlabeled data, the authors demonstrate the effectiveness of a task-agnostic approach on ImageNet. The key insight of their methodology is the utilization of a big network during pretraining and fine-tuning processes. They observe that the fewer labeled examples available, the more beneficial it is to employ a larger network for leveraging unlabeled data effectively. Following fine-tuning, the big network can be further refined and distilled into a smaller model with minimal loss in classification accuracy by reusing unlabeled examples in a task-specific manner. The proposed semi-supervised learning algorithm consists of three main steps: unsupervised pretraining of a big ResNet model using SimCLRv2 (a modification of SimCLR), supervised fine-tuning on a small number of labeled examples, and distillation with unlabeled data to enhance and transfer task-specific knowledge. Remarkably, this approach achieves an impressive 73.9% ImageNet top-1 accuracy with only 1% of the labels (equivalent to ≤13 labeled images per class) using ResNet-50—a tenfold improvement in label efficiency compared to previous state-of-the-art methods. Moreover, when provided with 10% of labels, ResNet-50 trained using their method achieves a remarkable 77.5% top-1 accuracy on ImageNet—outperforming standard supervised training that utilizes all available labels. The authors provide code and pretrained models for replication purposes at https://github.com/google-research/simclr. Overall, this study highlights the efficacy of employing big self-supervised models as strong semi-supervised learners in computer vision tasks like ImageNet classification. Their innovative approach showcases significant advancements in label efficiency and performance metrics compared to conventional supervised learning techniques.