# Learning Transferable Visual Models From Natural Language Supervision

**Authors:** Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever  
**arXiv:** [2103.00020](https://arxiv.org/abs/2103.00020)  
**Date:** 2021-02-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.00020v1/

## Key points

- State-of-the-art computer vision systems are limited by being trained to recognize a fixed set of predetermined object categories.
- Researchers at OpenAI have developed an alternative approach that uses natural language as a source of supervision for learning directly from raw text about images.
- Predicting which caption goes with which image is an efficient and scalable way to learn cutting-edge image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet.
- Natural language can be used to reference learned visual concepts or even describe new ones, enabling zero-shot transfer of the model to downstream tasks without requiring any specific training on those tasks.
- The researchers benchmarked their approach on over 30 different existing computer vision datasets spanning diverse tasks and achieved competitive results compared to fully supervised baselines without requiring any dataset-specific training.
- Their model transfers non-trivially to most tasks and often achieves competitive results compared to fully supervised baselines without using any of their training examples.
- The researchers have released their code and pre-trained model weights at https://github.com/OpenAI/CLIP for others to use and build upon.

## Summary

In the field of computer vision, state-of-the-art systems are typically trained to recognize a fixed set of predetermined object categories. However, this limited form of supervision restricts their generality and usability since additional labeled data is required to specify any other visual concept. To overcome this challenge, researchers at OpenAI have developed an alternative approach that leverages natural language as a source of supervision for learning directly from raw text about images. Specifically, they demonstrate that predicting which caption goes with which image is an efficient and scalable way to learn cutting-edge image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training on this large dataset, the researchers show that natural language can be used to reference learned visual concepts or even describe new ones. This enables zero-shot transfer of the model to downstream tasks without requiring any specific training on those tasks. To evaluate the performance of their approach, the researchers benchmarked it on over 30 different existing computer vision datasets spanning diverse tasks such as OCR, action recognition in videos, geo-localization, and fine-grained object classification. Remarkably, their model transfers non-trivially to most tasks and often achieves competitive results compared to fully supervised baselines without requiring any dataset-specific training. For example, they match the accuracy of ResNet-50 on ImageNet zero-shot without using any of its 1.28 million training examples. The researchers have released their code and pre-trained model weights at https://github.com/OpenAI/CLIP for others to use and build upon. Overall, this work demonstrates how natural language can serve as a powerful source of supervision for learning transferable visual models that generalize well across diverse computer vision tasks.