# An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

**Authors:** Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby  
**arXiv:** [2010.11929](https://arxiv.org/abs/2010.11929)  
**Date:** 2020-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.11929v2/

## Key points

- The Transformer architecture is widely used for natural language processing tasks and gaining popularity in computer vision.
- A recent study by Alexey Dosovitskiy and his team shows that a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks without the need for CNNs.
- This approach is called Vision Transformer (ViT) and involves pre-training on large amounts of data and transferring it to multiple mid-sized or small image recognition benchmarks such as ImageNet, CIFAR-100, and VTAB.
- ViT attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.
- ViT's success is due to its ability to capture long-range dependencies between image patches through self-attention mechanisms, allowing the model to learn meaningful representations of images without being constrained by the local receptive fields of convolutions.
- This study suggests that transformers have great potential for computer vision tasks and could potentially replace convolutional networks as the go-to architecture for image recognition at scale.

## Summary

The Transformer architecture has become the standard for natural language processing tasks and its application to computer vision is gaining traction. A recent study by Alexey Dosovitskiy and his team shows that reliance on convolutional networks (CNNs) is not necessary. They demonstrate that a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks without the need for CNNs. This approach, called Vision Transformer (ViT), involves pre-training on large amounts of data and transferring it to multiple mid-sized or small image recognition benchmarks such as ImageNet, CIFAR-100, and VTAB. ViT attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. The authors argue that ViT's success is due to its ability to capture long-range dependencies between image patches through self-attention mechanisms. This allows the model to learn meaningful representations of images without being constrained by the local receptive fields of convolutions. Overall, this study suggests that transformers have great potential for computer vision tasks and could potentially replace convolutional networks as the go-to architecture for image recognition at scale.