# A ConvNet for the 2020s

**Authors:** Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie  
**arXiv:** [2201.03545](https://arxiv.org/abs/2201.03545)  
**Date:** 2022-01-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2201.03545v2/

## Key points

- Vision Transformers (ViTs) have become the state-of-the-art image classification model, surpassing Convolutional Neural Networks (ConvNets).
- ViTs face challenges when applied to general computer vision tasks such as object detection and semantic segmentation.
- Hierarchical Transformers, such as Swin Transformers, reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks.
- A new study challenges the belief that the effectiveness of hybrid approaches is largely credited to the intrinsic superiority of Transformers rather than the inherent inductive biases of convolutions.
- The study introduces a family of pure ConvNet models dubbed ConvNeXt that compete favorably with Transformers in terms of accuracy and scalability while maintaining simplicity and efficiency.
- The researchers challenge common beliefs about convolution's importance in computer vision by modernizing ResNets towards Vision Transformer designs without introducing any new modules or architectural decisions beyond those used in traditional CNNs.
- Modern training techniques such as AdamW optimizer, data augmentation techniques, and regularization schemes significantly enhance the performance of a simple ResNet-50 model.

## Summary

In recent years, Vision Transformers (ViTs) have emerged as the state-of-the-art image classification model, surpassing Convolutional Neural Networks (ConvNets). However, ViTs face challenges when applied to general computer vision tasks such as object detection and semantic segmentation. Hierarchical Transformers, such as Swin Transformers, reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. While the effectiveness of these hybrid approaches is largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions, a new study challenges this belief. In "A ConvNet for the 2020s," researchers reexamine the design spaces and test the limits of what a pure ConvNet can achieve. They gradually "modernize" a standard ResNet toward the design of a vision Transformer and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability. They achieve 87.8% ImageNet top-1 accuracy and outperform Swin Transformers on COCO detection and ADE20K segmentation while maintaining the simplicity and efficiency of standard ConvNets. The study provides a trajectory going from ResNets to a ConvNet that bears resemblance to Transformers. The explorations investigate different levels of designs from Swin Transformer while maintaining network simplicity as a standard ConvNet. The conclusions for higher capacity models are consistent with those presented for ResNet-50/Swin-T complexity models. The researchers also challenge common beliefs about convolution's importance in computer vision by modernizing ResNets towards Vision Transformer designs without introducing any new modules or architectural decisions beyond those used in traditional CNNs. By using modern training techniques such as the AdamW optimizer, data augmentation techniques, and regularization schemes, they significantly enhance the performance of a simple ResNet-50 model. This enhanced training recipe increased the ResNet-50 model's performance from 76.1% to 78.8%, implying that a significant portion of the performance difference between traditional ConvNets and Vision Transformers may be due to training techniques. Overall, this study provides new observations and discussions that challenge common beliefs about convolution's importance in computer vision.