# Masked Autoencoders Are Scalable Vision Learners

**Authors:** Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick  
**arXiv:** [2111.06377](https://arxiv.org/abs/2111.06377)  
**Date:** 2021-11-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.06377v1/

## Key points

- The paper presents a simple and scalable approach to self-supervised learning for computer vision using masked autoencoders (MAE).
- The MAE approach involves masking random patches of the input image and reconstructing the missing pixels.
- The authors develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens.
- Masking a high proportion of the input image, such as 75%, yields a nontrivial and meaningful self-supervisory task.
- Their approach enables efficient and effective training of large models, leading to improved accuracy.
- A vanilla ViT-Huge model trained using their method achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data.
- Transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior.
- Simple self-supervised learning methods like MAE can provide scalable benefits in computer vision as well.
- Comparing with BEiT [2], their MAE approach achieves comparable results while being simpler and more computationally efficient.
- They demonstrate state-of-the-art performance on several benchmarks without relying on external data or supervision.

## Summary

This paper presents a simple and scalable approach to self-supervised learning for computer vision using masked autoencoders (MAE). The MAE approach involves masking random patches of the input image and reconstructing the missing pixels. The authors develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. They find that masking a high proportion of the input image, such as 75%, yields a nontrivial and meaningful self-supervisory task. The authors demonstrate that their approach enables efficient and effective training of large models, leading to improved accuracy. For instance, a vanilla ViT-Huge model trained using their method achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Moreover, transfer performance in downstream tasks outperforms supervised pre-training and shows promising scaling behavior. The authors note that while images and languages are signals of different nature, similar to NLP, simple self-supervised learning methods can provide scalable benefits in computer vision as well. Instead of attempting to remove objects from images, they remove random patches that most likely do not form a semantic segment. Likewise, their MAE reconstructs pixels rather than generating images. Comparing with BEiT [2], their MAE approach achieves comparable results while being simpler and more computationally efficient. Additionally, they demonstrate state-of-the-art performance on several benchmarks without relying on external data or supervision. Overall, this study highlights the potential for simple self-supervised learning methods like MAE to enable efficient training of large models in computer vision without relying on external supervision or data augmentation techniques.