# Swin MAE: Masked Autoencoders for Small Datasets

**Authors:** Zi'an Xu, Yin Dai, Fayu Liu, Weibing Chen, Yue Liu, Lifu Shi, Sheng Liu, Yuhang Zhou  
**arXiv:** [2212.13805](https://arxiv.org/abs/2212.13805)  
**Date:** 2022-12-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.13805v2/

## Key points

- Lack of large and well-annotated datasets hinders the development of deep learning models in medical image analysis.
- Unsupervised learning offers a solution by not requiring labeled data, but existing methods are often designed for large datasets.
- Swin MAE, developed by Zi'an Xu and team, combines masked autoencoders with Swin Transformer to enable unsupervised learning on small datasets in medical imaging.
- Swin MAE can extract meaningful semantic features from a few thousand medical images without pre-trained models and achieves comparable or superior performance to supervised models in transfer learning scenarios.
- The code implementation of Swin MAE is openly available on GitHub at https://github.com/Zian-Xu/Swin-MAE, providing a valuable resource for researchers and practitioners.

## Summary

In the field of medical image analysis, the development of deep learning models is often hindered by the lack of large and well-annotated datasets. <br>
Unsupervised learning presents a promising approach for addressing this challenge by not requiring labeled data. <br>
However, many existing unsupervised learning methods are designed for large datasets, making them less suitable for applications with limited data availability. <br>
To bridge this gap, a team of researchers led by Zi'an Xu introduced Swin MAE - a novel approach that leverages masked autoencoders with Swin Transformer as its backbone to enable unsupervised learning on small datasets. <br>
Despite working with only a few thousand medical images and without relying on pre-trained models, Swin MAE demonstrates the ability to extract meaningful semantic features directly from images. <br>
Remarkably, experimental results show that Swin MAE can achieve comparable or even slightly superior performance compared to supervised models trained on ImageNet using Swin Transformer when applied to downstream tasks in transfer learning scenarios. This highlights the effectiveness and potential of Swin MAE in enabling robust and efficient medical image analysis solutions. <br>
The code implementation of Swin MAE is openly available on GitHub at https://github.com/Zian-Xu/Swin-MAE - providing a valuable resource for researchers and practitioners looking to explore and utilize this innovative approach in their own work. <br>
The collaborative efforts of authors Zi'an Xu, Yin Dai, Fayu Liu, Weibing Chen, Yue Liu, Lifu Shi, Sheng Liu, and Yuhang Zhou have significantly contributed to advancing the capabilities of deep learning models in medical imaging applications.