# Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis

**Authors:** Yucheng Tang, Dong Yang, Wenqi Li, Holger Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, Ali Hatamizadeh  
**arXiv:** [2111.14791](https://arxiv.org/abs/2111.14791)  
**Date:** 2021-11-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2111.14791v2/

## Key points

- Introduction of a novel self-supervised learning framework for medical image analysis
- Proposal of a new 3D transformer-based model called Swin UNETR
- Utilization of a hierarchical encoder for self-supervised pre-training
- Focus on learning global and local representations transferable to downstream applications
- Training on 5,050 publicly available CT images for human anatomy pattern recognition
- Evaluation on BTCV Segmentation Challenge and MSD dataset
- Achieving state-of-the-art performance and ranking first on both datasets
- Superiority in medical image analysis compared to existing methods
- Potential for improving segmentation accuracy and advancing research in the field

## Summary

The authors of this paper introduce a novel self-supervised learning framework for medical image analysis, inspired by the success of Vision Transformers (ViTs) in self-supervised learning. They propose a new 3D transformer-based model called Swin UNEt TRansformers (Swin UNETR), which utilizes a hierarchical encoder for self-supervised pre-training. The model is designed to learn global and local representations that can be transferred to downstream applications. To train the Swin UNETR model, the authors use tailored proxy tasks that focus on learning the underlying patterns of human anatomy in medical images. They demonstrate the effectiveness of their approach by pre-training the model on 5,050 publicly available computed tomography (CT) images from various body organs. The performance of the pre-trained Swin UNETR models is evaluated by fine-tuning them on two challenging datasets: Beyond the Cranial Vault (BTCV) Segmentation Challenge and Medical Segmentation Decathlon (MSD) dataset. The BTCV dataset involves segmenting 13 abdominal organs, while the MSD dataset includes various segmentation tasks. Remarkably, the proposed Swin UNETR model achieves state-of-the-art performance and is ranked first on both public test leaderboards of the MSD and BTCV datasets. This demonstrates its superiority in medical image analysis tasks compared to other existing methods. Overall, this paper presents a promising self-supervised learning framework using Swin UNETR models for 3D medical image analysis. The results highlight its potential for improving segmentation accuracy and advancing research in this field.