# MDViT: Multi-domain Vision Transformer for Small Medical Image Segmentation Datasets

**Authors:** Siyi Du, Nourhan Bayasi, Ghassan Harmarneh, Rafeef Garbi  
**arXiv:** [2307.02100](https://arxiv.org/abs/2307.02100)  
**Date:** 2023-07-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.02100v1/

## Key points

- Medical image segmentation (MIS) is challenging due to complexity and variability of images.
- Vision transformers (ViTs) are promising for MIS but require larger training datasets compared to CNNs.
- Data-efficient ViTs neglect valuable knowledge from other datasets, leading to negative knowledge transfer (NKT.
- MDViT is a novel multi-domain ViT with domain adapters to leverage knowledge from multiple small data resources across domains.
- Mutual knowledge distillation in MDViT improves representation learning across domains.
- Experimental results show that MDViT outperforms state-of-the-art algorithms in skin lesion segmentation tasks while maintaining fixed model size.

## Summary

In the field of medical image segmentation (MIS), the task remains challenging due to the inherent complexity and variability of images. <br>
Vision transformers (ViTs) have emerged as a promising solution to enhance MIS; however, they require larger training datasets compared to convolutional neural networks. To address this limitation, data-efficient ViTs were introduced, but they are often trained using a single source of data, neglecting the valuable knowledge that could be leveraged from other available datasets. Combining datasets from different domains without consideration can lead to negative knowledge transfer (NKT), resulting in decreased model performance on domains with significant inter-domain heterogeneity. In response to these challenges, a novel approach called MDViT is proposed in this study. MDViT is the first multi-domain ViT that incorporates domain adapters to alleviate data-hunger issues and mitigate NKT by adaptively leveraging knowledge from multiple small data resources across different domains. Furthermore, to improve representation learning across domains, a mutual knowledge distillation paradigm is integrated into MDViT. This paradigm facilitates the transfer of knowledge between a universal network spanning all domains and auxiliary domain-specific branches. Experimental results conducted on four skin lesion segmentation datasets demonstrate that MDViT surpasses state-of-the-art algorithms in terms of segmentation performance while maintaining a fixed model size at inference time, even as additional domains are included. The findings highlight the effectiveness of MDViT in enhancing medical image segmentation tasks by effectively utilizing knowledge from diverse datasets. The code for MDViT is openly available at https://github.com/siyi-wind/MDViT. The authors of this innovative work are Siyi Du, Nourhan Bayasi, Ghassan Harmarneh, and Rafeef Garbi. The research has been accepted for presentation at the 26th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2023).