# CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection

**Authors:** Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A. Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, Zongwei Zhou  
**arXiv:** [2301.00785](https://arxiv.org/abs/2301.00785)  
**Date:** 2023-01-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.00785v1/

## Key points

- Public datasets are often small and partially labeled, limiting their usefulness in assessing anatomical structures and severe tumor subjects.
- The CLIP-Driven Universal Model for segmentation models has been introduced to address these limitations.
- This model uses embedding learned from Contrastive Language-Image Pre-training (CLIP) to better segment 25 organs and 6 types of tumors by exploiting the semantic relationship between abdominal structures.
- The Universal Model is developed from an assembly of 14 datasets with 3,410 CT scans and evaluated on 6,162 external CT scans from three datasets.
- It ranks first on the public leaderboard of the Medical Segmentation Decathlon (MSD) and achieves state-of-the-art results on Beyond The Cranial Vault (BTCV).
- Compared with dataset-specific models, it is computationally more efficient (six times faster), generalizes better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks.
- The design of CLIP embedding enables the Universal Model to be easily extended to new classes without catastrophically forgetting previously learned classes.
- With this breakthrough technology in organ segmentation and tumor detection, medical professionals can improve their assessments with greater accuracy and efficiency than ever before.

## Summary

The medical field has seen a surge in the use of public datasets to assess anatomical structures. However, these datasets are often small and partially labeled, and do not investigate severe tumor subjects. To address these limitations, a team of researchers has introduced the CLIP-Driven Universal Model for segmentation models. This model uses embedding learned from Contrastive Language-Image Pre-training (CLIP) to better segment 25 organs and 6 types of tumors by exploiting the semantic relationship between abdominal structures. The Universal Model is developed from an assembly of 14 datasets with 3,410 CT scans and evaluated on 6,162 external CT scans from three datasets. It ranks first on the public leaderboard of the Medical Segmentation Decathlon (MSD) and achieves state-of-the-art results on Beyond The Cranial Vault (BTCV). Compared with dataset-specific models, it is computationally more efficient (six times faster), generalizes better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks. The design of CLIP embedding enables the Universal Model to be easily extended to new classes without catastrophically forgetting previously learned classes. With this breakthrough technology in organ segmentation and tumor detection, medical professionals can improve their assessments with greater accuracy and efficiency than ever before.