# Multi-modal learning for predicting the genotype of glioma

**Authors:** Yiran Wei, Xi Chen, Lei Zhu, Lipei Zhang, Carola-Bibiane Schönlieb, Stephen J. Price, Chao Li  
**arXiv:** [2203.10852](https://arxiv.org/abs/2203.10852)  
**Date:** 2022-03-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.10852v1/

## Key points

- Isocitrate dehydrogenase (IDH) gene mutation as a crucial biomarker for diagnosing and predicting the prognosis of glioma
- Utilization of focal tumor image and geometric features combined with brain network features from MRI scans to enhance glioma genotype prediction
- Introduction of a multi-modal learning framework with three separate encoders to extract features from focal tumor images, tumor geometrics, and global brain networks
- Implementation of a self-supervised approach for generating brain networks from anatomical multi-sequence MRI scans to address limited diffusion MRI availability
- Design of a hierarchical attention module to extract tumor-related features from the brain network
- Introduction of a bi-level multi-modal contrastive loss function to align multi-modal features and bridge the domain gap between focal tumors and the global brain
- Proposal of a weighted population graph to integrate multi-modal features for more accurate genotype prediction
- Outperformance of baseline deep learning models by their model on experimental testing sets
- Validation through ablation experiments and visualized interpretation aligned with clinical knowledge, showcasing promising advancements in understanding and diagnosing glioma

## Summary

In their study, Wei et al. focus on the isocitrate dehydrogenase (IDH) gene mutation as a crucial biomarker for diagnosing and predicting the prognosis of glioma. They highlight the potential of enhancing glioma genotype prediction by combining focal tumor image and geometric features with brain network features obtained from MRI scans. The researchers propose a multi-modal learning framework that utilizes three separate encoders to extract features from focal tumor images, tumor geometrics, and global brain networks to address limitations in convolutional neural networks' ability to learn from non-Euclidean data. To overcome the challenge of limited diffusion MRI availability, they introduce a self-supervised approach for generating brain networks from anatomical multi-sequence MRI scans. Additionally, a hierarchical attention module is designed to extract tumor-related features from the brain network. The team also introduces a bi-level multi-modal contrastive loss function to align multi-modal features and bridge the domain gap between focal tumors and the global brain. They further propose a weighted population graph to integrate these multi-modal features for more accurate genotype prediction. Experimental results on the testing set demonstrate that their model outperforms baseline deep learning models. Ablation experiments validate the effectiveness of different components within their framework while visualized interpretation of results aligns well with clinical knowledge and undergoes further validation processes. In conclusion, this novel learning framework offers a promising approach for predicting the genotype of glioma, showcasing advancements in understanding and diagnosing this complex disease.