# Cross-modal Memory Networks for Radiology Report Generation

**Authors:** Zhihong Chen, Yaling Shen, Yan Song, Xiang Wan  
**arXiv:** [2204.13258](https://arxiv.org/abs/2204.13258)  
**Date:** 2022-04-28  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2204.13258v1/

## Key points

- Medical imaging is crucial for medical diagnosis in clinical practice
- Text reports of images are vital for understanding and guiding treatments
- Automating report generation helps radiologists and advances clinical automation
- Cross-modal mappings enhance radiology report generation
- Cross-modal Memory Networks (CMN) framework improves encoder-decoder models
- CMN captures alignment between images and texts for accurate reports
- CMN model demonstrates state-of-the-art performance on benchmark datasets
- CMN excels at aligning information from images and texts for improved accuracy

## Summary

Medical imaging is a crucial component of medical diagnosis in clinical practice. Text reports of images play a vital role in understanding and guiding subsequent treatments. Automating the generation of these reports not only helps alleviate the workload of radiologists but also advances clinical automation, a field gaining traction in the application of artificial intelligence to healthcare. Previous studies have predominantly focused on the encoder-decoder paradigm for text generation, but few have explored the significance of cross-modal mappings in enhancing radiology report generation. In response to this gap, a novel approach called Cross-modal Memory Networks (CMN) is introduced in this paper. The CMN framework enhances the traditional encoder-decoder model by incorporating a shared memory mechanism that captures the alignment between images and texts. This shared memory facilitates seamless interaction and generation across modalities, leading to more accurate and contextually relevant reports. Experimental results demonstrate the efficacy of the proposed CMN model, showcasing state-of-the-art performance on two widely used benchmark datasets: IU X-Ray and MIMIC-CXR. Further analysis confirms that the CMN model excels at aligning information from radiology images and texts, resulting in improved accuracy in generating reports with respect to clinical indicators. The authors - Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan - contribute significantly to advancing the field of radiology report generation through their innovative approach that leverages cross-modal memory networks for enhanced performance and efficiency in medical imaging analysis.