# Med-Flamingo: a Multimodal Medical Few-shot Learner

**Authors:** Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec  
**arXiv:** [2307.15189](https://arxiv.org/abs/2307.15189)  
**Date:** 2023-07-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.15189v1/

## Key points

- Introduction of Med-Flamingo, a multimodal few-shot learner for the medical domain
- Model based on OpenFlamingo-9B and pre-trained on medical image-text data
- Development of Med-Flamingo enables generative medical visual question answering (VQA)
- Human evaluation shows up to 20% improvement in clinician's rating for generative medical VQA
- Enables multimodal medical few-shot adaptations such as rationale generation
- Model, code, and evaluation app released for further research and development
- Highlights shortcomings in existing evaluation strategies for generative medical VQA models
- In-depth clinical evaluation study conducted using a dedicated evaluation app with medical raters
- Demonstrates effectiveness of Med-Flamingo in generative medical VQA tasks
- Addresses challenge of scarce data in many medical applications and potential for various clinical applications

## Summary

The paper introduces Med-Flamingo, a multimodal few-shot learner specifically designed for the medical domain. The model is based on OpenFlamingo-9B and continues pre-training on paired and interleaved medical image-text data from publications and textbooks. The main contribution of this work is the development of Med-Flamingo, which unlocks few-shot generative medical visual question answering (VQA) abilities. To assess the performance of Med-Flamingo, the authors conduct a human evaluation where physicians review the problems and blinded generations in an interactive app. The results show that Med-Flamingo improves performance in generative medical VQA by up to 20% in clinician's rating. Additionally, it enables multimodal medical few-shot adaptations such as rationale generation. The authors release their model, code, and evaluation app for further research and development. In addition to presenting Med-Flamingo, the paper also contributes to related works by highlighting shortcomings in existing evaluation strategies for generative medical VQA models. The authors conduct an in-depth clinical evaluation study using a dedicated evaluation app with medical raters. Overall, this paper presents a novel multimodal few-shot learner adapted to the medical domain and demonstrates its effectiveness in generative medical VQA tasks. It addresses the challenge of scarce data in many medical applications and shows promising potential for various clinical applications such as rationale generation and conditioning on retrieved multimodal context.