Capabilities of Gemini Models in Medicine
Authors: Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu-Yeu Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G. T. Barrett, Cathy Cheung, Basil Mustafa, Anil Palepu, Daniel McDuff, Le Hou, Tomer Golany, Luyang Liu, Jean-baptiste Alayrac, Neil Houlsby, Nenad Tomasev, Jan Freyberg, Charles Lau, Jonas Kemp, Jeremy Lai, Shekoofeh Azizi, Kimberly Kanada, SiWai Man, Kavita Kulkarni, Ruoxi Sun, Siamak Shakeri, Luheng He, Ben Caine, Albert Webson, Natasha Latysheva, Melvin Johnson, Philip Mansfield, Jian Lu, Ehud Rivlin, Jesper Anderson, Bradley Green, Renee Wong, Jonathan Krause, Jonathon Shlens, Ewa Dominowska, S. M. Ali Eslami, Claire Cui, Oriol Vinyals, Koray Kavukcuoglu, James Manyika, Jeff Dean, Demis Hassabis, Yossi Matias, Dale Webster, Joelle Barral, Greg Corrado, Christopher Semturs, S. Sara Mahdavi, Juraj Gottweis, Alan Karthikesalingam, Vivek Natarajan
Abstract
Excellence in a wide variety of medical applications poses considerable challenges for AI, requiring advanced reasoning, access to up-to-date medical knowledge and understanding of complex multimodal data. Gemini models, with strong general capabilities in multimodal and long-context reasoning, offer exciting possibilities in medicine. Building on these core strengths of Gemini, we introduce Med-Gemini, a family of highly capable multimodal models that are specialized in medicine with the ability to seamlessly use web search, and that can be efficiently tailored to novel modalities using custom encoders. We evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art (SoTA) performance on 10 of them, and surpass the GPT-4 model family on every benchmark where a direct comparison is viable, often by a wide margin. On the popular MedQA (USMLE) benchmark, our best-performing Med-Gemini model achieves SoTA performance of 91.1% accuracy, using a novel uncertainty-guided search strategy. On 7 multimodal benchmarks including NEJM Image Challenges and MMMU (health & medicine), Med-Gemini improves over GPT-4V by an average relative margin of 44.5%. We demonstrate the effectiveness of Med-Gemini's long-context capabilities through SoTA performance on a needle-in-a-haystack retrieval task from long de-identified health records and medical video question answering, surpassing prior bespoke methods using only in-context learning. Finally, Med-Gemini's performance suggests real-world utility by surpassing human experts on tasks such as medical text summarization, alongside demonstrations of promising potential for multimodal medical dialogue, medical research and education. Taken together, our results offer compelling evidence for Med-Gemini's potential, although further rigorous evaluation will be crucial before real-world deployment in this safety-critical domain.
The papers around it, by date and closeness
Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper
Papers around this one
Ranked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
MedGemma Technical ReportAndrew B. Sellergren, S. Kazemzadeh, Tiam Jaroensri et al. · 2025 · cited 532 times · Not summarized yet
Abstract
Artificial intelligence (AI) has significant potential in healthcare applications, but its training and deployment faces challenges due to healthcare's diverse data, complex tasks, and the need to preserve privacy. Foundation models that perform well on medical tasks and require less task-specific tuning data are critical to accelerate the development of healthcare AI applications. We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B. MedGemma demonstrates advanced medical understanding and reasoning on images and text, significantly exceeding the performance of similar-sized generative models and approaching the performance of task-spe…
-
Application of Gemini in Public Health Amid the Artificial Intelligence EraMoosa Tatar, Soheila Farokhi, Arash Azizian Foumani et al. · 2026 · IEEE Engineering Management Review · cited 4 times · Not summarized yet
Abstract
In December 2023, Google DeepMind unveiled its large language model, Gemini, which is shown to be capable of understanding and processing multimodal information. This article explores the applications of Gemini in public health. Unlike traditional models, Gemini integrates and analyzes information through diverse modalities (text, image, video, etc.) fostering new applications. Notably, Google Gemini is presented as the first model to surpass human experts on the Massive Multitask Language Understanding benchmark, which combines multiple subjects including science, technology, engineering, and mathematics, humanities, and others to test the knowledge and problem-solving abilities of artifici…
-
Potential of multimodal large language models for data mining of medical images and free-text reportsYutong Zhang, Yi Pan, Tianyang Zhong et al. · 2024 · Meta-Radiology · cited 52 times · Not summarized yet
Abstract
Medical images and radiology reports are essential for physicians to diagnose medical conditions . However, the vast diversity and cross-source heterogeneity inherent in these data have posed significant challenges to the generalizability of current data-mining methods for clinical decision-making. Recently, multimodal large language models (MLLMs), especially Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models, have revolutionized numerous domains, significantly impacting the medical field. In this study, we conducted a detailed evaluation of the performance of the Gemini series models (including Gemini-1.0-Pro-Vision, Gemini-1.5-Pro, and Gemini-1.5-Flash) and GPT series models (i…
-
An open vision-language model for diverse medical applicationsAndrew B. Sellergren, S. Kazemzadeh, Fereshteh Mahvar et al. · 2026 · Nature Medicine · Not summarized yet
Abstract
Artificial intelligence has high potential for impact in healthcare applications, but its training and deployment are challenging due to diverse data, a complex spectrum of possible tasks and important privacy needs. High-performing foundation models that enable data-efficient fine-tuning for diverse downstream tasks can meaningfully accelerate development in this domain. We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3. MedGemma demonstrates advanced medical understanding and reasoning across images and text and multiple medical imaging domains, exceeding the performance of similarly sized generative models while maintaining the general capab…
-
Towards Generalist Biomedical AITao Tu, Shekoofeh Azizi, Danny Driess et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 647 times · Not summarized yet
Abstract
Medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence (AI) systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery. To enable the development of these models, we first curate MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduce Med-PaLM Multimodal (Med-PaLM M), ou…
-
A Modular Multi-Agent Multimodal AI Framework Integrating Vision–Language Models and IoT Data for Explainable Clinical Decision SupportEt-Tousy Jamal, Chtara Malek, Zyane Abdellah · 2026 · 2026 6th International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · Not summarized yet
Abstract
Clinical decision support systems now provide more options because of the integration of medical imaging, clinical text, and contextual data utilizing contemporary visionlanguage models (VLMs). Performance, explainability, accessibility, and regulatory compliance are all trade-offs in any model. In this study, we assess four VLMs using multimodal medical benchmarks: MedGemma-4B-IT, LLaVA-Med, MedVLMR1, and GPT-4V. Because of its high multimodal capabilities, reasoning ability, integration of external knowledge, and modifiability, we have chosen MedGemma-4B-IT as the best opensource choice. MedGemma-4B-IT provides competitive baseline performance on standard medical VQA benchmarks by building…
-
Towards conversational artificial intelligence for disease managementValentin Liévin, Anil Palepu, Wei-Hung Weng et al. · 2026 · Nature · cited 18 times · Not summarized yet
Abstract
Although large language models have shown promise in diagnostic dialogue1, their capabilities for effective management reasoning, including disease progression, therapeutic response and safe medication prescription, have remained underexplored. We have advanced the previously demonstrated diagnostic capabilities of the Articulate Medical Intelligence Explorer (AMIE)1, 2–3 using a new large-language-model-based agentic system optimized for multivisit clinical management and dialogue. To ground the reasoning of AMIE in authoritative clinical knowledge, we leveraged the long-context capabilities of Gemini4, combining in-context retrieval with structured reasoning to align its output with up-to-…
-
Meissa: Multi-modal Medical Agentic IntelligenceYi-Xiong Chen, Xin-Yi Bai, Yue Pan et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
Abstract
Multi-modal large language models (MM-LLMs) have shown strong performance in medical image understanding and clinical reasoning. Recent medical agent systems extend them with tool use and multi-agent collaboration, enabling complex decision-making. However, these systems rely almost entirely on frontier models (e.g., GPT), whose API-based deployment incurs high cost, high latency, and privacy risks that conflict with on-premise clinical requirements. We present Meissa, a lightweight 4B-parameter medical MM-LLM that brings agentic capability offline. Instead of imitating static answers, Meissa learns both when to engage external interaction (strategy selection) and how to execute multi-step i…
Papers that cite it
-
Large Language Models lack essential metacognition for reliable medical reasoningMaxime Griot, Coralie Hemptinne, Jean M. Vanderdonckt et al. · 2025 · Nature Communications · cited 126 times · Not summarized yet
Abstract
Large Language Models have demonstrated expert-level accuracy on medical board examinations, suggesting potential for clinical decision support systems. However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored. To address this gap, we developed MetaMedQA, a benchmark incorporating confidence scores and metacognitive tasks into multiple-choice medical questions. We evaluated twelve models on dimensions including confidence-based accuracy, missing answer recall, and unknown recall. Despite high accuracy on multiple-choice questions, our study revealed significant metacognitive deficiencies across all tested models. Models consistently failed to rec…
-
Multimodal generative AI for interpreting 3D medical images and videosJung-Oh Lee, Hong-Yu Zhou, T. Berzin et al. · 2025 · npj Digital Medicine · cited 27 times · Not summarized yet
Abstract
This perspective proposes adapting video-text generative AI to 3D medical imaging (CT/MRI) and medical videos (endoscopy/laparoscopy) by treating 3D images as videos. The approach leverages modern video models to analyze multiple sequences simultaneously and provide real-time AI assistance during procedures. The paper examines medical imaging’s unique characteristics (synergistic information, metadata, and world model), outlines applications in automated reporting, case retrieval, and education, and addresses challenges of limited datasets, benchmarks, and specialized training.
-
Advancing conversational diagnostic AI with multimodal reasoningKhaled Saab, Chunjong Park, Tim Strother et al. · 2026 · Nature Medicine · cited 9 times · Not summarized yet
Abstract
Real-world clinical practice is inherently multimodal, relying on the synthesis of patient history with visual information such as medical imagery and clinical documents. Although large language models (LLMs) have shown promise in diagnostic dialogue, their evaluation has been largely restricted to text-only interactions, failing to capture the complexity of modern remote care delivery. Here we introduce a multimodal extension of the Articulate Medical Intelligence Explorer (multimodal AMIE), capable of gathering, interpreting and reasoning about multimodal data within a diagnostic conversation. To achieve this, we developed a state-aware dialogue framework that dynamically guides history-ta…
-
Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical KnowledgeMohammad R. Rezaei, Reza Saadati Fard, J. Parker et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 29 times · Not summarized yet
Abstract
Large Language Models (LLMs) have significantly advanced medical question-answering by leveraging extensive clinical data and medical literature. However, the rapid evolution of medical knowledge and the labor-intensive process of manually updating domain-specific resources pose challenges to the reliability of these systems. To address this, we introduce Agentic Medical Graph-RAG (AMG-RAG), a comprehensive framework that automates the construction and continuous updating of medical knowledge graphs, integrates reasoning, and retrieves current external evidence, such as PubMed and WikiSearch. By dynamically linking new findings and complex medical concepts, AMG-RAG not only improves accuracy…
-
Foundation Model for Medical Imaging: A Comprehensive ReviewLi-Cheng Jiao, Jingyi Yang, Rui-Yang Li et al. · 2026 · IEEE Transactions on Artificial Intelligence · cited 8 times · Not summarized yet
Abstract
The rapid development of highly flexible and reusable foundation models has introduced a new paradigm shift in the field of medical artificial intelligence (AI). Medical foundation models (MFMs) are built upon large-scale, diverse datasets and utilize multistage training strategies to thoroughly understand task representations, fitting the differences between various patterns, thereby achieving contextual reasoning abilities and broad applicability. MFMs can perform zero-shot inference on a variety of downstream tasks and also enhance the understanding of new tasks with task-specific prompts, without the need for extensive labeled data and retraining. To achieve more transparent and universa…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 3 November 2026.