Summarize Paper logo SummarizePaper.com

AI summary · arXiv: 2404.18416v1 · cs.AI

Capabilities of Gemini Models in Medicine

Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother,
and 58 more authorsChunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu-Yeu Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G. T. Barrett, Cathy Cheung, Basil Mustafa, Anil Palepu, Daniel McDuff, Le Hou, Tomer Golany, Luyang Liu, Jean-baptiste Alayrac, Neil Houlsby, Nenad Tomasev, Jan Freyberg, Charles Lau, Jonas Kemp, Jeremy Lai, Shekoofeh Azizi, Kimberly Kanada, SiWai Man, Kavita Kulkarni, Ruoxi Sun, Siamak Shakeri, Luheng He, Ben Caine, Albert Webson, Natasha Latysheva, Melvin Johnson, Philip Mansfield, Jian Lu, Ehud Rivlin, Jesper Anderson, Bradley Green, Renee Wong, Jonathan Krause, Jonathon Shlens, Ewa Dominowska, S. M. Ali Eslami, Claire Cui, Oriol Vinyals, Koray Kavukcuoglu, James Manyika, Jeff Dean, Demis Hassabis, Yossi Matias, Dale Webster, Joelle Barral, Greg Corrado, Christopher Semturs, S. Sara Mahdavi, Juraj Gottweis, Alan Karthikesalingam, Vivek Natarajan
Original paper Submitted to arXiv on 29 Apr. 2024 License: CC BY 4.0

How to read this summary? Illustrated guide

The essentials

  • Gemini models are promising in medical applications, excelling in multimodal and long-context reasoning.
  • Med-Gemini is a specialized family of highly capable multimodal models designed for medical tasks, leveraging web search functionalities and customizable for novel modalities.
  • Med-Gemini outperformed GPT-4 on 10 benchmarks and achieved an accuracy rate of 91.1% on the challenging MedQA benchmark.
  • Across seven multimodal benchmarks, Med-Gemini showed a 44.5% average relative improvement over GPT-4V.
  • Med-Gemini demonstrated its long-context capabilities in scenarios like scientific information synthesis and electronic health record dialogue.
  • Its potential extends to areas such as medical education, surgical training improvement, operating room efficiency optimization, and enhanced patient outcomes.
  • The performance of Med-Gemini underscores its potential utility in real-world healthcare settings, highlighting the transformative impact AI models can have on healthcare practices and outcomes.

Key points written by AI: they may contain errors, check them against the paper.

AI-generated keywords: Medical Applications Artificial Intelligence Multimodal Models Long-Context Reasoning Med-Gemini

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
In the realm of medical applications, achieving excellence with artificial intelligence (AI) presents a myriad of challenges that necessitate advanced reasoning, access to current medical knowledge, and the ability to comprehend complex multimodal data. Gemini models have emerged as promising contenders in this field, boasting strong capabilities in multimodal and long-context reasoning. To further enhance their utility in medicine, we introduce Med-Gemini - a specialized family of highly capable multimodal models designed specifically for medical tasks. These models are adept at leveraging web search functionalities and can be easily customized for novel modalities using custom encoders. Through rigorous evaluation on 14 medical benchmarks, Med-Gemini has demonstrated its prowess by establishing new state-of-the-art performance on 10 benchmarks and outperforming the GPT-4 model family on every comparable benchmark. Notably, on the challenging MedQA (USMLE) benchmark, our leading Med-Gemini model achieved an impressive accuracy rate of 91.1% through a novel uncertainty-guided search strategy. Across seven multimodal benchmarks including NEJM Image Challenges and MMMU (health & medicine), Med-Gemini showcased an average relative improvement of 44.5% over GPT-4V. The long-context capabilities of Med-Gemini were put to the test in various scenarios such as scientific information synthesis and electronic health record (EHR) dialogue. For instance, Figure 14 illustrates how Med-Gemini-M 1.5 effectively synthesized information from research articles to elucidate the mechanistic link between the FTO locus and obesity biology with supporting experimental evidence. Additionally, Figure 13 showcases how Med-Gemini-M 1.5 parsed extensive medical records to provide comprehensive summaries of patient conditions, demonstrating its potential to streamline clinician interactions with complex medical data. Moreover, the application of long-context capabilities in biomedicine extends beyond quantitative results to encompass areas such as medical education and clinical practice enhancement. For example, Med-Gemini's ability to identify surgical actions from videos holds promise for improving surgical training and optimizing operating room efficiency. Furthermore, its proficiency in analyzing surgical video dialogues could potentially enhance educational aids or assist clinicians during procedures for improved patient outcomes. In conclusion, the compelling performance of Med-Gemini across diverse medical tasks underscores its potential utility in real-world healthcare settings. While further rigorous evaluation is essential before deployment in safety-critical domains like medicine, these results highlight the transformative impact AI models like Med-Gemini can have on advancing healthcare practices and outcomes through innovative technology solutions.
Created on 01 May. 2024

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.
AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in
Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown) · Guides: Obsidian, Notion

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero Zotero guide
Citation alerts Citation alerts guide
Who works on it? The people and institutions who publish on its topic
Journal club Premium+ Slides to present it Propose for the journal club Journal club guide
Versions What changed between versions Versions guide
Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Very close
    MedGemma Technical Report
    Andrew B. Sellergren, S. Kazemzadeh, Tiam Jaroensri et al. · 2025 · cited 532 times · Not summarized yet
    Abstract

    Artificial intelligence (AI) has significant potential in healthcare applications, but its training and deployment faces challenges due to healthcare's diverse data, complex tasks, and the need to preserve privacy. Foundation models that perform well on medical tasks and require less task-specific tuning data are critical to accelerate the development of healthcare AI applications. We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B. MedGemma demonstrates advanced medical understanding and reasoning on images and text, significantly exceeding the performance of similar-sized generative models and approaching the performance of task-spe…

  2. Very close Cites this paper Uses this paper in depth
    Application of Gemini in Public Health Amid the Artificial Intelligence Era
    Moosa Tatar, Soheila Farokhi, Arash Azizian Foumani et al. · 2026 · IEEE Engineering Management Review · cited 4 times · Not summarized yet
    Abstract

    In December 2023, Google DeepMind unveiled its large language model, Gemini, which is shown to be capable of understanding and processing multimodal information. This article explores the applications of Gemini in public health. Unlike traditional models, Gemini integrates and analyzes information through diverse modalities (text, image, video, etc.) fostering new applications. Notably, Google Gemini is presented as the first model to surpass human experts on the Massive Multitask Language Understanding benchmark, which combines multiple subjects including science, technology, engineering, and mathematics, humanities, and others to test the knowledge and problem-solving abilities of artifici…

  3. Very close Cites this paper
    Potential of multimodal large language models for data mining of medical images and free-text reports
    Yutong Zhang, Yi Pan, Tianyang Zhong et al. · 2024 · Meta-Radiology · cited 52 times · Not summarized yet
    Abstract

    Medical images and radiology reports are essential for physicians to diagnose medical conditions . However, the vast diversity and cross-source heterogeneity inherent in these data have posed significant challenges to the generalizability of current data-mining methods for clinical decision-making. Recently, multimodal large language models (MLLMs), especially Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models, have revolutionized numerous domains, significantly impacting the medical field. In this study, we conducted a detailed evaluation of the performance of the Gemini series models (including Gemini-1.0-Pro-Vision, Gemini-1.5-Pro, and Gemini-1.5-Flash) and GPT series models (i…

  4. Very close Cites this paper
    An open vision-language model for diverse medical applications
    Andrew B. Sellergren, S. Kazemzadeh, Fereshteh Mahvar et al. · 2026 · Nature Medicine · Not summarized yet
    Abstract

    Artificial intelligence has high potential for impact in healthcare applications, but its training and deployment are challenging due to diverse data, a complex spectrum of possible tasks and important privacy needs. High-performing foundation models that enable data-efficient fine-tuning for diverse downstream tasks can meaningfully accelerate development in this domain. We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3. MedGemma demonstrates advanced medical understanding and reasoning across images and text and multiple medical imaging domains, exceeding the performance of similarly sized generative models while maintaining the general capab…

  5. Very close
    Towards Generalist Biomedical AI
    Tao Tu, Shekoofeh Azizi, Danny Driess et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 647 times · Not summarized yet
    Abstract

    Medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence (AI) systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery. To enable the development of these models, we first curate MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduce Med-PaLM Multimodal (Med-PaLM M), ou…

  6. Close
    A Modular Multi-Agent Multimodal AI Framework Integrating Vision–Language Models and IoT Data for Explainable Clinical Decision Support
    Et-Tousy Jamal, Chtara Malek, Zyane Abdellah · 2026 · 2026 6th International Conference on Innovative Research in Applied Science, Engineering and Technology (IRASET) · Not summarized yet
    Abstract

    Clinical decision support systems now provide more options because of the integration of medical imaging, clinical text, and contextual data utilizing contemporary visionlanguage models (VLMs). Performance, explainability, accessibility, and regulatory compliance are all trade-offs in any model. In this study, we assess four VLMs using multimodal medical benchmarks: MedGemma-4B-IT, LLaVA-Med, MedVLMR1, and GPT-4V. Because of its high multimodal capabilities, reasoning ability, integration of external knowledge, and modifiability, we have chosen MedGemma-4B-IT as the best opensource choice. MedGemma-4B-IT provides competitive baseline performance on standard medical VQA benchmarks by building…

  7. Close Cites this paper Uses this paper in depth
    Towards conversational artificial intelligence for disease management
    Valentin Liévin, Anil Palepu, Wei-Hung Weng et al. · 2026 · Nature · cited 18 times · Not summarized yet
    Abstract

    Although large language models have shown promise in diagnostic dialogue1, their capabilities for effective management reasoning, including disease progression, therapeutic response and safe medication prescription, have remained underexplored. We have advanced the previously demonstrated diagnostic capabilities of the Articulate Medical Intelligence Explorer (AMIE)1, 2–3 using a new large-language-model-based agentic system optimized for multivisit clinical management and dialogue. To ground the reasoning of AMIE in authoritative clinical knowledge, we leveraged the long-context capabilities of Gemini4, combining in-context retrieval with structured reasoning to align its output with up-to-…

  8. Close Cites this paper
    Meissa: Multi-modal Medical Agentic Intelligence
    Yi-Xiong Chen, Xin-Yi Bai, Yue Pan et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 6 times · Not summarized yet
    Abstract

    Multi-modal large language models (MM-LLMs) have shown strong performance in medical image understanding and clinical reasoning. Recent medical agent systems extend them with tool use and multi-agent collaboration, enabling complex decision-making. However, these systems rely almost entirely on frontier models (e.g., GPT), whose API-based deployment incurs high cost, high latency, and privacy risks that conflict with on-premise clinical requirements. We present Meissa, a lightweight 4B-parameter medical MM-LLM that brings agentic capability offline. Instead of imitating static answers, Meissa learns both when to engage external interaction (strategy selection) and how to execute multi-step i…

Papers that cite it

  1. Close
    Large Language Models lack essential metacognition for reliable medical reasoning
    Maxime Griot, Coralie Hemptinne, Jean M. Vanderdonckt et al. · 2025 · Nature Communications · cited 126 times · Not summarized yet
    Abstract

    Large Language Models have demonstrated expert-level accuracy on medical board examinations, suggesting potential for clinical decision support systems. However, their metacognitive abilities, crucial for medical decision-making, remain largely unexplored. To address this gap, we developed MetaMedQA, a benchmark incorporating confidence scores and metacognitive tasks into multiple-choice medical questions. We evaluated twelve models on dimensions including confidence-based accuracy, missing answer recall, and unknown recall. Despite high accuracy on multiple-choice questions, our study revealed significant metacognitive deficiencies across all tested models. Models consistently failed to rec…

  2. Related Uses this paper in depth
    Multimodal generative AI for interpreting 3D medical images and videos
    Jung-Oh Lee, Hong-Yu Zhou, T. Berzin et al. · 2025 · npj Digital Medicine · cited 27 times · Not summarized yet
    Abstract

    This perspective proposes adapting video-text generative AI to 3D medical imaging (CT/MRI) and medical videos (endoscopy/laparoscopy) by treating 3D images as videos. The approach leverages modern video models to analyze multiple sequences simultaneously and provide real-time AI assistance during procedures. The paper examines medical imaging’s unique characteristics (synergistic information, metadata, and world model), outlines applications in automated reporting, case retrieval, and education, and addresses challenges of limited datasets, benchmarks, and specialized training.

  3. Close
    Advancing conversational diagnostic AI with multimodal reasoning
    Khaled Saab, Chunjong Park, Tim Strother et al. · 2026 · Nature Medicine · cited 9 times · Not summarized yet
    Abstract

    Real-world clinical practice is inherently multimodal, relying on the synthesis of patient history with visual information such as medical imagery and clinical documents. Although large language models (LLMs) have shown promise in diagnostic dialogue, their evaluation has been largely restricted to text-only interactions, failing to capture the complexity of modern remote care delivery. Here we introduce a multimodal extension of the Articulate Medical Intelligence Explorer (multimodal AMIE), capable of gathering, interpreting and reasoning about multimodal data within a diagnostic conversation. To achieve this, we developed a state-aware dialogue framework that dynamically guides history-ta…

  4. Related Uses this paper in depth
    Agentic Medical Knowledge Graphs Enhance Medical Question Answering: Bridging the Gap Between LLMs and Evolving Medical Knowledge
    Mohammad R. Rezaei, Reza Saadati Fard, J. Parker et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 29 times · Not summarized yet
    Abstract

    Large Language Models (LLMs) have significantly advanced medical question-answering by leveraging extensive clinical data and medical literature. However, the rapid evolution of medical knowledge and the labor-intensive process of manually updating domain-specific resources pose challenges to the reliability of these systems. To address this, we introduce Agentic Medical Graph-RAG (AMG-RAG), a comprehensive framework that automates the construction and continuous updating of medical knowledge graphs, integrates reasoning, and retrieves current external evidence, such as PubMed and WikiSearch. By dynamically linking new findings and complex medical concepts, AMG-RAG not only improves accuracy…

  5. Related Uses this paper in depth
    Foundation Model for Medical Imaging: A Comprehensive Review
    Li-Cheng Jiao, Jingyi Yang, Rui-Yang Li et al. · 2026 · IEEE Transactions on Artificial Intelligence · cited 8 times · Not summarized yet
    Abstract

    The rapid development of highly flexible and reusable foundation models has introduced a new paradigm shift in the field of medical artificial intelligence (AI). Medical foundation models (MFMs) are built upon large-scale, diverse datasets and utilize multistage training strategies to thoroughly understand task representations, fitting the differences between various patterns, thereby achieving contextual reasoning abilities and broad applicability. MFMs can perform zero-shot inference on a variety of downstream tasks and also enhance the understanding of new tasks with task-specific prompts, without the need for extensive labeled data and retraining. To achieve more transparent and universa…

Go further

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 3 November 2026.

The illustrated guide to the papers around a paper

The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.