# Towards Expert-Level Medical Question Answering with Large Language Models

**Authors:** Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev  
**arXiv:** [2305.09617](https://arxiv.org/abs/2305.09617)  
**Date:** 2023-05-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.09617v1/

## Key points

- Recent advancements in AI systems have achieved remarkable milestones in various grand challenges, including Go and protein-folding.
- Large language models (LLMs) have played a significant role in the progress of retrieving medical knowledge, reasoning over it, and answering medical questions comparable to physicians.
- Med-PaLM was the first model to exceed a "passing" score in US Medical Licensing Examination-style questions on the MedQA dataset.
- Researchers presented Med-PaLM 2 which leverages a combination of base LLM improvements (PaLM 2), medical domain finetuning and prompting strategies that include a novel ensemble refinement approach.
- The new model scored up to 86.5% on the MedQA dataset setting a new state-of-the-art.
- Performance approaching or exceeding state-of-the-art was also observed across other clinical topics datasets like MedMCQA, PubMedQA and MMLU.
- Detailed human evaluations were performed on long-form questions relevant to clinical applications showing that physicians preferred Med-PaLM 2 answers compared to those produced by physicians on eight of nine axes pertaining to clinical utility (p < 0.001).
- Potential risks associated with using such models include inaccurate or irrelevant information, omission of critical information, evidence of demographic bias harm to patients and potential answer risks.

## Summary

Recent advancements in artificial intelligence (AI) systems have achieved remarkable milestones in various grand challenges, including Go and protein-folding. One such grand challenge has been the ability to retrieve medical knowledge, reason over it, and answer medical questions comparable to physicians. Large language models (LLMs) have played a significant role in this progress, with Med-PaLM being the first model to exceed a "passing" score in US Medical Licensing Examination-style questions on the MedQA dataset. To bridge this gap, researchers have presented Med-PaLM 2 which leverages a combination of base LLM improvements (PaLM 2), medical domain finetuning and prompting strategies that include a novel ensemble refinement approach. The new model scored up to 86.5% on the MedQA dataset setting a new state-of-the-art. Performance approaching or exceeding state-of-the-art was also observed across other clinical topics datasets like MedMCQA, PubMedQA and MMLU. Detailed human evaluations were performed on long-form questions relevant to clinical applications showing that physicians preferred Med-PaLM 2 answers compared to those produced by physicians on eight of nine axes pertaining to clinical utility (p < 0.001). To evaluate the potential impact of test set contamination on evaluation results researchers searched for overlapping text segments between multiple choice questions in MultiMedQA and the corpus used to train the base LLM underlying Med-PaLM 2. While further studies are necessary to validate the efficacy of these models in real world settings these results highlight rapid progress towards physician level performance in medical question answering. The potential risks associated with using such models include inaccurate or irrelevant information, omission of critical information, evidence of demographic bias harm to patients and potential answer risks.