# Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

**Authors:** Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, Eric Horvitz  
**arXiv:** [2311.16452](https://arxiv.org/abs/2311.16452)  
**Date:** 2023-11-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.16452v1/

## Key points

- Generalist foundation models like GPT-4 have impressive capabilities across various domains and tasks
- Previous studies on medical competency benchmarks relied on domain-specific training
- This study aims to explore GPT-4's capabilities on medical challenge benchmarks without specialized training
- Innovative prompting techniques can unlock deeper specialist capabilities in GPT-4
- GPT-4 surpasses previous leading results for medical benchmarks when guided by the prompting methods used in this study
- The prompting strategies explored are general-purpose and do not require domain expertise or curated content from experts
- Experimental design carefully controls for overfitting during the prompt engineering process
- Medprompt, a combination of several prompting strategies, achieves state-of-the-art performance on all nine benchmark datasets in the MultiMedQA suite
- Medprompt outperforms leading specialist models like Med-PaLM 2 while making significantly fewer calls to the model
- By using Medprompt, there is a 27% reduction in error rate on the MedQA dataset compared to previous best methods with specialist models, surpassing a score of 90% for the first time
- Medprompt has broad applicability beyond medical problems.

## Summary

Generalist foundation models like GPT-4 have shown impressive capabilities across various domains and tasks. However, there is a common belief that they cannot match the specialist abilities of fine-tuned models. Previous studies on medical competency benchmarks have relied on domain-specific training, such as BioGPT and Med-PaLM. In this study, we aim to explore GPT-4's capabilities on medical challenge benchmarks without any specialized training. Instead of using simple prompts to showcase the model's out-of-the-box abilities, we conduct a systematic investigation into prompt engineering. Our findings reveal that innovative prompting techniques can unlock deeper specialist capabilities in GPT-4. We demonstrate that GPT-4 surpasses previous leading results for medical benchmarks by a significant margin when guided by our prompting methods. The prompting strategies we explore are general-purpose and do not require domain expertise or curated content from experts. To ensure reliable results, our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, which combines several prompting strategies, and with this approach, GPT-4 achieves state-of-the-art performance on all nine benchmark datasets in the MultiMedQA suite. Remarkably, Medprompt outperforms leading specialist models like Med-PaLM 2 while making significantly fewer calls to the model. By steering GPT-4 with Medprompt, we achieve a 27% reduction in error rate on the MedQA dataset compared to previous best methods with specialist models, surpassing a score of 90% for the first time. Furthermore, we demonstrate that Medprompt has broad applicability beyond medical problems.