# MEDITRON-70B: Scaling Medical Pretraining for Large Language Models

**Authors:** Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, Antoine Bosselut  
**arXiv:** [2311.16079](https://arxiv.org/abs/2311.16079)  
**Date:** 2023-11-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.16079v1/

## Key points

- MEDITRON is a suite of open-source large language models (LLMs) designed for the medical domain.
- The LLMs have 7B and 70B parameters and are built upon Llama-2 using Nvidia's Megatron-LM distributed trainer.
- A comprehensively curated medical corpus, including PubMed articles, abstracts, and internationally recognized medical guidelines, was used to train these models.
- The training data covers a wide range of contexts in terms of geographic scope, resource settings, and target audiences.
- PubMed papers and abstracts were chosen to create the pre-training mix for MEDITRON due to their vast amount of biomedical textual data.
- Metadata information, non-English content, acknowledgments tables figures URLs were removed during preprocessing.
- In-text references were formatted using special tokens for accurate citations.
- Section headers were indicated with specific tokens for main sections and subsections.
- Experience replay was employed in the training process to prevent forgetting previously learned information.

## Summary

In this work, the researchers introduce MEDITRON, a suite of open-source large language models (LLMs) specifically designed for the medical domain. These LLMs have 7B and 70B parameters and are built upon Llama-2 using Nvidia's Megatron-LM distributed trainer. To train these models, a comprehensively curated medical corpus was used, which includes selected PubMed articles, abstracts, and internationally-recognized medical guidelines. The GUIDELINES corpus included in the training data covers a wide range of contexts in terms of geographic scope (from global to institutional), resource settings (high-, low-, and volatile-resource settings), and target audiences (clinicians or patients). It also incorporates various peer review processes ranging from UN bodies to publicly crowdsourced knowledge bases. To create the pre-training mix for MEDITRON, PubMed papers and abstracts were chosen due to their vast amount of biomedical textual data. The researchers collected 4.47M full-text papers from the PubMed Central Open Access Subset as well as 444,521 open-access full-text PubMed papers that were not found in the archive. Additionally, 16,209,047 PubMed and PubMed Central abstracts were collected with a knowledge cutoff date of August 2023. To preprocess the content from these sources metadata information references acknowledgments tables figures URLs and non-English content were removed while inline citations section headers figures tables mathematical formulas were identified using special tokens. In-text references were formatted with a similar methodology to the Galactica model to promote accurate citations by replacing them with [BIB_REF] tokens containing truncated titles and authors' last names while figure and table references were wrapped with [FIG_REF] tokens containing figure numbers and truncated captions. Section headers were indicated with '#' and '##' tokens for main sections and subsections respectively. Experience replay was employed in the training process to overcome catastrophic forgetting which helps prevent the model from forgetting previously learned information by including data from old tasks when training on new tasks.