# DataComp-LM: In search of the next generation of training sets for language models

**Authors:** Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muenninghoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton  
**arXiv:** [2406.11794](https://arxiv.org/abs/2406.11794)  
**Date:** 2024-06-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.11794v1/

## Key points

- DataComp for Language Models (DCLM) is a testbed designed to enhance language models through controlled dataset experiments.
- DCLM provides researchers with a standardized corpus of 240 trillion tokens from Common Crawl, pretraining recipes based on the OpenLM framework, and 53 downstream evaluations.
- Focus on data curation strategies like deduplication, filtering, and data mixing at varying model scales from 412 million to 7 billion parameters.
- Model-based filtering in assembling high-quality training sets is highlighted as significant.
- DCLM-Baseline enables training a 7 billion parameter language model from scratch with impressive results, including a 64% 5-shot accuracy on MMLU with 2.6 trillion training tokens.
- Outperforms previous state-of-the-art open-data language models like MAP-Neo while using less computational resources.
- Comparable performance to Mistral-7B-v0.3 and Llama 3-8B on MMLU tasks and across an average of 53 natural language understanding tasks while requiring less compute than Llama 3-8B.
- Importance of dataset design in training language models effectively is emphasized.
- Insights into data-centric benchmarks in the context of language models are provided along with details about the components of DCLM such as the raw text corpus (DCLM-POOL).
- DCLM serves as a valuable platform for advancing research in language modeling by facilitating controlled experiments and enabling systematic exploration of different data curation techniques.

## Summary

In this study, we present DataComp for Language Models (DCLM), a comprehensive testbed designed to enhance language models through controlled dataset experiments. The primary aim of DCLM is to improve language model performance by providing researchers with a standardized corpus of 240 trillion tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a diverse suite of 53 downstream evaluations. One key aspect of DCLM is its focus on data curation strategies such as deduplication, filtering, and data mixing at varying model scales ranging from 412 million to 7 billion parameters. Through extensive experimentation, we highlight the significance of model-based filtering in assembling high-quality training sets. The resulting dataset, DCLM-Baseline, enables training a 7 billion parameter language model from scratch to achieve impressive results. This includes a 64% 5-shot accuracy on MMLU with 2.6 trillion training tokens. Comparative analysis reveals that DCLM-Baseline outperforms previous state-of-the-art open-data language models like MAP-Neo by a substantial margin while utilizing significantly less computational resources. Furthermore, our baseline model demonstrates comparable performance to Mistral-7B-v0.3 and Llama 3-8B on MMLU tasks and achieves similar results across an average of 53 natural language understanding tasks while requiring substantially less compute than Llama 3-8B. Additionally, we delve into related work in the field of data curation for language models and highlight the importance of dataset design in training language models effectively. We also discuss open-source datasets curated by the community over the years and compare their performance with DCLM-Baseline. Furthermore, we provide insights into data-centric benchmarks in the context of language models and outline the components of DCLM, including the raw text corpus (DCLM-POOL) used in our benchmarking process. We detail the workflow involved in selecting competition scales, curating datasets through filtering and mixing strategies, training models with fixed hyperparameters, and evaluating model performance across various tasks. In conclusion, our study showcases how DCLM serves as a valuable platform for advancing research in language modeling by facilitating controlled experiments and enabling researchers to explore different data curation techniques systematically. While our findings represent significant progress in dataset design for training language models, there are limitations due to compute constraints that warrant further exploration and refinement in future studies.