# A Survey on Data Selection for Language Models

**Authors:** Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, William Yang Wang  
**arXiv:** [2402.16827](https://arxiv.org/abs/2402.16827)  
**Date:** 2024-02-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.16827v1/

## Key points

- Large language model success attributed to utilizing vast text datasets for unsupervised pre-training
- Filtering out irrelevant data improves model performance, reduces carbon footprint, and lowers financial costs
- Data selection methods crucial for determining training dataset content and effective sampling
- Limited resources hinder extensive research in data selection methods
- Comprehensive review of existing literature on data selection methods presented with a taxonomy of approaches used
- Review aims to accelerate progress in data selection and provide an entry point for researchers
- Study conducted by Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff,
- Bairu Hou,Liangming Pan,
- Haewon Jeong,
- Colin Raffel,
- Shiyu Chang,
- Tatsunori Hashimoto,
- and William Yang Wang

## Summary

The recent success of large language models can be attributed to the utilization of vast text datasets for unsupervised pre-training. However, training a model on all available data may not always be optimal due to varying data quality. Filtering out irrelevant data not only improves model performance but also reduces carbon footprint and financial costs associated with training. Data selection methods play a crucial role in determining which data points should be included in the training dataset and how to sample from them effectively. Despite the growing interest in data selection methods, limited resources hinder extensive research in this area. As a result, knowledge of effective practices is concentrated within a few organizations that do not always share their findings openly. To bridge this knowledge gap, a comprehensive review of existing literature on data selection methods has been presented, along with a taxonomy of approaches currently used. This review aims to accelerate progress in data selection by providing an entry point for both new and established researchers. By highlighting gaps in existing literature and proposing future research avenues, this work seeks to advance the field of data selection for language models. The study was conducted by Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff,
Bairu Hou,Liangming Pan,
Haewon Jeong,
Colin Raffel,
Shiyu Chang,
Tatsunori Hashimoto,
and William Yang Wang. For more detailed information on this topic and related research areas such as cross-lingual transfer learning and multi-task learning across multiple languages please refer to the full paper available at http://arxiv.org/pdf/2402.16827v1.