# DsDm: Model-Aware Dataset Selection with Datamodels

**Authors:** Logan Engstrom, Axel Feldmann, Aleksander Madry  
**arXiv:** [2401.12926](https://arxiv.org/abs/2401.12926)  
**Date:** 2024-01-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.12926v1/

## Key points

- Traditional dataset selection based on human notions of data quality may not always lead to improved model behavior
- Researchers have proposed a novel framework that reframes dataset selection as an optimization problem
- This method directly solves for the subset of data that maximizes model performance for target tasks using a learning algorithm
- The new approach provides a 2x compute multiplier over baseline methods by selecting datasets based on their ability to enhance model performance rather than perceived quality
- Models trained using this optimized dataset selection method demonstrate superior performance on benchmarks related to target tasks and show promise for improving generalization capabilities across various applications

## Summary

In the realm of training large-scale machine learning models, the selection of appropriate data is crucial for achieving optimal performance. Traditionally, this has been done by filtering for examples that align with human notions of data quality. However, this approach may not always lead to improved model behavior and can even have a negative impact on performance. To address this challenge, researchers have proposed a novel framework that reframes dataset selection as an optimization problem. By directly solving for the subset of data that maximizes model performance for target tasks using a learning algorithm, this method avoids subjective notions of data quality and instead focuses on how the learning process utilizes training data to predict outcomes on specific tasks. The resulting approach has shown significant improvements in language model (LM) performance across both pre-specified tasks and previously unseen tasks. This new approach provides a 2x compute multiplier over baseline methods by selecting datasets based on their ability to enhance model performance rather than perceived quality. This enhancement is particularly evident when evaluating models on diverse held-out benchmarks meant to simulate real-world scenarios. In comparison to existing dataset selection baselines, including methods that prioritize textual similarity with high-quality sources or rely on random selection, the new approach consistently outperforms these approaches. Models trained using this optimized dataset selection method demonstrate superior performance on benchmarks related to target tasks and show promise for improving generalization capabilities across various applications. Overall, this research highlights the importance of rethinking traditional approaches to dataset selection in machine learning and emphasizes the value of optimizing data choices based on their direct impact on model performance. By leveraging advanced optimization techniques and focusing on task-specific objectives, researchers can unlock new possibilities for enhancing the effectiveness and efficiency of large-scale machine learning models.