# Data Filtering Networks

**Authors:** Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, Vaishaal Shankar  
**arXiv:** [2309.17425](https://arxiv.org/abs/2309.17425)  
**Date:** 2023-09-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2309.17425v1/

## Key points

- Large training sets are crucial for advancements in language modeling and multimodal learning.
- Data filtering networks are developed to effectively filter large uncurated datasets.
- Quality of a network for data filtering differs from its performance on downstream tasks.
- New data filtering networks have been constructed to generate state-of-the-art image-text datasets.
- DFN-5B dataset enables training cutting-edge models within compute budgets and achieves high zero-shot transfer accuracy on ImageNet.
- Research findings can be used as a blueprint for creating high-quality datasets from publicly available data, democratizing access to such datasets.
- Release of DFN-2B will facilitate research on large image-text models and encourage collaboration in advancing machine learning capabilities.

## Summary

In recent years, large training sets have played a crucial role in the advancement of , particularly in the fields of language modeling and multimodal learning. The process of  for pre-training often involves collecting a vast amount of data from the Web and then filtering this dataset to create an actual training set using various heuristics. This study focuses on the development of a  to filter large uncurated datasets effectively. One key insight from this research is that the quality of a network for filtering data is different from its performance on downstream tasks. For example, a model that performs well on ImageNet may not necessarily produce high-quality training sets compared to a model with lower ImageNet accuracy trained on a smaller but higher-quality dataset. Building upon these insights, new <kd>data filtering networks</ kd > have been constructed to generate state-of-the-art image-text datasets. The best performing dataset, DFN-5B, has enabled the training of cutting-edge models within compute budgets. For instance, a Vision Transformer-H (ViT-H) trained on DFN-5B achieved an impressive 83.0% zero-shot transfer accuracy on ImageNet, outperforming models trained on other datasets such as LAION-2B, DataComp-1B, or OpenAI's WIT. Furthermore, the findings from this research can serve as a blueprint for creating high-quality datasets from scratch using only publicly available data. This approach contributes towards democratizing access to large high-quality datasets and promotes further research in < kd >dataset design advancements</ kd >. Additionally, the release of DFN-2B to the community will facilitate research on large image-text models and encourage collaboration in advancing machine learning capabilities. In summary, this study sheds light on the importance of effective <kd>data filtering networks</ kd > in generating top-tier datasets for machine learning applications and highlights the significance of quality over quantity in dataset construction. By leveraging these insights and advancements in <kd>dataset design</ kd >, researchers can continue to push boundaries in artificial intelligence and enhance model performance across various tasks and domains.