# Towards Sustainable Learning: Coresets for Data-efficient Deep Learning

**Authors:** Yu Yang, Hao Kang, Baharan Mirzasoleiman  
**arXiv:** [2306.01244](https://arxiv.org/abs/2306.01244)  
**Date:** 2023-06-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.01244v1/

## Key points

- Authors propose a scalable framework called CREST to improve efficiency and sustainability of learning deep models
- CREST is the first framework with theoretical guarantees for training non-convex models, specifically deep networks
- CREST models non-convex loss as quadratic functions and extracts a coreset for each sub-region, allowing for more efficient training
- CREST addresses faster convergence in stochastic gradient methods by extracting multiple mini-batch coresets from random subsets of data
- CREST enhances scalability and efficiency by excluding examples that have already been learned from the coreset selection pipeline
- Extensive experiments on various datasets confirm that CREST significantly speeds up training without sacrificing performance (1.7x to 2.5x speed improvements)
- Deep models benefit most from learning subsets with increasing difficulty levels, highlighting the effectiveness of CREST in guiding model training towards challenging examples
- Overall, CREST improves efficiency and sustainability through its theoretical guarantees, coreset extraction approach, and iterative mini-batch coreset generation

## Summary

In their paper titled "Towards Sustainable Learning: Coresets for Data-efficient Deep Learning," authors Yu Yang, Hao Kang, and Baharan Mirzasoleiman propose a scalable framework called CREST to improve the efficiency and sustainability of learning deep models. CREST is the first framework with rigorous theoretical guarantees that identifies valuable examples for training non-convex models, specifically deep networks. To ensure convergence to a stationary point of a non-convex function, CREST models the non-convex loss as a series of quadratic functions and extracts a coreset for each quadratic sub-region. This approach allows for more efficient training of deep networks by focusing on the most relevant examples. Additionally, CREST addresses the issue of faster convergence in stochastic gradient methods like mini-batch SGD. It achieves this by iteratively extracting multiple mini-batch coresets from larger random subsets of training data. This process ensures nearly-unbiased gradients with small variances, leading to improved convergence rates. Furthermore, CREST enhances scalability and efficiency by identifying and excluding examples that have already been learned from the coreset selection pipeline. This prevents redundant or unnecessary training on previously seen examples. The authors conducted extensive experiments on various deep networks trained on vision and NLP datasets such as CIFAR-10, CIFAR-100, TinyImageNet, and SNLI. The results confirm that CREST significantly speeds up training deep networks on large datasets without sacrificing performance. The proposed framework achieves speed improvements ranging from 1.7x to 2.5x while maintaining minimal loss in performance. By analyzing the learning difficulty of subsets selected by CREST, the authors demonstrate that deep models benefit most from learning subsets with increasing difficulty levels; this finding highlights the effectiveness of CREST in guiding model training towards challenging examples that contribute to better overall performance. Overall, this paper introduces an innovative framework called CREST which improves the efficiency and sustainability of learning deep models through its theoretical guarantees, coreset extraction approach and iterative mini-batch coreset generation; these features contribute to faster convergence rates and scalability while validating its effectiveness in accelerating training without compromising performance through experiments conducted on various datasets.