Dataset Distillation using Neural Feature Regression

AI-generated keywords: Dataset Distillation Neural Feature Regression Meta-Learning Synthetic Dataset Continual Learning

AI-generated Key Points

The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.

  • Dataset distillation aims to create a compact synthetic dataset retaining essential information from the original dataset.
  • The process is framed as a bi-level meta-learning problem with outer loop optimizing the meta-dataset and inner loop training a model on distilled data.
  • Authors propose neural Feature Regression with Pooling (FRePo) to address challenges like computing meta-gradients, reducing memory requirements, and faster training.
  • FRePo operates similar to truncated backpropagation through time using a pool of models to mitigate overfitting in dataset distillation tasks.
  • FRePo outperforms existing methods significantly on benchmark datasets such as CIFAR100, Tiny ImageNet, and ImageNet-1K.
  • High-quality distilled data can enhance downstream applications like continual learning and membership inference defense.
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Yongchao Zhou, Ehsan Nezhadarya, Jimmy Ba

Abstract: Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data. Meta-gradient computation is one of the key challenges in this formulation, as differentiating through the inner loop learning procedure introduces significant computation and memory costs. In this paper, we address these challenges using neural Feature Regression with Pooling (FRePo), achieving the state-of-the-art performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. The proposed algorithm is analogous to truncated backpropagation through time with a pool of models to alleviate various types of overfitting in dataset distillation. FRePo significantly outperforms the previous methods on CIFAR100, Tiny ImageNet, and ImageNet-1K. Furthermore, we show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense.

Submitted to arXiv on 01 Jun. 2022

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2206.00719v1

This paper's license doesn't allow us to build upon its content and the summarizing process is here made with the paper's metadata rather than the article.

In their paper titled "Dataset Distillation using Neural Feature Regression," authors Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba delve into the realm of dataset distillation. This process aims to create a compact synthetic dataset that retains essential information from the original dataset. The task is framed as a bi-level meta-learning problem, with the outer loop optimizing the meta-dataset and the inner loop involving training a model on the distilled data. A key challenge in this approach lies in computing meta-gradients. Differentiating through the inner loop learning procedure incurs significant computational and memory costs. To tackle these challenges, the authors propose a novel solution called neural Feature Regression with Pooling (FRePo). This method not only achieves state-of-the-art performance but also boasts an order of magnitude reduction in memory requirements and two orders of magnitude faster training compared to previous techniques. FRePo operates akin to truncated backpropagation through time by utilizing a pool of models to mitigate various forms of overfitting commonly encountered in dataset distillation tasks. The effectiveness of FRePo is demonstrated through extensive experiments on benchmark datasets such as CIFAR100, Tiny ImageNet, and ImageNet-1K. It outperforms existing methods significantly. Moreover, the authors showcase how high-quality distilled data can greatly enhance downstream applications like continual learning and membership inference defense. In conclusion, "Dataset Distillation using Neural Feature Regression" presents a cutting-edge approach to dataset distillation that not only addresses key challenges in meta-gradient computation but also showcases superior performance across various datasets and downstream tasks.
Created on 05 Jul. 2025

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.