# Sparse-to-Sparse Training of Diffusion Models

**Authors:** Inês Cardoso Oliveira, Decebal Constantin Mocanu, Luis A. Leiva  
**arXiv:** [2504.21380](https://arxiv.org/abs/2504.21380)  
**Date:** 2025-04-30  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2504.21380v1/

## Key points

- Authors introduce sparse-to-sparse training for diffusion models (DMs) to improve efficiency in training and inference.
- Sparse DMs can match or outperform dense counterparts while reducing trainable parameters and FLOPs.
- Three different methods (Static-DM, RigL-DM, MagRan-DM) are used to train sparse DMs from scratch on six datasets.
- Experimental results show the effectiveness of sparse-to-sparse training in DMs across various sparsity rates and dataset sizes.
- Safe and effective values for implementing sparse-to-sparse training in DMs are identified.
- Experiments with larger datasets like ImageNet-1k demonstrate the scalability of the proposed approach.

## Summary

In their paper "Sparse-to-Sparse Training of Diffusion Models," authors Inês Cardoso Oliveira, Decebal Constantin Mocanu, and Luis A. Leiva introduce a novel approach to training diffusion models (DMs) with the aim of improving both training and inference efficiency. DMs are known for their stable training dynamics and ability to generate high-quality samples in various image synthesis tasks, natural language processing, and temporal data modeling. However, they typically require significant computational resources for both training and inference stages. The authors propose the paradigm of sparse-to-sparse training for DMs, focusing on unconditional generation. They train sparse DMs from scratch using three different methods (Static-DM, RigL-DM, and MagRan-DM) on six datasets to investigate the impact of sparsity on model performance. The experimental results demonstrate that sparse DMs can match or even outperform their dense counterparts while significantly reducing the number of trainable parameters and floating-point operations (FLOPs). The study also identifies safe and effective values for implementing sparse-to-sparse training in DMs. Furthermore, the authors provide experimental details such as setting sparsity rates at {0.1, 0.25, 0.5, 0.75, 0.9}, exploration frequencies (∆Te), weight prune and regrowth ratios (p), and dataset sizes used for training models like CelebA-HQ and LSUN-Bedrooms. Additionally, experiments with larger datasets like ImageNet-1k are conducted to assess the scalability of the proposed approach. Overall,this research contributes to advancing the field of generative models by introducing a more efficient training method for diffusion models that can achieve comparable or superior performance while reducing computational overhead.