# Differential Privacy Meets Neural Network Pruning

**Authors:** Kamil Adamczewski, Mijung Park  
**arXiv:** [2303.04612](https://arxiv.org/abs/2303.04612)  
**Date:** 2023-03-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.04612v1/

## Key points

- Authors address the challenge of scalability in applying differential privacy to training deep neural network models
- They focus on the DP-SGD algorithm, which struggles with training moderately-sized neural network models when high privacy protection is desired
- Authors propose exploring dimensionality reduction techniques inspired by neural network pruning to improve scalability of DP-SGD
- Two modes of parameter updates: parameter freezing and parameter selection
- Parameter freezing involves pre-pruning the network and only updating remaining parameters using DP-SGD, reducing parameter space and improving efficiency
- Parameter selection involves selecting which parameters to update at each step of training and updating only those selected using DP-SGD
- Leveraging public data for freezing or selecting parameters aims to avoid additional privacy loss during these steps, success depends on how closely private and public data are related
- Experiments demonstrate that decreasing parameter space improves differentially private training performance
- Random selection and magnitude-based selection are two popular forms of pruning that do not rely on gradients and do not incur additional privacy loss; random selection performs equally well as magnitude-based selection in DP-SGD training
- Paper presents a novel approach for addressing scalability challenges in applying differential privacy to deep neural network models by incorporating dimensionality reduction techniques inspired by neural network pruning
- Offers insights into improving efficiency without compromising privacy protection by leveraging public data for freezing or selecting parameters while avoiding any additional privacy loss incurred during these steps.

## Summary

In the paper titled "Differential Privacy Meets Neural Network Pruning," authors Kamil Adamczewski and Mijung Park address the challenge of scalability in applying differential privacy to training deep neural network models. They focus on the widely-used training algorithm called differentially private stochastic gradient descent (DP-SGD), which struggles with training moderately-sized neural network models when a high level of privacy protection is desired. To improve the scalability of DP-SGD, the authors propose exploring dimensionality reduction techniques inspired by neural network pruning. They investigate how neural network pruning and differential privacy can work together through two modes of parameter updates: parameter freezing and parameter selection. In the first mode, parameter freezing, the authors pre-prune the network and only update the remaining parameters using DP-SGD. This approach helps reduce the parameter space and improve efficiency. In the second mode, parameter selection, they select which parameters to update at each step of training and update only those selected using DP-SGD. By leveraging public data for freezing or selecting parameters, they aim to avoid incurring additional privacy loss during these steps. The success of this paradigm relies on how closely private and public data are related. The authors conduct experiments that demonstrate how decreasing the parameter space improves differentially private training performance. Additionally, they study two popular forms of pruning that do not rely on gradients and do not incur additional privacy loss: random selection and magnitude-based selection. Their results show that random selection performs equally well as magnitude-based selection when it comes to DP-SGD training. Overall, this paper presents a novel approach for addressing scalability challenges in applying differential privacy to deep neural network models by incorporating dimensionality reduction techniques inspired by neural network pruning. It offers insights into improving efficiency without compromising privacy protection by leveraging public data for freezing or selecting parameters while avoiding any additional privacy loss incurred during these steps.