# When Does Re-initialization Work?

**Authors:** Sheheryar Zaidi, Tudor Berariu, Hyunjik Kim, Jörg Bornschein, Claudia Clopath, Yee Whye Teh, Razvan Pascanu  
**arXiv:** [2206.10011](https://arxiv.org/abs/2206.10011)  
**Date:** 2022-06-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2206.10011v1/

## Key points

- Re-initializing a neural network during training has been observed to improve generalization in recent works.
- This technique is not widely adopted in deep learning practice nor is it often used in state-of-the-art training protocols.
- The authors conducted an extensive empirical comparison of standard training with a selection of re-initialization methods by training over 15,000 models on a variety of image classification benchmarks.
- Re-initialization methods are consistently beneficial for generalization in the absence of any other regularization.
- When deployed alongside other carefully tuned regularization techniques such as data augmentation, weight decay, and learning rate schedules that resemble state-of-the-art training protocols, re-initialization methods offer little to no added benefit for generalization.
- Optimal generalization performance becomes less sensitive to the choice of hyperparameters under these conditions.
- Under label noise where other regularization techniques are not able to offer much help on learning tasks, re-initialization significantly improves upon standard training.
- Fixed-budget BANs do not improve performance compared to standard training in most cases but can serve as an important baseline for more sophisticated re-initialization methods.
- A deeper understanding of why re-initializations work or do not work well is missing and future work could explore online learning implications and extend the scope of study beyond specific datasets/architectures.

## Summary

In recent works, re-initializing a neural network during training has been observed to improve generalization. However, this technique is not widely adopted in deep learning practice nor is it often used in state-of-the-art training protocols. This raises the question of when re-initialization works and whether it should be used together with regularization techniques such as data augmentation, weight decay, and learning rate schedules. To answer this question, the authors conducted an extensive empirical comparison of standard training with a selection of re-initialization methods by training over 15,000 models on a variety of image classification benchmarks. The authors found that re-initialization methods are consistently beneficial for generalization in the absence of any other regularization. However, when deployed alongside other carefully tuned regularization techniques such as data augmentation, weight decay, and learning rate schedules that resemble state-of-the-art training protocols, re-initialization methods offer little to no added benefit for generalization. Nonetheless, optimal generalization performance becomes less sensitive to the choice of hyperparameters under these conditions. Surprisingly, under label noise where other regularization techniques are not able to offer much help on learning tasks, re-initialization significantly improves upon standard training. The authors also investigated the role of self-distillation and found that fixed-budget BANs do not improve performance compared to standard training in most cases but can serve as an important baseline for more sophisticated re-initialization methods. One limitation of this study is that although clear empirical trends were observed in when re-initialization works or does not work well under certain conditions and settings using specific datasets and architectures (CIFAR-10/100 and Tiny ImageNet), a deeper understanding of why it does or does not work is missing. Future work could explore online learning implications where Shrink & Perturb was first proposed and shown to be helpful or extend the scope of study beyond these specific datasets/architectures to investigate other tasks and data modalities which may provide further insight into why re-initializations work or do not work well in certain contexts.