# Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss

**Authors:** Sangyeon Park, Isaac Han, Seungwon Oh, Kyung-Joong Kim  
**arXiv:** [2502.01342](https://arxiv.org/abs/2502.01342)  
**Date:** 2025-02-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2502.01342v1/

## Key points

- The paper addresses the critical challenge of plasticity loss in neural network training, which hinders a model's ability to adapt to new tasks or shifts in data distribution.
- The proposed method, AID (Activation by Interval-wise Dropout), applies different dropout probabilities on each preactivation interval to generate subnetworks, effectively regularizing the network and preventing plasticity loss.
- Evaluation on standard image classification datasets like CIFAR10, CIFAR100, and TinyImageNet shows that AID maintains plasticity across benchmarks and enhances reinforcement learning performance.
- Comparison with Dropout in a warm-start learning experiment reveals that while Dropout improves generalizability, AID effectively mitigates plasticity loss by retaining a higher degree of plasticity in warm-start models.
- Overall findings suggest that AID is an effective method for preventing plasticity loss in neural networks and improving their adaptability to new tasks or changes in data distribution.

## Summary

, , , , 

The paper "Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss" addresses the critical challenge of plasticity loss in neural network training. This issue hinders a model's ability to adapt to new tasks or shifts in data distribution. To overcome this problem, the proposed method, AID (Activation by Interval-wise Dropout), is inspired by Dropout but introduces a novel approach by applying different dropout probabilities on each preactivation interval to generate subnetworks. Theoretical analysis shows that AID effectively regularizes the network, resulting in behavior similar to deep linear networks that do not suffer from plasticity loss. To evaluate the effectiveness of AID, various benchmarks were conducted on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. The results demonstrate that AID maintains plasticity across these benchmarks and enhances reinforcement learning performance in the Arcade Learning Environment benchmark. In a warm-start learning experiment inspired by previous research, models trained with vanilla settings, Dropout, and AID were compared after pre-training a RESNET-18 model on 10% of the training data for 1,000 epochs before continuing training on the full dataset. While Dropout appeared to improve generalizability in both warm-start and cold-start models, it was argued that this improvement stemmed from enhanced model generalization rather than mitigating plasticity loss. In contrast, AID showed a smaller performance improvement compared to the vanilla model but effectively mitigated plasticity loss as warm-start models trained with AID retained a higher degree of plasticity compared to those trained with Dropout. Overall, the findings suggest that AID is an effective method for preventing plasticity loss in neural networks and improving their adaptability to new tasks or changes in data distribution.