On the infinite-depth limit of finite-width neural networks

The authors of this paper investigate the infinite-depth limit of finite-width residual neural networks with random Gaussian weights. By fixing the width and allowing the depth to approach infinity, they demonstrate that the pre-activations converge in distribution to a zero-drift diffusion process through proper scaling. This is different from the infinite-width limit where pre-activations tend to weakly converge to a Gaussian random variable. The infinite-depth limit also yields diverse distributions depending on the chosen activation function. Two scenarios are identified where these distributions have distinct closed-form expressions. Interestingly, there is a change in regime for post-activation norms when transitioning from a width of 3 to 4. The study also compares two sequential limits: first approaching infinite depth followed by infinite width versus the more commonly explored path of infinite width preceding infinite depth. Understanding these limiting laws is crucial for designing resilient neural networks and gaining insights into overparameterized networks. While previous research has focused on the infinite-width limit, this work contributes by investigating the infinite-depth limit of finite-width neural networks. A key finding is that unlike in the case of infinite width where a Gaussian distribution is typically obtained under certain conditions regarding activation functions, behavior in the infinite-depth limit is notably influenced by activation function selection. Leveraging Itˆo's lemma allows for obtaining known distributions by carefully adjusting activation functions. Furthermore, an important characteristic is revealed regarding general width limits: there is a zero probability of process collapse for specific conditions in infinite-depth neural networks with various activation functions (including ReLU). This highlights their robustness and stability compared to other configurations.

Residual neural networks (ResNets) have gained widespread popularity in recent years due to their ability to achieve state-of-the-art performance on various deep learning tasks. These networks are known for their depth, which allows them to learn complex representations and handle a wide range of input data. However, as the depth of these networks increases, so does the risk of overfitting and instability. To address this issue, researchers have explored different limits of ResNets, including the infinite-width limit where the number of neurons tends towards infinity.

In this paper titled "The Infinite-Depth Limit of Finite-Width Residual Neural Networks with Random Gaussian Weights," authors investigate another important limit - the infinite-depth limit - where the width is fixed and only the depth approaches infinity. This study sheds light on how pre-activations behave in this scenario and how it differs from the well-studied infinite-width limit.

To understand this research better, let's first define some key terms. Pre-activation refers to the output before applying an activation function in a neural network layer. In contrast, post-activation refers to the output after applying an activation function. The authors use random Gaussian weights in their experiments, which means that each weight is drawn from a normal distribution with mean 0 and variance 1.

The main finding of this paper is that when approaching infinite depth while keeping width fixed at a finite value, pre-activations converge in distribution to a zero-drift diffusion process through proper scaling. This result is significantly different from what happens in the infinite-width limit where pre-activations tend to weakly converge to a Gaussian random variable.

Moreover, depending on which activation function is used in ResNet layers, there can be diverse distributions obtained at infinity-depth limits. The authors identify two scenarios where these distributions have distinct closed-form expressions: one for ReLU activations and another for sigmoid activations.

One interesting observation made by the authors is that there is a change in regime for post-activation norms when transitioning from a width of 3 to 4. This means that as the depth increases, there is a sudden shift in how post-activations behave, which can have significant implications for network design and performance.

The study also compares two sequential limits: first approaching infinite depth followed by infinite width versus the more commonly explored path of infinite width preceding infinite depth. The results show that these two paths lead to different limiting distributions, highlighting the importance of understanding both limits.

While previous research has focused on the infinite-width limit, this work contributes by investigating the infinite-depth limit of finite-width neural networks. A key finding is that unlike in the case of infinite width where a Gaussian distribution is typically obtained under certain conditions regarding activation functions, behavior in the infinite-depth limit is notably influenced by activation function selection.

To obtain known distributions at infinity-depth limits, the authors leverage Itˆo's lemma and carefully adjust activation functions. This approach allows them to obtain closed-form expressions for pre-activations and post-activations under different scenarios.

Furthermore, an important characteristic revealed by this study regarding general width limits is that there exists a zero probability of process collapse for specific conditions in ResNets with various activation functions (including ReLU). This highlights their robustness and stability compared to other configurations.

In conclusion, this paper provides valuable insights into how pre-activations behave at infinity-depth limits in ResNets with random Gaussian weights. By exploring this less-studied limit, researchers can gain a better understanding of overparameterized networks and design more resilient neural networks. The findings also highlight the importance of considering both sequential limits - approaching infinity depth followed by infinity width or vice versa - when studying ResNets' behavior at extreme depths.