Results of the summarizing process for the arXiv paper: 1912.02178v1

The study of generalization in deep networks has been a topic of great interest in recent years, with many theoretically and empirically motivated complexity measures proposed to understand it. To address the gap in knowledge on this topic, Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan and Samy Bengio conducted the first large-scale study of generalization in deep networks. The authors investigated over 40 complexity measures taken from both theoretical bounds and empirical studies by training more than 10,000 convolutional networks while systematically varying commonly used hyperparameters. The goal was to uncover potentially causal relationships between each measure and generalization through carefully controlled experiments. Surprisingly, some measures failed to provide accurate results while others showed promise for further research. This study provides valuable insights into the generalization of deep networks and highlights the need for larger scale studies that can generalize across different models and settings. The findings also emphasize the importance of carefully selecting appropriate complexity measures when evaluating generalization performance in deep learning models. Overall, this work contributes significantly to our understanding of how complex neural networks generalize their learned representations beyond their training data.

• Generalization in deep networks has been a topic of great interest in recent years
• Many complexity measures have been proposed to understand generalization
• Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan and Samy Bengio conducted the first large-scale study of generalization in deep networks
• Over 40 complexity measures were investigated by training more than 10,000 convolutional networks while systematically varying commonly used hyperparameters
• The goal was to uncover potentially causal relationships between each measure and generalization through carefully controlled experiments
• Some measures failed to provide accurate results while others showed promise for further research
• This study provides valuable insights into the generalization of deep networks and highlights the need for larger scale studies that can generalize across different models and settings
• The findings emphasize the importance of carefully selecting appropriate complexity measures when evaluating generalization performance in deep learning models
• Overall, this work contributes significantly to our understanding of how complex neural networks generalize their learned representations beyond their training data.

Exploring Generalization in Deep Networks: A Large-Scale Study

The study of generalization in deep networks has been a topic of great interest in recent years, with many theoretically and empirically motivated complexity measures proposed to understand it. However, there is still a gap in knowledge on this topic. To address this issue, Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan and Samy Bengio conducted the first large-scale study of generalization in deep networks. The authors investigated over 40 complexity measures taken from both theoretical bounds and empirical studies by training more than 10,000 convolutional networks while systematically varying commonly used hyperparameters. This paper provides valuable insights into the generalization of deep networks and highlights the need for larger scale studies that can generalize across different models and settings.

Overview of the Research

The goal of this research was to uncover potentially causal relationships between each measure and generalization through carefully controlled experiments. To do so, they trained multiple convolutional neural network (CNN) architectures using various hyperparameter values such as learning rate or batch size on CIFAR-10 datasets with different levels of noise added to them. They then measured the performance of these models using several complexity measures including Rademacher Complexity (RC), Neural Tangent Kernel (NTK) widths at initialization (W0), NTK widths at convergence (W∞), Lipschitz constant (Lc), spectral norm (SN) etc., which were obtained either from theoretical bounds or empirical studies.

Results & Findings

Surprisingly, some complexity measures failed to provide accurate results while others showed promise for further research. For example, RC provided an inaccurate estimation when compared against other methods such as W0 or W∞; however NTK widths at initialization had strong correlation with test accuracy across all noise levels tested indicating its potential usefulness for predicting model performance beyond training data sets. Additionally, Lc showed good correlation with test accuracy only under certain conditions such as low learning rates or high batch sizes suggesting that it may be useful for specific cases but not generally applicable across all scenarios studied here. Finally SN also showed good correlation with test accuracy but only when applied to shallow CNN architectures rather than deeper ones indicating its limited utility when dealing with complex models like those used here.

Conclusion

Overall, this work contributes significantly to our understanding of how complex neural networks generalize their learned representations beyond their training data by providing valuable insights into the relationship between various complexity measures and model performance on unseen data sets . The findings also emphasize the importance of carefully selecting appropriate complexity measures when evaluating generalization performance in deep learning models since some may fail to accurately predict model behavior while others may be better suited depending on specific circumstances such as architecture depth or hyperparameter values used during training .