# Understanding deep learning requires rethinking generalization

**Authors:** Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals  
**arXiv:** [1611.03530](https://arxiv.org/abs/1611.03530)  
**Date:** 2016-11-10  
**Summary page:** https://www.summarizepaper.com/en/paper/1611.03530v1/

## Key points

- Successful deep artificial neural networks exhibit a small difference between training and test performance despite their massive size.
- Conventional explanations attributing this to model family properties or regularization techniques fall short.
- State-of-the-art convolutional networks can fit random labeling of training data without explicit regularization, even with random noise instead of true images.
- Simple depth two neural networks achieve perfect finite sample expressivity when the number of parameters exceeds the number of data points in practical applications.
- The study challenges existing notions about model complexity and regularization techniques in deep learning, highlighting the need to rethink generalization.

## Summary

In their paper titled "Understanding deep learning requires rethinking generalization," authors Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals delve into the phenomenon of successful deep artificial neural networks exhibiting a small difference between training and test performance despite their massive size. The conventional belief attributes this small generalization error to either properties of the model family or the regularization techniques employed during training. However, through extensive systematic experiments, the authors demonstrate that these traditional explanations fall short in elucidating why large neural networks generalize effectively in practice. Their experiments specifically reveal that state-of-the-art convolutional networks for image classification, trained using stochastic gradient methods, are capable of easily fitting a random labeling of the training data. Surprisingly, this phenomenon persists even in the absence of explicit regularization and when true images are replaced with completely unstructured random noise. To support their experimental findings, the authors construct a theoretical framework showing that simple depth two neural networks achieve perfect finite sample expressivity when the number of parameters exceeds the number of data points – a common scenario in practical applications. By comparing their experimental results with traditional models, Zhang et al. highlight the need to rethink generalization in deep learning. Their study challenges existing notions about model complexity and regularization techniques by showcasing how large neural networks can effectively generalize without relying on conventional explanations. This research sheds new light on the inner workings of deep learning systems and calls for a reevaluation of current understanding in order to further advance this rapidly evolving field.