# Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth

**Authors:** Thao Nguyen, Maithra Raghu, Simon Kornblith  
**arXiv:** [2010.15327](https://arxiv.org/abs/2010.15327)  
**Date:** 2020-10-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2010.15327v2/

## Key points

- Scaling models by varying their architecture depth and width is important for achieving high performance in deep neural networks
- Limited understanding of how depth and width affect learned representations within these models
- Thao Nguyen, Maithra Raghu, and Simon Kornblith investigate this question in their paper "Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth"
- Larger capacity models exhibit a block structure in hidden representations when model capacity is large relative to training set size
- Block structure indicates underlying layers preserve and propagate dominant principal component of representations
- Features learned outside block structure are often similar across architectures with varying widths and depths, but block structure itself is unique to each model
- Wide and deep models exhibit distinctive error patterns across classes even when overall accuracy is similar
- Study provides insights into how neural network representations vary with width and depth for designing effective deep learning models.

## Summary

In the field of deep neural networks, scaling models by varying their architecture depth and width has been a key factor in achieving high performance for various tasks. However, there is limited understanding of how depth and width affect the learned representations within these models. In their paper, "Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth," authors Thao Nguyen, Maithra Raghu, and Simon Kornblith investigate this fundamental question. They find that larger capacity (wider or deeper) models exhibit a characteristic block structure in their hidden representations when model capacity is large relative to the size of the training set. This block structure indicates that underlying layers preserve and propagate the dominant principal component of their representations. Notably, features learned outside this block structure are often similar across architectures with varying widths and depths; however, the block structure itself is unique to each model. The authors also analyze output predictions from different model architectures and discover that even when overall accuracy is similar, wide and deep models exhibit distinctive error patterns across classes. Overall, this study provides important insights into how neural network representations vary with width and depth which can be used to design effective deep learning models.