# Width is Less Important than Depth in ReLU Neural Networks

**Authors:** Gal Vardi, Gilad Yehudai, Ohad Shamir  
**arXiv:** [2202.03841](https://arxiv.org/abs/2202.03841)  
**Date:** 2022-02-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2202.03841v1/

## Key points

- Width is less important than depth in ReLU neural networks
- Any target network with inputs in $\mathbb{R}^d$ can be approximated by a network with width $O(d)$
- Depth plays a more crucial role than width in capturing complex functions
- Networks with bounded weights and width at most $d+2$ can be constructed, but introduce an additional polynomial factor in the number of parameters compared to the target network
- Wide and shallow networks can be represented exactly using deep and narrow networks without increasing the number of parameters in certain cases
- The study provides insights into the relative importance of depth and width in neural networks' expressive power
- Practical implications for designing efficient models with reduced parameter counts while maintaining high performance levels

## Summary

In their paper titled "Width is Less Important than Depth in ReLU Neural Networks," authors Gal Vardi, Gilad Yehudai, and Ohad Shamir address an open question posed by Lu et al. (2017) regarding the expressive power of neural networks. They demonstrate that any target network with inputs in $\mathbb{R}^d$ can be approximated by a network with width $O(d)$, regardless of the target network's architecture. This result highlights the significance of depth over width in determining the expressive power of neural networks. The number of parameters in this width-$O(d)$ network is only larger by a linear factor compared to the target network, emphasizing the importance of depth in capturing complex functions. Previous depth separation theorems have shown that interchanging the roles of width and depth does not yield a similar approximation result. Therefore, it can be concluded that depth plays a more crucial role than width in capturing complex functions. The authors further extend their findings to constructing networks with bounded weights and networks with width at most $d+2$. The latter is particularly noteworthy as it approaches the minimal possible width based on previous lower bounds. However, both constructions introduce an additional polynomial factor in the number of parameters compared to the target network. Additionally, Vardi et al. present an exact representation scheme for wide and shallow networks using deep and narrow networks. Remarkably, this representation does not increase the number of parameters over the target network in certain cases. Overall, this study provides valuable insights into the relative importance of depth and width in neural networks' expressive power. The results contribute to our understanding of network architectures and offer practical implications for designing efficient models with reduced parameter counts while maintaining high performance levels.