# On the Expressive Power of Neural Networks

**Authors:** Jan Holstermann  
**arXiv:** [2306.0145](https://arxiv.org/abs/2306.0145)  
**Date:** 2023-05-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.0145v1/

## Key points

- The paper by Jan Holstermann focuses on the expressive power of neural networks.
- The universal approximation theorem states that wide shallow neural networks can approximate any continuous function on a compact set.
- Research has been done to determine optimal approximation rates for ReLU-networks in $L^p$-norms with $p \in [1,\infty)$ and to prove a universal approximation theorem for deep narrow ReLU-networks.
- Holstermann introduces a framework of two expressive powers to answer open questions regarding the expressive power of neural networks.
- The first expressive power counts the maximal number of linear regions of a function calculated by a ReLU-network, and Holstermann improves upon the best known bounds for this.
- The second expressive power measures how many times an input has to be transformed before it can be classified correctly by a neural network, and is entirely new.
- By improving existing bounds on expressive powers and introducing new ones, this work contributes to our understanding of how neural networks work and what they can do.

## Summary

The paper "On the Expressive Power of Neural Networks" by Jan Holstermann delves into the expressive power of neural networks, specifically focusing on the universal approximation theorem and its implications. The universal approximation theorem, first proven by George Cybenko in 1989, states that wide shallow neural networks can approximate any continuous function on a compact set. This sparked a lot of research, including determining optimal approximation rates for ReLU-networks in $L^p$-norms with $p \in [1,\infty)$ by Shen, Yang, and Zhang and proving a universal approximation theorem for deep narrow ReLU-networks by Kidger and Lyons. To answer open questions regarding the expressive power of neural networks such as whether there are any wide shallow ReLU-networks that cannot be well approximated by deep narrow ReLU-networks or if the universal approximation theorem still holds true for other norms like the Sobolev norm $W^{1,1}$, Holstermann introduces a framework of two expressive powers. The first one is well-known and counts the maximal number of linear regions of a function calculated by a ReLU-network; Holstermann improves upon the best known bounds for this expressive power. The second one is entirely new and measures how many times an input has to be transformed before it can be classified correctly by a neural network. By improving existing bounds on expressive powers and introducing new ones, this work contributes to our understanding of how neural networks work and what they can do. Holstermann's paper provides insights into both the limitations and capabilities of neural networks in approximating functions.