On the Expressive Power of Neural Networks
Authors: Jan Holstermann
Abstract
In 1989 George Cybenko proved in a landmark paper that wide shallow neural networks can approximate arbitrary continuous functions on a compact set. This universal approximation theorem sparked a lot of follow-up research. Shen, Yang and Zhang determined optimal approximation rates for ReLU-networks in $L^p$-norms with $p \in [1,\infty)$. Kidger and Lyons proved a universal approximation theorem for deep narrow ReLU-networks. Telgarsky gave an example of a deep narrow ReLU-network that cannot be approximated by a wide shallow ReLU-network unless it has exponentially many neurons. However, there are even more questions that still remain unresolved. Are there any wide shallow ReLU-networks that cannot be approximated well by deep narrow ReLU-networks? Is the universal approximation theorem still true for other norms like the Sobolev norm $W^{1,1}$? Do these results hold for activation functions other than ReLU? We will answer all of those questions and more with a framework of two expressive powers. The first one is well-known and counts the maximal number of linear regions of a function calculated by a ReLU-network. We will improve the best known bounds for this expressive power. The second one is entirely new.
The papers around it, by date and closeness
Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper
Papers around this one
Ranked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
On the Approximation Properties of Neural NetworksJonathan W. Siegel, Jinchao Xu · 2019 · Neural Networks (2020) · AI summary available
Abstract
We prove two new results concerning the approximation properties of neural networks. Our first result gives conditions under which the outputs of the neurons in a two layer neural network are linearly independent functions. Our second result concerns the rate of approximation of a two layer neural network as the number of neurons increases. We improve upon existing results in the literature by significantly relaxing the required assumptions on the activation function and by providing a better rate of approximation. We also provide a simplified proof that the class of functions represented by a two-layer neural network is dense in any compact set if the activation function is not a polynomial…
-
New Integral representations for the Fox-Wright functions and its applicationsKhaled Mehrez · 2017 · Classical Analysis and ODEs · AI summary available
Abstract
Our aim in this paper is to derive several new integral representations of the Fox-Wright functions. In particular, we give new Laplace and Stieltjes transform for this special functions under a special restriction on parameters. From the positivity conditions for the weight in these representations, we found sufficient conditions to be imposed on the parameters of the Fox-Wright functions that it be completely monotonic. As applications, we derive a class of function related to the Fox H-functions is positive definite and an investigation of a class of the Fox H-function is non-negative. Moreover, we extended the Luke's inequalities and we establish a new Tur\'an type inequalities for the F…
-
Identities for the Ln-transform, the L2n-transform and the P2n transform and their applicationsNese Dernek, Fatih Aylikci · 2014 · Classical Analysis and ODEs · AI summary available
Abstract
In the present paper, the authors introduce several new integral transforms including the Ln-transform, the L2n-transform and P2n-transform generalizations of the classical Laplace transform and the classical Stieltjes transform as respectively. It is shown that the second iterate of the L2n-transform is essentially the P2n-transform. Using this relationship, a few new Parseval-Goldstein type identities are obtained. The theorem and the lemmas that are proven in this article are new useful relations for evaluating infinite integrals of special functions. Some related illustrative examples are also given.
-
Perturbations of Cauchy differencesEszter Gselmann, Tomasz Małolepszy, Janusz Matkowski · 2026 · Classical Analysis and ODEs · AI summary available
Abstract
This paper investigates functional equations arising from perturbations of Cauchy differences. We study equations of the form \[ f(x+y)-f(x)-f(y)=B(x,y) \quad \text{or} \quad f(xy)-f(x)f(y) = B(x,y) \] where $B$ is a biadditive mapping, and also more general cases where the inhomogeneity depends on unknown functions \begin{align*} f(x+y)-f(x)-f(y)&= αx y \\[2.5mm] f(x+y)-f(x)-f(y)&= α(x y)\\[2.5mm] f(x+y)-f(x)-f(y)&= α(x)α(y). \end{align*} Our results extend previous work on the bilinearity of the Cauchy exponential difference by Alzer and Matkowski. We characterize solutions under various structural and regularity assumptions, including additive and exponential Cauchy differences, and show …
-
New elementary formulas for any derivative of any rational functionShahar Nevo, Irina Raichik · 2020 · Classical Analysis and ODEs · AI summary available
Abstract
In this paper we introduce elementary and completely explicit formulas for the derivative of any order of any function of the type 1/p, where p is a polynomial with known zeros.
-
Phase retrieval for solutions of the Schr{ö}dinger equationsPhilippe Jaming · 2025 · SampTA 2025, Jul 2025, Vienne, Austria · AI summary available
Abstract
In this note we present several questions about the phase retrieval problem for the Schr{\"o}dinger equation. Some partial answers are given as well as some of the heuristics behind these questions.
-
Multiple basic hypergeometric transformation formulas arising from the balanced duality transformationYasushi Kajihara · 2013 · Classical Analysis and ODEs · Not summarized yet
Abstract
Some multiple hypergeometric transformation formulas arising from the balanced du- ality transformation formula are discussed through the symmetry. Derivations of some transformation formulas with different dimensions are given by taking certain limits of the balanced duality transformation. By combining some of them, some transformation formulas for $A_n$ basic hypergeometric series is given. They include some generalizations of Watson, Sears and ${}_8 W_7$ transformations.
-
Orthogonal Fourier Analysis on DomainsMihail N. Kolountzakis · 2024 · Classical Analysis and ODEs · AI summary available
Abstract
In this paper we go over the history of the Fuglede or Spectral Set Conjecture as it has developed over the last 30 years or so. We do not aim to be exhaustive and we do not cover important areas of development such as the results on the problem in classes of finite groups or the version of the problem that focuses on spectral measures instead of sets. The selection of the material has been strongly influenced by personal taste, history and capabilities. We are trying to be more descriptive than detailed and we point out several open questions.
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
For now only among the papers summarized on this site: search again to add the prior work, the follow-ups and Semantic Scholar’s recommendations.