REVIEW 3 cited by
The Power of Depth for Feedforward Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We show that there is a simple (approximately radial) function on $\reals^d$, expressible by a small 3-layer feedforward neural networks, which cannot be approximated by any 2-layer network, to more than a certain constant accuracy, unless its width is exponential in the dimension. The result holds for virtually all known activation functions, including rectified linear units, sigmoids and thresholds, and formally demonstrates that depth -- even if increased by 1 -- can be exponentially more valuable than width for standard feedforward neural networks. Moreover, compared to related results in the context of Boolean functions, our result requires fewer assumptions, and the proof techniques and construction are very different.
Forward citations
Cited by 3 Pith papers
-
A Measure-Theoretic Analysis of Reasoning: Structural Generalization and Approximation Limits
Applies optimal transport to bound OOD generalization error in Transformers via Lipschitz continuity and TC^0 circuit depth lower bounds for Dyck-k backtracking, supported by evaluations on 54 configurations.
-
Towards a Comparative Framework for Compositional AI Models
A categorical framework for compositional generalisation is applied to DisCoCirc models, showing quantum circuits outperform neural networks on systematicity while neural models overfit more.
-
Iterate to Accelerate: A Unified Framework for Iterative Reasoning and Feedback Convergence
The paper claims a unified O(1/t^2) acceleration theorem and a feedback-versus-feedforward separation, but the proof steps are invalid and the main results are false as stated.
Discussion (0). Continue with ORCID to comment.