REVIEW 4 cited by
Benefits of depth in neural networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
For any positive integer $k$, there exist neural networks with $\Theta(k^3)$ layers, $\Theta(1)$ nodes per layer, and $\Theta(1)$ distinct parameters which can not be approximated by networks with $\mathcal{O}(k)$ layers unless they are exponentially large --- they must possess $\Omega(2^k)$ nodes. This result is proved here for a class of nodes termed "semi-algebraic gates" which includes the common choices of ReLU, maximum, indicator, and piecewise polynomial functions, therefore establishing benefits of depth against not just standard networks with ReLU gates, but also convolutional networks with ReLU and maximization gates, sum-product networks, and boosted decision trees (in this last case with a stronger separation: $\Omega(2^{k^3})$ total tree nodes are required).
Forward citations
Cited by 4 Pith papers
-
On Universality of Deep Equivariant Networks
Deep equivariant networks are universal over the entry-wise separable regime once depth stabilizes separation or a convolutional readout is added, unifying prior architecture-specific results.
-
Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations
A task ma+nb mod p is representable by a z^k holomorphic network iff m+n=k; non-representable tasks cannot be memorised at any width.
-
Discrete Functional Geometry of ReLU Networks via ReLU Transition Graphs
The paper claims ReLU Transition Graphs of ReLU networks are expanders whose spectral gap, region entropy, and edge KL divergence bound generalization and capacity; the proofs are sketches, and the empirical checks are weak.
-
Iterate to Accelerate: A Unified Framework for Iterative Reasoning and Feedback Convergence
The paper claims a unified O(1/t^2) acceleration theorem and a feedback-versus-feedforward separation, but the proof steps are invalid and the main results are false as stated.
Discussion (0). Continue with ORCID to comment.