REVIEW 3 cited by
On the inductive bias of infinite-depth ResNets and the bottleneck rank
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We compute the minimum-norm weights of a deep linear ResNet, and find that the inductive bias of this architecture lies between minimizing nuclear norm and rank. This implies that, with appropriate hyperparameters, deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.
Forward citations
Cited by 3 Pith papers
-
Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks
Develops general framework for representation costs of parametric models, proving that depth-L ReLU networks induce p-normable quasi-Banach spaces with p=2/L.
-
Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers
Neural collapse is globally optimal in deep regularized ResNets and transformers, with the approximation improving as depth grows.
-
Differentially Private Natural Gradient Descent
DP-NGD enables second-order optimization under differential privacy by decoupling curvature estimation onto public data, performing isotropic DP operations in a whitened space, and dynamically clamping curvature eigen...
Discussion (0). Continue with ORCID to comment.