REVIEW 5 cited by
Nonparametric regression using deep neural networks with ReLU activation function
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to $\log n$-factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential network parameters exceeding the sample size. The analysis gives some insights into why multilayer feedforward neural networks perform well in practice. Interestingly, for ReLU activation function the depth (number of layers) of the neural network architectures plays an important role and our theory suggests that for nonparametric regression, scaling the network depth with the sample size is natural. It is also shown that under the composition assumption wavelet estimators can only achieve suboptimal rates.
Forward citations
Cited by 5 Pith papers
-
Dimension independent bounds for general shallow networks
An abstract theorem gives dimension-independent, smoothness-improving approximation rates for shallow kernel networks, covering ReLU networks, RBFs, manifold learning, and quasirandom integration.
-
Dimension-independent rates for structured neural density estimation
Neural density estimators that factor over a known Markov random field achieve dimension-independent L1 rates n^{-1/(4+r)} (and optimally n^{-1/(2+r)}), where r is the maximum clique size.
-
Convergence Rates of Variational Inference in Sparse Deep Learning
Variational inference for sparse deep ReLU networks achieves near-minimax rates for Hölder regression functions, matching exact Bayesian inference.
-
Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recovery
When the regression function is (s+α)-Hölder on a d-dimensional manifold, the empirical risk minimizer over deep ReLU networks achieves mean squared error n^{-2(s+α)/(2(s+α)+d)} log^3 n.
-
Unifying Learning Dynamics and Generalization in Transformers Scaling Law
Claims a two-stage transformer scaling law (exponential then C^{-1/6}) with matching bounds, but the lower bounds are missing, the exponent is inconsistent (-1/7 vs -1/6), and the law is an artifact of hand-set M = Θ(...
Discussion (0). Continue with ORCID to comment.