REVIEW 2 major objections 4 minor 12 references
Estimation of a function of low local dimensionality by deep neural networks
T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper proves that least-squares sparse deep neural network estimators can converge at a rate set by the local dimensionality of the regression function, not by the ambient input dimension, when the function is low-dimensional on each…
desk verdict Solid new rate result for locally low-dimensional DNN regression, but the 'smooth transitions' in the motivation are not actually covered by the theorem's boundary-mass condition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing construction is a sparse neural network class: sums of $M^*$ small fully connected sigmoidal networks, with weights bounded by $n^{c_2}$ and the number $M^*$ chosen by sample splitting. The proof works through an oracle inequality (Theorem 2 in the Supplement): up to $(\log n)^3$, the DNN error is bounded by the best error any linear combination of basis functions from $B^*_{n,M,K_1}$ could achieve, where each basis function is a product of a MARS-style truncated linear power (the hinge functions used in multivariate adaptive regression splines) and a tensor-product B-spline. This basis can exactly represent polytope indicators as differences of truncated linear functions and approximates smooth local pieces by B-splines, so the theorem reduces to a sparse-approximation problem in local dimension $d^*$. Approximation lemmas show each such basis product is representable by a DNN to accuracy $1/n^3$, and a covering-number bound controls the complexity cost $I/n$.
What would settle it
Take $d=2$ and let $X$ be concentrated in a strip of width $n^{-1/2}$ around the diagonal of $[0,1]^2$, with $m(x)=x_1$ on one side of the diagonal and $m(x)=x_2$ on the other. Then the boundary mass is of order $n^{-1/2}$, not $c_5/n$, so the error term from misclassifying boundary points in the proof becomes $n^{-1/2}$; a simulation measuring the $L_2$ error versus $n$ would show the stated $n^{-2p/(2p+d^*)}$ rate fails for this distribution.
Extended reading notes
Core claim
The central discovery is that low local dimensionality—not just global low intrinsic dimensionality—is enough for deep networks to avoid the curse of dimensionality. Under Definition 3, a function has local dimensionality $d^*$ if it is sandwiched between two sums of the form $\sum_k f_k(x_{J_k})$ times indicators of polytopes, where each $f_k$ is $(p,C)$-smooth and depends on at most $d^*$ coordinates, and the $P_X$-mass of the boundary strips between the outer and inner polytopes is at most $c_5/n$. Theorem 1 states that the sparse neural network least-squares estimator reaches the $d^*$-dimensional rate $n^{-2p/(2p+d^*)}$ up to logarithmic and constant factors, with an error bound independent of $d$. This extends earlier dimension-reduction results for hierarchical composition models and for data lying on low-dimensional manifolds to a setting where the input distribution itself need not be concentrated on any low-dimensional manifold.
Load-bearing premise
The assumption that the probability mass of the transition strips between local regions is at most $c_5/n$, and that the strips shrink at least as fast as $c_6/n^{c_7}$, is the load-bearing premise; if those boundaries carry more probability, the theorem's stated rate does not follow.
Editorial extensions
If this is right
- The convergence rate for regression functions with local dimensionality $d^*$ is $n^{-2p/(2p+d^*)}$ up to log factors, so when $d^* \ll d$ the estimator circumvents the curse of dimensionality.
- The class covered by Theorem 1 contains every $(p,C)$-smooth function that depends on at most $d^*$ input components, so the rate is minimax-optimal up to logarithmic factors in that subclass by the classical lower bound.
- The assumption does not require the covariate distribution to live on a low-dimensional set; the function itself may be globally smooth or piecewise smooth with polytope regions and thin boundary strips.
- The same sparse-network estimates also satisfy an oracle version of the MARS error bound, meaning the DNN inherits any future improvement in approximating locally low-dimensional functions by truncated-power/B-spline products.
- Numerical experiments and the bike-sharing data experiment show error reductions against MARS and other nonparametric estimators, consistent with the theoretical rate advantage.
Reading between the lines
- Editorial inference: the boundary-mass condition is likely the real price of smoothness; on distributions where transitions carry mass $n^{-a}$ with $a<1$, the theorem's proof suggests the error would degrade, and one can test this by simulating a boundary strip of width $n^{-1/2}$.
- Editorial inference: the oracle inequality suggests that data-driven selection of the number of network blocks $M^*$ should adapt to the unknown local structure, so a formal adaptive-rate theorem for unknown $d^*$ is a natural next step.
- Editorial inference: because the construction approximates B-splines and truncated hinges, the same dimension-reduction argument should transfer to ReLU activations and to other spline bases, connecting the result to practical architectures.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies least-squares regression with sparse feedforward neural networks using a smooth squashing activation function. It introduces a notion of "local dimensionality" (Definition 3) based on squeezing the regression function between two sums of the form sum_k f_k(x_J_k) 1_{P_k}(x), where the sets P_k are polytopes and the squeeze is performed on shrunken and expanded versions of the polytopes, with a control on the P_X-measure of the intervening boundary bands. Theorem 1 states that, under this assumption with P_X-border c5/n and borders δ_i,k bounded below by c6/n^{c7}, the L2 error of the sparse neural network estimate is bounded by c8 (log n)^3 2^{K1} K2 n^{-2p/(2p+d*)}, so the rate is independent of the input dimension d. The proof in the supplement proceeds through an oracle inequality for sparse networks relative to a MARS/B-spline basis (Theorem 2), approximation of polytope indicators by truncated linear functions, approximation of smooth f_k by tensor-product B-splines, and control of the boundary bands by the probability condition. The paper also contains simulations on locally low-dimensional functions in d=10 and a bike-sharing data experiment, comparing the estimate with fully connected networks, a hierarchical-interaction network, nearest neighbor, RBF, and MARS.
Significance. If the main theorem is correct, the paper makes a useful contribution to the statistical theory of deep networks: it identifies a structural condition on the regression function, rather than on the covariate distribution alone, under which the curse of dimensionality is circumvented and the d*-dimensional rate is achieved. The supplement is a genuine strength: it contains complete proofs of the oracle inequality, covering number bounds, and approximation lemmas for B-splines and MARS basis functions, and the final rate is benchmarked against Stone's minimax rate. The main caveats are that the formal P_X-border condition substantially narrows the "smooth transition" class advertised in the introduction, and that one step in the proof of Theorem 2 needs a more careful justification. These issues are addressable without changing the central mathematical claim, but they affect the paper's interpretation and the completeness of the proof as written.
major comments (2)
- [Supplement C, proof of Theorem 2] The application of Lemma 8 with "J = d" is not justified as written. Lemma 8 approximates linear combinations of products of B-splines over a fixed set of J coordinates, but the basis functions B_i in B^*_{n,M,K1} may have different subsets J1 of coordinates. A single B-spline is not identically 1 on [-A,A], and rewriting a B_i with |J1| < d as a product over d coordinates via the partition-of-unity property would multiply the number of terms by (K+M)^{d-|J1|}, which would destroy the needed bound when d > d*. The intended componentwise argument -- approximate each B_i separately with J=|J1_i| and then embed the resulting networks into the larger class F(L,r,α_n) -- should be stated explicitly. With that fix, the error bound I max|w_i|/n^3 remains valid. This point is load-bearing because Theorem 2 underlies the oracle inequality used to prove Theorem 1.
- [Section 1.4, Definition 3, and Supplement D] The paper presents Definition 3 as allowing "smooth transitions" between local regimes, and Figure 1 advertises a globally smooth function with smooth transitions, but Theorem 1's P_X-border condition excludes fixed-width smooth transitions under a continuous design. Outside the union of boundary bands ∪_k ((P_k)^{δ_k} \(P_k)_{δ_k}), the squeeze in Definition 3 forces m(x)=∑_k f_k(x_{J_k}) 1_{(P_k)_{δ_k}}(x). Therefore the entire transition zone of a smooth f must be contained in the boundary bands. If a transition strip has width w>0 and P_X has density bounded below on that strip, then P_X(∪_k boundary bands) ≥ c w > 0, contradicting the assumption P_X(∪_k boundary bands) ≤ c5/n for all n > c5/(c w). Thus the theorem covers only functions whose transition zones have P_X-mass O(1/n), not the fixed-width smooth transitions promised in the introduction. The motivation, Figure 1, and the real-data discussion should be qualified accordingly. The proof in Supplement D does show that the boundary contribution is O(P_X(boundary)), so the weaker condition P_X(boundary) ≤ C n^{-2p/(2p+d*)} would suffice for the stated rate, but this does not remove the gap between the formal assumption and the advertised examples.
minor comments (4)
- [Supplement D, Step 1] The displayed formula for γ_{k,1} appears to assume that the first coordinate of a_k is nonzero; for a general a_k with zero first coordinate one should choose any coordinate with a nonzero component, and the resulting γ's still lie in the required polynomial bounds. The text should be corrected to avoid this implicit assumption.
- [Equation (20) and proof of Theorem 2] The notation J is used both for the number of B-spline factors in Lemma 8 and for the index subset J1 in the definition of B^*_{n,M,K1}; this makes the phrase "Lemma 8 with J = d" very difficult to parse. Please separate these notations and clarify which parameter is being fixed.
- [Theorem 1, architecture formula] The formula for L is printed as "L = 3K1 + d · (M + 2)−1", which can be misread. It should be parenthesized as L = 3K1 + d(M+2) − 1, matching Lemma 8.
- [Section 1.4 / Table 3] The data set name "Capital Bike Sharing" is misspelled as "Captial Bike Sharing" in two places. The values in Table 3 would also benefit from a sentence explaining the sample splitting into the 500 training/testing points and the remaining evaluation points.
Circularity Check
No meaningful circularity: the rate in Theorem 1 follows from approximation and oracle-inequality arguments; the PX-border condition is a substantive scope limitation, not an input that reproduces the claimed rate.
full rationale
The derivation is self-contained and anchored to external benchmarks. Theorem 1's bound is obtained by an oracle inequality (Theorem 2, Supplement) bounding the sparse-network risk by the best approximation error in a B-spline/truncated-power basis, by approximating polytope indicators with such basis functions (Supplement D, step 1), by approximating (p,C)-smooth local components by d*-dimensional tensor-product B-splines (step 2), and by controlling the integral over the polytope border bands using the assumed PX-border c5/n (step 3). The rate n^{-2p/(2p+d*)} emerges from choosing K = ceil(n^{1/(2p+d*)}) basis splines and using smoothness error O(K^{-p}); it is not equal to a fitted parameter or to the border-mass bound. The PX-border condition in Definition 3 is part of the assumed function class, not a consequence of the conclusion: for a fixed function with a positive-width transition strip and PX density bounded below near the strip, the squeeze condition forces the strip into the border bands, so PX(border) is positive and cannot be <= c5/n for large n. Thus Theorem 1 narrows the class relative to the 'smooth transitions' motivation in Section 1.4. That is a substantive scope/correctness caveat, but not circularity. The self-citations (Bauer and Kohler 2019, Kohler and Krzyzak 2017, Kohler and Langer 2020) are contextual; the load-bearing Lemma 9 is a standard oracle inequality whose complete proof is cited rather than reproduced, and the novel approximation lemmas 1-8 are proved from scratch. No fitted parameter is relabeled as a prediction, and no uniqueness theorem is imported.
Assumptions & free parameters
free parameters (2)
- Network architecture constants L and r =
L = 3K1 + d(M+2)-1, r = 2^{M-1}*16 + sum_{k=2}^M 2^{M-k+1} + d + 5
- Weight bound alpha_n =
c1 * n^{c2} for generic constants c1, c2
assumptions (6)
- domain assumption The regression function m has local dimensionality d* as defined in Definition 3, i.e., it is squeezed between two sums of f_k(x_{J_k}) times indicators of polytopes.
- domain assumption Each local function f_k is (p,C)-smooth and bounded.
- domain assumption The PX-border of the polytope boundaries is at most c5/n and borders delta_{i,k} >= c6/n^{c7}.
- domain assumption supp(X) is contained in [-A,A]^d and Y has an exponential tail: E(exp(c4 |Y|^2)) < infinity.
- domain assumption The activation function sigma is 2-admissible (Definition 4), e.g., the logistic squasher.
- standard math Standard results: B-spline approximation theorems (e.g., Theorems 15.1 and 15.2 in Györfi et al. 2002) and Stone's minimax lower bound for (p,C)-smooth functions.
Cite this review
Pith. "Pith review of Estimation of a function of low local dimensionality by deep neural networks." pith.science (2026). https://pith.science/paper/6YWRU7KG
@misc{pith2026190811140,
author = {Pith},
title = {Pith review of: Estimation of a function of low local dimensionality by deep neural networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/6YWRU7KG}},
note = {Machine review of arXiv:1908.11140}
}
abstract
Deep neural networks (DNNs) achieve impressive results for complicated tasks like object detection on images and speech recognition. Motivated by this practical success, there is now a strong interest in showing good theoretical properties of DNNs. To describe for which tasks DNNs perform well and when they fail, it is a key challenge to understand their performance. The aim of this paper is to contribute to the current statistical theory of DNNs. We apply DNNs on high dimensional data and we show that the least squares regression estimates using DNNs are able to achieve dimensionality reduction in case that the regression function has locally low dimensionality. Consequently, the rate of convergence of the estimate does not depend on its input dimension $d$, but on its local dimension $d^*$ and the DNNs are able to circumvent the curse of dimensionality in case that $d^*$ is much smaller than $d$. In our simulation study we provide numerical experiments to support our theoretical result and we compare our estimate with other conventional nonparametric regression estimates. The performance of our estimates is also validated in experiments with real data.
Figures
Reference graph
Works this paper leans on
-
[1]
Bagirov, A. M., Clausen, C., and Kohler, M. (2009). Estimati on of a regression function by maxima of minima of linear functions. IEEE Trans. Information Theory , 55(2):833–845. Bauer, B., Devroye, L., Kohler, M., Krzyżak, A., and Walk, H. (2017). Nonparamet- ric estimation of a function from noiseless observations at random points. Journal of Multivariat...
work page 2009
-
[2]
· J + 3K1)2 52 such that |f (x) − fbasis(x)| ≤ 1 n3 for all x ∈ [−a, a]d. Proof. Set fk(x) = Bjk,M,t k(x(k)) for k = 1, . . . , J and fk(x) = max { d∑ j=1 αk,j · (x(j) − γk,j ), 0 } for k = J + 1, . . . , J + K1. By Lemma 5 there exists a neural network fBjk ,M,t k ∈ F ( M + 1, 2M −1 · 16 + M∑ k=2 2M −k+1, c25 · R2 B ) , satisfying |fBjk ,M,t k − fk(x)| ≤...
work page 2002
-
[3]
Corollary 1 concerns the result of Theorem 2, if we choose J1 = ∅ in (20)
By combining our proofs with the techniques introduced in Sc hmidt-Hieber (2020) it is possible to show that a similar result also holds for neural networks using the ReLU-function σReLU (x) = max {0, x} as activation function. Corollary 1 concerns the result of Theorem 2, if we choose J1 = ∅ in (20). This results in a basis B∗ n,K 1, which consists of al...
work page 2020
-
[4]
Let f : Rd → R be a (p, C)–smooth function
Choose A ≥ 1 such that supp(X) ⊆ [−A, A]d. Let f : Rd → R be a (p, C)–smooth function. If the spline degree M ∈ N fulfills the condition M ≥ q and we choose a knot sequence tv,k = −A + k · A K (v ∈ { 1, . . . , d}, k ∈ Z) for some fixed K ∈ N, standard results from the theory of B-splines (cf., e.g., T heorems 15.1 and 15.2 in Györfi et al. (2002) and Theore...
work page 2002
-
[6]
Then fsq(x) = R2 σ′′(tσ,sq ) · ( σ (2x R + tσ,sq ) − 2 · σ (x R + tσ,sq ) + σ(tσ,sq ) ) ∈ F (1, 2, c15 · R2) satisfies for any x ∈ [−a, a]: |fsq(x) − x2| ≤ 5 · ‖σ′′′‖∞ · a3 3 · |σ′′(tσ,sq )| · 1 R . Proof. The result follows in a straightforward way from the proof of Theorem 2 in Scarselli and Tsoi (1998). For the sake of completeness we provide neverthele...
work page 1998
-
[7]
· 9·‖σ ′′ ‖∞·(an)2 2·|σ ′ (tσ.id )| we get from (29) that |f t id(x) − x| ≤ t∑ k=1 |f k id(x) − f k−1 id (x)| = t∑ k=1 |fid(f k−1 id (x)) − f k−1 id (x)| ≤ t · 9 · ‖σ ′′ ‖∞ · (an)2 2 · |σ ′ (tσ.id )| · 1 R . For R > M · 9·‖σ ′′ ‖∞·(an)2 2·|σ ′ (tσ.id )| we can conclude, that ⏐ ⏐ ⏐ ⏐f l id ( x − tj tj+l − tj )⏐ ⏐ ⏐ ⏐ ≤ 3an and ⏐ ⏐ ⏐ ⏐f l id ( tj+l+1 − x tj...
work page 2002
-
[9]
Assume that the distribution of (X, Y ) satisfies (24) for some constant c4 > 0 and that the regression function m is 57 bounded in absolute value. Let ˜mn be the least squares estimate ˜mn(·) = arg min f ∈Fn 1 n n∑ i=1 |Yi − f (Xi)|2 based on some function space Fn and set mn = Tβ n ˜mn. Then mn satisfies for any n > 1 E ∫ |mn(x) − m(x)|2PX(dx) ≤ c29 · log...
work page 2009
-
[10]
Assume that the squashing func- tion σ used in F (sparse) M ∗,L,r,α is Lipschitz continuous. Then log ( N (ǫ, F (sparse) M ∗,L,r,α , ‖ · ‖∞, [−a,a ]d) ) ≤ c33 · log(n) · M ∗ holds for any n > 1 and a constant c33 > 0 independent of n and M ∗. Proof. The result follows in a straightforward way from the proof of Lemma 9 in Bauer and Kohler (2019). For the s...
work page 2019
Show all 12 references
-
[11]
(2017) we can conclude, that for an y k = 1,
· max k=1,...,M ∗, i=1,...,r |¯c(L) k,i | From Lemma 5 in Bauer et al. (2017) we can conclude, that for an y k = 1, . . . , M∗ |f (L+1) k (x) − ¯f (L+1) k (x)| ≤ (L +
2017
-
[37]
Estimation of a function of low local dimensionality by deep neural networks
Schaal, S. and Vijayakumar, S. (1997). Local dimensionalit y reduction for locally weighted learning. In CIRA, pages 220–225. IEEE Computer Society. Schmidhuber, J. (2015). Deep learning in neural networks: A n overview. Neural Networks, 61:85–117. 29 Schmidt-Hieber, J. (2019)...
1997 arXiv
-
[208]
and Krzyżak, A
Kohler, M. and Krzyżak, A. (2017). Nonparametric regressio n based on hierarchical inter- action models. IEEE Trans. Information Theory , 63(3):1620–1630. Kohler, M. and Langer, S. (2020). On the rate of convergence o f fully connected deep neural network regression estimates....
2017 arXiv
-
[242]
and Gama, J
Fanaee-T, H. and Gama, J. (2013). Event labeling combining e nsemble detectors and background knowledge. Progress in Artificial Intelligence , pages 1–15. Friedman, J. H. (1991). Multivariate adaptive regression s plines. The Annals of Statistics , 19(1):1–67. Györfi, L., Kohler...
2013
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.