REVIEW 2 major objections 5 minor 38 references
A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data
T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper proves that for supervised deep learning with exponential-family responses, the test error of a ReLU network is governed by the entropic dimension of the covariate distribution rather than the ambient dimension, yielding error…
desk verdict A useful framework and a solid ambient-dimension result, but the headline entropic-dimension rate rests on an unproved self-citation and the minimax lower bound has a support-overlap error. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing objects are the Bregman divergence attached to the exponential family, the entropic dimension of the covariate measure, and an oracle inequality that splits the error into approximation and generalization terms. Maximum likelihood for an exponential family is equivalent to minimizing $d_\varphi(y\|\mu(f(x)))$, where $\varphi$ is the conjugate of the log-partition function, and the smoothness and strong-convexity assumptions let Lemma 7 compare the excess risk with $\|f-f_0\|^2_{L^2(\lambda)}$ up to constants. The entropic dimension, $\bar d_{2\beta}(\lambda)=\limsup_{\varepsilon\downarrow 0}\log N_\varepsilon(\lambda,\varepsilon^{2\beta})/\log(1/\varepsilon)$, counts how many $\varepsilon$-balls are needed to cover all but an $\varepsilon^{2\beta}$ fraction of the covariate mass; this number, rather than the support dimension, enters the network approximation bound and fixes the sample-size exponent. The generalization side is controlled by pseudodimension bounds for ReLU networks together with localization lemmas, while a companion approximation result supplies $\varepsilon$ error with $O(\varepsilon^{-d^*/\beta}\log(1/\varepsilon))$ weights whenever $d^*>\bar d_{2\beta}(\lambda)$.
What would settle it
Take the approximation theorem cited from the companion paper and test it on a concrete case: let $\lambda$ concentrate on a self-similar set with known $2\beta$-entropic dimension, choose a $\beta$-Hölder $f_0$, and compute the best $L^2(\lambda)$ approximation error achievable by ReLU networks with $W$ weights. If the error decays slower than $W^{-\beta/d^*}$ for some $d^*>\bar d_{2\beta}(\lambda)$, or if the theorem needs assumptions beyond A1–A3 (for example absolute continuity or a density bound), then the proof of Theorem 12 is incomplete.
Extended reading notes
Core claim
The central claim is that minimizing the Bregman divergence over ReLU networks learns the conditional mean at a rate determined by the entropic dimension of the covariates. Theorem 12 states that under Assumptions A1–A3, for any $d^*>\bar d_{2\beta}(\lambda)$, a network of depth $O(\log n)$ and $O(n^{d^*/(2\beta+d^*)}\log n)$ weights satisfies $\|\hat f-f_0\|^2_{L^2(\lambda)}\lesssim n^{-2\beta/(2\beta+d^*)}(\log n)^5$ with probability at least $1-3\exp(-n^{d^*/(2\beta+d^*)})$. Since $\bar d_{2\beta}(\lambda)\le \dim_M(\lambda)$, this is faster than rates based on the Minkowski dimension of the support; for regular supports it recovers manifold-based rates without uniform-sharpness assumptions. In the bounded-density case, Theorem 8 supplies an explicit polynomial factor $d^{2\lfloor\beta\rfloor(\beta+d)/(2\beta+d)}$ in the ambient dimension, and Theorem 10 shows the exponent $n^{-2\beta/(2\beta+d)}$ is minimax, so the deep learner is nearly optimal.
Load-bearing premise
The load-bearing premise is a result borrowed from the authors' own companion work: for a $\beta$-Hölder function and any $d^*$ above the entropic dimension, a ReLU network with about $\epsilon^{-d^*/\beta}\log(1/\epsilon)$ weights can approximate the function in $L^2(\lambda)$ to error $\epsilon$; if that result fails for the measures covered by Assumptions A1–A3, Theorem 12's low-dimensional rate collapses to the slower ambient-dimension rate.
Editorial extensions
If this is right
- In the bounded-density case, the explicit bound $\tilde{\mathcal{O}}(d^{2\lfloor\beta\rfloor(\beta+d)/(2\beta+d)} n^{-2\beta/(2\beta+d)})$ shows that the ambient-dimension dependence is polynomial, so high-dimensional input spaces do not automatically destroy the guarantee.
- The minimax lower bound of Theorem 10 means that no estimator, deep or otherwise, can improve the $n^{-2\beta/(2\beta+d)}$ exponent in the bounded-density case except by logarithmic factors, so the ReLU learner's rate is essentially the best possible.
- Because $\bar d_{2\beta}(\lambda)\le \dim_M(\lambda)$, any distribution whose mass concentrates in subregions inherits a strictly faster rate than Minkowski-dimension bounds, with strict improvement possible even on simple examples.
- When the covariate support is a $\tilde d$-regular set, the rate becomes $\tilde{\mathcal{O}}(n^{-2\beta/(2\beta+\tilde d)})$, recovering manifold-based nonparametric regression results without requiring uniform-sharpness assumptions on the manifold.
- Since the setup includes classification through the Bernoulli exponential family, low-dimensional structure speeds up the binary cross-entropy learning rate just as it does for Gaussian regression.
Reading between the lines
- An extension the paper leaves implicit is that the oracle-inequality-plus-localization proof mechanism is not tied to exponential families: any loss whose population risk is sandwiched between constants times $\|f-f_0\|^2_{L^2(\lambda)}$ and whose stochastic gradient terms are sub-Gaussian would plausibly inherit the same entropic-dimension rate; this conjecture is mine, not the paper's.
- Because $\bar d_{2\beta}(\lambda)$ measures concentration of mass, any preprocessing step that makes covariate mass concentrate more, such as feature extraction, embedding, or subsampling, would lower the rate exponent; the paper does not analyze such data-dependent transformations.
- A fully self-contained version of this work would need the companion approximation theorem proved in the same document; checking that theorem on covariates that are singular or have unbounded density is the direct way to test whether Theorem 12's assumptions are sufficient.
- Estimating the entropic dimension from data and plugging it into the network-size choice would turn the guarantee into an adaptive procedure, but the paper does not address estimation of $\bar d_{2\beta}(\lambda)$.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies supervised deep learning with exponential-family responses and analyzes the expected test error of the maximum-likelihood estimator, which is equivalent to minimizing a Bregman divergence. The main upper bound (Theorem 8) shows that under a bounded density on the covariates, ReLU networks with suitably chosen size achieve an L2(λ) error of order d^{2⌊β⌋(β+d)/(2β+d)} n^{-2β/(2β+d)} (log n)^5, and Theorem 10 gives a matching minimax lower bound under a lower-bounded density. The central contribution is Theorem 12, which replaces the ambient dimension d by an entropic dimension d^* > \bar d_{2β}(λ) and claims the rate n^{-2β/(2β+d^*)} (log n)^5 for intrinsically low-dimensional data. The proofs combine a Bregman oracle inequality, localization-based generalization bounds, pseudo-dimension estimates, and ReLU approximation results.
Significance. If the results are correct, the entropic-dimension rate in Theorem 12 is a genuine improvement over existing intrinsic-dimension bounds such as those of Nakada and Imaizumi (2020), and the explicit polynomial dependence on d in Theorem 8 is a useful complement to the usual rate-in-n statements. The paper also covers classification and other exponential-family losses in a unified way, and the localization machinery in Lemmas 17–20 is presented in enough detail to be checked. The main caveats are that the proof of Theorem 12 delegates its entire entropic-dimension content to an unproved self-citation, and the proof of Theorem 10 contains an incorrect disjoint-support claim; these are load-bearing issues that need to be addressed before the results can be fully trusted.
major comments (2)
- [6.2 (Theorem 10)] The minimax lower bound construction places bumps h_δ of radius δ=1/m on centers spaced 1/m apart. Adjacent bumps therefore overlap on sets of positive Lebesgue measure, so the asserted identity ||f_ω – f_ω'||²_{L2(λ)} = ||ω – ω'||₁ ∫ h_δ² dx is invalid: the expansion contains cross terms δ_ξ δ_η ∫ h_δ(·–θ_ξ)h_δ(·–θ_η) dx that can have either sign. Consequently the claimed separation constant δ^{2β} is not established, and the subsequent KL bound in (11), which also uses the linear-in-||ω–ω'||₁ expression, does not follow. The argument can be repaired by taking δ = c/m with c < 1/2 so that the bump supports are disjoint, but as written the proof of the minimax rate is incorrect.
- [6.1.4 (Theorem 12)] The rate improvement in Theorem 12 rests entirely on the invocation 'by Chakraborty and Bartlett (2024b, Theorem 18), we can find f* ∈ F ...' for a ReLU network approximation of a β-Hölder function in L2(λ) with O(ε^{-d*/β} log(1/ε)) weights. This theorem is neither stated nor proved in the present manuscript, and it is a self-citation to an unpublished preprint (arXiv:2401.15801). The approximation lemma proved in the appendix (Lemma 21) treats only the ambient-dimension case, so the claimed entropic-dimension rate cannot be verified from the text. To make the paper self-contained, the authors should state the precise approximation theorem they need (including the bounded-output condition) and provide a proof in an appendix.
minor comments (5)
- [A.1 (Lemma 7)] The proof of Lemma 7 loses the squares in the strong-convexity and Lipschitz constants: the display after (12) gives R ≤ τ2 σ1 ||f0–f̂||², but the correct bound is R ≤ τ2 σ1² ||f0–f̂||², and similarly (15) should read R ≥ τ1 σ2² ||f0–f̂||². Since the constants do not affect the rates, the lemma should be restated as R ≍ ||f̂–f0||²_{L2(λ)} or corrected to σ2²/σ1 and σ1²/σ2.
- [6.1.4] The network size expressions contain typos: 'Lǫ ≼ ǫ log(1/ǫ)' and 'Wǫ ≼ ǫ ǫ^{-d⋆/β} log(1/ǫ)' have spurious factors of ǫ on the right-hand sides, and the first term in the following display reads 'ǫ n^{-2α}', which should presumably be n^{-2α}. These typos make the calibration of ε and α hard to follow.
- [Appendix B (Lemma 21)] The statement 'f ∈ Hβ(R, R, C)' should be 'f ∈ Hβ(R^d, R, C)' (or Hβ([0,1]^d, R, C)), since the proof works on [0,1]^d with the d-dimensional Lebesgue measure.
- [6.2] In the last display of the proof of Theorem 10, '≿ n n^{-2β/(2β+d)}' contains a stray factor n; the intended conclusion is δ^{2β} ≍ n^{-2β/(2β+d)}.
- [Section 3] The text refers to the 'Bergman divergence' where 'Bregman divergence' is meant (see also Definition 5).
Circularity Check
No circular-by-construction step; Theorem 12's entropic-dimension rate is a genuine statistical derivation, though it imports the key approximation theorem from the authors' own prior preprint.
full rationale
The derivation chain is not circular in the sense of the enumerated patterns. Lemma 7 relates excess risk to L2(λ) estimation error through smoothness and strong-convexity constants, Lemmas 17–20 provide a standard oracle/localization decomposition, and no data-dependent parameter is fitted and then renamed a prediction. The bounded-density result (Theorem 8) is proved using Lemma 21, whose proof is contained in Appendix B; the minimax lower bound (Theorem 10) is a standard Fano argument. The only load-bearing self-citation is in the proof of Theorem 12, where the paper invokes 'Chakraborty and Bartlett (2024b, Theorem 18)' to obtain a ReLU network approximating f0 in L2(λ) with ε^{-d*/β} weights. That theorem is neither stated nor proved in this manuscript, so Theorem 12's advertised improvement over Minkowski-dimension rates depends on the correctness of an unpublished same-author result. This is a verification risk and a limitation on independence, but it is not a circular reduction: the cited approximation statement is parameter-free, its assumptions do not include the statistical rate being proved, and the remaining proof (localization, pseudo-dimension, bias-variance tradeoff) is carried out here. Under the rule that self-citation is not circular unless the argument reduces to an unverified self-citation chain, this dependency warrants a low score rather than an allegation of constructed circularity.
Assumptions & free parameters
assumptions (9)
- domain assumption A1: samples are i.i.d. from P in (3) and λ([0,1]^d)=1.
- domain assumption A2: f0 ∈ Hβ(Rd,R,C).
- domain assumption A3: Ψ is σ1-smooth and σ2-strongly convex.
- domain assumption A4: λ has Lebesgue density bounded above by \bar b_λ.
- domain assumption A5: λ has Lebesgue density bounded below by b_λ.
- standard math Bregman representation of exponential families (Banerjee et al. 2005, Theorem 4) and strong convexity/smoothness duality (Kakade et al. 2009, Theorem 6).
- standard math Pseudo-dimension bound for ReLU networks O(WL log W) (Bartlett et al. 2019, Theorem 6).
- ad hoc to paper For β-Hölder f0 and d* > \bar d_{2β}(λ), a ReLU network with O(ε^{-d*/β} log(1/ε)) weights approximates f0 in L2(λ) to error ε.
- ad hoc to paper Product functions on [-1,1]^d can be implemented by ReLU networks with O(m) weights and L∞ error 2^{-m}.
Cite this review
Pith. "Pith review of A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data." pith.science (2026). https://pith.science/paper/E4SHNPUA
@misc{pith2026241209779,
author = {Pith},
title = {Pith review of: A Statistical Analysis for Supervised Deep Learning with Exponential Families for Intrinsically Low-dimensional Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/E4SHNPUA}},
note = {Machine review of arXiv:2412.09779}
}
abstract
Recent advances have revealed that the rate of convergence of the expected test error in deep supervised learning decays as a function of the intrinsic dimension and not the dimension $d$ of the input space. Existing literature defines this intrinsic dimension as the Minkowski dimension or the manifold dimension of the support of the underlying probability measures, which often results in sub-optimal rates and unrealistic assumptions. In this paper, we consider supervised deep learning when the response given the explanatory variable is distributed according to an exponential family with a $\beta$-H\"older smooth mean function. We consider an entropic notion of the intrinsic data-dimension and demonstrate that with $n$ independent and identically distributed samples, the test error scales as $\tilde{\mathcal{O}}\left(n^{-\frac{2\beta}{2\beta + \bar{d}_{2\beta}(\lambda)}}\right)$, where $\bar{d}_{2\beta}(\lambda)$ is the $2\beta$-entropic dimension of $\lambda$, the distribution of the explanatory variables. This improves on the best-known rates. Furthermore, under the assumption of an upper-bounded density of the explanatory variables, we characterize the rate of convergence as $\tilde{\mathcal{O}}\left( d^{\frac{2\lfloor\beta\rfloor(\beta + d)}{2\beta + d}}n^{-\frac{2\beta}{2\beta + d}}\right)$, establishing that the dependence on $d$ is not exponential but at most polynomial. We also demonstrate that when the explanatory variable has a lower bounded density, this rate in terms of the number of data samples, is nearly optimal for learning the dependence structure for exponential families.
Reference graph
Works this paper leans on
-
[1]
Anthony, M. and Bartlett, P. (1999). Neural network learning: Theoretical foundations . Cambridge University Press
work page 1999
-
[2]
S., Ghosh, J., and Lafferty, J
Banerjee, A., Merugu, S., Dhillon, I. S., Ghosh, J., and Lafferty, J. (2005). Clustering with bregman divergences. Journal of machine learning research , 6(10)
work page 2005
-
[3]
L., Harvey, N., Liaw, C., and Mehrabian, A
Bartlett, P. L., Harvey, N., Liaw, C., and Mehrabian, A. (2019). Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research , 20(1):2285--2301
work page 2019
-
[4]
Bousquet, O. (2002). Concentration Inequalities and Empirical Processes Theory Applied to the Analysis of Learning Algorithms . PhD thesis, Biologische Kybernetik
work page 2002
-
[5]
Chakraborty, S. and Bartlett, P. (2024a). A statistical analysis of wasserstein autoencoders for intrinsically low-dimensional data. In The Twelfth International Conference on Learning Representations
work page 2024
-
[6]
Chakraborty, S. and Bartlett, P. L. (2024b). On the statistical properties of generative adversarial models for low intrinsic data dimension. arXiv preprint arXiv:2401.15801
arXiv 2024
-
[7]
Chen, M., Jiang, H., Liao, W., and Zhao, T. (2019). Efficient approximation of deep relu networks for functions on low dimensional manifolds. Advances in neural information processing systems , 32
work page 2019
-
[8]
Chen, M., Jiang, H., Liao, W., and Zhao, T. (2022). Nonparametric regression on low-dimensional manifolds using deep relu networks: Function approximation and statistical recovery. Information and Inference: A Journal of the IMA , 11(4):1203--1253
work page 2022
Show all 38 references
-
[9]
Cover, T. M. and Thomas, J. A. (2005). Elements of Information Theory . John Wiley & Sons, Hoboken, NJ
2005
-
[10]
Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems , 2(4):303--314
1989
-
[11]
a henb \
Donahue, J., Kr \"a henb \"u hl, P., and Darrell, T. (2017). Adversarial feature learning. In International Conference on Learning Representations
2017
-
[12]
Dudley, R. M. (1969). The speed of mean glivenko-cantelli convergence. The Annals of Mathematical Statistics , 40(1):40--50
1969
-
[13]
Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural networks , 4(2):251--257
1991
-
[14]
Huang, J., Jiao, Y., Li, Z., Liu, S., Wang, Y., and Yang, Y. (2022). An error analysis of generative adversarial networks for learning distributions. Journal of Machine Learning Research , 23(116):1--43
2022
-
[15]
Jiao, Y., Shen, G., Lin, Y., and Huang, J. (2021). Deep nonparametric regression on approximately low-dimensional manifolds. arXiv preprint arXiv:2104.06708
2021 arXiv
-
[16]
Kakade, S., Shalev-Shwartz, S., Tewari, A., et al. (2009). On the duality of strong convexity and strong smoothness: Learning applications and matrix regularization. Unpublished Manuscript, http://ttic. uchicago. edu/shai/papers/KakadeShalevTewari09. pdf , 2(1):35
2009
-
[17]
Kolmogorov, A. N. and Tikhomirov, V. M. (1961). -entropy and -capacity of sets in function spaces. Translations of the American Mathematical Society , 17:277--364
1961
-
[18]
Lehmann, E. L. and Casella, G. (2006). Theory of Point Estimation . Springer Science & Business Media
2006
-
[19]
Lu, J., Shen, Z., Yang, H., and Zhang, S. (2021). Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis , 53(5):5465--5506
2021
-
[20]
and Pontil, M
Maurer, A. and Pontil, M. (2021). Concentration inequalities under sub-gaussian and sub-exponential conditions. Advances in Neural Information Processing Systems , 34:7588--7597
2021
-
[21]
and Imaizumi, M
Nakada, R. and Imaizumi, M. (2020). Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research , 21(174):1--38
2020
-
[22]
Paul, D., Chakraborty, S., Das, S., and Xu, J. (2021). Uniform concentration bounds toward a unified framework for robust clustering. Advances in Neural Information Processing Systems , 34:8307--8319
2021
-
[23]
and Voigtlaender, F
Petersen, P. and Voigtlaender, F. (2018). Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks , 108:296--330
2018
-
[24]
Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T. (2020). The intrinsic dimension of images and its impact on learning. In International Conference on Learning Representations
2020
-
[25]
C., Rodemich, E
Posner, E. C., Rodemich, E. R., and Rumsey Jr, H. (1967). Epsilon entropy of stochastic processes. The Annals of Mathematical Statistics , pages 1000--1020
1967
-
[26]
Schmidt-Hieber, J. (2020). Nonparametric regression using deep neural networks with ReLU activation function . The Annals of Statistics , 48(4):1875 -- 1897
2020
-
[27]
Shen, Z., Yang, H., and Zhang, S. (2019). Nonlinear approximation via compositions. Neural Networks , 119:74--84
2019
-
[28]
Shen, Z., Yang, H., and Zhang, S. (2022). Optimal approximation rate of relu networks in terms of width and depth. Journal de Math \'e matiques Pures et Appliqu \'e es , 157:101--135
2022
-
[29]
Suzuki, T. (2018). Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. arXiv preprint arXiv:1810.08033
2018 arXiv
-
[30]
and Nitanda, A
Suzuki, T. and Nitanda, A. (2021). Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic besov space. Advances in Neural Information Processing Systems , 34:3609--3621
2021
-
[31]
Telgarsky, M. J. and Dasgupta, S. (2013). Moment-based uniform deviation bounds for k -means and friends. Advances in Neural Information Processing Systems , 26
2013
-
[32]
Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation . Springer Series in Statistics. Springer, Springer New York, NY, 1 edition. Published: 26 November 2008
2009
-
[33]
Uppal, A., Singh, S., and P \'o czos, B. (2019). Nonparametric density estimation & convergence rates for gans under besov ipm losses. Advances in neural information processing systems , 32
2019
-
[34]
Vershynin, R. (2018). High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press
2018
-
[35]
Wainwright, M. J. (2019). High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press
2019
-
[36]
and Bach, F
Weed, J. and Bach, F. (2019). Sharp asymptotic and finite-sample rates of convergence of empirical measures in wasserstein distance. Bernoulli , 25(4A):2620 -- 2648
2019
-
[37]
and Barron, A
Yang, Y. and Barron, A. (1999). Information-theoretic determination of minimax rates of convergence. Annals of Statistics , pages 1564--1599
1999
-
[38]
Yarotsky, D. (2017). Error bounds for approximations with deep relu networks. Neural Networks , 94:103--114
2017
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.