REVIEW 2 major objections 5 minor 300 references
Diffusion models cannot overfit training data and still generalize unless the sample size grows exponentially with dimension.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-12 07:49 UTC pith:HOIBCMWX
load-bearing objection Clean architecture-independent impossibility of simultaneous small train/test score-matching losses, plus exact linear asymptotics that isolate why score matching lacks the alignment that makes regression benignly overfit. the 2 major comments →
Benign Overfitting Does Not Occur in Diffusion Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Unless the sample size grows exponentially with the data's lower Rényi dimension, the population and empirical score-matching losses cannot be simultaneously small; consequently the population loss follows a classical U-shaped curve in model complexity rather than double descent. The structural reason is that score matching admits no target-covariance alignment of the kind that makes residual error small in ordinary regression.
What carries the argument
A Fisher-information / total-variation lower bound (Lemma 3.1 and Theorem 3.1) that forces the empirical and population measures at small noise to be close, together with an exact asymptotic comparison of linear random-feature score matching versus multi-output regression that isolates the missing alignment.
Load-bearing premise
The quantitative exponential sample-size lower bound needs the time-weighting density to be non-decreasing on an interval and both horizon and ambient dimension to be large enough; the rate also collapses if the data measure has zero lower Rényi dimension.
What would settle it
Train a single score network on a data distribution of known positive intrinsic dimension d' with sample size well below the exponential threshold of Theorem 3.1, drive the empirical score-matching loss to near zero, and check whether the population ESM loss (or reverse-process KL) remains bounded away from zero as claimed; a simultaneous drop of both losses would refute the impossibility result.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that benign overfitting and double descent, which are central to modern deep-learning theory, do not occur for diffusion models under practical sample sizes. Architecture-independent impossibility results (Lemma 3.1, Theorem 3.1) show that the empirical and population explicit score-matching losses cannot be simultaneously small unless n grows exponentially with the lower Rényi dimension of the data; consequently the population risk is U-shaped in model complexity. A linear random-feature analysis (Proposition 4.1) makes the mechanism precise by contrasting score matching with ordinary regression: the latter benefits from alignment of the target with the empirical covariance, while the former does not. Time-smoothness of the score and early stopping are identified as implicit regularizers that keep the population loss controlled (Propositions 5.1–5.2). High-dimensional U-Net experiments on a CIFAR-10 subset corroborate the U-shaped curves and the regularizing effect of time range and early stopping.
Significance. If correct, the result cleanly separates the generalization theory of score-based generative models from the benign-overfitting paradigm that has dominated supervised deep learning. The architecture-free lower bound of Theorem 3.1 is especially valuable: it rests only on Girsanov, Pinsker and Rényi dimension, applies to any score network, and already covers the ELBO and variance weightings used in practice. The linear-RFNN asymptotics supply an explicit, Marchenko–Pastur-based explanation of why the null-space contribution explodes as t→0, and the identification of time-smoothness and early stopping as built-in regularizers is both theoretically clean and experimentally supported. Together these contributions motivate a distinct theoretical program for diffusion models and give practitioners concrete guidance on when memorization becomes inevitable.
major comments (2)
- [Section 3.2, Theorem 3.1] Theorem 3.1 and Remark 3.1: the exponential lower bound is stated for weightings whose density is non-decreasing on [ϵ,T]. While ELBO and variance weightings satisfy this, many practical schedules (e.g., cosine or truncated-SNR) do not. A short extension or counter-example for non-monotone ϖ would clarify the scope of the impossibility claim.
- [Section 4, Proposition 4.1] Proposition 4.1 and the surrounding comparison with regression: the argument that score matching admits “no alignment” relies on the isotropic Gaussian assumption (Assumption 4.1) and on the exact ERM solution AW=−Σ̂t−1. For non-Gaussian or low-rank data the residual on ker(Σ̂) need not explode as σt−2; a brief remark on the robustness of this mechanism would strengthen the claim that the phenomenon is generic rather than Gaussian-specific.
minor comments (5)
- [Figure 1, Appendix C.1] Figure 1 caption and Appendix C.1: the random-feature diffusion experiment uses a Monte-Carlo approximation of the DSM expectation (Ng=10). A one-sentence statement of the resulting bias would help readers assess the plotted train/test curves.
- [Appendix A.1] Definition A.1 and Remark A.2: the lower Rényi dimension is introduced without a pointer to the more common Hausdorff or Minkowski dimensions used in the manifold-hypothesis literature; a short cross-reference would improve accessibility.
- [Proposition 4.1] Equation (7) and the O(1/ψd) remainder: the asymptotic is stated for ψd,ψp>1; the transition regime ψp≈ψd is left implicit. A sentence noting continuity of the limiting expressions would remove a minor ambiguity.
- [Figure 2] Figure 2 (right panel): the nearest-train-image comparison is shown only for NF=8. Displaying the same panel for a larger NF (e.g., 64) would make the memorization claim visually complete.
- [Title page and Appendix A] Typographical: “Umut S ¸ims ¸ekli” and several occurrences of “R ´enyi” retain LaTeX spacing artifacts; these should be cleaned for the camera-ready version.
Circularity Check
No significant circularity: impossibility bounds and linear asymptotics are derived from first principles without self-definitional or fitted-input reductions.
full rationale
The central claims rest on Lemma 3.1 / Theorem 3.1 (architecture-independent lower bounds via relative Fisher information, Girsanov, Pinsker and the lower Rényi dimension) and on Proposition 4.1 / 5.1 / 5.2 (exact Marchenko–Pastur asymptotics of an explicitly solvable linear ERM problem). None of these steps defines a quantity in terms of the target phenomenon and then re-labels it as a prediction; no free parameter is fitted to data and then used to “predict” a closely related observable; and the few self-citations (Farghly et al. 2026a,b; Dupuis et al. 2025) appear only as contextual motivation or complementary generalization bounds, never as load-bearing uniqueness theorems that force the present conclusions. The modeling idealizations (Gaussian data, linear features, non-decreasing density of ϖ) are stated explicitly and the resulting U-shaped excess-risk curves follow by direct calculation, not by construction. Consequently the derivation chain is self-contained and non-circular.
Axiom & Free-Parameter Ledger
free parameters (2)
- ridge λ in random-feature experiments
- time-horizon T and early-stop ϵ
axioms (4)
- standard math Marchenko-Pastur theorem for empirical covariance spectra under i.i.d. entries with 4+ε moments
- standard math Girsanov theorem relating path-space KL to integrated score error
- domain assumption Data distribution ν has finite second moments and positive lower Rényi dimension d'
- ad hoc to paper Score network is a two-layer linear random-feature model with Gaussian weights (Assumption 4.1)
read the original abstract
Benign overfitting and double descent have come to shape our understanding of generalization in deep learning, establishing that overfitting is not only compatible with good generalization but can actively benefit it. Diffusion models share much of the machinery of standard deep learning, so it is natural to assume that they also exhibit these properties. In this work, we show that this assumption is largely incorrect. We first establish fundamental impossibility results showing that, unless the sample size grows exponentially with the data dimension, overfitting and good generalization cannot occur simultaneously. Consequently, the population loss follows a classical U-shaped curve in model complexity rather than exhibiting double descent. Analyzing a simplified setting, we identify a key difference between regression and score matching: regression benefits from an alignment between the target and the empirical covariance; score matching admits no such alignment, leaving overfitting irreparably harmful. We further identify implicit regularization stemming from time-smoothness of the score and early stopping during training as mechanisms that prevent such overfitting and verify our findings with high-dimensional image generation experiments. Our results reveal that generalization in diffusion models is governed by mechanisms distinct from those of traditional regression, motivating the development of new theory.
Figures
Reference graph
Works this paper leans on
-
[1]
Random Features for Large-Scale Kernel Machines , volume =
Rahimi, Ali and Recht, Benjamin , booktitle =. Random Features for Large-Scale Kernel Machines , volume =
-
[2]
The Eleventh International Conference on Learning Representations , year=
Statistical Efficiency of Score Matching: The View from Isoperimetry , author=. The Eleventh International Conference on Learning Representations , year=
-
[3]
The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=
Diffusion Models and the Manifold Hypothesis: Log-Domain Smoothing is Geometry Adaptive , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year=
-
[4]
EurIPS 2025 Workshop on Principles of Generative Modeling (PriGM) , year=
Generalization Dynamics of Linear Diffusion Models , author=. EurIPS 2025 Workshop on Principles of Generative Modeling (PriGM) , year=
2025
-
[5]
Marchenko, V. A. and Pastur, L. A. , year=. Distribution of Eigenvalues for some Sets of Random Matrices , volume=. Mathematics of the USSR-Sbornik , publisher=
-
[6]
2023 , eprint=
Six Lectures on Linearized Neural Networks , author=. 2023 , eprint=
2023
-
[7]
ICLR 2026 Workshop on Geometry-grounded Representation Learning and Generative Modeling , year=
Manifold Generalization Provably Proceeds Memorization in Diffusion Models , author=. ICLR 2026 Workshop on Geometry-grounded Representation Learning and Generative Modeling , year=
2026
-
[8]
Comment on: ``
Otto, Felix and Villani, C. Comment on: ``. 2001 , journal =
2001
-
[9]
Procaccia and P
I. Procaccia and P. Grassberger and H. G. E. Hentschel , title =. Dynamical Systems and Chaos , year =
-
[10]
On Rigorous Mathematical Definitions of Correlation Dimension and Generalized Spectrum for Dimensions , author =. 1993 , month = may, journal =. doi:10.1007/BF01058436 , urldate =
-
[11]
The Fourteenth International Conference on Learning Representations , year=
Provable Separations between Memorization and Generalization in Diffusion Models , author=. The Fourteenth International Conference on Learning Representations , year=
-
[12]
Denoising
George, Anand Jerry and Veiga, Rodrigo and Macris, Nicolas , year =. Denoising
-
[13]
Reversible
Aldous, David and Allen Fill, James , year =. Reversible
-
[14]
Rudolf, Daniel , year =. Explicit Error Bounds for. Dissertationes Mathematicae , volume =. 1108.3201 , primaryclass =
-
[15]
Nakkiran, Preetum and Kaplun, Gal and Bansal, Yamini and Yang, Tristan and Barak, Boaz and Sutskever, Ilya , year =. Deep. ICLR 2020 , eprint =
2020
-
[16]
Fractional
Tristani, Isabelle , year =. Fractional. Commun. Math. Sci. 13 , eprint =
-
[17]
Proceedings of the Thirty-Second Conference on Learning Theory , pages =
Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLT , author=. Proceedings of the Thirty-Second Conference on Learning Theory , pages =. 2019 , volume =
2019
-
[18]
The Cifar-10 Dataset , author =
-
[19]
2009 , url=
Learning Multiple Layers of Features from Tiny Images , author=. 2009 , url=
2009
-
[20]
Hodgkinson, Liam and Mahoney, Michael W. , year =. Multiplicative Noise and Heavy Tails in Stochastic Optimization , booktitle =. 2006.06293 , primaryclass =
Pith/arXiv arXiv 2006
-
[21]
Journal of Machine Learning Research , volume=
Topology of deep neural networks , author=. Journal of Machine Learning Research , volume=
-
[22]
Very Deep Convolutional Networks for Large-Scale Image Recognition , booktitle =
Karen Simonyan and Andrew Zisserman , editor =. Very Deep Convolutional Networks for Large-Scale Image Recognition , booktitle =
-
[23]
He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , booktitle =
-
[24]
North American Chapter of the Association for Computational Linguistics , year=
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author=. North American Chapter of the Association for Computational Linguistics , year=
-
[25]
Language
Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and. Language. Advances in. 2020 , volume =
2020
-
[26]
and Monro, S
Robbins, H. and Monro, S. , journal =
-
[27]
Bottou, L. The. Advances in. 2007 , volume =
2007
-
[28]
, year =
Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E. , year =. Communications of the ACM , volume =
-
[29]
Kallenberg, Olav , year =. Foundations of. doi:10.1007/978-3-030-61871-1 , urldate =
-
[30]
2023 IEEE 6th International Conference on Pattern Recognition and Artificial Intelligence (PRAI) , pages=
Deep neural networks architectures from the perspective of manifold learning , author=. 2023 IEEE 6th International Conference on Pattern Recognition and Artificial Intelligence (PRAI) , pages=. 2023 , organization=
2023
-
[31]
arXiv preprint arXiv:2204.08624 , year=
Topology and geometry of data manifold in deep learning , author=. arXiv preprint arXiv:2204.08624 , year=
-
[32]
Understanding Machine Learning: From Theory to Algorithms , publisher=
Shalev-Shwartz, Shai and Ben-David, Shai , year=. Understanding Machine Learning: From Theory to Algorithms , publisher=
-
[33]
Understanding Machine Learning: From Theory to Algorithms , publisher=
Bach, Francis , series =. Understanding Machine Learning: From Theory to Algorithms , publisher=
-
[34]
Theory of Probability and its Applications , year =
Vladimir Vapnik and Alexey Chervonenkis , title =. Theory of Probability and its Applications , year =
-
[35]
Doklady Akademii Nauk USSR , year =
Vladimir Vapnik and Alexey Chervonenkis , title =. Doklady Akademii Nauk USSR , year =
-
[36]
Kuru, Nurdan and Birbil,. Differentially. 2022 , month = jun, journal =. 2008.01989 , primaryclass =
Pith/arXiv arXiv 2022
-
[37]
and Blei, David M
Mandt, Stephan and Hoffman, Matthew D. and Blei, David M. , year =. A. International
-
[38]
2023 , eprint=
Equivalences of Geometric Ergodicity of Markov Chains , author=. 2023 , eprint=
2023
-
[39]
Pertti Mattila , title =
-
[40]
John Mackay and Jeremy Tyson , title =
-
[41]
Estimating
Herrera, Calypso and Krach, Florian and Teichmann, Josef , year =. Estimating. Estimating Full Lipschitz Constants of Deep Neural Networks , eprint =
-
[42]
Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks , author =. 2018 , month = jan, journal =. 1710.11029 , primaryclass =
Pith/arXiv arXiv 2018
-
[43]
Chaudhuri, Kamalika and Monteleoni, Claire and Sarwate, Anand D. , year =. Differentially. Journal of Machine Learning Research , volume =. 0912.0071 , primaryclass =
-
[44]
Annals of Mathematics and Artificial Intelligence , volume=
Topological measurement of deep neural networks using persistent homology , author=. Annals of Mathematics and Artificial Intelligence , volume=. 2022 , publisher=
2022
-
[45]
and Tweedie, Richard L
Meyn, Sean P. and Tweedie, Richard L. , year =. Markov
-
[46]
and Braun, H
Riedmiller, M. and Braun, H. , booktitle=. A direct adaptive method for faster backpropagation learning: the RPROP algorithm , year=
-
[47]
Xiao, Han and Rasul, Kashif and Vollgraf, Roland , year =. Fashion-. 1708.07747 , primaryclass =
-
[48]
High-dimensional limit theorems for SGD: Effective dynamics and critical scaling , booktitle=
Gerard Ben Arous and Reza Gheissari and Aukosh Jagannath , year=. High-dimensional limit theorems for SGD: Effective dynamics and critical scaling , booktitle=. 2206.04030 , archivePrefix=
-
[49]
Alfonsi, Aur. Evolution of the. 2016 , month = dec, number =. doi:10.48550/arXiv.1606.02994 , urldate =. 1606.02994 , primaryclass =
-
[50]
and Orbanz, Peter , year =
Zhou, Wenda and Veitch, Victor and Austern, Morgane and Adams, Ryan P. and Orbanz, Peter , year =. Non-Vacuous. International
-
[51]
2024 , keywords =
Alquier, Pierre , journal=. 2024 , keywords =
2024
-
[52]
Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimization , volume =
Amir, Idan and Livni, Roi and Srebro, Nati , booktitle =. Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimization , volume =
-
[53]
Advances in Neural Information Processing Systems (NeurIPS) , year =
Ron Amit and Baruch Epstein and Shay Moran and Ron Meir , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =
-
[54]
Andreeva, Rayna and Dupuis, Benjamin and Sarkar, Rik and Birdal, Tolga and Simsekli, Umut , year =. Topological. 38th. 2407.08723 , primaryclass =
-
[55]
ICML 2023 Workshop on Topological, Algebraic and Geometric Learning , year =
Rayna Andreeva and Katharina Limbeck and Bastian Rieck and Rik Sarkar , title =. ICML 2023 Workshop on Topological, Algebraic and Geometric Learning , year =
2023
-
[56]
Metric Space Magnitude for Evaluating the Diversity of Latent Representations , volume =
Limbeck, Katharina and Andreeva, Rayna and Sarkar, Rik and Rieck, Bastian , booktitle =. Metric Space Magnitude for Evaluating the Diversity of Latent Representations , volume =. doi:10.52202/079017-3937 , editor =
-
[57]
Deep Learning in Optical Metrology: A Review , shorttitle =
Zuo, Chao and Qian, Jiaming and Feng, Shijie and Yin, Wei and Li, Yixuan and Fan, Pengfei and Han, Jing and Qian, Kemao and Chen, Qian , year =. Deep Learning in Optical Metrology: A Review , shorttitle =. Light: Science & Applications , volume =
-
[58]
1999 , publisher =
Neural Network Learning: Theoretical Foundations , author =. 1999 , publisher =
1999
-
[59]
Andrieu, Christophe and Lee, Anthony and Power, Sam and Wang, Andi Q. , year =. Poincar. 2208.05239 , publisher =
-
[60]
On Logarithmic
An. On Logarithmic. 2000 , journal =
2000
- [61]
-
[62]
Asadi, Amir R. and Abbe, Emmanuel and Verd. Chaining. 2019 , month = jul, number =. 1806.03803 , primaryclass =
Pith/arXiv arXiv 2019
-
[63]
Analysis and
Bakry, Dominique and Gentil, Ivan and Ledoux, Michel , year =. Analysis and
-
[64]
Probability Measures on Groups , author =
Functional. Probability Measures on Groups , author =. 2004 , pages =
2004
-
[65]
2024 , eprint=
Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent , author=. 2024 , eprint=
2024
-
[66]
Schertzer, D. and Larchev, M. and Duan, J. and Yanovsky, V. V. and Lovejoy, S. , year =. Fractional. Journal of Mathematical Physics , volume =. math/0409486 , pages =
-
[67]
Freitag, Eberhard and Rusam, Rolf , year =. Complex
-
[68]
Lang, Serge , year =. Complex
-
[69]
Distributions and
Grubb, Gerd , year =. Distributions and
-
[70]
International Conference on Learning Representations , year=
A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima , author=. International Conference on Learning Representations , year=
-
[71]
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages =
Stable behaviour of infinitely wide deep neural networks , author =. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages =. 2020 , editor =
2020
-
[72]
Stochastic
Wojtowytsch, Stephan , year =. Stochastic. Journal of Nonlinear Science , volume =
-
[73]
Adamczak, Rados. Modified Log-. 2022 , month = apr, journal =. 2007.10209 , primaryclass =
Pith/arXiv arXiv 2022
-
[74]
International Conference on Learning Representations , year=
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. International Conference on Learning Representations , year=
-
[75]
SIAM Journal on Mathematics of Data Science , volume=
KL convergence guarantees for score diffusion models under minimal data assumptions , author=. SIAM Journal on Mathematics of Data Science , volume=. 2025 , publisher=
2025
-
[76]
1982 , issn =
Reverse-time diffusion equation models , journal =. 1982 , issn =
1982
-
[77]
Jung, Paul and Lee, Hoil and Lee, Jiho and Yang, Hongseok , year =. alpha-. Advances in Applied Probability , Volume 55 , Issue 4 , eprint =
-
[78]
A Fractal Dimension for Measures via Persistent Homology , author =. 2020 , journal =. 1808.01079 , primaryclass =
Pith/arXiv arXiv 2020
-
[79]
Frontiers in Artificial Intelligence , volume=
Topology applied to machine learning: From global to local , author=. Frontiers in Artificial Intelligence , volume=. 2021 , publisher=
2021
-
[80]
Stability
Banerjee, Arindam and Chen, Tiancong and Li, Xinyan and Zhou, Yingxue , year =. Stability. arXiv.org , urldate =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.