REVIEW 3 major objections 5 minor 3 cited by
Optimizing Noise Schedules of Generative Models in High Dimensionss
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read With correctly dilated noise schedules, a generative probability-flow ODE for Gaussian mixtures and Curie-Weiss models becomes discretizable with a constant number of steps instead of the $\Theta(\sqrt{d})$ steps a uniform grid requires.
desk verdict The VP/VE dichotomy and time-dilation idea are genuinely new and likely correct, but the headline Θ_d(1)-step claim is not actually proved because the iterated limit order does not control the Euler error uniformly in the step size. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the stochastic interpolant and its probability-flow ODE, $\dot X_\tau = b_\tau(X_\tau)$ with $b_\tau(x)=\mathbb{E}[\dot I_\tau \mid I_\tau=x]$, together with a time-dilation $\tau(t)$ that acts as a non-uniform noise schedule. The paper specializes to $\alpha_\tau=1-\tau$, $\beta_\tau=\tau$ (and the VP variant $\sqrt{1-\tau^2}$), with noise scale $c=1$ for VP and $c=\sqrt{d}$ for VE, and chooses $\tau(t)$ so that the speciation window has constant length in $t$: for VP, $\tau$ reaches $\kappa/\sqrt{d}$ at $t=1/2$; for VE, the final window $\tau\in[1-\kappa/\sqrt{d},1]$ is stretched to $t\in[1/2,1]$. The argument reduces the high-dimensional ODE to two low-dimensional objects: the magnetization $M_t=r\cdot X_t/d$ (or $\mu_t=r\cdot X_t/\sqrt{d}$ in the first VP phase) that settles the mode, and the orthogonal Gaussian fluctuations that carry $\sigma^2$. Each phase's limiting ODE is identified with a known one-dimensional interpolant transport, which is what allows explicit formulas for the endpoints.
What would settle it
Fix a small step size such as $\Delta t=0.01$ and simulate the dilated VE probability-flow ODE for a Gaussian mixture with known $p=0.8$ and $\sigma^2=0.25$ at dimensions $d=10^4,10^6,10^8$; if, with the step size held fixed, the empirical magnetization at $t=1/2$ does not approach $p\delta_1+(1-p)\delta_{-1}$ and the coordinate variance at $t=1$ does not approach $\sigma^2$ as $d$ grows, then the double limit is not uniform and the $\Theta_d(1)$-step conclusion fails in the finite-step regime it claims.
Extended reading notes
Core claim
The central discovery is that the two phases of generation correspond to two distinct features, and each can be stretched to constant duration by rescaling time near the speciation transition. Starting from the stochastic interpolant $I_\tau = c\alpha_\tau z + \beta_\tau a$ with $z$ Gaussian and $a$ drawn from the data, the paper proves for the Gaussian mixture $p\mathcal{N}(r,\sigma^2 I_d)+(1-p)\mathcal{N}(-r,\sigma^2 I_d)$ with $|r|^2=d$ that a uniform grid has VP's speciation time $\tau_s=1/\sqrt{d}$ and therefore cannot resolve $p$ with $\Theta_d(1)$ steps, while VE captures $p$ but drives the orthogonal fluctuations to zero in the $d\to\infty$ limit, losing $\sigma^2$. The dilated VP schedule $\tau(t)=2\kappa t/\sqrt{d}$ on $[0,1/2]$ followed by a linear ramp, and the dilated VE schedule $\tau(t)=(1-\kappa/\sqrt{d})2t$ followed by a $\kappa/\sqrt{d}$ window, both produce limiting two-phase ODEs: a first phase with a $\tanh$ drift that settles the magnetization $M_t$ to $p\delta_1+(1-p)\delta_{-1}$ and a second phase that transports the orthogonal component to variance $\sigma^2$ (for VP) or to the mode distribution (for VE). For the Curie-Weiss model the same dilated VE schedule resolves both the mode asymmetry and the discrete $\{\pm 1\}$ spin distribution. Because the limiting ODE is independent of $d$, the paper concludes that $\Theta_d(1)$ discretization points suffice, whereas uniform grids need $\Theta(\sqrt{d})$.
Load-bearing premise
The load-bearing premise is that the asymptotic limits commute in the order $d$ then step size, so that errors of order $1/\sqrt{d}$ can be discarded before the discretization error is taken to zero; if those errors are not uniform in the schedule parameters, the claim of $\Theta_d(1)$ steps may hold only in a narrower regime than stated.
Editorial extensions
If this is right
- Under a uniform grid, VP and VE solve complementary halves of the Gaussian-mixture problem: VP captures $\sigma^2$, VE captures $p$, and neither captures both with $o(\sqrt{d})$ uniform steps.
- The dilated schedules given in equations (4) and (5) make both $p$ and $\sigma^2$ recoverable with $\Theta_d(1)$ steps for the Gaussian mixture, and the same dilated VE schedule works for the Curie-Weiss model.
- The first-phase limiting ODEs for the Gaussian mixture and Curie-Weiss model coincide up to a factor $m$, so the early phase is blind to low-level model details; only the second phase distinguishes the two data distributions.
- In practice, a non-uniform grid with half the points spent before $\tau=\kappa/\sqrt{d}$ and half after reproduces the output that a uniform grid needs $\sim\sqrt{d}$ times more points to achieve.
- Real-image experiments on CelebA-HQ show the same directional pattern: more VP steps improve high-level features without fixing low-level quality, while more VE steps improve low-level features without fixing high-level diversity.
Reading between the lines
- The time-dilation construction suggests a general recipe for hierarchical data: identify each feature's speciation time and stretch the schedule so every critical window has constant duration; the paper demonstrates this only on two test distributions, but the mechanism is stated in terms of generic two-phase ODEs.
- If the order-of-limits issue can be tightened, the $\Theta_d(1)$ statement becomes a proof that schedule design, not just score estimation or solver order, is the dominant factor in the step-count cost of sampling in high dimension.
- A natural test is to apply the dilated VP and VE schedules to pretrained image samplers with fixed step budgets and measure feature-level KL divergences; the paper's CelebA experiment suggests VP and VE would swap which feature improves with steps.
- The finite-$\kappa$ correction is unexplored: the theorems recover $p$ and $\sigma^2$ only as $\kappa\to\infty$, so the optimal dilation strength for a given feature tolerance and dimension is an open quantitative question.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies probability flow ODE generative models built from stochastic interpolants, focusing on two high-dimensional test distributions: a two-mode Gaussian mixture and the Curie-Weiss model. It shows that with uniform noise schedules the VP interpolant captures the low-level feature (mode variance σ²) but misses the high-level feature (mode asymmetry p), while the VE interpolant does the opposite. The authors then introduce piecewise-linear time dilations specific to VP and VE and prove, in a combined limit d→∞ followed by Δt→0 (and, for some statements, κ→∞), that the dilated schedules recover both p and σ² and that the resulting probability flow ODE can be discretized with a number of steps that is constant in the dimension, Θ_d(1), rather than Θ_d(√d). The paper also reports numerical experiments on the GM and CW models and on CelebA-HQ with pretrained VP/VE SDEs.
Significance. The paper identifies a clean mechanism for a practically relevant phenomenon: the complementary failure modes of VP and VE schedules and the possibility of curing them by time dilation. The reduction of the high-dimensional probability flow ODE to one-dimensional limiting ODEs in Lemmas 2–5 is elegant, and the explicit limiting equations for the magnetization and the orthogonal variance are concrete and falsifiable. The GM and CW results are stated with full proofs in the appendices, and the numerical experiments support the qualitative phase structure. If the technical gaps in the limit interchanges are closed or properly qualified, the Θ_d(1)-step claim would be a valuable theoretical contribution to the diffusion-model literature.
major comments (3)
- [§3.2, Theorem 1 proof, Eq. (44)] The Θ_d(1)-step conclusion requires a two-limit argument that is not supplied. The proof derives the limiting ODE by dropping O(1/√d) terms and then lets Δt→0, but to justify the sentence 'we can discretize the ODE with Δt ∈ Θ_d(1) and get accurate estimation' one must show that, for fixed Δt, the d→∞ limit of the Euler discretization of the d-dimensional system equals the Euler discretization of the limiting ODE, with errors that are controlled uniformly in Δt. Equation (44) asserts precisely such a decomposition, µ_{t=1/2} = θ + O(1/√d) + o_Δt(1), but no uniform-in-Δt bound or exchange-of-limits lemma is proved. As written, the theorem establishes the double limit for the continuous-time solution and separately that the Euler method converges for the limiting ODE; it does not by itself establish the claimed O_d(1)-step guarantee for the original discretized dynamics.
- [§3.2, Theorems 1 and 2] The recovery of both features is stated in a combined limit that also sends κ to infinity, but this qualification is not carried through the abstract or the Θ_d(1)-step claim. For fixed κ, Theorem 2 gives lim_{Δt→0} lim_{d→∞} σ_1^{κ,Δt,d} = κ σ / √(κ²+σ²), which is strictly smaller than σ, and Theorem 1 gives M_1 ∼ p_κ δ_1 + (1−p_κ)δ_{−1} with p_κ ≠ p for finite κ. Thus a fixed schedule with a fixed κ does not exactly recover both features in the d→∞ limit; the paper should state explicitly that κ must be sent to infinity (or chosen large) and should quantify the resulting bias, e.g., as O(1/κ). This is a load-bearing qualification for the headline claim that a Θ_d(1)-step discretization captures both p and σ².
- [§A, Lemma 2 and §3.2, Eqs. (4)–(5)] The time dilations (4) and (5) are only piecewise C¹, with a kink at t = 1/2, while Lemma 2 requires α_τ, β_τ ∈ C²([0,1]). The composed coefficients α_t = 1−τ(t) and β_t = τ(t) are therefore not C², and the velocity field of the interpolant may be discontinuous at the kink. The proofs treat the two phases separately, but the paper does not justify that the probability flow ODE, or its Euler discretization, is well-defined across the kink, nor that the phase-wise limiting ODEs combine to give the stated limiting dynamics. A regularity lemma for absolutely continuous or piecewise smooth time changes, or an explicit smoothing argument, is needed.
minor comments (5)
- [Title] The title contains a typo: 'Dimensionss' should be 'Dimensions'.
- [Figure 2 caption] The caption reads 'uniformly discretized with step size d = 10^6, Δt = 0.01'; this should be 'dimension d = 10^6, step size Δt = 0.01'.
- [Figure 5 caption] The caption begins 'or different number of discretization steps'; it should begin 'For different number of discretization steps'.
- [Theorem 1 statement] The quantity p_κ is used in the statement M_1 ∼ p_κ δ_1 + (1−p_κ)δ_{−1} before it is defined; the definition 'p_κ is such that lim_{κ→∞} p_κ = p' should appear before this display.
- [Appendix C, Eq. (25)] In the proof of Proposition 3, the display 'µ_τ/d = α_τ Z + √d β_τ m' is dimensionally inconsistent; from X_τ = α_τ z + β_τ a and µ_τ = r·X_τ/√d one obtains µ_τ = α_τ Z + √d β_τ m. The subsequent comparison α_τ ≈ √d β_τ follows from the corrected equation.
Circularity Check
No significant circularity: the VP/VE dichotomy and the dilation results are derived from the exact interpolant ODE, and self-citations are tool citations rather than load-bearing conclusions.
full rationale
The paper's derivation chain is self-contained: Propositions 1 and 2 solve the exact probability-flow ODE from Lemma 2, which is the conditional-expectation velocity field of the stochastic interpolant, and obtain the VP/VE dichotomy by taking d→∞ at fixed step size and then Δt→0. The dichotomy is an output of those limits, not an input. Proposition 3 re-derives the VP speciation time (1/√d) from the same ODE, so the earlier related-work citation to Biroli et al. (2024) is contextual rather than load-bearing. The dilated-schedule theorems (Theorems 1-3) plug the explicit time dilations (4)-(5) into the velocity fields and identify the limiting ODEs; the terminal laws pδ1+(1-p)δ-1 and the recovered variances σ² are outputs of those limiting ODEs, not fitted parameters renamed as predictions. The stochastic-interpolant lemmas cited from Albergo et al. (2023), which share an author with the present paper, are exact parameter-free identities with stated assumptions and are used as computational tools; the paper's central claim about the number of discretization steps is not defined in terms of those lemmas. The one substantive concern raised by a skeptical reading is not circularity: the Θ_d(1)-step statement is justified through the iterated limit lim_{Δt→0} lim_{d→∞} without a uniform-in-Δt error bound, so there may be a limit-exchange or uniformity gap, but that is a rigor issue rather than a reduction of the conclusion to the input. Score 2 reflects minor, non-load-bearing self-citation rather than actual circularity.
Assumptions & free parameters
free parameters (2)
- kappa (κ) =
3 in Section 4 experiments; general in theorems
- phase split at t=1/2 =
1/2
assumptions (6)
- domain assumption The exact velocity field bτ(x)=E[İτ|Iτ=x] is known, with no score estimation error.
- domain assumption The limits d→∞ and Δt→0 commute in the stated order, and error terms O(1/√d) vanish uniformly.
- ad hoc to paper The time dilations τ(t) in (4) and (5) may be treated as smooth schedules compatible with α,β∈C².
- domain assumption The tanh approximations in the Curie-Weiss analysis (linearization and sign replacement) hold with error o_κ(1) uniformly in t and d.
- domain assumption For the Gaussian mixture, σ²=Θ(1) and |r|²=d, giving the two-scale structure.
- standard math The law of large numbers applies to the Curie-Weiss spins in the second phase, replacing empirical averages with expectations.
Cite this review
Pith. "Pith review of Optimizing Noise Schedules of Generative Models in High Dimensionss." pith.science (2026). https://pith.science/paper/LUJCFBAK
@misc{pith2026250100988,
author = {Pith},
title = {Pith review of: Optimizing Noise Schedules of Generative Models in High Dimensionss},
year = {2026},
howpublished = {\url{https://pith.science/paper/LUJCFBAK}},
note = {Machine review of arXiv:2501.00988}
}
abstract
Recent works have shown that diffusion models can undergo phase transitions, the resolution of which is needed for accurately generating samples. This has motivated the use of different noise schedules, the two most common choices being referred to as variance preserving (VP) and variance exploding (VE). Here we revisit these schedules within the framework of stochastic interpolants. Using the Gaussian Mixture (GM) and Curie-Weiss (CW) data distributions as test case models, we first investigate the effect of the variance of the initial noise distribution and show that VP recovers the low-level feature (the distribution of each mode) but misses the high-level feature (the asymmetry between modes), whereas VE performs oppositely. We also show that this dichotomy, which happens when denoising by a constant amount in each step, can be avoided by using noise schedules specific to VP and VE that allow for the recovery of both high- and low-level features. Finally we show that these schedules yield generative models for the GM and CW model whose probability flow ODE can be discretized using $\Theta_d(1)$ steps in dimension $d$ instead of the $\Theta_d(\sqrt{d})$ steps required by constant denoising.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 3 Pith papers
-
Optimal Scheduling of Dynamic Transport
For a fixed transport map with symmetric positive definite Jacobian, the closed-form optimal schedule minimizing the uniform spatial Lipschitz constant of the flow velocity is an exponential two-phase curve that achie...
-
Classifier-Free Guidance: From High-Dimensional Analysis to Generalized Guidance Forms
CFG's distortion of the target distribution vanishes as data dimension grows, and a power-law generalization improves fidelity and diversity in high-dimensional generative models.
-
Open Materials Generation with Stochastic Interpolants
By tuning stochastic interpolants for periodic crystals and coupling them with discrete flow matching, OMatG sets new state-of-the-art results on crystal structure prediction and de novo materials generation.
Reference graph
Works this paper leans on
-
[1]
S., Boffi, N
Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. (2023). Stochastic interpolants: A unifying framework for flows and diffusions
2023
-
[2]
Albergo, M. S. and Vanden-Eijnden, E. (2023). Building normalizing flows with stochastic interpolants
work page 2023
- [3]
-
[4]
D., Doucet, A., and Deligiannidis, G
Benton, J., Bortoli, V. D., Doucet, A., and Deligiannidis, G. (2024). Nearly d -linear convergence bounds for diffusion models via stochastic localization
work page 2024
-
[5]
Biroli, G., Bonnaire, T., de Bortoli, V., and Mézard, M. (2024). Dynamical regimes of diffusion models
work page 2024
-
[6]
Biroli, G. and Mézard, M. (2023). Generative diffusion in very large dimensions. Journal of Statistical Mechanics: Theory and Experiment , 2023(9):093402
work page 2023
-
[7]
Chen, S., Chewi, S., Lee, H., Li, Y., Lu, J., and Salim, A. (2023). The probability flow ode is provably fast. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems , volume 36, pages 68552--68575. Curran Associates, Inc
2023
-
[8]
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y., and Rombach, R. (2024). Scaling rectified flow transformers for high-resolution image synthesis
work page 2024
Show all 23 references
-
[9]
Gatmiry, K., Kelner, J., and Lee, H. (2024). Learning mixtures of gaussians using diffusion models
2024
-
[10]
Ddpm celeba-hq 256
Google (2021a). Ddpm celeba-hq 256. https://huggingface.co/google/ddpm-celebahq-256
2021
-
[11]
Ncsn++ celeba-hq 256
Google (2021b). Ncsn++ celeba-hq 256. https://huggingface.co/google/ncsnpp-celebahq-256
2021
-
[12]
Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840--6851
2020
-
[13]
Karras, T., Aila, T., Laine, S., and Lehtinen, J. (2018). Progressive growing of gans for improved quality, stability, and variation
2018
-
[14]
and Chen, S
Li, M. and Chen, S. (2024). Critical windows: non-asymptotic theory for feature emergence in diffusion models
2024
-
[15]
Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. (2023). Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations
2023
-
[16]
Liu, X., Gong, C., and Liu, Q. (2022). Flow straight and fast: Learning to generate and transfer data with rectified flow
2022
-
[17]
and Ambrogioni, L
Raya, G. and Ambrogioni, L. (2023). Spontaneous symmetry breaking in generative diffusion models
2023
-
[18]
Sclocchi, A., Favero, A., and Wyart, M. (2024). A phase transition in diffusion models reveals the hierarchical nature of data
2024
-
[19]
and Ozpinar, A
Serengil, S. and Ozpinar, A. (2024). A benchmark of facial recognition pipelines and co-usability performances of modules. Journal of Information Technologies , 17(2):95--107
2024
-
[20]
Shah, K., Chen, S., and Klivans, A. (2023). Learning mixtures of gaussians using the ddpm objective
2023
-
[21]
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256--2265. PMLR
2015
-
[22]
and Ermon, S
Song, Y. and Ermon, S. (2020). Improved techniques for training score-based generative models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 12438--12448. Curran Associates, Inc
2020
-
[23]
P., Kumar, A., Ermon, S., and Poole, B
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. (2021). Score-based generative modeling through stochastic differential equations
2021
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.