Pith. sign in

REVIEW 3 major objections 4 minor 61 references

VarDiU: A Variational Diffusive Upper Bound for One-Step Diffusion Distillation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes VarDiU, a variational upper bound on the diffusive KL divergence whose gradient can be estimated unbiasedly, and reports improved one-step generation quality and training stability over Diff-Instruct on a 40-Gaussian toy.

desk verdict A clean variational trick for diffusion distillation, but the 'unbiased gradient' is only unbiased for the upper bound, not the DiKL, and the toy evidence doesn't yet carry the weight. read the letter →

arxiv 2508.20646 v1 pith:ZFYYLL5Y submitted 2025-08-28 cs.LG

Leyang Wang , Mingtian Zhang , Zijing Ou , David Barber This is my paper · ORCID
classification cs.LG
keywords diffusiondistillationone-stepgenerationvariationalupperboundunbiasedgradientestimationdenoisingscorematchingdiffusiveKLdivergencenormalizingflowsimplicitgenerativemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most one-step diffusion distillation methods minimize a diffusive divergence by estimating the student's score through denoising score matching, and those estimates are imperfect, so the gradient is biased. This paper proposes VarDiU, an upper bound on the diffusive KL divergence whose gradient can be estimated without that score-matching bias: it needs only the teacher's score and a variational approximation to the student's posterior over latent codes. Because the student is an implicit one-step generator, the joint entropy term in the bound is constant, so no entropy estimation is needed. On a 2D mixture-of-40-Gaussians benchmark, VarDiU reports higher log-density and lower maximum mean discrepancy than Diff-Instruct under true-score and empirical-score settings, with tighter confidence intervals and faster convergence. If the result carries beyond the toy setting, it offers a way to train one-step generators that does not depend on how accurately a separate score network has been learned.

What carries the argument

The mechanism is the diffusive variational upper bound $U^{(t)}(\theta,\varphi)$, which avoids student-score estimation: unlike the DiKL gradient, it does not require the unavailable student score $\nabla_{x_t}\log p_{\theta}^{(t)}(x_t)$; instead it uses the variational posterior's log-density and the teacher score. The two load-bearing pieces are Proposition 3.1, which makes the joint entropy $H(p_{\theta}^{(t)}(x_t,z))$ a $\theta$-independent constant for Gaussian conditionals, so the objective needs no entropy network, and the score-reparameterization identity of Equation (11), which converts the unknown teacher density into a stop-gradient operation on the available teacher score. A neural spline flow posterior $q_{\varphi}(z|x_t)$ is used to make the bound tight by approximating the true posterior more closely than a Gaussian can.

What would settle it

On the 2D 40-Gaussian mixture, compute the gap $U^{(t)}(\theta,\varphi)-\mathrm{KL}(p_{\theta}^{(t)}\|p_d^{(t)})$ during training for both the Gaussian and flow posteriors; if the converged gap stays large with the Gaussian posterior and the flow posterior does not reduce it, then the unbiased estimator is not minimizing the claimed divergence and the quality gains would have to be attributed to the schedule or variance-reduction tricks.

Watch

Extended reading notes

Core claim

VarDiU's central claim is that the diffusive KL divergence $\mathrm{DiKL}(p_{\theta}\|p_d)=\int_0^1 \omega(t)\mathrm{KL}(p_{\theta}^{(t)}(x_t)\|p_d^{(t)}(x_t))\,dt$ can be minimized through the variational upper bound $U^{(t)}(\theta,\varphi)=\mathrm{KL}(p_{\theta}^{(t)}(x_t|z)p(z)\|p_d^{(t)}(x_t)q_{\varphi}^{(t)}(z|x_t))$, which holds for any variational posterior by Jensen's inequality and becomes tight exactly when $q_{\varphi}^{(t)}(z|x_t)=p_{\theta}^{(t)}(z|x_t)$. For an implicit student $p_{\theta}(x_0)=\int\delta(x_0-g_{\theta}(z))p(z)\,dz$ with a Gaussian conditional $p_{\theta}^{(t)}(x_t|z)=\mathcal{N}(x_t;g_{\theta}(z),\sigma_t^2 I)$, the joint entropy $H(p_{\theta}^{(t)}(x_t,z))$ is independent of $\theta$, so the bound reduces to a cross-entropy-like term that can be evaluated with only the teacher's score, using the gradient identity $\nabla_{\theta}\int p_{\theta}^{(t)}(x_t)\log p_d^{(t)}(x_t)\,dx_t = \nabla_{\theta}\int p_{\theta}^{(t)}(x_t)[x_t^{\top}\nabla_{x_t}\log p_d^{(t)}(x_t)]_{\mathrm{sg}}\,dx_t$. The paper further uses a normalizing-flow posterior to tighten the bound and interprets the objective through an information-maximization lens: minimizing the variational conditional entropy maximizes the student's marginal entropy. The reported experiments on a 2D 40-Gaussian mixture show that VarDiU, especially with a neural spline flow posterior, achieves higher sample log-density and lower MMD than Diff-Instruct when the teacher score is exact or empirical, and trains more stably and quickly.

Load-bearing premise

The training objective is faithful only when the auxiliary variational distribution over latent codes, given a noisy sample, closely matches the student's true distribution over latent codes; otherwise the upper bound is loose and the gradient can point away from the true diffusive divergence the method is meant to minimize.

Editorial extensions

If this is right

  • Distillation no longer depends on an accurately learned student score, so one source of gradient bias in methods like Diff-Instruct is removed.
  • The bound's tightness is controlled by the variational posterior, so investing in a more flexible posterior family directly improves the fidelity of the training signal.
  • Because the loss only needs the teacher score, the method applies whenever a pretrained diffusion model's score is available, without training an extra student score network.
  • With an annealed noise schedule that moves from coarse to fine scales, VarDiU reports stable convergence under a fixed generator-gradient budget and in wall-clock time.
  • On the 40-Gaussian benchmark, VarDiU achieves higher log-density and lower MMD than Diff-Instruct with 10 score steps, with smaller variance across runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test outside the paper is to scale VarDiU to image generation with a fixed teacher; if the toy gains persist, the method would remove the memory and compute overhead of a student score network in large distillation runs.
  • The information-maximization view suggests VarDiU could be adapted to any implicit generative model with an accessible teacher score, not only diffusion teachers, since the joint-entropy-constant property depends only on the Gaussian conditional form.
  • An ablation separating the unbiased bound from the annealed noise schedule and symmetric sampling would isolate how much of the reported stability comes from unbiasedness rather than variance reduction; the paper does not provide that decomposition.
  • The looseness of the bound under imperfect variational posteriors may matter more at high dimension; measuring the exact bound gap on a tractable high-dimensional target would show how much flexibility the posterior needs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes VarDiU, a variational upper bound for one-step diffusion distillation. The objective DiU (Eq. 9) upper-bounds the diffusive KL divergence (DiKL) by introducing an auxiliary variational posterior q_phi(z|x_t), with tightness when the posterior matches the true posterior. The authors show that the joint entropy of the diffused student is independent of the generator parameters (Prop. 3.1), derive a score-based gradient estimator (Eq. 11 and Appendix B), extend the variational family to normalizing flows (Sec. 3.2), and report experiments on a 2D mixture of 40 Gaussians comparing VarDiU with Diff-Instruct under true, empirical, and learned teacher scores. The paper claims that VarDiU yields higher generation quality and a more efficient and stable training procedure.

Significance. If the claims hold, the contribution is meaningful: it removes the separately learned student score network used by Diff-Instruct/VSD, provides an unbiased gradient estimator for a well-defined variational upper bound, and the entropy-constancy observation is clean and useful. The derivations in Appendices B and D appear correct, and the upper-bound inequality is a standard and valid variational argument. The practical significance is, however, not yet established: the evaluation is restricted to a low-dimensional toy, the learned-score setting (the one relevant to real distillation) shows mixed results, and the variational gap that controls the difference between the optimized objective and DiKL is never measured.

major comments (3)
  1. [Section 3.1 / Appendix A / Eq. (24)] The claimed unbiased gradient is for the upper bound U^{(t)}, not for the DiKL being distilled. From Eq. (24), U^{(t)} = KL(p_theta^{(t)} || p_d^{(t)}) + E_{x_t ~ p_theta^{(t)}} KL(p_theta^{(t)}(z|x_t) || q_phi^{(t)}(z|x_t)), and the second term is non-negative and theta-dependent. If q_phi cannot represent the true posterior, minimizing DiU may select generators with artificially simple posteriors rather than generators with lower DiKL, so the objective actually optimized can differ materially from the intended divergence. The manuscript's own Section 3.2 concedes that simple Gaussian posteriors 'struggle to approximate complex posteriors', yet no measurement of the residual gap is reported. I ask for an estimate of the gap (for example, using a stronger flow or a learned lower bound on the conditional KL) and a sensitivity analysis with respect to flow length, because the central claim that VarDiU is a faithful substitute for DiKL rests on this unverified tightness assumption.
  2. [Section 4 / Appendix F.2] The empirical comparison uses different training schedules for the two methods, so the reported gains in stability and efficiency are not attributable solely to the objective. VarDiU anneals rho from 0.1 to rho_end while Diff-Instruct keeps rho fixed at 1.5, and in the learned-score setting the noise bounds also differ (sigma_min = 1.5 for VarDiU vs sigma_min = 1.1 for Diff-Instruct). Annealing the noise distribution changes the effective weighting over sigma and can by itself stabilize training. The paper should either match the schedules or include an ablation in which both Diff-Instruct and VarDiU receive annealed and fixed schedules, and report the resulting MMD and log-density trajectories.
  3. [Section 4 / Table 1c] The abstract's claim that VarDiU 'achieves higher generation quality' than Diff-Instruct is not supported in the learned-score setting, which is the setting relevant to practical distillation from a pre-trained diffusion model. In Table 1c, VarDiU-NSF improves log-density (-7.89 vs -8.47) but has worse log-MMD (-5.68 vs -5.82) than Diff-Instruct with 10 score steps, and the paper itself describes the learned-score comparisons as unreliable. Combined with the fact that the experiments are on a 2D mixture of 40 Gaussians only, the current evidence does not establish that the method transfers to realistic one-step distillation tasks. At minimum, the abstract and conclusion should be qualified, and higher-dimensional experiments, or a clear argument for why the toy is predictive, are needed before the central practical claim can be accepted.
minor comments (4)
  1. [Section 3.1 / Eq. (12)] The notation U^{(t)} .= -E[x_t^T [score]_{sg} + log q_phi] is imprecise: as a value, this expression is not equal to U^{(t)} up to a constant for a non-Gaussian teacher density; only the gradient with respect to (theta, phi) coincides with the gradient of U^{(t)} under reparameterization. Please state explicitly that Eq. (12) is a gradient-equivalent surrogate, not a value-equivalent estimator of the bound.
  2. [Section 3.1 and Abstract] The 'unbiased' claim should be qualified: it holds conditional on the teacher score being exact. In the learned-score experiments, the estimator inherits the teacher's bias, as the paper partly acknowledges later; a sentence making this qualification explicit in the abstract or introduction would avoid overstatement.
  3. [Appendix F] Training details are incomplete for reproducibility: no code is provided, no random seeds are reported, and the number of score-matching inner steps for Diff-Instruct is described as 1, 5, or 10 without specifying the batch size or update rule for those inner steps. Please provide code or a complete configuration list.
  4. [Page 4, paragraph after Eq. (10)] The sentence 'The second term of Equation 9 cannot be directly optimised via automatic differentiation because the teacher density p_d is unknown' is slightly misleading; the issue is that the unnormalized teacher density is unavailable, and the score-based surrogate resolves this for gradient estimation. Consider rewording to avoid implying that automatic differentiation is the obstacle.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: VarDiU's bound and gradient estimator are derived from Jensen's inequality and reparametrization, with no fitted input presented as a prediction.

full rationale

The paper's central derivation is self-contained. The variational upper bound DiU is obtained from a Jensen/KL decomposition proved in Appendix A, and the paper explicitly shows U^(t) = KL(p_theta^(t) || p_d^(t)) + E[KL(p_theta^(t)(z|x_t) || q_phi^(t)(z|x_t))], so the bound is a genuine upper bound rather than a restatement of the target by construction. The 'unbiased gradient' claim concerns the chosen upper-bound objective, not the DiKL itself, and the paper states this distinction clearly; any looseness of the bound is a variational tightness question, not a circularity. The entropy constancy result (Proposition 3.1) and the score-based gradient identity (Eq. 11) are proved from explicit assumptions, not imported as predictions. The only self-citation to the earlier RKL upper bound [56] is not load-bearing because the inequality is re-derived in Appendix A. The empirical comparisons to Diff-Instruct are external evaluations on a toy task and do not encode their own success into the formulation. No fitted parameter is renamed as a prediction, no uniqueness theorem is invoked, and no ansatz is smuggled in via citation. The paper is therefore free of significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No invented entities. The method introduces an auxiliary variational encoder q_phi, but that is a standard variational component rather than a new physical or model entity. All free parameters are training hyperparameters; none are fitted to make the derivation produce a specific result.

free parameters (3)
  • Annealing schedule for rho = rho_init=0.1, increment 0.01 per 1000 epochs, rho_end 5.0 for true score and 2.0 for learned/dataset settings
    The noise-level schedule is annealed for VarDiU but fixed for Diff-Instruct; this is a method hyperparameter tuned per setting and is a confound in the comparison.
  • Noise schedule bounds sigma_min and sigma_max = True score: 0.1/20. Learned score: 1.5/40 for VarDiU, 1.1/40 for Diff-Instruct. Dataset: 0.65/40 for both.
    Different sigma_min for VarDiU versus Diff-Instruct in the learned-score setting; schedule choices affect training dynamics and the comparison.
  • Normalizing flow length = 4
    Depth of the Neural Spline Flow for VarDiU-NSF, chosen without sensitivity analysis.
assumptions (5)
  • domain assumption The teacher diffusion model provides an accurate score function for the marginal p_d(t)(x_t).
    The unbiased gradient result in Eq. (11) and Appendix B holds for the score that is plugged in; with a learned EDM score the estimate is not unbiased for the true data score, as the paper acknowledges in Section 4.
  • domain assumption The student's diffused conditional p_theta(x_t|z) is Gaussian with fixed covariance sigma_t^2 I, making the joint entropy independent of theta.
    Proposition 3.1 and Appendix D use this to drop the entropy term. It holds for implicit generators with Gaussian diffusion noise but would fail for learned covariances or non-Gaussian transition kernels.
  • domain assumption The variational posterior q_phi(z|x_t) can be made close to the true posterior p_theta(z|x_t).
    Tightness of the bound in Eq. (9) and Appendix A requires q_phi(z|x_t) = p_theta(z|x_t) almost everywhere; otherwise the objective is a loose upper bound.
  • standard math Reparametrization and interchange of expectation and gradient are valid for the generator network.
    Appendix B assumes differentiability of F_theta and appropriate regularity conditions for the gradient of E[log p_d].
  • standard math The empirical kernel-density score is a consistent estimator of the true score.
    Appendix F.3 relies on the law of large numbers and Gaussian kernel smoothing for the empirical score estimator.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VarDiU: A Variational Diffusive Upper Bound for One-Step Diffusion Distillation." pith.science (2026). https://pith.science/paper/ZFYYLL5Y

@misc{pith2026250820646,
  author       = {Pith},
  title        = {Pith review of: VarDiU: A Variational Diffusive Upper Bound for One-Step Diffusion Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZFYYLL5Y}},
  note         = {Machine review of arXiv:2508.20646}
}
read the original abstract

Recently, diffusion distillation methods have compressed thousand-step teacher diffusion models into one-step student generators while preserving sample quality. Most existing approaches train the student model using a diffusive divergence whose gradient is approximated via the student's score function, learned through denoising score matching (DSM). Since DSM training is imperfect, the resulting gradient estimate is inevitably biased, leading to sub-optimal performance. In this paper, we propose VarDiU (pronounced /va:rdju:/), a Variational Diffusive Upper Bound that admits an unbiased gradient estimator and can be directly applied to diffusion distillation. Using this objective, we compare our method with Diff-Instruct and demonstrate that it achieves higher generation quality and enables a more efficient and stable training procedure for one-step diffusion distillation.

Figures

Figures reproduced from arXiv: 2508.20646 by the authors.

Figure 1
Figure 1. The three rows correspond to the true score, training data, and learned score settings. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. The left figure shows log-MMD over 1M generator steps, reflecting the efficiency of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Density of sigma v.s different ρ. Here σmin = 0.1, σmax = 20. Unlike Variance Preserving (VP;[42, 15]) schedule which must satisfy α 2 t + σ 2 t = 1, we employ the schedule used in EDM [17] which is a popular class of diffusion models, where we set αt = 1 for all t and we sample the time variable t ∼ Unif[0, 1] and define the corresponding noise levels as σt = σmin + t ρ (σmax − σmin). (56) A visualisation of the re… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Log-density trajectory with true score seeting. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Comparison of samples log-density and log-MMD trajectories with learned score. [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Comparison of samples log-density and log-MMD trajectories under setting of given [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 43 canonical work pages

  1. [1]

    F. Bao, C. Li, J. Sun, J. Zhu, and B. Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07309, 2022

    arXiv 2022

  2. [2]

    F. Bao, C. Li, J. Zhu, and B. Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. arXiv preprint arXiv:2201.06503, 2022

    arXiv 2022

  3. [3]

    Barber and F

    D. Barber and F. Agakov. The im algorithm: a variational approach to information maximization.Advances in neural information processing systems, 2004

    work page 2004

  4. [4]

    Berthelot, A

    D. Berthelot, A. Autef, J. Lin, D. A. Yap, S. Zhai, S. Hu, D. Zheng, W. Talbott, and E. Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.04248, 2023

    arXiv 2023

  5. [5]

    V . D. Bortoli, A. Galashov, J. S. Guntupalli, G. Zhou, K. Murphy, A. Gretton, and A. Doucet. Distributional diffusion models with scoring rules, 2025

    work page 2025

  6. [6]

    Botev and A

    Z. Botev and A. Ridder. Variance reduction. Wiley statsRef: Statistics reference online, 136:476, 2017

    work page 2017

  7. [7]

    De Bortoli, J

    V . De Bortoli, J. Thornton, J. Heng, and A. Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 2021

    work page 2021

  8. [8]

    L. Dinh, D. Krueger, and Y . Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014

    arXiv 2014

Show all 61 references
  1. [9]

    Durkan, A

    C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. Advances in Neural Information Processing Systems, 2019

    2019

  2. [10]

    B. Efron. Tweedie’s formula and selection bias. Journal of the American Statistical Association , 106(496):1602–1614, 2011

    2011

  3. [11]

    Goodfellow, J

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Generative adversarial nets. Advances in neural information processing systems, 2014

    2014

  4. [12]

    Gretton, K

    A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The journal of machine learning research, (1):723–773, 2012

    2012

  5. [13]

    J. He, W. Chen, M. Zhang, D. Barber, and J. M. Hernández-Lobato. Training neural samplers with reverse diffusive kl divergence. In International Conference on Artificial Intelligence and Statistics. PMLR, 2025

    2025

  6. [14]

    J. Heek, E. Hoogeboom, and T. Salimans. Multistep consistency models. arXiv preprint arXiv:2403.06807, 2024

    2024 arXiv

  7. [15]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020

    2020

  8. [16]

    F. Huszár. Variational inference using implicit distributions. arXiv preprint arXiv:1702.08235, 2017

    2017 arXiv

  9. [17]

    Karras, M

    T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 2022

    2022

  10. [18]

    Kim, C.-H

    D. Kim, C.-H. Lai, W.-H. Liao, N. Murata, Y . Takida, T. Uesaka, Y . He, Y . Mitsufuji, and S. Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In International Conference on Learning Representations, 2024

    2024

  11. [19]

    Kobyzev, S

    I. Kobyzev, S. J. Prince, and M. A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(11):3964–3979, 2020

    2020

  12. [20]

    D. P. Kroese, T. Taimre, and Z. I. Botev. Handbook of monte carlo methods. John Wiley & Sons, 2013

    2013

  13. [21]

    Li and J

    L. Li and J. He. Bidirectional consistency models. arXiv preprint arXiv:2403.18035, 2024

    2024 arXiv

  14. [22]

    Lipman, R

    Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023

    2023

  15. [23]

    L. Liu, Y . Ren, Z. Lin, and Z. Zhao. Pseudo numerical methods for diffusion models on manifolds.arXiv preprint arXiv:2202.09778, 2022

    2022 arXiv

  16. [24]

    X. Liu, C. Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023. 7

    2023

  17. [25]

    C. Lu, Y . Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 2022

    2022

  18. [26]

    C. Lu, Y . Zhou, F. Bao, J. Chen, C. Li, and J. Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. Machine Intelligence Research, pages 1–22, 2025

    2025

  19. [27]

    S. Luo, Y . Tan, L. Huang, J. Li, and H. Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023

    2023 arXiv

  20. [28]

    W. Luo, T. Hu, S. Zhang, J. Sun, Z. Li, and Z. Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 2023

    2023

  21. [29]

    C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023

    2023

  22. [30]

    L. I. Midgley, V . Stimper, G. N. Simm, B. Schölkopf, and J. M. Hernández-Lobato. Flow annealed importance sampling bootstrap. In International Conference on Learning Representations, 2023

    2023

  23. [31]

    A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning. PMLR, 2021

    2021

  24. [32]

    Z. Ou, M. Zhang, A. Zhang, T. Z. Xiao, Y . Li, and D. Barber. Improving probabilistic diffusion models with optimal covariance matching. International Conference on Learning Representations, 2025

    2025

  25. [33]

    Rezende and S

    D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning. PMLR, 2015

    2015

  26. [34]

    H. E. Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pages 388–394. Springer, 1992

    1992

  27. [35]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022

    2022

  28. [36]

    Salimans and J

    T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022

    2022

  29. [37]

    Salimans, T

    T. Salimans, T. Mensink, J. Heek, and E. Hoogeboom. Multistep distillation of diffusion models via moment matching. arXiv preprint arXiv:2406.04103, 2024

    2024 arXiv

  30. [38]

    Sohl-Dickstein, E

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. PMLR, 2015

    2015

  31. [39]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021

    2021

  32. [40]

    Song and P

    Y . Song and P. Dhariwal. Improved techniques for training consistency models. InInternational Conference on Learning Representations, 2024

    2024

  33. [41]

    Y . Song, P. Dhariwal, M. Chen, and I. Sutskever. Consistency models. InInternational Conference on Machine Learning. PMLR, 2023

    2023

  34. [42]

    Song and S

    Y . Song and S. Ermon. Generative modeling by estimating gradients of the data distribution.Advances in Neural Information Processing Systems, 2019

    2019

  35. [43]

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  36. [44]

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. 2021

    2021

  37. [45]

    Stimper, D

    V . Stimper, D. Liu, A. Campbell, V . Berenz, L. Ryll, B. Schölkopf, and J. M. Hernández-Lobato. normflows: A pytorch package for normalizing flows. Journal of Open Source Software, 8(86):5361, 2023

    2023

  38. [46]

    E. G. Tabak and C. V . Turner. A family of nonparametric density estimation algorithms.Communications on Pure and Applied Mathematics, 66(2):145–164, 2013

    2013

  39. [47]

    E. G. Tabak and E. Vanden-Eijnden. Density estimation by dual ascent of the log-likelihood. Commun. Math. Sci., 8(1):217–233, 2010. 8

    2010

  40. [48]

    P. Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011

    2011

  41. [49]

    Z. Wang, C. Lu, Y . Wang, F. Bao, C. Li, H. Su, and J. Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 2024

    2024

  42. [50]

    Z. Xiao, K. Kreis, and A. Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804, 2021

    2021 arXiv

  43. [51]

    S. Xie, Z. Xiao, D. Kingma, T. Hou, Y . N. Wu, K. P. Murphy, T. Salimans, B. Poole, and R. Gao. Em distillation for one-step diffusion models. Advances in Neural Information Processing Systems, 2024

    2024

  44. [52]

    Y . Xu, W. Nie, and A. Vahdat. One-step diffusion models withf-divergence distribution matching. arXiv preprint arXiv:2502.15681, 2025

    2025 arXiv

  45. [53]

    T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, and B. Freeman. Improved distribution matching distillation for fast image synthesis. Advances in neural information processing systems, 2024

    2024

  46. [54]

    L. Yu, T. Xie, Y . Zhu, T. Yang, X. Zhang, and C. Zhang. Hierarchical semi-implicit variational inference with application to diffusion model acceleration. Advances in Neural Information Processing Systems, 2024

    2024

  47. [55]

    S. Zhai, R. Zhang, P. Nakkiran, D. Berthelot, J. Gu, H. Zheng, T. Chen, M. A. Bautista, N. Jaitly, and J. Susskind. Normalizing flows are capable generative models. arXiv preprint arXiv:2412.06329, 2024

    2024 arXiv

  48. [56]

    Zhang, T

    M. Zhang, T. Bird, R. Habib, T. Xu, and D. Barber. Variational f-divergence minimization. arXiv preprint arXiv:1907.11891, 2019

    1907 arXiv

  49. [57]

    Zhang, W

    M. Zhang, W. Chen, J. He, Z. Ou, J. M. Hernández-Lobato, B. Schölkopf, and D. Barber. Towards training one-step diffusion models without distillation. arXiv preprint arXiv:2502.08005, 2025

    2025

  50. [58]

    Zhang, P

    M. Zhang, P. Hayes, T. Bird, R. Habib, and D. Barber. Spread divergence. In International Conference on Machine Learning. PMLR, 2020

    2020

  51. [59]

    L. Zhou, S. Ermon, and J. Song. Inductive moment matching. In International Conference on Machine Learning, 2025

    2025

  52. [60]

    M. Zhou, H. Zheng, Y . Gu, Z. Wang, and H. Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. In International Conference on Learning Representations, 2025

    2025

  53. [61]

    VarDiU: A Variational Diffusive Upper Bound for One-Step Diffusion Distillation

    M. Zhou, H. Zheng, Z. Wang, M. Yin, and H. Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024. 9 Appendix for “VarDiU: A Variational Diffusive Upper B...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.