REVIEW 2 major objections 5 minor 1 cited by
Score-of-Mixture Training: Training One-Step Generative Models Made Simple via Score Estimation of Mixture Distributions
T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper shows that one-step generative models can be trained from scratch in a stable way by minimizing alpha-skew Jensen--Shannon divergences, with the needed mixture-distribution scores learned by ordinary denoising score matching…
desk verdict A clean, well-executed one-step generative training framework that generalizes reverse-KL distillation to α-skew JSD; the central claim holds and the paper deserves serious refereeing despite the disclosed heuristics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the $\alpha$-skew Jensen--Shannon divergence, $D^{(\alpha)}_{\mathrm{JSD}}(q_\theta, p) = \frac{1}{\alpha}\mathrm{KL}(q_\theta \,\|\, \alpha p + (1-\alpha) q_\theta) + \frac{1}{1-\alpha}\mathrm{KL}(p \,\|\, \alpha p + (1-\alpha) q_\theta)$, whose gradient with respect to the generator parameters is expressed solely through scores of the mixture distribution minus the fake score. The load-bearing identity is Proposition 3.2: this mixture score is the minimizer of a convex combination of two denoising score-matching losses, one on real data and one on fake data, so it can be learned without access to normalizing constants. Conditioning the score network on both $\alpha$ and the noise level $t$ gives a single amortized model whose output, substituted into the gradient expression, updates the generator. In distillation, the explicit parameterization $s^{\mathrm{exp}}_\psi(x_t;\alpha, t) = D_\psi(x_t;\alpha, t)\,s_p(x_t;t) + (1-D_\psi(x_t;\alpha, t))\,s^{\mathrm{fake}}_\psi(x_t;t)$, with $D_\psi$ a sigmoid of a learned log-density-ratio, replaces the amortized model and makes the discriminator trainable through the same mixture score-matching objective.
What would settle it
On a two-dimensional Gaussian-mixture target, compute the exact mixture score and run two identical SMT trainings, one using the exact score in the generator update and one using the amortized network, then compare the achieved $\alpha$-skew JSD or sample quality; if the amortized run cannot match the oracle run, the score-approximation premise is falsified.
Extended reading notes
Core claim
The central claim is that minimizing the $\alpha$-skew Jensen--Shannon divergence between the generated distribution $q_\theta$ and the data distribution $p$, averaged over noise levels, is a practical and stable objective for one-step generative models. Proposition 3.1 gives the generator gradient as $\nabla_\theta D^{(\alpha)}_{\mathrm{JSD}}(q_\theta, p) = \frac{1}{\alpha}\mathbb{E}_{q(z)}[\nabla_\theta g_\theta(z)\,(s_{\theta;0}(x) - s_{\theta;\alpha}(x))|_{x=g_\theta(z)}]$, so the only quantity needed beyond the fake score is the score of the mixture distribution $s_{\theta;\alpha}(x) = \nabla_x \log(\alpha p(x) + (1-\alpha) q_\theta(x))$. Proposition 3.2 shows this mixture score is the minimizer of a weighted combination of denoising score-matching losses on real and fake samples, so it can be learned by an amortized network conditioned on both $\alpha$ and the noise level. In distillation, an explicit parameterization writes the mixture score as a sigmoid-weighted blend of the teacher score and a learned fake score, which simultaneously yields the log-density-ratio estimate needed for a GAN-style regularizer. The paper reports that this recipe yields stable training and FID scores competitive with or better than prior one-step methods on CIFAR-10 and ImageNet 64x64.
Load-bearing premise
The load-bearing premise is that the amortized score network is accurate enough, at every sampled $\alpha$ and noise level, that plugging its outputs into the generator update moves the model downhill on the divergence; if that approximation fails, the training objective loses its grounding.
Editorial extensions
If this is right
- One-step generators can be trained from scratch without simulating reverse diffusion or a probability-flow ODE, and the paper reports stable loss and gradient-norm curves throughout training.
- The from-scratch method SMT reaches FID 3.23 on ImageNet 64x64 with 296M parameters, matching iCT-deep with half the model size; the distillation method SMD reaches FID 1.48, outperforming several reverse-KL distillation baselines.
- Because the objective spans the full range of $\alpha$, it combines mode-covering and mode-seeking behaviors, avoiding the support-mismatch failure of pure forward or reverse KL while retaining the reverse-KL limit as a special case.
- The score learning objective uses both real and fake samples, which the paper argues removes the need for expensive ODE anchoring, reverse-process simulation, or lengthy finetuning stages used in some baselines.
- The distillation variant learns its discriminator through the mixture score-matching objective itself, so no separate discriminator loss is needed for score training, only an optional GAN-type regularizer on the generator.
Reading between the lines
- The paper samples $\alpha$ uniformly but never exploits the schedule; a natural untested extension would anneal $\alpha$ from 0 to 1 during training to shift from mode-covering to mode-seeking behavior as the generator improves.
- Because the method only requires a denoiser backbone, the same mixture-score objective should transfer to latent-space diffusion models, text, or audio if a suitable denoiser architecture exists, a direction the paper lists as future work.
- In low-dimensional problems the true mixture score is computable, so the amortized score network can be audited directly against the oracle; that comparison would show whether remaining sample-quality gaps come from score approximation or from the divergence objective itself.
- The paper trains only one-step samplers, but the learned mixture score at intermediate noise levels could plausibly be reused for a few-step refinement scheme, extending the framework beyond the one-step regime it currently claims.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Score-of-Mixture Training (SMT), a method for training one-step generative models by minimizing a family of α-skew Jensen–Shannon divergences between real and generated noisy distributions. The key idea is to estimate the score of the mixture distribution αp_t + (1−α)q_{θ,t} via denoising score matching, and to use this amortized score in the generator gradient. A distillation variant (SMD) leverages a pretrained diffusion model by parameterizing the mixture score explicitly with a learned discriminator and fake score. The authors report competitive FID on CIFAR-10 and ImageNet 64×64, with stable training curves, and release code. The mathematical framework is clearly laid out, but the implemented objective includes additional heuristics (an adaptive weighting and a GAN regularizer) that are not part of the pure divergence-minimization derivation.
Significance. If the claims hold, this is a valuable contribution to one-step generative modeling. The framework connects statistical divergence minimization with multi-scale denoising score matching, avoiding the need to simulate a reverse diffusion process or a probability-flow ODE. The paper includes formal propositions with proofs, a released codebase, and empirical results that are competitive with or better than strong baselines such as iCT and DMD2. The alternating training scheme is intuitive and the ablation studies help identify which components are responsible for the final performance. However, the current experimental validation covers the full pipeline (including the adaptive weighting and GAN regularizer), so the specific contribution of the pure α-JSD objective is less directly supported than the narrative suggests.
major comments (2)
- [Appendix A.1 / Proposition 3.1] The proof of Proposition 3.1 contains a display that is dimensionally inconsistent: the first expectation is written as E_{q_θ(x)}[∇_θ log q_θ(x) αp(x)+(1−α)q_θ(x)], which is a vector multiplied by a scalar density with no division or logarithm. The subsequent cancellation of terms leading to the final expression is not demonstrated. Since the generator gradient in Eq. (6) relies on this proposition, the proof must be rewritten carefully. The result itself appears to be a standard identity, but the paper as written does not provide a rigorous derivation.
- [Sec. 3.5 / Eqs. (9)–(11) / Fig. 2b] The actual training objective implemented in the paper is not exactly the α-JSD gradient derived in Eq. (6). The adaptive weight w_α in Eq. (10) and the GAN regularizer in Eq. (11) are additional heuristics, and the ablations in Fig. 2b show that these components are important for the reported FID (e.g., the curve with random α and DMD weighting converges noticeably worse than the full method). The paper should clearly state that the empirical evaluation validates the full pipeline, and it should provide a more direct measurement of the pure α-JSD objective's performance, or explicitly frame the method as a practical approximation to the theoretical objective. The concluding claim in Sec. 6 should be qualified accordingly.
minor comments (5)
- [Sec. 3.5] The description of the α-sampling is ambiguous: Eq. (6) divides by α, yet p(α) is said to be uniform over [0,1]. Please clarify that α > 0 for generator updates, or explain how α = 0 is handled in the generator gradient.
- [Corollary 4.1] Corollary 4.1 is not proved in the appendix or the main text. Since it is a corollary of Proposition A.2, a short proof would improve the paper's rigor.
- [Algorithm 1] The discriminator loss L_disc(ψ) is referenced in Algorithm 1 but not defined in the main text; the reader is directed to Appendix D.2. For the main text to be self-contained, include the explicit discriminator objective or at least the final expression used.
- [Table 3] Several hyperparameters that are likely important for reproducibility are not listed in Table 3, including the GAN regularizer weight λ and the score training sub-iteration count. Please add them or mention them in the text.
- [Appendix A.1] The proof of Proposition 3.1 also omits the intermediate steps for the term E_{q_θ}[∇_θ log q_θ(x) log(q_θ(x)/m_θ(x))], which is needed for the cancellation. A complete derivation would make the proof easier to verify.
Circularity Check
No significant circularity: the mixture-score estimator and generator gradient are derived from first principles and validated against external FID benchmarks.
full rationale
The derivation chain is self-contained. Proposition 3.1 (Sec. 3.1, Eq. 4) expresses the gradient of the alpha-skew JSD in terms of the score of the mixture s_{theta;alpha}; Proposition 3.2 (Sec. 3.3, Eq. 7) shows the weighted score-matching objective is minimized exactly at that mixture score, with the proof in Appendix A.2. The sample-only DSM objective in Eq. (8) is a standard Tweedie/MMSE estimator (Proposition A.1), so the score model is trained on an externally defined target (the mixture score of real p and current fake q_theta), not on the reported FID. The generator update in Eq. (6) substitutes the amortized score for the true mixture score; this is the usual critic-approximation assumption in score-based and adversarial training and is disclosed, not a fitted parameter renamed as a prediction. The adaptive weighting and GAN regularizer in Eqs. (9)-(11) are heuristics; their contribution is tested by ablation in Fig. 2b, and the final FID numbers (Table 2) are external benchmark comparisons. The warmup stage (Sec. 3.5) uses a data-only denoising objective, so it does not make the 'training from scratch' claim circular. No load-bearing self-citation or imported uniqueness theorem appears; citations such as Song & Dhariwal (2024b) for Fourier embeddings are non-central. The Limitations paragraph in Sec. 6 concerns few-step extension and other modalities, and does not undermine the derivation. Verdict: no significant circularity.
Assumptions & free parameters
free parameters (5)
- w_alpha(xt,t) adaptive weighting =
heuristic form from Eq. (10)
- GAN regularizer weight lambda =
not reported in Table 3
- warmup training steps =
15k (CIFAR-10), 40k (ImageNet)
- alpha sampling grid density and 25% alpha=0 rate =
1000 points, 25% zero
- score training sub-iterations =
5
assumptions (4)
- standard math E_{q_theta}[nabla_theta log q_theta] = 0
- domain assumption The amortized score network can represent the mixture score s_{theta;alpha,t} at all alpha and t
- standard math Tweedie's formula and denoising score matching equivalence
- domain assumption Alternating training converges to a useful fixed point
Cite this review
Pith. "Pith review of Score-of-Mixture Training: Training One-Step Generative Models Made Simple via Score Estimation of Mixture Distributions." pith.science (2026). https://pith.science/paper/IJHWFYFP
@misc{pith2026250209609,
author = {Pith},
title = {Pith review of: Score-of-Mixture Training: Training One-Step Generative Models Made Simple via Score Estimation of Mixture Distributions},
year = {2026},
howpublished = {\url{https://pith.science/paper/IJHWFYFP}},
note = {Machine review of arXiv:2502.09609}
}
abstract
We propose Score-of-Mixture Training (SMT), a novel framework for training one-step generative models by minimizing a class of divergences called the $\alpha$-skew Jensen--Shannon divergence. At its core, SMT estimates the score of mixture distributions between real and fake samples across multiple noise levels. Similar to consistency models, our approach supports both training from scratch (SMT) and distillation using a pretrained diffusion model, which we call Score-of-Mixture Distillation (SMD). It is simple to implement, requires minimal hyperparameter tuning, and ensures stable training. Experiments on CIFAR-10 and ImageNet 64x64 show that SMT/SMD are competitive with and can even outperform existing methods.
Figures
Figures from the paper (9 more)
Forward citations
Cited by 1 Pith paper
-
Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis
A new adversarial distribution matching loss for diffusion distillation gives one-step and few-step generators that match or exceed prior distillation methods on SDXL, SD3, and CogVideoX.
Reference graph
Works this paper leans on
-
[3]
Table 3.Hyperparameters used for training one-step generators with Score-of-Mixture Training and Distillation. Hyperparameter CIFAR-10 ImageNet64×64 Scratch Distillation Scratch Distillation Generator learning rate 1e-4 5e-5 5e-6 2e-6 Score learning rate 5e-4 5e-5 5e-5 2e-6 Score learning rate decay cosine None cosine None Batch size 280 280 280 280 Diffu...
work page 2020
-
[4]
Overview of Score-of-Mixture Distillation.Top:To update the generator weights, the fake image is diffused at noise level t and then used to compute the gradient of the α-skew divergence with the explicitly parametrized amortized score model using Eq.(15). Bottom:Amortized score model training involves computing the score of the mixture distribution on bot...
work page 2012
-
[5]
DPM- solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
Lu, C., Zhou, Y ., Bao, F., Chen, J., Li, C., and Zhu, J. DPM- solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. InAdv. Neural Inf. Proc. Syst., volume 35, pp. 5775–5787, 2022a. Lu, C., Zhou, Y ., Bao, F., Chen, J., Li, C., and Zhu, J. DPM- Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Aroun...
-
[6]
Nielsen, F. A Family of Statistical Symmetric Diver- gences based on Jensen’s Inequality.arXiv preprint arXiv:1009.4004,
-
[7]
Xie, S., Xiao, Z., Kingma, D. P., Hou, T., Wu, Y . N., Murphy, K. P., Salimans, T., Poole, B., and Gao, R. EM Distil- lation for One-Step Diffusion Models.arXiv Preprint arXiv:2405.16852,
-
[8]
12 Score-of-Mixture Training: Training One-Step Generative Models Made Simple A.2. Proof of Proposition 3.2 Proof of Proposition 3.2.We can write the objectiveL(ψ;α)as L(ψ;α) = Z n (αp(x) + (1−α)q θ(x))∥sψ(x;α)∥ 2 −2(αp(x)s p(x) + (1−α)q θ(x))⊺sψ(x;α) o dx+C = Z (αp(x) + (1−α)q θ(x)) sψ(x;α)− αp(x)sp(x) + (1−α)q θ(x)sqθ (x) αp(x) + (1−α)q θ(x) 2 dx+C ′. H...
work page 2015
-
[10]
and conditional normalizing flows trained with Gaussian conditional flow matching (Lipman et al., 2023), are also particular instances of (Gaussian) diffusion models with specialized noise schedules and weighted loss functions, as show in (Kingma & Gao, 2024). Formulation.We take the following unified view in our definition of DPMs as inspired by (Kingma & Gao,
work page 2023
-
[47]
We thus call the general divergence for α∈(0,1) the α-Le Cam distance
in the literature (Polyanskiy & Wu, 2019). We thus call the general divergence for α∈(0,1) the α-Le Cam distance. In the GAN literature, this is known as the LSGAN objective (Mao et al., 2017). As we revealed, our discriminator training in distillation can also be done separately using the α-Le Cam-distance-based objective. However, we conjecture that our...
work page 2019
Show all 15 references
-
[1956]
This process generates noise-image pairs, which are then used to further supervise the generator’s training
to express it in terms of a pretrained denoiserf ϕ and a denoiser for the fake samplesf ψ, ∇θLDMD(θ) =E q(z)p(t)q(ϵ) [wDMD(xt,x, t)∇θgθ(z)(fψ(xt;t)−f ϕ(xt;t))| x=gθ(z)], where an adaptive weight is used to ensure that the scale of the gradient is roughly uniform across noise l...
2023
-
[2009]
Dhariwal, P
doi: 10.1109/ CVPR.2009.5206848. Dhariwal, P. and Nichol, A. Diffusion Models Beat GAns On Image Synthesis. InAdv. Neural Inf. Proc. Syst., volume 34, pp. 8780–8794,
2009
-
[2014]
Several years later, Ho et al
based on the principles of thermodynamic diffusion with a Markov-chain variational posterior that maximizes the evidence lower bound (ELBO). Several years later, Ho et al. (2020) re-introduced DPMs (DDPMs) with modern neural network architectures and a simplified loss function...
2020
-
[2017]
A., Zhai, S., Hu, S., Zheng, D., Talbott, W., and Gu, E
Berthelot, D., Autef, A., Lin, J., Yap, D. A., Zhai, S., Hu, S., Zheng, D., Talbott, W., and Gu, E. TRACT: De- noising Diffusion Models with Transitive Closure Time- Distillation.arXiv Preprint arXiv:2303.04248,
-
[2021]
Karras, T., Aittala, M., Aila, T., and Laine, S
doi: 10.1109/TPAMI.2020.2970919. Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the Design Space of Diffusion-based Generative Models. InAdv. Neural Inf. Proc. Syst., volume 35, pp. 26565– 26577, 2022a. Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating...
2020
-
[2023]
Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.arXiv preprint arXiv:2209.03003,
Liu, X., Gong, C., and Liu, Q. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.arXiv preprint arXiv:2209.03003,
-
[2025]
Unlike distillation techniques, consistency models can also be trained from scratch
have shown that similar performance can be achieved by using theℓ 2 distance or a pseudo-Huber norm. Unlike distillation techniques, consistency models can also be trained from scratch. Assume that s=t−δt, δt→0 . Then, the sampling step can be approximated using Tweedie’s form...
1956
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.