Direct Distributional Optimization for Provable Alignment of Diffusion Models

Atsushi Nitanda; Kazusato Oko; Ryotaro Kawata; Taiji Suzuki

REVIEW 2 major objections 4 minor 58 references

Direct Distributional Optimization for Provable Alignment of Diffusion Models

T0 review · 2 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Aligning diffusion models can be made provable at $O(1/K)$ without log-Sobolev constants.

desk verdict The DA + h-transform framework is original and technically solid, but the DPO/KTO headline examples violate the bounded-derivative assumption, so the convergence theorems don't actually cover the advertised applications. read the letter →

arxiv 2502.02954 v3 pith:HL7RGZ24 submitted 2025-02-05 cs.LG

Ryotaro Kawata , Kazusato Oko , Atsushi Nitanda , Taiji Suzuki This is my paper · ORCID

classification cs.LG

keywords dualaveragingDoobh-transformdirectpreferenceoptimizationreinforcementlearningfromhumanfeedbackKahneman-Tverskydiffusionmodelsisoperimetry-freesamplingdistributional

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that preference alignment of a diffusion model can be solved directly as a regularized optimization over probability distributions, with rigorous guarantees that existing mean-field methods lack. The algorithm never evaluates the model density: it runs dual averaging to update a potential $f_k$, stores the current aligned distribution implicitly as $q^{(k)} \propto \exp(-f_k) p_{\mathrm{ref}}$, and samples the final $q^{(K)}$ by adding a Doob $h$-transform correction to the pretrained score. The central theoretical claim is an end-to-end total-variation bound on the aligned sampler whose rate does not involve the log-Sobolev constant, meaning the sampling cost of the shifted distribution does not degrade with multimodality when the reference score is accurate. A sympathetic reader would care because the method targets the true DPO, RLHF, and KTO objectives rather than an upper-bound surrogate, and it comes with $O(1/K)$ convergence for convex losses and a stationarity guarantee for nonconvex losses.

What carries the argument

The engine of the method is the dual-averaging update over the space of densities: the ideal next distribution is $\hat q^{(k+1)} \propto \exp(-\bar g^{(k)}) p_{\mathrm{ref}}$, where $\bar g^{(k)}$ is a weighted average of past functional derivatives $\delta F/\delta q(q^{(j)})$; in practice a network $f_{k+1}$ is fit to $\bar g^{(k)}$ by regression on samples from $p_{\mathrm{ref}}$, and a one-step recurrence lets the algorithm carry only one potential. Sampling is then carried by Doob's $h$-transform: the score of the aligned density is $\nabla \log q_t(x) = \nabla \log p_t(x) + \nabla \log \mathbb{E}[\exp(-f^*(X^{\leftarrow}_T)) \mid X^{\leftarrow}_t = x]$, so only a correction term $u$ is added to the pretrained score network. The proof of the sampling bound transfers the score-discretization analysis of diffusion models to the tilted density $q \propto \rho p$ by controlling the smoothness of $\nabla \log q_t$ in terms of the smoothness of $\nabla \log p_t$ and the bounded density ratio $\rho$.

What would settle it

Take a two-mode Gaussian mixture where the ideal dual-averaging distributions $\hat q^{(k)}$ can be computed in closed form; measure the fitted distribution's total-variation distance to $\hat q^{(k)}$ and the KL discrepancy $D_{\mathrm{KL}}(\hat q^{(k)}\|\hat q^{(k+1)})$ at each $k$. If the discrepancy does not fall as $O(1/K)$, or falls only while the total-variation fitting error is pushed to zero, the load-bearing assumption on the fitted potentials is exposed.

Watch

Extended reading notes

Core claim

The paper's central claim is that the optimal aligned distribution $\hat q_{\mathrm{opt}} \propto \exp(-\hat f_{\mathrm{opt}}) p_{\mathrm{ref}}$ can be approached without any Markov-chain inner loop or any isoperimetric assumption. Theorem 1 gives $O(1/K)$ weighted convergence of the regularized objective for convex $F$; Theorem 2 gives, for nonconvex $F$, convergence of the KL discrepancy between successive ideal dual-averaging iterates at rate $O(1/K)$, which is interpreted as vanishing variance of the functional derivative; and Theorem 3 bounds the final sampling error in total variation by the score error, the density-ratio error, a discretization term, and an exponentially decaying initial-condition term, with no log-Sobolev constant. These results jointly assert that if the reference score is accurate, the complexity of sampling from the aligned distribution is independent of isoperimetric conditions.

Load-bearing premise

The proofs assume that the neural-network fits are close enough to the ideal dual-averaging iterates and that the gradient expectations are exact; the practical implementation approximates both with regression and Monte Carlo samples, so the proved guarantee may not bind for the algorithm as run.

Editorial extensions

If this is right

The paper's Theorem 1 gives $O(1/K)$ weighted convergence of the regularized objective for convex $F$, with an additive $O(\epsilon_{\mathrm{TV}})$ term that vanishes when the fitted potentials are exact.
The paper's Theorem 2 gives $\min_{1\le k\le K} D_{\mathrm{KL}}(\hat q^{(k)}\|\hat q^{(k+1)}) = O(1/K)$ even for nonconvex $F$, which is interpreted as convergence of the dual variable to a constant.
The paper's Theorem 3 bounds the final sampler's total-variation error by score error, discretization, density-ratio error, and an exponentially decaying initialization term, with no log-Sobolev constant.
The framework directly optimizes the true DPO, RLHF, and KTO objectives rather than the upper-bound or ELBO surrogates used by earlier diffusion fine-tuning methods.
Because the method only needs samples from $p_{\mathrm{ref}}$ and the learned potential, the same convergence and sampling bounds apply in principle to any differentiable distributional objective, including the covariate-shift and climate-tracking problems listed in the conclusion.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Pith inference: the same $O(1/K)$ dual-averaging updates could make diffusion models into a generic sampler for any mean-field distributional optimization, since the inner loop never needs to mix a Markov chain; the isoperimetry-free guarantee would then extend beyond alignment to any differentiable functional $F$.
Pith inference: a testable extension is to learn the Doob correction $u(x,t)$ with a network instead of nested Monte Carlo; Theorem 4 predicts the $u$-error should enter the total-variation bound through $\epsilon_{\rho,l}$, so one could measure whether a learned $u$ actually reduces the final sampler's total-variation error at small step size.
Pith inference: the bound suggests a regularization-dependent trade-off not explored in the experiments; as $\beta$ shrinks the target concentrates and the bounded-ratio constant $C_\rho$ should grow, so the sampling complexity of the correction term would be expected to rise even though the dual-averaging convergence rate does not.
Pith inference: the same machinery could run on covariate-shift density-ratio estimation by choosing $F$ as a divergence between shifted and reference distributions; the paper lists that as future work, and the framework's avoidance of density evaluations is exactly what the setting needs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things about this paper. First, the core idea is genuinely new: it replaces the usual Langevin-with-LSI approach by dual averaging plus Doob's h-transform to align diffusion models, and it derives direct functional derivatives of the population DPO and KTO losses. Second, the main theorems do not cover the headline examples. The stress-test is right: Assumption 1(i)/2(i) requires a global bound on the functional derivative, but the DPO derivative in Eq. (2) contains 1/(dq/dpref(x)), which blows up when the density ratio goes to zero. Since the space P contains densities with arbitrarily small ratios on the relevant sets, the sup over P is infinite. KTO has the same 1/r prefactor. So Theorems 1 and 2 apply only to functionals with uniformly bounded first variation — bounded-reward RLHF qualifies, DPO/KTO do not. The clipping used in the experiments is not part of the theory, so it doesn't fix the gap.

What is solid: the functional derivative derivations are careful; the h-transform sampling error analysis (Theorems 3 and 4) is substantial and uses Bismut-Elworthy-Li to avoid LSI, which is a real technical step beyond prior work. The DA convergence proof is a reasonable adaptation of Nitanda et al., though it leans on their Lemma 1 without proof. The experiments are honest but thin: one synthetic GMM baseline, no error bars, and the Phase 2 sampling is O(L^2) with 30000 Monte Carlo samples — expensive, as the authors acknowledge.

Other soft spots: the assumption that the implemented q^(k) is within TV distance epsilon_TV of the ideal DA iterate is simply assumed, with no procedure that guarantees it. The functional derivatives are estimated by Monte Carlo from a few thousand points, and no finite-sample error enters the main bounds. The paper's own appendix notes that the practical comparison is left for future work, so it does not overstate its empirical case.

The paper deserves a serious referee. The framework is original and the technical content is substantial enough to engage with, but the revision needs to either restrict the domain (e.g., densities with bounded density ratios) and re-prove the theorems under that restriction, or add a finite-sample analysis that connects the actual algorithm to the guarantees. As it stands, the provable-alignment claim for DPO/KTO is not supported. I would not cite it as a provable method for DPO, but the h-transform analysis may be useful for other work.

Recommendation: send it to peer review, expect major revision. The idea is worth engaging with, not desk-rejecting.

Referee Report

2 major / 4 minor

Summary. The paper proposes a distributional optimization framework for aligning diffusion models. It formulates alignment as minimizing F(q)+βDKL(q∥pref) over densities, solves this by a dual averaging (DA) scheme that iteratively constructs Gibbs distributions, and then samples from the final distribution by approximating its score through Doob's h-transform. The authors prove an O(1/K) convergence rate for convex F, a stationarity result for nonconvex F, and an end-to-end TV error bound for the sampling process that does not involve isoperimetric constants. They claim the framework applies to RLHF, DPO, and KTO, and they report experiments on a 2D Gaussian mixture and on image generation tasks.

Significance. If the stated guarantees were valid for the claimed applications, the paper would make a useful step toward isoperimetry-free alignment of diffusion models, complementing mean-field Langevin and particle dual averaging methods that rely on log-Sobolev inequalities. The h-transform discretization analysis is nontrivial and the appendix contains detailed proofs. However, the central advertised applications to DPO and KTO are not covered by the convergence theorems, and the practical algorithm is not shown to satisfy the assumptions under which the theorems are proved, so the significance as a provable alignment method for those objectives is severely diminished.

major comments (2)

[§3, Eq. (2), Propositions 2–3, Assumptions 1(i)/2(i)] Theorems 1 and 2 are conditional on the condition TV(ˆq(k), q(k)) ≤ ε_TV for all k, but Algorithm D.1 provides no mechanism that guarantees this bound. The algorithm trains f_{k+1} by minimizing an MSE objective against a finite-sample target, yet no regression error or sample-complexity analysis is supplied to show that the resulting distribution q(k) ∝ exp(−fk)pref is within ε_TV of the ideal DA iterate ˆq(k). Therefore the convergence theorems do not apply to the algorithm as actually implemented.
[§5 and §6] The convergence analysis assumes that the expectations E_q[δF/δq] appearing in the DA updates are evaluated exactly, while the experiments compute these expectations using empirical averages over finite datasets (e.g., 1000 points for the GMM and 6400 for the image tasks). No concentration or bias analysis is provided, so the end-to-end error bound in Theorem 3 does not cover the finite-sample Monte Carlo error that is present in the reported algorithm.

minor comments (4)

[Throughout] The term "Lipshitz" should be "Lipschitz" throughout the paper, including Assumptions 1 and 2.
[Theorem 2 statement] The display for Theorem 2 contains a typo: "min_{k=1,...,K} { c_kDKL(ˆq(k+1)∥ˆq(k) }" has mismatched braces and the inequality "≤( ˜L1(ˆq(1) − L(ˆqopt)) + ..." is missing a closing parenthesis.
[Appendix D.1] The phrase "the reference objectives 2 can be (and were) analytically calculated" should refer to "the reference objective in Eq. (2)" to avoid confusion.
[Figure 3 caption] The caption lists "iter=2" twice; one of these should presumably read "iter=1".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence and sampling bounds are derived from stated assumptions rather than re-imported from their conclusions.

full rationale

The derivation chain is not circular. The DA convergence results (Theorems 1 and 2) are proved in Appendix A from explicit assumptions on the functional derivative (boundedness, Lipschitz continuity in TV, convexity or lower boundedness) plus the TV approximation condition TV(qhat(k), q(k)) <= eps_TV. The proof is a telescoping-sum argument over the dual-averaging potential V_k, with the only imported ingredient being Lemma 1 from the authors' earlier PDA paper, an elementary inequality for KL-regularized linear functionals whose proof is omitted. That lemma does not itself assert the O(1/K) convergence, so the citation supports a technical inequality rather than re-importing the theorem being proved. The h-transform sampling bound (Theorem 3) is a new derivation: its right-hand side explicitly contains the input errors (score error epsilon, correction-term error epsilon_rho,l, discretization terms, and an exponentially damped initial-condition term), and Theorem 4 analyzes epsilon_rho,l instead of assuming the target TV bound. The aligned-score identity (Lemma 12) is the standard Doob h-transform relation and is derived from the backward-process definition. No fitted parameter is renamed as a prediction: f_k is iteratively regressed to the DA potential, and final alignment quality is reported as the directly optimized DPO objective, not as a value forced by construction. The one substantive weakness is that Assumptions 1(i) and 2(i) require a global bound on delta F / delta q, while the DPO and KTO functional derivatives (Eq. (2) and Proposition 3) contain the reciprocal density-ratio factor integral(e^{-f} d p_ref) / e^{-f(x)} and are therefore unbounded over the whole space of densities; this is an applicability gap in the headline examples, not a circularity, because the theorems do not assume the DPO/KTO conclusion and the experimental clipping of the derivative is not part of the theory.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

The central results rest on standard diffusion model assumptions plus several ad hoc conditions that connect the idealized DA theory to the practical algorithm. The most consequential ad hoc entries are the TV approximation error bound on the learned potentials and the assumption of exact functional derivative evaluations; both are load-bearing for the convergence theorems but not enforced by the implemented procedure.

free parameters (5)

beta (KL regularization coefficient) = 0.04 (GMM and butterflies), 0.01 (Head CT)
Chosen by hand in experiments; controls the trade-off between the objective and proximity to the reference distribution.
beta' (Dual Averaging hyperparameter) = 0.04 to 0.2
Controls DA learning speed; the paper reports the best result at beta' = 0.04.
gamma (DPO inverse temperature) = 0.1 (GMM, Head CT), 1.0 (butterflies)
Scales the log-density ratio in the DPO objective; no sensitivity analysis is provided.
Clipping thresholds for the functional derivative = +/-20 (butterflies), +/-5 (Head CT)
Ad hoc stabilization used when training the DA potential; not part of the theory.
Monte Carlo sample count for h-transform correction = 30000 (GMM), 128 (images)
Used in Phase 2 to estimate the conditional expectation defining u; the finite-sample error appears only in the appendix, not in the main theorems.

assumptions (7)

domain assumption F is differentiable with first variation deltaF/delta q bounded and Lipschitz in TV distance (Assumptions 1 and 2).
Required for the DA convergence theorems; not verified for DPO or KTO objectives in general, and boundedness can fail for unbounded rewards.
domain assumption There exists a unique minimizer q_opt of L(q) = F(q) + beta DKL(q || pref).
Stated in Section 2 without proof; needed to define the convergence target of the DA algorithm.
ad hoc to paper The implemented iterates satisfy TV(qhat^(k), q^(k)) <= epsilon_TV for all k.
Used in Theorems 1 and 2, but Algorithm D.1 only trains f_k by MSE regression and provides no guarantee of this TV bound.
domain assumption The density ratio rho* is bounded with C_rho^{-1} <= rho* <= C_rho and nabla log rho* is L_rho-smooth (Assumption 3.2).
Needed for the h-transform score decomposition and the smoothness bound in Lemma 8; can fail for strongly concentrated alignment rewards.
domain assumption Score estimation error is bounded by epsilon and the correction term error by epsilon_{rho,l} (Assumptions 3.3 and 3.4).
These are inputs to the end-to-end bound, not quantities derived from the method itself; the practical Monte Carlo error in u is not in the main statement.
ad hoc to paper The functional derivative expectations are evaluated exactly with respect to pref.
The algorithm replaces these expectations by empirical averages over a few thousand samples, but no finite-sample error analysis is supplied.
standard math Lemma 1 from Nitanda et al. (2021) is used without proof.
This published lemma provides the strong convexity of the regularized linearized objective; the paper cites it rather than reproving it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Direct Distributional Optimization for Provable Alignment of Diffusion Models." pith.science (2026). https://pith.science/paper/HL7RGZ24

@misc{pith2026250202954,
  author       = {Pith},
  title        = {Pith review of: Direct Distributional Optimization for Provable Alignment of Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HL7RGZ24}},
  note         = {Machine review of arXiv:2502.02954}
}

abstract

We introduce a novel alignment method for diffusion models from distribution optimization perspectives while providing rigorous convergence guarantees. We first formulate the problem as a generic regularized loss minimization over probability distributions and directly optimize the distribution using the Dual Averaging method. Next, we enable sampling from the learned distribution by approximating its score function via Doob's $h$-transform technique. The proposed framework is supported by rigorous convergence guarantees and an end-to-end bound on the sampling error, which imply that when the original distribution's score is known accurately, the complexity of sampling from shifted distributions is independent of isoperimetric conditions. This framework is broadly applicable to general distribution optimization problems, including alignment tasks in Reinforcement Learning with Human Feedback (RLHF), Direct Preference Optimization (DPO), and Kahneman-Tversky Optimization (KTO). We empirically validate its performance on synthetic and image datasets using the DPO objective.

Figures

Figures reproduced from arXiv: 2502.02954 by the authors.

**Figure 2.** In the simplest phase 2 that we present (Algorithm D.2), we used significant computational resources, with O(L 2 ) time complexity. To estimate the correction term u for each sample simultaneously by Monte Carlo with N samples, O(N) memory space is required. However, this Doob’s h-transform technique itself has been used in image generation (Uehara et al., 2024a;b), Bayesian samping (Heng et al., 2024), and filteri… view at source ↗

**Figure 2.** Left and Middle. The smoothed loss during optimization for Gaussian Mixture Model in Diffusion-DPO with/without regularization (left) and ours (middle). “True Objective”: the true DPO loss (Rafailov et al., 2023) whose target point was µw = [2.5, 0]. “Upperbound”: An approximate upperbound of “Objective” optimized by Diffusion-DPO (Wallace et al., 2024). Right. Aligned samples by Doob’s h-transform. “Ref.” represent… view at source ↗

Figures from the paper (12 more)

**Figure 3.** Figure 3: Left. Examples of aligned image generation. Our goal was to generate light-colored butterflies. “iter=2”: ours with k = 2 DA iterations, “iter=2”: ours with k = 1 DA iteration. “Ref.”: samples from pref. Right. Tilt-corrected Head CT image generation. “iter=3”: ours wi…

**Figure 4.** Figure 4: Left. Pre-train MSE loss of denoising score matching. The minimum losses until the current epoch were plotted. Right. The histogram of 20000 samples from the pre-trained DDPM. that the reference objectives2 can be (and were) analytically calculated because q/pref = 1. …

**Figure 5.** Figure 5: (For Reference) Losses without smoothing corresponding to Figure 2. “D-DPO”: [PITH_FULL_IMAGE:figures/full_fig_p045_5.png]

**Figure 6.** Figure 6: The heatmap of the potential fk in the kth loop in Dual Averaging for Gaussian Mixture Model. The target point was [2.5, 0]. Note that fk is the negated log-density ratio: the aligned density is exp(−fk)pref. Algorithm D.2 Doob’s h-transform (A simplest implementation)…

**Figure 8.** Figure 8: The pre-trained Model The model samples the images of butterflies of 32 × 32 pixels. Number of the sampling step was 1000. The objective The target color was [0.9, 0.9, 0.9] in RGB. The reward is visualized in [PITH_FULL_IMAGE:figures/full_fig_p047_8.png]

**Figure 7.** Figure 7: Left. Output images of pref sorted by the distance from the target color (= −reward). Right. Images sorted by the learned potential in k = 2. Doob’s h-transform We sampled the aligned images with 1000 diffusion steps. The guidance term of Doob’s h-transform was calcula…

**Figure 8.** Figure 8: Left. The smoothed loss during DA for image generation alignment from k = 1. “Objective”: DPO objective. “Regularized Objective”: “Objective” + βDKL(q∥pref), β = 0.05 Right. Examples of aligned image generation. “iter=2”: ours with k = 2 DA iterations, “iter=2”: ours …

**Figure 10.** Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p048_10.png]

**Figure 9.** Figure 9: Left. The MSE loss in pretraining the Autoencoder. The solid blue line and the dashed red represent the validation loss and the train loss. Right. The MSE loss in pretraining the diffusion model. The objective The reward was defined to be −|predicted angle| of the pre-…

**Figure 10.** Figure 10: Top. Outputs of pre-trained latent diffusion. Bottom. Decoded images of the outputs. epochs was 5. We compared the learned potential with the reference reward in [PITH_FULL_IMAGE:figures/full_fig_p049_10.png]

**Figure 11.** Figure 11: Left. Output CT images of the pre-trained model, sorted by the absolute values of estimated angles (= −reward). Right. Output CT Images sorted by the trained potential in the 3rd loop in DA. Doob’s h-transform We sampled the aligned images with 1000 diffusion steps. T…

**Figure 12.** Figure 12: Left. The smoothed loss during DA for tilt correction from k = 1. “Objective”: DPO objective. The target point was [2.5, 0]. “Regularized Objective”: “Objective” + βDKL(q∥pref), β = 0.01 Right. Tilt-corrected Head CT image generation. “iter=3”: ours with k = 3 DA iter…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 34 canonical work pages

[1]

Analysis and geometry of Markov diffusion operators, volume 348 of Grundlehren der mathematischen Wissenschaften

Dominique Bakry, Ivan Gentil, Michel Ledoux, et al. Analysis and geometry of Markov diffusion operators, volume 348 of Grundlehren der mathematischen Wissenschaften. Springer, 2014

work page 2014
[2]

Large deviations and the M alliavin calculus , volume 45 of Progress in Mathematics

Jean-Michel Bismut. Large deviations and the M alliavin calculus , volume 45 of Progress in Mathematics. Birkh\"auser Boston, Inc., Boston, MA, 1984

work page 1984
[3]

Training diffusion models with reinforcement learning, 2024

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning, 2024. URL https://arxiv.org/abs/2305.13301

arXiv 2024
[4]

Entropic fictitious play for mean field optimization problem

Fan Chen, Zhenjie Ren, and Songbo Wang. Entropic fictitious play for mean field optimization problem. Journal of Machine Learning Research, 24 0 (211): 0 1--36, 2023 a

work page 2023
[5]

Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions

Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pp.\ 4735--4763. PMLR, 2023 b

work page 2023
[6]

Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, 2023 c . URL https://openreview.net/forum?id=zyLVMgsZ0U_

work page 2023
[7]

Mean-field langevin dynamics : Exponential convergence and annealing

L \'e na \" c Chizat. Mean-field langevin dynamics : Exponential convergence and annealing. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=BDqzLH1gEm

work page 2022
[8]

Nicolas Chopin, Andras Fulop, Jeremy Heng, and Alexandre H. Thiery. Computational doob h-transforms for online filtering of discretely observed diffusions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro...

work page 2023

Show all 58 references

[9]

Directly fine-tuning diffusion models on differentiable rewards, 2024

Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differentiable rewards, 2024. URL https://arxiv.org/abs/2309.17400

2024 arXiv
[10]

Elworthy and X.M

K.D. Elworthy and X.M. Li. Formulae for the derivatives of heat semigroups. Journal of Functional Analysis, 125 0 (1): 0 252--286, 1994

1994
[11]

Kto: Model alignment as prospect theoretic optimization, 2024

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024. URL https://arxiv.org/abs/2402.01306

2024 arXiv
[12]

Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models

Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt,...

2023
[13]

Diffusion models for counterfactual generation and anomaly detection in brain images

Alessandro Fontanella, Grant Mair, Joanna Wardlaw, Emanuele Trucco, and Amos Storkey. Diffusion models for counterfactual generation and anomaly detection in brain images. IEEE Transactions on Medical Imaging, pp.\ 1--1, 2024. doi:10.1109/TMI.2024.3460391

2024
[14]

Diffusion Schrödinger Bridges for Bayesian Computation

Jeremy Heng, Valentin De Bortoli, and Arnaud Doucet. Diffusion Schrödinger Bridges for Bayesian Computation . Statistical Science, 39 0 (1): 0 90 -- 99, 2024. doi:10.1214/23-STS908. URL https://doi.org/10.1214/23-STS908

2024 doi
[15]

Denoising diffusion probabilistic models

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.\ 6840--6851. Curran Associates, Inc., 2020. URL http...

2020
[16]

Logarithmic sobolev inequalities and stochastic ising models

Richard Holley and Daniel Stroock. Logarithmic sobolev inequalities and stochastic ising models. Journal of statistical physics, 46 0 (5-6): 0 1159--1194, 1987

1987
[17]

Mean-field langevin dynamics and energy landscape of neural networks

Kaitong Hu, Zhenjie Ren, David S i s ka, and ukasz Szpruch. Mean-field langevin dynamics and energy landscape of neural networks. In Annales de l'Institut Henri Poincare (B) Probabilites et statistiques, volume 57, pp.\ 2043--2065. Institut Henri Poincar \'e , 2021

2021
[18]

The hugging face diffusion models course, 2022

HuggingFace. The hugging face diffusion models course, 2022. https://huggingface.co/course, 2022

2022
[19]

Aligning diffusion models by optimizing human utility, 2024 a

Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility, 2024 a . URL https://arxiv.org/abs/2404.04465

2024 arXiv
[20]

Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding, 2024 b

Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Shuiwang Ji, Aviv Regev, Sergey Levine, and Masatoshi Uehara. Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding, 2024 b . URL h...

2024 arXiv
[21]

Diffusion model-based probabilistic downscaling for 180-year east asian climate reconstruction

Fenghua Ling, Zeyu Lu, Jing-Jia Luo, Lei Bai, Swadhin K Behera, Dachao Jin, Baoxiang Pan, Huidong Jiang, and Toshio Yamagata. Diffusion model-based probabilistic downscaling for 180-year east asian climate reconstruction. npj Climate and Atmospheric Science, 7 0 (1): 0 131, 2024

2024
[22]

Johansson

Changxin Liu, Xuyang Wu, Xinlei Yi, Yang Shi, and Karl H. Johansson. Rate analysis of dual averaging for nonconvex distributed optimization. IFAC-PapersOnLine, 56 0 (2): 0 5209--5214, 2023 a . ISSN 2405-8963. doi:https://doi.org/10.1016/j.ifacol.2023.10.117. URL https://www.sc...

2023 doi
[23]

Polyak-- ojasiewicz inequality on the space of measures and convergence of mean-field birth-death processes

Linshan Liu, Mateusz B Majka, and ukasz Szpruch. Polyak-- ojasiewicz inequality on the space of measures and convergence of mean-field birth-death processes. Applied Mathematics & Optimization, 87 0 (3): 0 48, 2023 b

2023
[24]

Medical mnist classification, 2017

Arturo Polanco Lozano. Medical mnist classification, 2017. URL https://github.com/apolanco3225/Medical-MNIST-Classification

2017
[25]

Implicit diffusion: Efficient optimization through stochastic sampling, 2024

Pierre Marion, Anna Korba, Peter Bartlett, Mathieu Blondel, Valentin De Bortoli, Arnaud Doucet, Felipe Llinares-López, Courtney Paquette, and Quentin Berthet. Implicit diffusion: Efficient optimization through stochastic sampling, 2024. URL https://arxiv.org/abs/2402.05468

2024 arXiv
[26]

A mean field view of the landscape of two-layer neural networks

Song Mei, Andrea Montanari, and Phan\-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. In Proceedings of the National Academy of Sciences of the United States of America, volume 33, pp.\ E7665–E7671, 2018

2018
[27]

An introduction to evolution PDEs , Chapter 0: On the Gronwall lemma, 2019

St \'e phane Mischler. An introduction to evolution PDEs , Chapter 0: On the Gronwall lemma, 2019. URL https://www.ceremade.dauphine.fr/ mischler/Enseignements/M2evol2018/chap0.pdf

2019
[28]

Primal-dual subgradient methods for convex problems

Yurii Nesterov. Primal-dual subgradient methods for convex problems. Mathematical Programming, 120 0 (1): 0 221--259, Aug 2009

2009
[29]

Two-layer neural network on infinite dimensional data: global optimization guarantee in the mean-field regime

Naoki Nishikawa, Taiji Suzuki, Atsushi Nitanda, and Denny Wu. Two-layer neural network on infinite dimensional data: global optimization guarantee in the mean-field regime. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Informati...

2022
[30]

Improved particle approximation error for mean field neural networks

Atsushi Nitanda. Improved particle approximation error for mean field neural networks. arXiv preprint 2405.15767, 2024

2024 arXiv
[31]

Particle dual averaging: Optimization of mean field neural network with global convergence rate analysis

Atsushi Nitanda, Denny Wu, and Taiji Suzuki. Particle dual averaging: Optimization of mean field neural network with global convergence rate analysis. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processin...

2021
[32]

Convex analysis of the mean field langevin dynamics

Atsushi Nitanda, Denny Wu, and Taiji Suzuki. Convex analysis of the mean field langevin dynamics. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Pr...

2022
[33]

Primal and dual analysis of entropic fictitious play for finite-sum problems

Atsushi Nitanda, Kazusato Oko, Denny Wu, Nobuhito Takenouchi, and Taiji Suzuki. Primal and dual analysis of entropic fictitious play for finite-sum problems. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proce...

2023
[34]

Particle stochastic dual coordinate ascent: Exponential convergent algorithm for mean field neural network optimization

Kazusato Oko, Taiji Suzuki, Atsushi Nitanda, and Denny Wu. Particle stochastic dual coordinate ascent: Exponential convergent algorithm for mean field neural network optimization. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?i...

2022
[35]

Training language models to follow instructions with human feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

2022
[36]

Direct preference optimization: Your language model is secretly a reward model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neu...

2023
[37]

L. C. G. Rogers and David Williams. Diffusions, Markov Processes and Martingales. Cambridge Mathematical Library. Cambridge University Press, 2 edition, 2000

2000
[38]

Stochastic dual coordinate ascent methods for regularized loss minimization

Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research, 14 0 (1), 2013

2013
[39]

Promptable counterfactual diffusion model for unified brain tumor segmentation and generation with mris, 2024

Yiqing Shen, Guannan He, and Mathias Unberath. Promptable counterfactual diffusion model for unified brain tumor segmentation and generation with mris, 2024. URL https://arxiv.org/abs/2407.12678

2024 arXiv
[40]

Deep unsupervised learning using nonequilibrium thermodynamics

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of ...

2015
[41]

Generative modeling by estimating gradients of the data distribution

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, ...

2019
[42]

Score-based generative modeling through stochastic differential equations

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=PxTIG12RRHS

2021
[43]

Direct importance estimation for covariate shift adaptation

Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul Von B \"u nau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60: 0 699--746, 2008

2008
[44]

Convergence of mean-field langevin dynamics: time-space discretization, stochastic gradient, and variance reduction

Taiji Suzuki, Denny Wu, and Atsushi Nitanda. Convergence of mean-field langevin dynamics: time-space discretization, stochastic gradient, and variance reduction. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Proce...

2023
[45]

Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond, 2024

Wenpin Tang. Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond, 2024. URL https://arxiv.org/abs/2403.06279

2024
[46]

Direct density ratio estimation for large-scale covariate shift adaptation

Yuta Tsuboi, Hisashi Kashima, Shohei Hido, Steffen Bickel, and Masashi Sugiyama. Direct density ratio estimation for large-scale covariate shift adaptation. Journal of Information Processing, 17: 0 138--155, 2009

2009
[47]

Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review, 2024 a

Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review, 2024 a . URL https://arxiv.org/abs/2407.13734

2024 arXiv
[48]

Fine-tuning of continuous-time diffusion models as entropy-regularized control, 2024 b

Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous-time diffusion models as entropy-regularized control, 2024 b . URL https://arxiv.org/abs/2402.15194

2024 arXiv
[49]

Score-based generative modeling in latent space

Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp.\ 11287--11302. Curran Associates...

2021
[50]

Diffusers: State-of-the-art diffusion models

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers, 2022

2022
[51]

Diffusion model alignment using direct preference optimization

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and...

2024
[52]

Minimizing convex functionals over space of probability measures via KL divergence gradient flow

Rentian Yao, Linjun Huang, and Yun Yang. Minimizing convex functionals over space of probability measures via KL divergence gradient flow. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceedings of The 27th International Conference on Artificial Intelligence and...

2024
[53]

Yao, and Wenpin Tang

Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, and Wenpin Tang. Scores as actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning, 2024. URL https://arxiv.org/abs/2409.08400

2024 arXiv
[54]

Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020. URL https://arxiv.org/abs/1909.08593

2020 arXiv
[55]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[56]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[57]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[58]

dvipdf"] 0

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [1]

Analysis and geometry of Markov diffusion operators, volume 348 of Grundlehren der mathematischen Wissenschaften

Dominique Bakry, Ivan Gentil, Michel Ledoux, et al. Analysis and geometry of Markov diffusion operators, volume 348 of Grundlehren der mathematischen Wissenschaften. Springer, 2014

work page 2014

[2] [2]

Large deviations and the M alliavin calculus , volume 45 of Progress in Mathematics

Jean-Michel Bismut. Large deviations and the M alliavin calculus , volume 45 of Progress in Mathematics. Birkh\"auser Boston, Inc., Boston, MA, 1984

work page 1984

[3] [3]

Training diffusion models with reinforcement learning, 2024

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning, 2024. URL https://arxiv.org/abs/2305.13301

arXiv 2024

[4] [4]

Entropic fictitious play for mean field optimization problem

Fan Chen, Zhenjie Ren, and Songbo Wang. Entropic fictitious play for mean field optimization problem. Journal of Machine Learning Research, 24 0 (211): 0 1--36, 2023 a

work page 2023

[5] [5]

Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions

Hongrui Chen, Holden Lee, and Jianfeng Lu. Improved analysis of score-based generative modeling: User-friendly bounds under minimal smoothness assumptions. In International Conference on Machine Learning, pp.\ 4735--4763. PMLR, 2023 b

work page 2023

[6] [6]

Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions

Sitan Chen, Sinho Chewi, Jerry Li, Yuanzhi Li, Adil Salim, and Anru Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In The Eleventh International Conference on Learning Representations, 2023 c . URL https://openreview.net/forum?id=zyLVMgsZ0U_

work page 2023

[7] [7]

Mean-field langevin dynamics : Exponential convergence and annealing

L \'e na \" c Chizat. Mean-field langevin dynamics : Exponential convergence and annealing. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=BDqzLH1gEm

work page 2022

[8] [8]

Nicolas Chopin, Andras Fulop, Jeremy Heng, and Alexandre H. Thiery. Computational doob h-transforms for online filtering of discretely observed diffusions. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro...

work page 2023