Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A KL-regularized stochastic-control reformulation of diffusion fine-tuning admits a policy-iteration algorithm that converges to the unique optimal control at a linear rate and preserves regularity throughout training.

desk verdict Genuine theoretical contribution with a solid contraction proof; the experiments don't instantiate the theorem and the parametric rate is conjectural, but the core convergence guarantee holds up and deserves review. read the letter →

arxiv 2412.18164 v4 pith:DXUKLRVW submitted 2024-12-24 cs.LG math.OC

Yinbin Han , Meisam Razaviyayn , Renyuan Xu This is my paper · ORCID
classification cs.LGmath.OC MSC 93E2049L2090C40
keywords stochasticcontroldiffusionmodelfine-tuningKLregularizationpolicyiterationlinearconvergenceDDPMregularityrewardalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to place diffusion-model fine-tuning on a provable footing by recasting it as a discrete-time stochastic control problem. The pre-trained denoising diffusion model acts as a fixed reference; the fine-tuned score at each step is the control, a terminal reward scores the final sample, and a per-step Kullback–Leibler penalty keeps the new dynamics close to the pre-trained one. Because the KL term between Gaussian conditionals collapses to a weighted squared distance between control and pre-trained score, the one-step Bellman objective becomes strongly concave once the regularization coefficient is large enough. That concavity yields a unique optimal control, and the paper's policy-iteration algorithm (PI-FT) converges to it globally at rate $O((1-\lambda)^M)$ after $M$ sweeps, while every iterate provably keeps the same Lipschitz and gradient-Lipschitz regularity as the optimum. A sympathetic reader should care because this is presented as the first convergence guarantee of its kind for diffusion fine-tuning, and the proof shows the regularity is generated by the algorithm rather than assumed.

What carries the argument

The load-bearing object is the Bellman operator for the KL-regularized problem, viewed as a fixed-point map on controls. Lemma 2.5 converts the KL divergence between two Gaussian transitions into $\frac{(1-\alpha_t)^2}{2\alpha_t\sigma_t^2}\lVert u_t(y)-s_t^{\mathrm{pre}}(y)\rVert_2^2$, so the right-hand side of the Bellman equation becomes a strongly concave function of $u_t(y)$ whenever $\beta_t>\sigma_t^2 L^{\bar V}_{1,t+1}$. The first-order condition then gives the implicit optimal-control equation (18), and the same equation defines the inner-loop update (35) of Algorithm 1: $u_t^{(m+1)}(y)=s_t^{\mathrm{pre}}(y)+\frac{\sqrt{\alpha_t}\sigma_t^2}{(1-\alpha_t)\beta_t}\mathbb{E}[\nabla V_{t+1}^{(m_{t+1})}(\tfrac{1}{\sqrt{\alpha_t}}(y+(1-\alpha_t)u_t^{(m)}(y))+\sigma_tW_t)]$. The proof propagates regularity backward by differentiating under the Gaussian expectation via an integration-by-parts identity, so the smoothing effect of Gaussian noise turns Lipschitz and gradient-Lipschitz assumptions into the same properties for the next value function and control.

What would settle it

Take $d=1$ with constant $\alpha_t$ and $\sigma_t$, a smooth bounded reward with known Lipschitz constants, and a gradient-Lipschitz pre-trained score; solve the fixed-point equation (18) by fine-grid collocation to get $u_t^*$, run Algorithm 1 with $m_t=M$, and measure $\sup_y\lVert u_t^{(M)}(y)-u_t^*(y)\rVert_2$. If the supremum does not shrink like $\Theta((1-\lambda)^M)$ up to the stated constants, Theorem 3.1 fails. Alternatively, running the same experiment with $r(y)=\|y\|^2$, which violates global gradient-Lipschitzness of the reward, tests the necessity of the smoothness assumption.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is Theorem 3.1: under global smoothness of the expected reward and the pre-trained scores, if each regularization weight $\beta_t$ is chosen so that $1 - \sigma_t^2 L^{\bar V}_{1,t+1}/\beta_t \ge \lambda_t > 0$, then the controls returned by Algorithm 1 satisfy $\lVert u_t^{(m_t)}(y)-u_t^*(y)\rVert_2 \le \left((1-\lambda_t)^{m_t}L^{V^*}_{0,t+1}+\lambda_t^{-1}E_{t+1}\right)\frac{\sqrt{\alpha_t}(1-\lambda_t)}{(1-\alpha_t)L^{V^*}_{1,t+1}}$, with $E_t$ bounded by a sum of geometrically decaying terms of the form $C_{2,k}(1-\lambda_k)^{m_k+1}$. With $m_t=M$ for all $t$, this gives $\lVert u_t^{(M)}-u_t^*\rVert_2 = O((1-\lambda)^M)$, uniformly in the state. The same backward induction proves existence and uniqueness of the optimal control, characterized by the implicit equation $u_t^*(y)=s_t^{\mathrm{pre}}(y)+\frac{\sqrt{\alpha_t}\sigma_t^2}{(1-\alpha_t)\beta_t}\mathbb{E}[\nabla V_{t+1}^*(\tfrac{1}{\sqrt{\alpha_t}}(y+(1-\alpha_t)u_t^*(y))+\sigma_tW_t)]$, and shows that both the optimal value function and every iterate of the algorithm remain Lipschitz and gradient-Lipschitz throughout training.

Load-bearing premise

The load-bearing premise is that the expected reward and every pre-trained score are globally smooth, meaning bounded slopes and bounded slope changes across all of $\mathbb{R}^d$, and that the reward function is known exactly; if any of these fails (a learned reward, a score network with unbounded gradients, or unbounded reward curvature), the contraction and regularity estimates in Theorems 2.8 and 3.1 no longer hold, and the paper explicitly leaves the unknown-reward case to future work.

Editorial extensions

If this is right

  • With $M$ updates per time step, the returned controls are within $O((1-\lambda)^M)$ of the unique optimal control uniformly over states, so fine-tuning quality is controlled by a single geometric factor determined by the regularization schedule.
  • At no iteration do the value functions or controls lose regularity: their Lipschitz and gradient-Lipschitz constants stay bounded by those of the optimal solution, so the optimization landscape cannot become wilder along the trajectory.
  • The regularization coefficient $\beta_t$ (equivalently $\lambda$) sets an explicit trade-off: larger $\beta_t$ gives faster convergence and keeps the fine-tuned model closer to the pre-trained one, matching the paper's numerical experiments.
  • In the linear-parameterization extension, the sketch identifies a unique stationary point of the expected-return objective under a realizability assumption and conjectures a linear rate for policy gradient, so the same concavity structure transfers from abstract control space to parameter space.
  • In continuous time, the optimal control takes the same form $u_t^*(y)=s_t^{\mathrm{pre}}(y)+\frac{\sqrt{\alpha_t}\sigma_t^2}{\beta_t(1-\alpha_t)}\partial_y V(t,y)$ and the HJB equation mirrors the discrete-time recursion, pointing to an analogous theory in the continuous-time limit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper assumes the reward is known exactly; a natural extension it leaves implicit is estimating $\mathbb{E}[\nabla V_{t+1}^*]$ from samples and treating estimation error as an additive $E_{t+1}$ term, which would turn the guarantee into $O((1-\lambda)^M + \text{estimation error})$ for a stochastic variant of PI-FT.
  • Because the contraction argument only requires strong concavity of the one-step objective, the framework should extend to other penalties whose Hessian is lower-bounded, not just KL; the paper hints at this in its continuous-time discussion but does not prove it.
  • A testable prediction is that on real score networks with large but bounded Lipschitz constants, observed convergence degrades monotonically with the ratio $\sigma_t^2 L^{\bar V}_{1,t+1}/\beta_t$, so measuring that ratio before training would predict how many inner iterations $M$ suffice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies fine-tuning of pre-trained diffusion models through a finite-horizon stochastic control problem with linear dynamics and KL regularization between conditional Gaussian transition kernels. The main theoretical results are: well-posedness and Lipschitz/gradient-Lipschitz regularity of the optimal value function and optimal control under global smoothness assumptions on the expected reward and the pre-trained score; a policy iteration algorithm (PI-FT); and a linear convergence guarantee for that algorithm in functional space, with explicit constants and preservation of regularity for all iterates. The paper also sketches a linear-parameterization extension and a continuous-time limit, and reports Stable Diffusion/ImageReward experiments.

Significance. If the main theorem is correct, the contraction/regularity-propagation argument is a genuine contribution: it gives explicit constants, proves that the iterates maintain Lipschitz and gradient-Lipschitz regularity rather than assuming it, and obtains a global linear convergence rate for a non-concave, continuous-state control problem. The paper is also honest about the known-reward assumption and about the fact that the continuous-time and parametric sections are exploratory. The code availability is a further positive feature. The main reservations concern a terminal-value gap in the proof of Theorem 2.8, the fact that Section 5 does not instantiate the theorem's assumptions or the algorithm's exact update rule, and a small but real indexing error in Algorithm 1.

major comments (3)
  1. [Section 2.2, proof of Theorem 2.8] The base case of the backward induction is not valid as written. The proof states that Assumption 2.6 implies V*_T(y) = R(y) is L_r0-Lipschitz and L_r1-gradient Lipschitz, but Assumption 2.6 is stated for r(y) = E[R(y)], not for R(y). If R is genuinely stochastic (as suggested by the notation R ∈ L1(Rd,F0)), then R(y) need not be Lipschitz with the constants L_r0,L_r1. This is load-bearing because every subsequent regularity step depends on the terminal case. The repair is local: define the terminal value by V*_T(y) = r(y), replace R(Y_T) by r(Y_T) in (8)-(9), or explicitly assume the reward is a known deterministic function. Algorithm 1 already initializes V_T = r, so the intended theory is clear, but Theorem 2.8 as stated is not fully proved.
  2. [Section 5 and Theorem 3.1] The experimental section does not instantiate Theorem 3.1, and its claims should be reframed. The experiments use Stable Diffusion with LoRA and AdamW, which do not implement the functional update (35)-(36); they use ImageReward, a learned neural reward, which is not the known expected reward r of Assumption 2.6; and the global Lipschitz and gradient-Lipschitz constants of ImageReward and of the score network are neither verified nor plausible for functions on all of R^d. Consequently, the observation 'a linear convergence rate is observed' in Figure 3a and the statement that the experiments 'align with our theoretical analysis' are heuristic claims, not tests of the theorem. The paper should either present Section 5 as a proof-of-concept under an idealized reward model, or add explicit caveats that the assumptions of Theorem 3.1 are not checked and that the implemented update differs from the analyzed one.
  3. [Algorithm 1, lines 4-8] The loop header 'for m = 1, ..., m_t - 1' is not consistent with the rest of the paper. If m_t = 1, no iteration is executed and u_t^{(m_t)} is never defined, yet line 8 uses u_t^{(m_t)}. The intended loop is presumably 'for m = 0, ..., m_t - 1' so that u_t^{(m_t)} is produced after m_t updates. This is an indexing error, but it affects the reproducibility of the algorithm as printed.
minor comments (5)
  1. [Section 2.1, Eq. (3)] The notation 'y 0 ∼ N(0, Id)' should read 'Y_0 ∼ N(0, I_d)' for consistency with the state notation.
  2. [Lemma 2.5] The phrase 'divergencd' should be 'divergence'.
  3. [Section 4.1] The parametric section is a roadmap and explicitly conjectures a linear rate. The Conclusion should not state that the framework has been extended to parametric settings as a proven result; the current wording overstates the status of Section 4.1.
  4. [Section 4.2] The continuous-time HJB derivation in (78)-(79) is formal and should be labeled as such, since no existence, uniqueness, or regularity theorem is proved for the continuous-time equation.
  5. [Section 2.2] The proof of Theorem 2.8 invokes the dynamic programming principle and measurable selection without stating or proving these steps. A remark or a reference for the measurable-selection argument would help readers verify the passage from the supremum in (8) to the pointwise Bellman equation (9).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the convergence and regularity results follow from the stated smoothness assumptions through contraction and backward-induction arguments, with no fitted parameter or self-citation used to force the conclusion.

full rationale

The paper is not circular in its central derivation. Theorem 2.8 constructs the Lipschitz constants in (14)-(17) by backward induction from Assumptions 2.6 and 2.7, and the condition 1 - sigma_t^2/beta_t LbarV_{1,t+1} >= lambda_t > 0 is a stated sufficient condition on the regularization coefficient, not a parameter fitted to the desired conclusion. The fixed-point characterization (18) follows from first-order optimality in the Bellman equation, and the contraction property (67) is derived from the gradient Lipschitzness of V*_{t+1} and the choice of beta_t; it is a proved inequality, not an assumed one. Lemma 3.2 establishes regularity of the iterated controls and value functions from the regularity of the input bV_{t+1} via explicit recursions and the integration-by-parts formula, so no regularity is imported as an ansatz. The error recursions in Lemmas 3.3 and 3.4 and Theorem 3.1 are also derived rather than assumed. The only self-citations in the paper, e.g., [29] and [30], appear in related-work and motivation contexts and are not load-bearing for the convergence proof. The principal limitations are that the theory assumes a known, globally smooth expected reward and globally smooth pre-trained score, while the experiments use ImageReward, LoRA, and AdamW; this is an applicability gap, not circularity, because the empirical section does not claim to instantiate the theorem's exact functional update. Overall, no circular step was found.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central theorem rests on smoothness and known-reward assumptions, the Gaussian smoothing identity, and the choice of sufficiently large KL penalties. The control-as-score language is interpretive rather than a new physical entity. The free parameters are regularization, contraction-rate, and iteration-count hyperparameters, not constants fitted to data to force the result.

free parameters (3)
  • beta_t (KL regularization coefficient) = 0.01, 0.1, 1.0 in experiments; theory requires it to be sufficiently large
    User-chosen coefficient in the objective (2) that makes the Bellman step strongly concave and controls how close the fine-tuned model stays to the pretrained model.
  • lambda_t (contraction rate parameter) = not specified; lambda_t in (0,1)
    User-chosen lower bound in the condition 1 - sigma_t^2/beta_t * L >= lambda_t. It sets the linear rate (1-lambda_t) and trades off against the required size of beta_t.
  • m_t (inner iterations per timestep) = set to M in the simplified rate statement
    Algorithm input controlling the number of inner control updates at each timestep; the convergence bound is stated as a function of m_t.
assumptions (5)
  • domain assumption The expected reward r(y) = E[R(y)] is Lr0-Lipschitz and Lr1-gradient Lipschitz.
    Assumption 2.6. Provides the terminal regularity of V*_T and the gradient-Lipschitz control needed for the strong concavity and contraction estimates.
  • domain assumption Each pretrained score function s_t is Ls0,t-Lipschitz and Ls1,t-gradient Lipschitz.
    Assumption 2.7. Used throughout Theorem 2.8 and Lemma 3.2 to propagate Lipschitz and gradient-Lipschitz bounds for controls and value functions.
  • domain assumption The expected reward r is known exactly and R is in L1; no reward estimation error is modeled.
    Stated in Section 2.1 and Remark 2.3. The unknown-reward case is explicitly left for future work, so the convergence theorem does not cover the effect of a learned reward model.
  • standard math Gaussian smoothing and integration by parts: E[grad V(z + sigma W)] is differentiable with derivative E[Hess V(z + sigma W)], and the identity E[grad V(z + sigma W) W^T] = sigma E[Hess V(z + sigma W)] holds.
    Invoked in Theorem 2.8 Step 2 and Lemma 3.2 Step 1 via [22, Theorem 2.27]. Requires the Gaussian noise scale sigma_t to be positive.
  • domain assumption The fine-tuned process follows the linear Gaussian dynamics (3), the pretrained DDPM is exactly described by (1), and Markovian policies are sufficient for optimality.
    Assumed in Section 2.1. The Bellman recursion, the KL simplification in Lemma 2.5, and the pathwise KL interpretation in Remark 2.4 all depend on this generative model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence." pith.science (2026). https://pith.science/paper/DXUKLRVW

@misc{pith2026241218164,
  author       = {Pith},
  title        = {Pith review of: Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXUKLRVW}},
  note         = {Machine review of arXiv:2412.18164}
}
read the original abstract

Diffusion models have emerged as powerful tools for generative modeling, demonstrating exceptional capability in capturing target data distributions from large datasets. However, fine-tuning these massive models for specific downstream tasks, constraints, and human preferences remains a critical challenge. While recent advances have leveraged reinforcement learning algorithms to tackle this problem, much of the progress has been empirical, with limited theoretical understanding. To bridge this gap, we propose a stochastic control framework for fine-tuning diffusion models. Building on denoising diffusion probabilistic models as the pre-trained reference dynamics, our approach integrates linear dynamics control with Kullback-Leibler regularization. We establish the well-posedness and regularity of the stochastic control problem and develop a policy iteration algorithm (PI-FT) for numerical solution. We show that PI-FT achieves global convergence at a linear rate. Unlike existing work that assumes regularities throughout training, we prove that the control and value sequences generated by the algorithm maintain the regularity. Additionally, we explore extensions of our framework to parametric settings and continuous-time formulations, and demonstrate the practical effectiveness of the proposed PI-FT algorithm through numerical experiments. Our code is available at https://github.com/yinbinhan/fine-tuning-of-diffusion-models.

Figures

Figures reproduced from arXiv: 2412.18164 by the authors.

Figure 1
Figure 1. Comparison of the ImageReward Score among three models: pre-trained Stable Diffusion (red), [PITH_FULL_IMAGE:figures/full_fig_p025_1.png] view at source ↗
Figure 2
Figure 2. Visual comparison of images generated by the original Stable Diffusion model (pre-trained), DPOK [PITH_FULL_IMAGE:figures/full_fig_p026_2.png] view at source ↗
Figure 3
Figure 3. (a) Gradient norm (logarithmic scale) of PI-FT during training. The curves are smoothed using [PITH_FULL_IMAGE:figures/full_fig_p026_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Randomly generated samples from PI-FT model with different KL regularization coefficients. [PITH_FULL_IMAGE:figures/full_fig_p027_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Is RL fine-tuning harder than regression? A PDE learning approach for diffusion models

    cs.LG 2025-09 conditional novelty 7.0 of 10

    For diffusion model fine-tuning, RL value estimation reduces to a variational inequality whose solution satisfies a supervised-learning oracle inequality with self-mitigating statistical error.

  2. Statistical guarantees for continuous-time policy evaluation: blessing of ellipticity and new tradeoffs

    cs.LG 2025-02 conditional novelty 6.0 of 10

    For continuous-time policy evaluation, the LSTD estimator's H1 error scales as the square root of (approximation error plus m/T), with a trajectory length that can be nearly linear in the number of basis functions whe...

Reference graph

Works this paper leans on

81 extracted references · 53 canonical work pages · cited by 2 Pith papers

  1. [41]

    On convergence analysis of policy iteration algorithms for entropy-regularized stochastic control problems

    Jin Ma, Gaozhan Wang, and Jianfeng Zhang. On convergence analysis of policy iteration algorithms for entropy-regularized stochastic control problems. arXiv preprint arXiv:2406.10959 , 2024

    arXiv 2024

  2. [1]

    A green colored rabbit

    Alekh Agarwal, Sham M. Kakade, Jason Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22:98:1–98:76, 2021. 25 Figure 2: Visual comparison of images generated by the original Stable Diffusion model (pre-trained), DPOK model, and PI-FT model (ours). P...

    2021

  3. [2]

    A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications

    Laith Alzubaidi, Jinshuai Bai, Aiman Al-Sabaawi, Jose Santamar ´ ıa, Ahmed Shihab Albahri, Bashar Sami Nayyef Al-dabbagh, Mohammed A Fadhel, Mohamed Manoufali, Jinglan Zhang, Ali H Al-Timemy, et al. A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications. Journal of Big Data , 10(1):46, 2023

    work page 2023

  4. [3]

    A systematic review on data scarcity problem in deep learning: solution and applications

    Ms Aayushi Bansal, Dr Rewa Sharma, and Dr Mamta Kathuria. A systematic review on data scarcity problem in deep learning: solution and applications. ACM Computing Surveys (Csur) , 54(10s):1–29, 2022

    work page 2022

  5. [4]

    An optimal control perspective on diffusion-based generative modeling

    Julius Berner, Lorenz Richter, and Karen Ullrich. An optimal control perspective on diffusion-based generative modeling. 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=oYIjw37pTP

    work page 2024

  6. [5]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf , 2(3):8, 2023

    2023

  7. [6]

    Four wolves in the park

    Jalaj Bhandari and Daniel Russo. On the linear convergence of policy gradient methods for finite MDPs. In International Conference on Artificial Intelligence and Statistics , pages 2386–2394. PMLR, 2021. 26 Figure 4: Randomly generated samples from PI-FT model with different KL regularization coefficients. Images from a single text prompt: “Four wolves in...

    work page 2021

  8. [7]

    Global optimality guarantees for policy gradient methods

    Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. Operations Research, 2024

    work page 2024

Show all 81 references
  1. [8]

    Training diffusion models with reinforcement learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations ,

  2. [9]

    LQR through the lens of first order methods: Discrete-time case

    Jingjing Bu, Afshin Mesbahi, Maryam Fazel, and Mehran Mesbahi. LQR through the lens of first order methods: Discrete-time case. arXiv preprint arXiv:1907.08921 , 2019

    1907 arXiv

  3. [10]

    Fast global convergence of natural policy gradient methods with entropy regularization

    Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research, 2021

    2021

  4. [11]

    Emu: Enhancing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807 , 2023

    2023 arXiv

  5. [12]

    Diffusion models beat GANs on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems , 34:8780–8794, 2021

    2021

  6. [13]

    Jovanovi´ c

    Dongsheng Ding, Kaiqing Zhang, Tamer Ba¸ sar, and Mihailo R. Jovanovi´ c. Natural policy gradient primal-dual method for constrained Markov decision processes. In NeurIPS, 2020

    2020

  7. [14]

    Tutorial on variational autoencoders

    Carl Doersch. Tutorial on variational autoencoders. arXiv preprint arXiv:1606.05908 , 2016

    2016 arXiv

  8. [15]

    Adjoint matching: Fine- tuning flow and diffusion generative models with memoryless stochastic optimal control

    Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine- tuning flow and diffusion generative models with memoryless stochastic optimal control. In Inter- national Conference on Learning Representations , 2025. URL https://openreview.net...

    2025

  9. [16]

    Generative Adversarial Network (GAN): A general review on different variants of GAN and applications

    M Durgadevi et al. Generative Adversarial Network (GAN): A general review on different variants of GAN and applications. In 2021 6th International Conference on Communication and Electronics Systems (ICCES) , pages 1–8. IEEE, 2021

    2021

  10. [17]

    Optimizing DDPM Sampling with Shortcut Fine-Tuning

    Ying Fan and Kangwook Lee. Optimizing DDPM Sampling with Shortcut Fine-Tuning. In International Conference on Machine Learning , pages 9623–9639. PMLR, 2023

    2023

  11. [18]

    Reinforcement learning for fine-tuning text-to-image diffusion models

    Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems , 36, 2024. 27

    2024

  12. [19]

    Stochastic policy gradient methods: Improved sample complexity for fisher-non-degenerate policies

    Ilyas Fatkhullin, Anas Barakat, Anastasia Kireeva, and Niao He. Stochastic policy gradient methods: Improved sample complexity for fisher-non-degenerate policies. In International Conference on Machine Learning, pages 9827–9869. PMLR, 2023

    2023

  13. [20]

    Global convergence of policy gradient methods for the linear quadratic regulator

    Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467–1476. PMLR, 2018

    2018

  14. [21]

    Understanding the limita- tions of conditional generative models

    Ethan Fetaya, Joern-Henrik Jacobsen, Will Grathwohl, and Richard Zemel. Understanding the limita- tions of conditional generative models. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=r1lPleBFvH

    2020

  15. [22]

    Real analysis: modern techniques and their applications , volume 40

    Gerald B Folland. Real analysis: modern techniques and their applications , volume 40. John Wiley & Sons, 1999

    1999

  16. [23]

    Single-timescale actor-critic provably finds glob- ally optimal policy

    Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Single-timescale actor-critic provably finds glob- ally optimal policy. In International Conference on Learning Representations , 2021. URL https: //openreview.net/forum?id=pqZV_srUVmK

    2021

  17. [24]

    Current strategies to address data scarcity in artificial intelligence-based drug discovery: A comprehen- sive review

    Amit Gangwal, Azim Ansari, Iqrar Ahmad, Abul Kalam Azad, and Wan Mohd Azizi Wan Sulaiman. Current strategies to address data scarcity in artificial intelligence-based drug discovery: A comprehen- sive review. Computers in Biology and Medicine , 179:108734, 2024

    2024

  18. [25]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835–10866. PMLR, 2023

    2023

  19. [26]

    Reward-directed score-based diffusion models via q-learning

    Xuefeng Gao, Jiale Zha, and Xun Yu Zhou. Reward-directed score-based diffusion models via q-learning. arXiv preprint arXiv:2409.04832 , 2024

    2024 arXiv

  20. [27]

    Fast policy learning for linear quadratic regulator with entropy regularization

    Xin Guo, Xinyu Li, and Renyuan Xu. Fast policy learning for linear quadratic regulator with entropy regularization. arXiv preprint arXiv:2311.14168 , 2023

    2023

  21. [28]

    Policy gradient methods for the noisy linear quadratic regulator over a finite horizon

    Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods for the noisy linear quadratic regulator over a finite horizon. SIAM Journal on Control and Optimization , 59(5):3359–3391, 2021

    2021

  22. [29]

    Policy gradient converges to the globally optimal policy for nearly linear-quadratic regulators

    Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Policy gradient converges to the globally optimal policy for nearly linear-quadratic regulators. arXiv preprint arXiv:2303.08431 , 2023

    2023 arXiv

  23. [30]

    Neural network-based score estimation in diffusion models: Optimization and generalization

    Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Neural network-based score estimation in diffusion models: Optimization and generalization. In International Conference on Learning Representations ,

  24. [31]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021. URL https://openreview.net/forum?id= qw8AKxfYbI

    2021

  25. [32]

    URL https://openreview.net/forum?id=h8GeqOxtd4

  26. [33]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  27. [34]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , volume 33, pages 6840–6851, 2020

    2020

  28. [35]

    Neural tangent kernel: Convergence and general- ization in neural networks

    Arthur Jacot, Franck Gabriel, and Cl´ ement Hongler. Neural tangent kernel: Convergence and general- ization in neural networks. Advances in neural information processing systems , 31, 2018. 28

    2018

  29. [36]

    Estimation of non-normalized statistical models by score matching

    Aapo Hyv¨ arinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005

    2005

  30. [37]

    Aligning text-to-image models using human feedback

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mo- hammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 , 2023

    2023 arXiv

  31. [38]

    Provably efficient reinforcement learning with linear function approximation

    Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory , pages 2137–2143. PMLR, 2020

    2020

  32. [39]

    Neural trust region/proximal policy optimization attains globally optimal policy

    Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in Neural Information Processing Systems , 32, 2019

    2019

  33. [40]

    Towards non-asymptotic convergence for diffusion- based generative models

    Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Towards non-asymptotic convergence for diffusion- based generative models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=4VGEeER6W9

    2024

  34. [42]

    An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods

    Yanli Liu, Kaiqing Zhang, Tamer Ba¸ sar, and Wotao Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. In NeurIPS, 2020

    2020

  35. [43]

    Global expo- nential convergence of gradient methods over the nonconvex landscape of the linear quadratic regulator

    Hesameddin Mohammadi, Armin Zare, Mahdi Soltanolkotabi, and Mihailo R Jovanovi´ c. Global expo- nential convergence of gradient methods over the nonconvex landscape of the linear quadratic regulator. In 2019 IEEE 58th Conference on Decision and Control (CDC) , pages 7474–7479....

    2019

  36. [44]

    Bartlett, and Martin J

    Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter L. Bartlett, and Martin J. Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. Journal of Machine Learning Research , 21:21:1–21:51, 2019

    2019

  37. [45]

    On Bellman equations for continuous-time policy evaluation I: discretiza- tion and approximation

    Wenlong Mou and Yuhua Zhu. On Bellman equations for continuous-time policy evaluation I: discretiza- tion and approximation. arXiv preprint arXiv:2407.05966 , 2024

    2024 arXiv

  38. [46]

    Improved sample complexity analysis of natural policy gra- dient algorithm with general parameterization for infinite horizon discounted reward markov decision processes

    Washim U Mondal and Vaneet Aggarwal. Improved sample complexity analysis of natural policy gra- dient algorithm with general parameterization for infinite horizon discounted reward markov decision processes. In International Conference on Artificial Intelligence and Statistics...

    2024

  39. [47]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨ uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023

    2023 arXiv

  40. [48]

    Sora: Creating video from text

    OpenAI. Sora: Creating video from text. https://openai.com/sora, 2024

    2024

  41. [49]

    Hierarchical text- conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022

    2022 arXiv

  42. [50]

    Random features for large-scale kernel machines

    Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007

    2007

  43. [51]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  44. [52]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨ orn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684–10695, 2022

    2022

  45. [53]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015

    2015

  46. [54]

    Low-rank adaptation for fast text-to-image diffusion fine-tuning

    Simo Ryu. Low-rank adaptation for fast text-to-image diffusion fine-tuning. 2023. URL https:// github.com/cloneofsimo/lora. 29

    2023

  47. [55]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Con- ference on Learning Representations, 2021. URL https://openreview.net/forum?id=Px...

    2021

  48. [56]

    Generative modeling by estimating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019

    2019

  49. [57]

    Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models

    Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. Exploration-exploitation trade-off for continuous-time episodic reinforcement learning with linear-convex models. arXiv preprint arXiv:2112.10264, 2021

    2021 arXiv

  50. [58]

    Support vector machines

    Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008

    2008

  51. [59]

    Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond

    Wenpin Tang. Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond. arXiv preprint arXiv:2403.06279 , 2024

    2024

  52. [60]

    Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning

    Lukasz Szpruch, Tanut Treetanthiploet, and Yufei Zhang. Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning. SIAM Journal on Control and Optimization , 62(1):135–166, 2024

    2024

  53. [61]

    Feedback efficient online fine- tuning of diffusion models

    Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Sergey Levine, and Tommaso Biancalani. Feedback efficient online fine- tuning of diffusion models. In Forty-first International Conference on Machine Learning...

    2024

  54. [62]

    Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review

    Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding reinforcement learning-based fine-tuning of diffusion models: A tutorial and review. arXiv preprint arXiv:2407.13734, 2024

    2024 arXiv

  55. [63]

    Diffusion model alignment using direct pref- erence optimization

    Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct pref- erence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision a...

    2024

  56. [64]

    Bridging model-based optimization and generative modeling via conservative fine-tuning of diffusion models

    Masatoshi Uehara, Yulai Zhao, Ehsan Hajiramezanali, Gabriele Scalia, G¨ okcen Eraslan, Avantika Lal, Sergey Levine, and Tommaso Biancalani. Bridging model-based optimization and generative modeling via conservative fine-tuning of diffusion models. arXiv preprint arXiv:2405.196...

    2024 arXiv

  57. [65]

    Neural policy gradient methods: Global optimality and rates of convergence

    Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=BJgQfkSYDS

    2020

  58. [66]

    Reinforcement learning in continuous time and space: A stochastic control approach

    Haoran Wang, Thaleia Zariphopoulou, and Xunyu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. The Journal of Machine Learning Research, 21(1):8145–8178, 2020

    2020

  59. [67]

    On the convergence rates of policy gradient methods

    Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research, 23(282):1–36, 2022. 30

    2022

  60. [68]

    De novo design of protein structure and function with rfdiffusion

    Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089–1100, 2023

    2023

  61. [69]

    Doubly robust off-policy actor-critic: Convergence and optimality

    Tengyu Xu, Zhuoran Yang, Zhaoran Wang, and Yingbin Liang. Doubly robust off-policy actor-critic: Convergence and optimality. In ICML, 2021

    2021

  62. [70]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems , 36:15903–15935, 2023

    2023

  63. [71]

    Some fine properties of backward stochastic differential equations

    Jianfeng Zhang. Some fine properties of backward stochastic differential equations . Purdue University, 2001

    2001

  64. [72]

    Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence

    Wenhao Zhan, Shicong Cen, Baihe Huang, Yuxin Chen, Jason D Lee, and Yuejie Chi. Policy mirror descent for regularized reinforcement learning: A generalized framework with linear convergence. SIAM Journal on Optimization , 33(2):1061–1091, 2023

    2023

  65. [73]

    Variational policy gradient method for reinforcement learning with general utilities

    Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. Advances in Neural Information Processing Systems, 33:4572–4583, 2020

    2020

  66. [74]

    Backward stochastic differential equations

    Jianfeng Zhang. Backward stochastic differential equations . Springer, 2017

    2017

  67. [75]

    Scores as actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning

    Hanyang Zhao, Haoxian Chen, Ji Zhang, David D Yao, and Wenpin Tang. Scores as actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning. arXiv preprint arXiv:2409.08400, 2024

    2024 arXiv

  68. [76]

    Provably efficient actor-critic for risk-sensitive and robust adversarial RL: A linear-quadratic case

    Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Provably efficient actor-critic for risk-sensitive and robust adversarial RL: A linear-quadratic case. In International Conference on Artificial Intelligence and Statistics , pages 2764–2772. PMLR, 2021

    2021

  69. [77]

    A policy gradient framework for stochastic optimal control problems with global convergence guarantee

    Mo Zhou and Jianfeng Lu. A policy gradient framework for stochastic optimal control problems with global convergence guarantee. arXiv preprint arXiv:2302.05816 , 2023

    2023 arXiv

  70. [78]

    Adding conditional control to diffusion models with reinforcement learning

    Yulai Zhao, Masatoshi Uehara, Gabriele Scalia, Tommaso Biancalani, Sergey Levine, and Ehsan Haji- ramezanali. Adding conditional control to diffusion models with reinforcement learning. arXiv preprint arXiv:2406.12120, 2024

    2024 arXiv

  71. [79]

    Solving time-continuous stochastic optimal control problems: Algorithm design and convergence analysis of actor-critic flow

    Mo Zhou and Jianfeng Lu. Solving time-continuous stochastic optimal control problems: Algorithm design and convergence analysis of actor-critic flow. arXiv preprint arXiv:2402.17208 , 2024. 31

    2024 arXiv

  72. [80]

    Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees

    Mo Zhou and Jianfeng Lu. Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees. Journal of Machine Learning Research , 24(222):1–34, 2023

    2023

  73. [2024]

    URL https://openreview.net/forum?id=YCWjhGrJFD

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.