Pith. sign in

REVIEW 4 major objections 6 minor 26 references

LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LC-GRPO keeps stochastic RL rollouts as sharp as test-time ODE sampling by adding one Langevin correction per step.

desk verdict Candidly: the method is genuinely useful and the experiments are decent, but the paper's theoretical guarantee applies to a smaller step-size schedule than the one implemented, and the missing code and error bars make the empirical claims hard to verify. read the letter →

arxiv 2608.05600 v1 pith:XGJTF76I submitted 2026-08-06 cs.LG cs.AIcs.CV

Yingqing Guo , Hui Yuan , Zijian He , Mengdi Wang , Zheng Ding This is my paper · ORCID
classification cs.LGcs.AIcs.CV MSC 68T0760H1060J60
keywords flowmatchinggrouprelativepolicyoptimizationLangevincorrectiontrain-inferencegapreinforcementlearningtext-to-imagegenerationtext-to-videoWassersteindistance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LC-GRPO is a reinforcement-learning recipe for flow-based generative models that fixes a mismatch between training and inference: training rollouts are stochastic, while test-time sampling is a deterministic ODE, and naive stochastic rollouts turn blurry as exploration noise grows. The paper replaces each rollout step with the same ODE Euler step used at inference, followed by one Langevin correction step that adds stochasticity while targeting the marginal distribution at the new timestep. Because the score for that correction is read off from the flow model's own velocity, no extra score network is needed, and the resulting transition remains a Gaussian whose density is tractable for the Group Relative Policy Optimization (GRPO) objective. The paper proves the corrected rollout is closer to the target marginal than both the ODE step and the matched reverse-SDE step, and reports higher rewards on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo.

What carries the argument

The central object is the per-step transition kernel $$p_\$\theta$(x_{t-\$\Delta$ t}\mid x_t,c)=\mathcal{N}(m_\$\theta$(x_t),2\epsilon_t I_d),$$ with mean $m_\theta(x_t)=x'+\epsilon_t s_\theta(x',t-\Delta t)$, where $x'=x_t-\Delta t\,v_\theta(x_t,t)$ is the ODE Euler output. The score is recovered from the flow velocity by Tweedie's formula, $$s(x,t)=-\frac{x+(1-t)v_\$\theta$(x,t)}{t},$$ so no auxiliary score network is trained. The Langevin step size obeys $\sqrt{2\epsilon_t}=\eta(t-\Delta t)$, making $\eta$ the exploration knob; the theorem's contraction conditions (Assumption 1 and the step-size bounds in Eq. 9) are what make the correction provably contract toward $p_{t-\Delta t}$. This kernel carries the argument because it keeps transport identical to inference while adding a tunable, likelihood-tractable source of stochasticity.

What would settle it

Compare LC-GRPO rollouts with deterministic ODE samples on the same prompts at matched noise: if the Langevin-corrected rollouts are not closer, in Wasserstein distance or in a perceptual image-quality metric, to ODE outputs than standard SDE rollouts are, or if the reward gains vanish when the score estimate is replaced by a numerically verified exact score, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that stochastic exploration for flow-model reinforcement learning need not be purchased with sampling accuracy. The paper's sampler decouples transport from exploration: each rollout transition first takes the exact ODE Euler step the inference-time sampler would take, then applies a single Langevin step targeting the marginal at the new time, using a score obtained from the flow velocity. This makes the transition an isotropic Gaussian, so GRPO's likelihood ratio stays tractable, while the sample stays close to the deterministic ODE trajectory. Theorem 3 shows that one Langevin step strictly decreases the Wasserstein error of an imperfect Euler step, and Theorem 4 shows that, at matched noise, the corrected step beats the standard Euler-Maruyama SDE step; experiments confirm smaller train-evaluation reward gaps and higher rewards across image and video models.

Load-bearing premise

The practical algorithm assumes the score recovered from the model's velocity output is the exact score of the target distribution at each step, and that the fixed step size $\sqrt{2\epsilon_t}=\eta(t-\Delta t)$ stays inside the theorem's contraction regime, which depends on an Euler error the algorithm never measures.

Editorial extensions

If this is right

  • Training rewards become a more faithful signal: LC-GRPO rollouts are close to test-time ODE samples, so the reward measured during training is closer to the reward obtained at evaluation.
  • Exploration can be increased without the usual blur: because the Langevin step size controls exploration and the ODE part preserves the inference trajectory, larger $\eta$ should accelerate optimization without degrading rollout quality.
  • No extra score model or architectural change is required; the correction costs one extra velocity evaluation per step and is compatible with any flow model, as shown on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo.
  • The same GRPO objective and baselines are preserved, so the method is a drop-in sampler replacement for existing flow-based GRPO pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The predictor-corrector split is not tied to GRPO itself: any stochastic-policy update that needs a tractable transition density for flow models could reuse the same ODE-then-Langevin kernel, provided the score estimate is good enough.
  • Because the theory assumes the exact score, the practical method inherits a hidden dependency on the velocity model's accuracy; a testable safeguard would be to track the score residual during training and shrink $\eta$ when it grows.
  • The comparison condition in Theorem 4 is stated but not verified on the trained models, so part of the empirical gain could come from the Langevin sampler injecting more total noise than the SDE at the same per-step $\eta$; an ablation that matches total noise variance would cleanly separate the two explanations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes LC-GRPO for reinforcement-learning fine-tuning of flow-based generative models. Each rollout transition first takes an inference-aligned ODE Euler step and then applies one Langevin correction targeting the marginal distribution at the new timestep, with the score obtained from the learned flow velocity via Tweedie's formula. The transition remains an isotropic Gaussian with a tractable likelihood, so it can be used as the stochastic policy in GRPO. The authors prove in Appendix A that, under strong log-concavity and exact scores, one Langevin correction step reduces the W2 error of an ODE Euler step (Theorem 3) and can beat a matched reverse-SDE Euler–Maruyama step under an explicit comparison condition (Theorem 4). Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo report consistent reward improvements and a smaller training–evaluation gap relative to Flow-GRPO, DanceGRPO, and CPS.

Significance. The core idea of separating deterministic transport from stochastic exploration is attractive and practically motivated: it directly targets the known degradation of SDE rollouts at high exploration noise while keeping the policy likelihood tractable. The appendix contains self-contained proofs with explicit assumptions, which is stronger than a purely empirical contribution. The experiments cover two text-to-image models and one text-to-video model, with an explicit attempt to match or favor baseline compute budgets. However, the theoretical claims are conditional on exact scores and on a step-size regime that the implemented algorithm does not satisfy, and the final transition of Algorithm 1 is not covered by the Gaussian-policy formulation. As a result, the asserted provable guarantees do not currently apply to the method as evaluated. The contribution is promising and repairable, but the gap between theory and implementation is substantial.

major comments (4)
  1. [Section 4, Eq. (7), Algorithm 1, and Appendix A, Theorem 4] The implemented Langevin step size is set by sqrt(2*epsilon_t)=eta*(t-Delta t), giving epsilon_t=eta^2*(t-Delta t)^2/2, which is O(1) as Delta t -> 0 for any fixed t. Theorem 4, however, assumes epsilon <= Lambda*h with h=Delta t, and its matched-noise comparison sets epsilon_t = sigma_t^2 * h / 2 with sigma_t = eta*sqrt(t/(1-t)). Therefore the leading-order expansion in Eq. (12) and the strict comparison in Eq. (13) do not apply to the sampler evaluated in Section 5. Theorem 3's condition (9) also depends on the unknown W2 error epsilon of the Euler step and is not verified for the implemented schedule. The conclusion that the implemented LC-GRPO sampler is "provably more accurate" is therefore not established by the theorems; the theorems cover a nearby, smaller-step-size sampler. Please either change Algorithm 1 to use an epsilon_t=O(h) schedule (for example the matched choice epsilon_t=sigma_t^2 h/2), or extend the proofs to the O(1) schedule and verify the required inequalities.
  2. [Section 4, Eq. (7), Algorithm 1 line 5, and Appendix A Theorems 3-4] The score used in the correction is computed from the learned velocity as s_theta(x', t-Delta t) = -[x'+(1-t+Delta t) v_theta(x', t-Delta t)]/(t-Delta t), whereas Theorems 3 and 4 assume the exact score nabla log p_r. The Tweedie identity is exact for the true marginal velocity, so this is not circular, but the manuscript provides no bound on the effect of replacing the true velocity with v_theta. Since Langevin dynamics is invariant for p_r only when the exact score is used, the 'distribution-preserving' property is only approximate in practice. Please add a score-error term to the contraction estimate (for example in Lemma 1 or in the statement of Theorem 3), or provide an empirical diagnostic showing that the learned-score error is small in the tested regimes.
  3. [Algorithm 1, lines 8-10, and Section 4 MDP formulation] The final transition from t_{N-1} to t_N=0 is a deterministic ODE step, but Section 4 states that the transition kernel p_theta(x_{t-Delta t}|x_t,c) is an isotropic Gaussian N(m_theta(x_t), 2 epsilon_t I). For the final action the transition density is a Dirac measure, and the likelihood ratio p_theta/p_theta_old in the GRPO objective (Eq. (4)) is not defined when theta differs from theta_old. The paper should either apply the Langevin correction at the final step with a well-defined score, or exclude the final transition from the likelihood ratio with an explicit adjustment of the 1/T normalization, or provide a regularized density for the deterministic final step.
  4. [Appendix B and Tables 1-2] The baseline comparisons vary the noise level eta and the KL coefficient beta across methods; for example, in the OCR setting Flow-GRPO uses eta=0.7 and beta=0.04 while LC-GRPO uses eta=0.8 and beta=4e-3, and in video generation eta differs across DanceGRPO (0.25), CPS (0.8), and LC-GRPO (0.6). Since eta directly controls exploration and beta controls quality preservation, the reported reward improvements may be attributable to hyperparameter selection rather than to the Langevin correction mechanism itself. Please include an ablation in which LC-GRPO is run with the same eta and beta as each baseline, or a noise-level sweep demonstrating that the advantage is robust to both hyperparameters.
minor comments (6)
  1. [Appendix A, Theorem 3] The symbol epsilon is used both for the W2 error of the Euler step and for the Langevin step size; Eq. (9) and the proof are therefore ambiguous. Please rename one of the two quantities (for example, use gamma or delta for the step size).
  2. [Section 4, paragraph after Eq. (7)] The claim that the sum of the noise variance is larger for Langevin than for the SDE is stated without a derivation. With sqrt(2 epsilon_t)=eta(t-Delta t) and the SDE noise sigma_t sqrt(Delta t)=eta sqrt(t Delta t/(1-t)), the comparison depends on the time schedule and on the number of steps; please show the calculation explicitly.
  3. [Tables 1-2] The experimental tables report point estimates without error bars or significance tests. Given the relatively small numbers of prompts per epoch, repeating each run with a few seeds and reporting standard deviations would materially strengthen the empirical claims.
  4. [References and Figure 1] There are formatting inconsistencies: the entry for Wasti et al. lacks a year, several references mix preprint and venue styles, and the legend in Figure 1 ("FLUX.1-Dev Flow-GRPO CPS Ours") is difficult to parse. Please clean these up.
  5. [Section 4 and Appendix A] The informal Theorem 1 and Theorem 2 in the main text are not explicitly cross-referenced to the formal Theorem 3 and Theorem 4 in Appendix A. Adding explicit cross-references would help readers connect the statements.
  6. [Appendix A, Theorems 3-4] The theorems analyze a single transition, while the GRPO objective is over a full multi-step rollout. The paper should state clearly that the theoretical guarantee is per-transition and does not by itself bound the total rollout error after N steps.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Wasserstein-improvement theorems are self-contained conditional proofs; the main issue is a theory-practice step-size mismatch, which is a correctness gap, not a circular reduction.

full rationale

The central derivation chain is not circular. Theorem 3 is a conditional contraction bound: given strong log-concavity and Lipschitz score (Assumption 1), it proves W2(law(x_lc), p_r) < W2(law(x_ode), p_r) whenever the step size satisfies Eq. 9, and the proof in Appx. A.1 is a self-contained Langevin contraction argument that does not assume its conclusion. Theorem 4 derives the leading-order Wasserstein errors (Eq. 12) from Taylor expansion and Otto calculus, with an explicit comparison condition Eq. 13; no fitted parameter is renamed as a prediction, and the condition is stated as a hypothesis rather than silently assumed. The Tweedie score identity s(x,t) = -(x + (1-t)v(x,t))/t is a mathematical consequence of the flow-matching interpolation, not an input-output loop. Self-citations (Guo et al. 2024, 2026; Yuan et al. 2023; Ding & Ye 2025) appear only in related-work context and do not carry the proof. The main validity concern is a theory-implementation gap rather than circularity: Algorithm 1 sets sqrt(2 epsilon_t) = eta(t - Delta t), which is O(1) for fixed t as Delta t -> 0, whereas Theorem 4 assumes epsilon <= Lambda h and its matched-noise comparison sets epsilon = sigma_t^2 Delta t / 2; the provable guarantee therefore covers a nearby smaller-step-size sampler with exact scores, not exactly the implemented schedule. That is a correctness or robustness gap, not an equivalence of the claim to its inputs. Empirical comparisons against Flow-GRPO, DanceGRPO, and CPS use independent external reward and quality benchmarks, so results are not forced by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The practical method relies on the score-from-velocity identity and a hand-tuned noise level η, while the theoretical guarantees require strong log-concavity, smoothness, and an unverified comparison condition. No new physical entities are introduced.

free parameters (2)
  • Noise level η for LC-GRPO Langevin step = 0.8 (OCR SD3.5), 0.7 (HPS SD3.5), 0.9 (FLUX combined), 0.6 (video)
    Controls the Langevin step size via √(2ϵ_t)=η(t-Δt) in Section 4. Tuned per task in Appendix B and differs across methods; the theorem's step-size condition is not verified for this choice.
  • KL coefficient β = 0.04 Flow-GRPO OCR, 1e-4 CPS OCR, 4e-3 LC-GRPO OCR, 0 otherwise
    Set separately per method and task in Appendix B. Affects the reward-quality trade-off and makes the baseline comparison not fully matched.
assumptions (5)
  • domain assumption Target marginal p_r is α-strongly log-concave with L-Lipschitz score (Assumption 1).
    Used in Lemma 1 and Theorem 3. Real flow-model marginals are not guaranteed to satisfy strong log-concavity.
  • domain assumption Regularity: v and log p are C^{2,4} with polynomial-growth derivatives and finite moments (Assumption 2).
    Used in Theorem 4 and Lemma 4 to justify Taylor expansions and Wasserstein asymptotic expansions.
  • ad hoc to paper Comparison condition Eq 13: ⟨E, dot{s}⟩_{p_t} < σ²/2 ||dot{s}||²_{p_t}.
    The strict improvement of LC over the SDE step in Theorem 4 holds only if this explicit condition holds; the paper does not test it in experiments.
  • domain assumption Acceleration of ODE trajectories is bounded by K (Assumption 4).
    Used in Lemma 2 to bound the one-step Euler error in Wasserstein distance.
  • standard math Tweedie's formula gives the score as s(x,t) = -(x + (1-t)v(x,t))/t.
    Identity under the flow-matching interpolation convention; used in Eq 7 and Algorithm 1 to recover the score from the flow velocity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction." pith.science (2026). https://pith.science/paper/XGJTF76I

@misc{pith2026260805600,
  author       = {Pith},
  title        = {Pith review of: LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGJTF76I}},
  note         = {Machine review of arXiv:2608.05600}
}
read the original abstract

Flow-based generative models are typically sampled by solving a deterministic ordinary differential equation (ODE), whereas online reinforcement learning requires stochastic rollouts for policy exploration and optimization. Existing GRPO methods for flow models therefore replace the inference-time ODE with a stochastic differential equation (SDE) during training. Although the ODE and SDE share the same marginal distributions in continuous time, their finite-step discretizations can differ substantially. In particular, SDE rollouts often become blurry as the exploration noise increases, creating a mismatch between the samples used for reinforcement learning and those generated by the test-time ODE sampler. We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction. Each rollout transition first takes an inference-aligned ODE Euler step and then applies a stochastic Langevin correction targeting the marginal distribution at the resulting timestep. The required score is recovered directly from the flow velocity, requiring no additional score model, while the resulting transition remains an isotropic Gaussian with a tractable likelihood for policy optimization. We theoretically show that, under suitable conditions, one Langevin correction step reduces the Wasserstein error of an imperfect ODE Euler step. At a matched randomness level, we further show that the proposed transition can be more accurate than the standard Euler--Maruyama discretization of the reverse SDE. Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate that LC-GRPO consistently improves reward optimization across text-to-image and text-to-video tasks, preserves generation quality, and substantially narrows the gap between stochastic training rollouts and deterministic test-time ODE inference.

Figures

Figures reproduced from arXiv: 2608.05600 by the authors.

Figure 1
Figure 1. Performance of LC-GRPO. Top: Langevin correction narrows the gap between stochastic RL rollouts and ODE inference at test time, as shown at step 0, enabling LC-GRPO to achieve the smallest training–evaluation gap. Bottom: Evaluation results on DrawBench (Saharia et al., 2022). alent reverse-time stochastic differential equation (SDE) during rollout generation. In continuous time, the ODE and SDE share the same margi… view at source ↗
Figure 2
Figure 2. Sampling Method Comparison. As the noise level1 increases, SDE-based sampling degrades sample quality, whereas ODE sampling with Langevin correction preserves sample clarity. Training–Inference Gap from SDE Rollouts. During inference, although the ODE (Eq. 1) and SDE (Eq. 2) reach the same marginal distribution at each timestep in continuous time (Kloeden et al., 2012), their finite-step numerical samplers can diffe… view at source ↗
Figure 3
Figure 3. Qualitative Comparison. Left: SD3.5-M-based models trained with OCR reward; Right: FLUX.1-Dev-based models trained with HPS-v2.1 and CLIP score. Prompts are taken from OCR and DrawBench, respectively. applying Langevin correction without CFG achieves better performance on SD3.5-M while also reducing computation. 0 100 200 300 400 500 600 Training Steps 0.30 0.32 0.34 0.36 0.38 HPS Reward Stop-grad Grad (a) FLUX.1-De… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Effect of design choices in the Langevin correction step. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison based on HunyuanVideo trained with the visual quality metric in [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Additional qualitative comparison based on SD3.5-M trained with OCR reward. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Additional qualitative comparison based on FLUX.1-Dev trained with HPS-v2.1 reward, [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 3 canonical work pages

  1. [2]

    gummy bear zone

    + h2 2 ∥A∥, where the first term iso(h 2)becauseµ h andT #pshare the same weak expansion, and the second uses R ∥x−T(x)∥ 2p= h4 4 ∥A∥2. Lower bound.Testing the hypothesis withf=−ϕgives Z (−ϕ)dµ h − Z (−ϕ)dp= h2 2 Z ∥A∥2 p dx+o(h2) = h2 2 ∥A∥2 +o(h 2). On the other hand, integrating∇ϕalong theW 2-geodesic fromptoµ h and applying the first-order (Otto) calc...

    work page 2001

  2. [4]

    Densegrpo: From sparse to dense reward for flow matching model alignment.arXiv preprint arXiv:2601.20218,

    Haoyou Deng, Keyu Yan, Chaojie Mao, Xiang Wang, Yu Liu, Changxin Gao, and Nong Sang. Densegrpo: From sparse to dense reward for flow matching model alignment.arXiv preprint arXiv:2601.20218,

    arXiv

  3. [10]

    Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192,

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192,

    arXiv

  4. [11]

    Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802,

    Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Zhong, and Liefeng Bo. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802,

    arXiv

  5. [12]

    The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning

    Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, et al. The mirage of optimizing training policies: Monotonic inference policies as the real objective for llm reinforcement learning.arXiv preprint arXiv:2606.29526,

    work page Pith review arXiv

  6. [13]

    Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling.arXiv preprint arXiv:2210.02747,

    arXiv

  7. [14]

    Flow-grpo: Training flow matching models via online rl.Ad- vances in neural information processing systems, 38:40783–40818, 2026a

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl.Ad- vances in neural information processing systems, 38:40783–40818, 2026a. Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia,...

    arXiv

  8. [15]

    De- feating the training-inference mismatch via fp16.arXiv preprint arXiv:2510.26788,

    Penghui Qi, Zichen Liu, Xiangxin Zhou, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. De- feating the training-inference mismatch via fp16.arXiv preprint arXiv:2510.26788,

    arXiv

Show all 26 references
  1. [16]

    Fp8-rl: A practical and stable low-precision stack for llm reinforcement learning.arXiv preprint arXiv:2601.18150,

    Zhaopeng Qiu, Shuang Yu, Jingqi Zhang, Shuai Zhang, Xue Huang, Jingyi Yang, and Junjie Lai. Fp8-rl: A practical and stable low-precision stack for llm reinforcement learning.arXiv preprint arXiv:2601.18150,

    arXiv

  2. [17]

    Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020a

    11 Preprint Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020a. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through sto...

    2010 arXiv

  3. [18]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314,

    arXiv

  4. [19]

    Coefficients-preserving sampling for reinforcement learning with flow matching.arXiv preprint arXiv:2509.05952,

    Feng Wang and Zihao Yu. Coefficients-preserving sampling for reinforcement learning with flow matching.arXiv preprint arXiv:2509.05952,

  5. [20]

    No more train-inference mismatch: Bitwise consistent on-policy reinforcement learning with vllm and torchtitan

    Bram Wasti, Wentao Ye, Teja Rao, Michael Goin, et al. No more train-inference mismatch: Bitwise consistent on-policy reinforcement learning with vllm and torchtitan. 2025.URl: https://blog. vllm. ai/2025/11/10/bitwise-consistent-train-inference. html. Chenfei Wu, Jiahao Li, Ji...

    2025 arXiv

  6. [21]

    Human preference score: Better aligning text-to-image models with human preference

    Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. InProceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2096–2105,

  7. [22]

    Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818,

    Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation.arXiv preprint arXiv:2505.07818,

    arXiv

  8. [23]

    Your efficient rl framework secretly brings you off-policy rl training, august 2025.URL https://fengyao

    Feng Yao, Liyuan Liu, Dinghuai Zhang, Chengyu Dong, Jingbo Shang, and Jianfeng Gao. Your efficient rl framework secretly brings you off-policy rl training, august 2025.URL https://fengyao. notion. site/off-policy-rl,

    2025

  9. [24]

    Diffusionnft: Online diffusion reinforcement with forward process.arXiv preprint arXiv:2509.16117,

    Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming-Yu Liu. Diffusionnft: Online diffusion reinforcement with forward process.arXiv preprint arXiv:2509.16117,

    arXiv

  10. [25]

    Manifold-aware exploration for reinforcement learning in video generation.arXiv preprint arXiv:2603.21872,

    Mingzhe Zheng, Weijie Kong, Yue Wu, Dengyang Jiang, Yue Ma, Xuanhua He, Bin Lin, Kaixiong Gong, Zhao Zhong, Liefeng Bo, et al. Manifold-aware exploration for reinforcement learning in video generation.arXiv preprint arXiv:2603.21872,

  11. [2005]

    Training diffusion models with reinforcement learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. InInternational Conference on Learning Representations, volume 2024, pp. 4965–4987,

    2024

  12. [2012]

    Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603,

    arXiv

  13. [2021]

    Treegrpo: Tree-advantage grpo for online rl post-training of diffusion models.arXiv preprint arXiv:2512.08153,

    Zheng Ding and Weirui Ye. Treegrpo: Tree-advantage grpo for online rl post-training of diffusion models.arXiv preprint arXiv:2512.08153,

  14. [2022]

    Directly fine-tuning diffusion models on differentiable rewards

    Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. InInternational Conference on Learning Representations, volume 2024, pp. 4793–4822,

    2024

  15. [2023]

    Gradient guidance for diffusion models: An optimization perspective.arXiv preprint arXiv:2404.14743,

    Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusion models: An optimization perspective.arXiv preprint arXiv:2404.14743,

    arXiv

  16. [2024]

    Diffusion posterior sampling for general noisy inverse problems.arXiv preprint arXiv:2209.14687,

    Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems.arXiv preprint arXiv:2209.14687,

    arXiv

  17. [2025]

    Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767,

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment.arXiv preprint arXiv:2304.06767,

    arXiv

  18. [2026]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. InProceedings of the 2021 conference on empirical methods in natural language processing, pp. 7514–7528,

    2021

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.