REVIEW 5 major objections 6 minor 38 references
VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A Monte Carlo-trained value function replaces sparse final rewards with dense differentiable supervision across the diffusion trajectory for stable RL fine-tuning with differentiable or non-differentiable rewards.
desk verdict VARD is a genuine new combination of known ideas, with honest limitations, but the non-differentiable protein experiment is weaker than the rest and needs a redo or a reframing before the headline claim is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the value function, also called a process reward model: a network $V_\phi(x_{T-t}, c)$ that maps any intermediate noisy state, together with its prompt, to the expected final reward of all trajectories that pass through that state. It is trained by Monte Carlo regression (Eq. 8) on final rewards of trajectories from the fixed pretrained policy, which lets the network generalize the final reward backward across the denoising chain. During fine-tuning it becomes the differentiation engine: the policy climbs the predicted values in Eq. (9), turning a sparse or non-differentiable reward into a dense, smooth signal. The companion mechanism is Lemma 1, which states that the gradient of the squared distance between the updated and pretrained samples equals $2\sigma^2$ times the gradient of the KL divergence between the two Gaussian denoising steps, so the KL anchor costs nothing extra in the same backprop. The value function receives lightweight online updates (learning rates $10^{-6}$ to $5\times10^{-6}$) to track the moving policy, and for differentiable rewards it is implemented as the reward model itself with LoRA adapters.
What would settle it
Two concrete checks would settle the claim. First, freeze the value function after pretraining (no online updates) and compare reward curves with full VARD: if the frozen variant matches it, the online recalibration is doing no work, and if it collapses, calibration is the load-bearing part. Second, replace the learned value with the true final reward backpropagated through every denoising step on a differentiable reward: if that dense final-reward baseline matches VARD, the learned value adds nothing beyond densification. A diagnostic third check is to measure Eq. (8)'s prediction error on states sampled from the moving policy as training progresses and ask whether rewards keep improving while this error grows large.
Extended reading notes
Core claim
The paper's central claim is that a Monte Carlo-trained value function can carry the entire reward signal of diffusion-model fine-tuning. The value function $V_\phi$ is trained by Eq. (8), regressing the final rewards of complete trajectories sampled from the pretrained policy onto each intermediate state, and it is then used in the fine-tuning objective of Eq. (9): $J(\theta) = -\mathbb{E}[V_\phi(x_{T-t}, c) + \eta \|x_{T-t} - x^0_{T-t}\|^2]$, where the second term anchors the updated policy to the pretrained one through a per-step squared-distance surrogate for the KL divergence (Lemma 1). Backpropagating through the value function gives the policy a dense, smooth gradient at every denoising step, and the same mechanism serves non-differentiable rewards because the value function, not the reward, is what is differentiated. The authors report steeper reward-versus-sample curves than DRaFT, ReFL, and DDPO on Aesthetic Score, PickScore, and ImageReward; more $\beta$-sheet-rich and structurally diverse proteins from the FoldFlow-2 base model; and resistance to reward hacking, with prompt alignment and low FID to the base model preserved even at high reward.
Load-bearing premise
The load-bearing premise is that the value function stays accurate for the states the updated policy visits: it is pretrained on trajectories of the frozen pretrained model and only lightly updated during fine-tuning, and the paper's own Limitation section names value-function accuracy as the primary risk, so if $V_\phi$ becomes miscalibrated on the moving policy's states, the dense gradient in Eq. (9) optimizes a stale surrogate rather than the true reward.
Editorial extensions
If this is right
- Any reward that can be scored on final samples, differentiable or not, becomes trainable by stable backpropagation once distilled into the value function, removing the differentiability constraint that limits final-step backpropagation methods.
- Fine-tuning needs far fewer reward evaluations, since the reported reward-versus-sample curves rise faster than DDPO, ReFL, and DRaFT, so a given reward level is reached with less sampling.
- The per-step KL anchor keeps the optimized policy near the pretrained distribution, which the experiments tie to reduced reward hacking, lower per-prompt FID to the base model, and higher HPSv2 scores on unseen prompts.
- The recipe transfers across domains (text-to-image and protein backbones) and across base models (SD 1.4 and SDXL with LoRA), suggesting the value-function mechanism rather than a task-specific trick delivers the gains.
- Trajectory-level supervision generalizes beyond the training reward: the VARD-trained model scores highest on all four HPDv2 test splits when trained with HPSv2 on the training split.
Reading between the lines
- Reasoning from the paper's own limitation section, the method's ceiling is the value function's calibration: if $V_\phi$'s prediction error grows on the states the updated policy visits during fine-tuning, the dense gradient in Eq. (9) optimizes a stale surrogate. A direct test is to plot that held-out error against training step and ask whether reward gains track value accuracy.
- The protein experiment is a sharper test than the image experiments because the value function there is trained on clean SCOPe structures rather than on generated trajectories (Appendix A.4); if offline-curated value pretraining matches on-policy pretraining elsewhere, value learning could be decoupled from the generative model entirely.
- Because Lemma 1's KL equivalence assumes Gaussian conditional steps with shared covariance, the anchor's justification weakens for non-Gaussian samplers such as DDIM or distilled few-step models; an explicit KL estimator is the natural extension for those samplers.
- The trajectory-wise reward alignment claim presumes that a scalar expected value adequately summarizes the future; for heavy-tailed or multi-modal rewards, a distributional or quantile value head would test whether the mean is the right statistic.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes VARD (Value-based Reinforced Diffusion), a method for fine-tuning pretrained diffusion models with reinforcement learning. VARD first trains a value function (also called a process reward model) to predict the expected final reward from intermediate diffusion states, using Monte Carlo regression on trajectories of the frozen pretrained policy (Eq. 8). It then fine-tunes the diffusion policy by backpropagating through this value function with a KL-regularization term that penalizes deviation from the pretrained model (Eq. 9). The method is evaluated on non-differentiable rewards (protein secondary-structure diversity, image compressibility) and differentiable rewards (Aesthetic Score, PickScore, ImageReward, HPSv2), with additional experiments on unseen-prompt generalization, cross-reward validation, and prior preservation. The central claim is that dense value-based supervision yields faster convergence, better sample efficiency, and stronger reward alignment than final-step backpropagation (DRaFT, ReFL) or policy-gradient (DDPO) baselines, while staying close to the pretrained distribution.
Significance. If the central claims hold, VARD would be a useful contribution to the growing literature on RL fine-tuning of diffusion models, particularly for non-differentiable rewards where policy-gradient methods are sample-inefficient. The idea of using a learned value function to provide dense, differentiable supervision throughout the diffusion trajectory is natural and potentially impactful. The paper includes comparisons on both protein and image domains, a generalization study, and a discussion of reward hacking. However, the current demonstration is weakened by a likely sign error in the main objective, a post hoc change to the protein reward weights, and the acknowledged but unaddressed issue of value-function calibration on the moving policy's states. If these are corrected, the method could be a solid contribution; as presented, the evidence does not yet firmly establish the advantages claimed.
major comments (5)
- [Eq. (9), Section 3.2] The KL regularization term in Eq. (9) has the wrong sign. The objective is written as J(θ) = -E[Vφ(x_{T-t}, c) + η·||x_{T-t} - x^0_{T-t}||²]. Minimizing J with respect to θ is equivalent to maximizing E[V] + η·E[||x - x^0||²], which would push the fine-tuned model away from the pretrained model rather than keeping it close. The text states that the KL term 'serves to prevent model collapse' by 'minimizing the gap' between the outputs, and Lemma 1 is invoked to justify this. The correct objective for KL-penalized reward maximization should be J(θ) = -E[V] + η·E[||x - x^0||²], i.e., a minus sign before the η term. This is load-bearing: all experiments and the KL ablation (Figure 10) depend on the actual objective, and if the sign in the text is a typo while the implementation uses the corrected sign, the paper needs to state this explicitly. Please correct the equation and clarify the implemented objective.
- [Section 7, Eq. (8), Appendix A.4, Tables 2-3] The central assumption that the value function Vφ, trained on trajectories of the frozen pretrained policy p_{θ0}, remains an accurate predictor of expected final reward on states visited by the moving policy p_θ is not validated. The paper's own Limitation section (Section 7) names value-function accuracy as the primary limitation, and the online update schedules are very light: value learning rates of 1e-6 to 5e-6 over 30 to 1,000 training steps (Tables 2-3), with only 8 value-pretraining steps for differentiable rewards. For the protein experiment, Appendix A.4 states that the value function was trained on the SCOPe dataset rather than on FoldFlow-2 trajectories, so it does not follow Eq. (8) at all. Without evidence that Vφ is calibrated on the fine-tuned policy's states, the dense supervision in Eq. (9) may simply be optimizing a stale surrogate, and the claimed advantage over final-step backpropagation is not established. Please provide calibration plots (e.g., predicted value vs. Monte Carlo return on held-out states from the fine-tuned policy) or otherwise demonstrate that the value function tracks the moving policy's state distribution.
- [Appendix A.4, Table 4, Section 4.1] The protein secondary-structure reward weight w_b was changed from 2 to 5 after observing that the original-weight FoldFlow-2-ReFT baseline underperformed the base model. Table 4 shows that with w_b=2, FoldFlow-2-ReFT has a lower reward than FoldFlow-2-base (0.76 vs. 0.83), and only with w_b=5 does the baseline outperform base (1.27 vs. 0.85). This post hoc adjustment, made 'so that the reward of FoldFlow-2-ReFT can be higher' than the base, is not a principled a priori choice and directly affects the comparison between VARD and the baseline. The claim that the original w_b=2 is a 'typo' is not substantiated. Please report results for both weight settings, or justify the chosen weights independently of the outcomes.
- [Table 1, Appendix A.4] Table 1 compares cross-prompt generalization scores but does not disclose that DDPO and PRDP are trained with 512,000 sampled images, while ReFL, DRaFT-1, DRaFT-LV, and VARD are trained with 160,000 (as stated in Appendix A.4). Since the paper emphasizes sample efficiency, the comparison should hold the sample budget fixed or visibly mark the sample counts in the table. As presented, the reader cannot assess whether VARD's advantage stems from the method or from a different training budget.
- [Eq. (7), Section 3.2] The definition of the value function is notationally unclear. The expression Vφ(x_{T-t}, c) := E_{t∈[0,T]}[Σ_{t'=t}^T R(...) | x_{T-t'}] appears to condition on x_{T-t'} for a range of t', which is not the standard conditional expectation given the current state x_{T-t}. Presumably the expectation is over future denoising steps given x_{T-t} and c; please rewrite this definition with explicit randomness and conditioning to avoid confusion.
minor comments (6)
- [Throughout] The notation (x, x^0)_{T-t} ∼ (p_θ, p_{θ0})(·|x_{T-t+1}, c) in Eq. (9) is compressed; it would be clearer to define the two samples separately, as done in the text following the equation.
- [Section 2.1] There are minor typographical errors, e.g., 'predifined' for 'predefined', and 'approximates' for 'approximate'.
- [Abstract] The abstract contains the typo 'expection' instead of 'expectation'.
- [Section 3.2] The statement 'This can also be supported by Lemma 1' would benefit from a brief explanation of how the lemma connects the MSE term to a KL divergence, since the derivation is not immediate.
- [Figure 6] The caption 'Reward v.s. sampled images' is ambiguous; it should say something like 'Reward as a function of the number of training samples.'
- [Appendix A.4] The sentence 'we refer to the FoldFlow-2-ReFT as FoldFlow-2-base after the reinforced fine-tuning introduced in the original FoldFlow-2 paper' is confusing; it seems to conflate the baseline and the fine-tuned model. Please clarify the naming.
Circularity Check
No material circularity: VARD's value function is a fitted surrogate, but the paper's central claims are validated by external reward models, cross-reward checks, and FID distance; the single self-citation is background and not load-bearing.
full rationale
The derivation chain is a standard actor-critic construction. Equation (8) fits V_phi by Monte Carlo regression to final rewards on pretrained-policy trajectories; Equation (9) then optimizes that value estimate (plus an MSE surrogate for KL, whose gradient identity is proved in Lemma 1 and Appendix A.2). This is not a circular reduction: the paper's reported successes are measured on the actual reward models (Aesthetic, PickScore, ImageReward), on HPSv2 cross-reward validation, and on FID distance to the pretrained model, none of which are V_phi's own predictions. The admitted limitation in Section 7 that value-function accuracy 'critically influences the entire fine-tuning process' is an approximation-risk statement, not an admission that the objective equals its input. The non-differentiable protein demonstration is weakened by training the value function on SCOPe rather than FoldFlow-2 trajectories (Appendix A.4) and by adjusting wb from 2 to 5 after the ReFT baseline underperformed (Table 4); these are experimental-validity concerns, not construction-level circularity. The only self-citation (Zhuang et al. 2023) appears in the offline-RL background sentence and is not load-bearing. Hence score 2 reflects a minor self-citation and approximate-surrogate caveats, with no identified circular step.
Assumptions & free parameters
free parameters (4)
- KL regularization weight (eta) =
0.1 (protein), 1 (compressibility), 100 (Aesthetic), 0.5 (PickScore), 20 (ImageReward)
- Protein secondary-structure reward weights (w_a, w_b, w_c) =
w_a=1, w_b=5, w_c=0.5 (w_b raised from 2)
- Image fine-tuning horizon =
final 10 of 50 denoising steps for images; all 50 for protein
- Value-pretraining budget for differentiable rewards =
8 training steps (batch 32)
assumptions (4)
- domain assumption Conditional distributions p_theta and p_theta0 are Gaussians with identical covariance sigma^2 I, making the gradient of the sample MSE equal to 2 sigma^2 times the gradient of the KL (Lemma 1).
- domain assumption The denoising process is an MDP with deterministic transitions and a sparse terminal reward (Eq. 6).
- domain assumption A value function regressed on the pretrained model's trajectories remains an accurate predictor of expected final reward on the intermediate states visited by the updated policy during fine-tuning.
- domain assumption Diffusion trajectories are consistent enough that intermediate states carry reliable information about the final reward.
Cite this review
Pith. "Pith review of VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL." pith.science (2026). https://pith.science/paper/3LLDM5CB
@misc{pith2026250515791,
author = {Pith},
title = {Pith review of: VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL},
year = {2026},
howpublished = {\url{https://pith.science/paper/3LLDM5CB}},
note = {Machine review of arXiv:2505.15791}
}
read the original abstract
Diffusion models have emerged as powerful generative tools across various domains, yet tailoring pre-trained models to exhibit specific desirable properties remains challenging. While reinforcement learning (RL) offers a promising solution,current methods struggle to simultaneously achieve stable, efficient fine-tuning and support non-differentiable rewards. Furthermore, their reliance on sparse rewards provides inadequate supervision during intermediate steps, often resulting in suboptimal generation quality. To address these limitations, dense and differentiable signals are required throughout the diffusion process. Hence, we propose VAlue-based Reinforced Diffusion (VARD): a novel approach that first learns a value function predicting expection of rewards from intermediate states, and subsequently uses this value function with KL regularization to provide dense supervision throughout the generation process. Our method maintains proximity to the pretrained model while enabling effective and stable training via backpropagation. Experimental results demonstrate that our approach facilitates better trajectory guidance, improves training efficiency and extends the applicability of RL to diffusion models optimized for complex, non-differentiable reward functions.
Figures
Figures from the paper (17 more)
Reference graph
Works this paper leans on
-
[1]
Atom level enzyme active site scaffolding using rfdiffusion2
Woody Ahern, Jason Yim, Doug Tischer, Saman Salike, Seth Woodbury, Donghyo Kim, Indrek Kalvet, Yakov Kipnis, Brian Coventry, Han Altae-Tran, et al. Atom level enzyme active site scaffolding using rfdiffusion2. bioRxiv, pages 2025–04,
work page 2025
-
[2]
arXiv preprint arXiv:2405.15489,
-
[3]
Unseen prompt generalization. As referenced in the main text, Table 1 includes metrics from the DRaFT [Clark et al., 2023] and PRDP [Deng et al., 2024] papers. To provide full context for these comparisons, we detail relevant aspects of their experimental settings here. Notably, DDPO and PRDP are trained using 512,000 sampled images. In contrast, other me...
work page 2023
-
[5]
Diffuseq: Sequence to sequence text generation with diffusion models
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933,
-
[7]
Dealing with sparse rewards in reinforcement learning.arXiv preprint arXiv:1910.09281,
Joshua Hare. Dealing with sparse rewards in reinforcement learning.arXiv preprint arXiv:1910.09281,
arXiv 1910
-
[9]
Sequence-augmented se (3)-flow matching for conditional protein backbone generation
Guillaume Huguet, James Vuckovic, Kilian Fatras, Eric Thibodeau-Laufer, Pablo Lemos, Riashat Islam, Cheng-Hao Liu, Jarrid Rector-Brooks, Tara Akhound-Sadegh, Michael Bronstein, et al. Sequence-augmented se (3)-flow matching for conditional protein backbone generation. arXiv preprint arXiv:2405.20313,
-
[10]
URL https://arxiv. org/abs/2411.15247. John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873):583–589,
-
[11]
Aligning text-to-image models using human feedback
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192,
Show all 38 references
-
[12]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,
2005 arXiv
-
[13]
Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding
Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gokcen Eraslan, Surag Nair, Tommaso Biancalani, Shuiwang Ji, Aviv Regev, Sergey Levine, et al. Derivative-free guidance in continuous and discrete diffusion models with soft value-based decoding. arXiv preprint arXiv:2408.08252,
-
[15]
Flow matching for generative modeling
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747,
-
[16]
Flow straight and fast: Learning to generate and transfer data with rectified flow
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003,
-
[17]
Ilya Loshchilov and Frank Hutter
URL https://openreview.net/forum?id= Aye5wL6TCn. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,
-
[19]
Exploring the limit of outcome reward for learning mathematical reasoning
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.06781,
-
[20]
Reward model learning vs
Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban, Georgios Tzannetos, Goran Radanovi´c, and Adish Singla. Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences. arXiv preprint arXiv:2403.01857,
-
[21]
Sdxl: Improving latent diffusion models for high-resolution image synthesis
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952,
-
[22]
Multisample flow matching: Straightening flows with minibatch couplings
Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky TQ Chen. Multisample flow matching: Straightening flows with minibatch couplings. arXiv preprint arXiv:2304.14772,
-
[23]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov
URL https://laion.ai/blog/ laion-aesthetics/. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[26]
Math-shepherd: Verify and reinforce llms step-by-step without human annotations
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935,
-
[27]
Diffusion language models are versatile protein learners
Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402.18567,
-
[28]
Focus-n-fix: Region-aware fine-tuning for text-to-image generation
Xiaoying Xing, Avinab Saha, Junfeng He, Susan Hao, Paul Vicol, Moonkyung Ryu, Gang Li, Sahil Singla, Sarah Young, Yinxiao Li, et al. Focus-n-fix: Region-aware fine-tuning for text-to-image generation. arXiv preprint arXiv:2501.06481,
-
[29]
Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang, Qunlin Jin, Shurun Li, et al. Visionreward: Fine-grained multi-dimensional human preference learning for image and video generation. arXiv preprint arXiv:2412.21059,
-
[30]
Se (3) diffusion model with application to protein backbone generation
Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277,
-
[31]
Towards controllable diffusion models via reward-guided exploration
Hengtong Zhang and Tingyang Xu. Towards controllable diffusion models via reward-guided exploration. arXiv preprint arXiv:2304.07132,
-
[32]
Large-scale reinforcement learning for diffusion models
Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. In European Conference on Computer Vision, pages 1–17. Springer, 2024a. Ziyi Zhang, Li Shen, Sen Zhang, Deheng Ye, Yong Luo, Miaojing Shi, Bo Du, and Dacheng Tao. Al...
-
[33]
Behavior proximal policy optimization
Zifeng Zhuang, Kun Lei, Jinxin Liu, Donglin Wang, and Yilang Guo. Behavior proximal policy optimization. arXiv preprint arXiv:2302.11312,
-
[34]
15 A Appendix / supplemental material A.1 Preliminaries on protein backbone generation A.1.1 Protein backbone parameterization The parameterization of a protein backbone for generative modeling commonly follows principles similar to those used in AlphaFold2 [Jumper et al., 202...
2021
-
[35]
Given these components, the flow matching objective for SO(3) can be formulated as: LSO(3)(θ) = Et∼U (0,1),q(R0,R1),Rt∼ρt(Rt|R0,R1) ∥vθ(t, Rt) − ut(Rt|R0, R1)∥2 SO(3)
to ρ1 (data at t = 1), a common expression for the target vector field, representing the velocity at Rt pointing towards R1, is given by [Bose et al., 2023]: ut(Rt|R0, R1) = logRt (R1) t , (13) where logRt (R1) is an element of the Lie algebra so(3) in the tangent space at Rt....
2023
-
[36]
Be- sides, for experiments with protein generation, care- ful readers may find that in the original FoldFlow-2 paper [Bose et al., 2023], wb is set to be 2 instead of
2023
-
[38]
Color intensity correlates with η magnitude (darker corresponds to higher values)
19 Exp4: Ablation FID-trained steps PickScore ImageRewardAestheticScore S A h c Sco P ckSco m Figure 10: Training trend of FID (left) and reward (right) under KL regularization. Color intensity correlates with η magnitude (darker corresponds to higher values). Higher η configu...
1956
-
[1999]
Fine-tuning of continuous-time diffusion models as entropy-regularized control
13 Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous-time diffusion models as entropy-regularized control. arXiv preprint arXiv:2402.15194, 202...
-
[2015]
Denoising diffusion implicit models
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a. Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic dif...
2010 arXiv
-
[2017]
Improve mathematical reasoning in language models by automated process supervision
12 Liangchen Luo, Yinxiao Liu, Rosanne Liu, Samrat Phatale, Meiqi Guo, Harsh Lara, Yunxuan Li, Lei Shu, Yun Zhu, Lei Meng, et al. Improve mathematical reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592,
-
[2020]
Does rlhf scale? exploring the impacts from data, model, and method
Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du, Aohan Zeng, Xiao Liu, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. Does rlhf scale? exploring the impacts from data, model, and method. arXiv preprint arXiv:2412.06000,
-
[2022]
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,
-
[2023]
Se (3)-stochastic flow matching for protein backbone generation
Avishek Joey Bose, Tara Akhound-Sadegh, Guillaume Huguet, Kilian Fatras, Jarrid Rector-Brooks, Cheng-Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein, and Alexander Tong. Se (3)-stochastic flow matching for protein backbone generation. arXiv preprint arXiv:23...
-
[2024]
Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control
Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint arXiv:2409.08861,
-
[2025]
Training diffusion models with reinforcement learning
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301,
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.