REVIEW 4 major objections 3 minor 2 cited by
Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization
T0 review · 4 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper claims that online reward-weighted conditional flow matching makes the model's data distribution converge to a Dirac delta at the maximum-reward sample, and that adding a Wasserstein-2 distance penalty converts this collapse…
desk verdict A useful likelihood-free RL fine-tuning recipe with a correct W2 bound, but the induced-distribution theorems rest on an assumed exponential tilt and a bogus proportionality step. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is the observation that a weighted regression loss is a change of measure: putting $w(x_1)$ inside the CFM expectation replaces the data distribution $q$ with $p_{\mathrm{new}}\propto w q$, so reward maximization becomes iterative reweighting. The brake is the Wasserstein-2 upper bound $W_2^2(p_1^{\theta_1},p_1^{\theta_2})\le e^{2L}\int_0^1\mathbb{E}_{x\sim p_s^{\theta_1}}[\|v_{\theta_1}(s,x)-v_{\theta_2}(s,x)\|^2]\,ds$, obtained by coupling the two flows through the same initial noise and using a differential inequality on the trajectory discrepancy; replacing the integral with a Monte Carlo average turns the penalty into a per-sample loss term $\alpha\|v_{\theta_{\mathrm{ft}}}(t,x)-v_{\theta_{\mathrm{ref}}}(t,x)\|^2$. Composing the reweighting and the penalty across epochs produces the closed-form recursion $q^n_\theta\propto w q^{n-1}_\theta\exp(-\beta D^{n-1})$, from which collapse and trade-off follow.
What would settle it
Run one epoch of reward-weighted conditional flow matching on a toy Gaussian flow with a known reward, then compare the empirical distribution of generated $x_1$ with $w(x_1)q(x_1)/Z$ while recording per-sample fitting losses; if the discrepancy tracks variation in $L_{\mathrm{cfm}}(x_1)$, the constancy assumption fails and Theorem 1's distributional claim would need a correction term.
Extended reading notes
Core claim
Under a reward-weighting function $w(x_1)\propto r(x_1)$, the loss $L_{\mathrm{RW\text{-}CFM}}(\theta)=\mathbb{E}[w(x_1)\|v_\theta(t,x)-u_t(x|x_1)\|^2]$ is equivalent to running standard conditional flow matching on the reweighted distribution $p_{\mathrm{new}}(x_1)=w(x_1)q(x_1)/Z$ (Theorem 1). If each epoch samples from the current model, the induced distribution after $N$ epochs is $q^N_\theta(x_1)=w(x_1)^N q(x_1)/Z_N$, so with strictly positive weights the limit is the Dirac delta $\delta(x_1-x^*_1)$ at the maximizer of $w$ (Theorem 2 and Lemma 1). Adding the Wasserstein-2 regularizer with coefficient $\alpha$ changes the recursion to $q^n_\theta(x_1)\propto w(x_1)q^{n-1}_\theta(x_1)\exp(-\beta D^{n-1}(x_1))$, where $D^{n-1}$ is the mean squared difference between the fine-tuned and reference vector fields along the flow; with the exponential weight $w(x_1)=\exp(\tau r(x_1))$ this yields $q^N_\theta(x_1)\propto \exp(\tau N r(x_1)-\beta\sum_{n=1}^N D^{n-1}(x_1))q(x_1)$ (Theorems 4 and 5). The paper also proves the squared Wasserstein-2 distance between two flow models is bounded by $e^{2L}\int_0^1 \mathbb{E}_{x\sim p_s^{\theta_1}}\|v_{\theta_1}(s,x)-v_{\theta_2}(s,x)\|^2\,ds$, making the regularizer computable by Monte Carlo (Theorem 3).
Load-bearing premise
The argument stands on the assumption that once the flow-matching model fits its training data well, per-sample fitting errors are roughly equal, so the only forces reweighting the learned distribution are the reward multiplier and the Wasserstein penalty; the exponential-tilt formula $q\exp(-\gamma\tilde L)$ is the template that makes every theorem go through.
Editorial extensions
If this is right
- Unregularized online reward-weighted fine-tuning of a flow-matching model converges to a delta distribution at the maximum-reward sample, so reward alone is not enough for stable alignment (Lemma 1).
- With the W2 penalty, the induced distribution after $N$ epochs is proportional to $w(x_1)^N q(x_1)\exp(-\beta\sum_{n=1}^N D^{n-1}(x_1))$, giving an explicit reward-distance trade-off controlled by $\alpha$ and $\tau$ (Theorems 4 and 5).
- Tuning the regularization strength recovers both extremes: $\alpha\to 0$ reproduces collapse, while $\alpha\to\infty$ keeps the model essentially at the reference distribution (Corollaries 11 and 12).
- The method needs no likelihoods, no filtered preference datasets, and no differentiable reward, because the regularizer is a Monte Carlo estimate from vector fields (Theorem 3 and Corollary 10).
- The update rule matches the form of advantage-weighted regression and KL-regularized policy optimization, so standard RL intuition about exploration-exploitation applies to flow matching fine-tuning (Appendix C.9).
Reading between the lines
- Beyond the paper: the core mechanism is a property of iterative reweighting, not of flow matching specifically—any generative model trained by minimizing a per-sample regression loss should show the same collapse-to-delta behavior when samples are drawn from the improving model, so the W2-style bound may transfer to score-based diffusion or autoregressive models.
- Beyond the paper: the W2 upper bound contains the factor $e^{2L}$, and for high-dimensional image models the Lipschitz constant $L$ of the reference vector field is likely large, so the practical value of $\alpha$ that prevents collapse may correspond to a much smaller true Wasserstein radius than the theory suggests.
- Beyond the paper: a direct testable extension is to measure the per-sample fitting loss $L_{\mathrm{cfm}}(x_1)$ during fine-tuning; if high-reward samples are systematically harder to fit, the factorization $q^N\propto w^N q\exp(-\beta\sum D^{n-1})$ should be corrected by a reward-dependent fitting error term.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ORW-CFM-W2, an online reward-weighted conditional flow matching method with Wasserstein-2 regularization for fine-tuning flow-based generative models toward arbitrary rewards. The central theoretical claims are: (i) unregularized online reward weighting drives the learned distribution to q_N ∝ w^N q, which collapses to a delta at the maximum-reward point (Theorems 1-2, Lemma 1); (ii) adding W2 regularization yields an induced distribution of the form q_N ∝ exp(τ N r − β Σ D_n) q, giving a controllable reward-diversity trade-off (Theorems 4-5); and (iii) the W2 distance between two flow-matching models is bounded by an integral of vector-field deviations (Theorem 3). Experiments on MNIST, CIFAR-10 compression, and Stable Diffusion 3 text-image alignment are presented as empirical support.
Significance. If the theoretical claims were established, this would be a valuable contribution: a likelihood-free, reward-weighted RL method for continuous flow matching with a tractable divergence penalty and an explicit reward-diversity trade-off. The W2 bound in Theorem 3 (App. C.4) appears correct and is a useful standalone result; the proof via flow-map coupling and Gronwall's inequality is sound, and the paper deserves credit for that part. The empirical results, especially the SD3 experiments, suggest that the algorithmic recipe can improve CLIP alignment while retaining diversity. However, the core theoretical results on induced distributions are not supported: Theorem 6 assumes the exponential-tilt form it claims to derive, and Theorem 7's factorization to w exp(−βD) is algebraically invalid under its own Assumption 4. Because the paper's headline claim is to be "theoretically sound," and because the reward-diversity trade-off theorems depend directly on these steps, the main theoretical contribution is not currently established.
major comments (4)
- [App. C.1, Theorem 6, Eq. (27)] Theorem 6 is asserted rather than proved. The claim that minimizing L_total(θ) = E_{x1∼q}[\tilde L(x1)] induces p_new(x1) ∝ q(x1) exp(−γ \tilde L(x1)) for any per-sample loss is not derived from the loss-minimization objective. The 'RL perspective' proof simply postulates the tilted distribution p_new ∝ q exp(τ r) as the target; the energy perspective defines an EBM with that Boltzmann form; and the Bayesian perspective requires \tilde L to be a negative log-likelihood. None of these establishes the general theorem. Since Theorem 7, Theorem 4, and Theorem 5 all inherit this step, their conclusions are conditional on an unproved assumption, not consequences of the stated losses.
- [App. C.5, Assumption 4 and Eqs. (114)-(115)] The factorization p_new(x1) ∝ q(x1) w(x1) exp(−βD(x1)) does not follow from Assumption 4. Starting from Eq. (114), p_new ∝ q exp(−γ[w(x1)L_cfm(x1) + αD(x1)]), if L_cfm ≈ c is a constant, the factor is exp(−γ c w(x1)), which is not proportional to w(x1) for a generic weighting function. If L_cfm is negligible, then exp(−γ w L_cfm) ≈ 1 and the result is q exp(−βD) with no w factor at all. The remark that 'w(x1)L_cfm(x1) becomes proportional to w(x1)' confuses smallness of the product with proportionality; algebraically, exp(−γ c w) is not w. This invalidates Theorem 7 and, through recursion, Theorem 4 and Theorem 5.
- [App. C.6, Theorem 4, Eqs. (129)-(130)] The recursive step q_n ∝ q_{n−1} exp(−γ[w L_cfm + αD_{n−1}]) → q_n ∝ w q_{n−1} exp(−βD_{n−1}) repeats the same unsupported simplification. Even if Theorem 6 were accepted, the replacement of exp(−γ w L_cfm) by w requires Assumption 4, and Assumption 4 does not deliver that replacement, as noted above. Consequently Theorem 5 and the corollaries describing the exponential-weighting case (App. C.7) are not proven. The claimed reward-diversity trade-off is thus a conjecture about the assumed tilted form, not a derived property of the ORW-CFM-W2 loss.
- [App. C.5, Assumption 4] Assumption 4 is load-bearing and unmeasured. The paper offers no experiment or estimate showing that w(x1)L_cfm(x1) is approximately constant or negligible relative to αD(x1) during fine-tuning, even though this assumption is the bridge to all closed-form induced-distribution results. Without empirical evidence or a valid derivation, the closed-form expressions in Theorems 4, 5, and 7 remain unsupported for the actual training regime.
minor comments (3)
- [Sec. 5 and App. A.2] The experimental plots report single runs without error bars or confidence intervals; Table 1 is computed from 64 samples without repeated seeds, which makes the quantitative trade-off claims harder to assess.
- [App. C.6, Theorem 4] The notation for the divergence term is ambiguous: Eq. (10) evaluates the W2 regularizer at the current parameters θft, while D^{n−1}(x1) in Theorem 4 is defined using θ_{n−1}. The relationship between these quantities should be clarified.
- [App. G.2] The cross-reference 'Fig. G.2' should be 'Fig. 12'; several other internal references to figures in the appendix are similarly imprecise.
Circularity Check
The induced-distribution theorems are assumed, not derived: Theorem 6 posits the exponential-tilt form and Assumption 4 inserts the w-factor that Theorems 4 and 5 need.
-
self definitional
[Appendix C.1, Theorem 6, 'Proof of Theorem 6, from RL Perspective', Eqs. (28)-(30)]
"In fact, most previous RLHF ... have found that assume a reward model r(x) that captures human preferences, with the goal of modifying the base generative model q(x) such that it generates the following tilted distribution: pnew(x1) = p∗(x1) ∝ q(x1) exp(τ ∗ r(x1)) ... just set the reward model as the negative of per-sample loss ˜L(x1), namely: r(x1) = −˜L(x1) ... and we can obtain: pnew(x1) = p∗(x1) ∝ q(x1) exp(−γ ˜L(x1))"
Theorem 6 claims that any per-sample loss L̃ induces pnew ∝ q exp(−γL̃). The proof does not derive this from the flow-matching or optimization dynamics; instead it begins by assuming the target is the tilted distribution q exp(τ r), then substitutes r = −L̃. The conclusion is therefore the starting assumption with L̃ renamed as −r. The energy and Bayesian proofs similarly posit the Boltzmann form or identify L̃ with a negative log-likelihood. Since Theorem 6 is the load-bearing premise for Theorems 7, 4, and 5, the later 'predictions' of delta collapse and reward-diversity trade-off inherit this assumed exponential-tilt form rather than deriving it from the loss.
-
other
[Appendix C.5, Theorem 7, Assumption 4 and Eq. (115)]
"Assumption 4. Assuming Lcfm(x1) is approximately constant or negligible compared to D(x1), namely: • When the model fits the data well, Lcfm(x1) is small. (i.e., w(x1) dominates) • The term w(x1)Lcfm(x1) becomes proportional to w(x1). ... Thus, we can write: pnew(x1) ∝ q(x1)w(x1) exp(−βD(x1))"
Starting from pnew ∝ q exp(−γ[w Lcfm + αD]), the only algebraic route to q w exp(−βD) is to require exp(−γ w Lcfm) ∝ w. Assumption 4 asserts exactly that 'w Lcfm becomes proportional to w', but this does not follow from Lcfm being small or constant. If Lcfm ≈ 0, the exponential factor is ≈ 1 and no w factor appears; if Lcfm ≈ c, the factor is exp(−γ c w), which is not proportional to w for a generic weighting function. The desired factorization is thus inserted by assumption, and the reward-diversity trade-off theorems built on it are forced by this assumption.
1 more flagged steps
-
other
[Appendix C.6, Proof of Theorem 4, Eqs. (127)-(130)]
"Similarly, we can assume that the probability of x1 being selected for training at epoch n is proportional to exp(−γ ˜Ln(x1)). Therefore: qn θ(x1) ∝ qn−1 θ(x1) exp(−γ ˜Ln(x1)) ... Assuming Ln cfm(x1) is approximately constant or negligible compared to αDn−1(x1) (i.e., well learned), we simplify: qn θ(x1) ∝ w(x1)qn−1 θ(x1) exp(−βDn−1(x1))"
The recursive update that defines the whole W2-regularized online evolution is introduced by 'we can assume', not derived from the ORW-CFM-W2 loss or from flow-matching dynamics. The update is the same exponential-tilt ansatz from Theorem 6, and the simplification to a w(x1) factor repeats the invalid Assumption 4 step. Theorems 4 and 5, which constitute the paper's central controllable-collapse and reward-diversity trade-off claims, therefore reduce to this assumed update rule rather than following from the proposed objective.
full rationale
The paper's main theoretical contribution is the induced-distribution analysis: unregularized online reward weighting collapses to a delta (Theorems 1-2, Lemma 1), and W2 regularization yields the trade-off q_N ∝ exp(τ N r − β Σ D) q (Theorems 4-5). The load-bearing step is Theorem 6, which asserts that any per-sample loss L̃ induces p_new ∝ q exp(−γ L̃). Its three proofs either assume the tilted distribution as the RL target, posit a Boltzmann/EBM form, or identify L̃ with a negative log-likelihood; none derive the exponential tilt from the actual optimization objective. Theorem 7 then needs to factor q exp(−γ[w L_cfm + αD]) into q w exp(−βD); Assumption 4 asserts the needed proportionality, but algebraically a negligible or constant L_cfm does not produce a multiplicative w factor. Theorem 4 repeats the assumed exponential-tilt update and the same simplification, and Theorem 5 inherits it. Thus the central 'predictions' of delta collapse and controllable reward-diversity trade-off are forced by the assumed exponential-tilt form, not by the flow matching dynamics. The paper does contain one genuinely self-contained result, Theorem 3's W2 bound, which follows from a Gronwall argument with Lipschitz continuity, and the experiments compare against external baselines; these provide independent support for the algorithm as a practical method. However, the theoretical derivation chain for the induced distributions—the paper's central claim—reduces by definition and by tailored assumption. No load-bearing self-citation chain is present, so the score reflects definitional circularity rather than citation-based circularity.
Assumptions & free parameters
free parameters (3)
- τ (temperature) =
user-chosen (e.g., 0, 0.01, 1, 10)
- α (W2 regularization coefficient) =
user-chosen (e.g., 0, 0.3, 0.8, 1, 10)
- β (regularization strength in induced distribution) =
β=γα, with γ unspecified
assumptions (6)
- ad hoc to paper p_new ∝ q exp(-γ L̃) for any per-sample loss (Theorem 6)
- ad hoc to paper The CFM fitting loss w(x1)L_cfm(x1) is approximately constant or negligible vs αD(x1) (Assumption 4)
- domain assumption At each epoch the model perfectly learns the distribution implied by the loss
- domain assumption The reference vector field is Lipschitz in x with constant L, and the two models share the same initial noise x0
- domain assumption w(x1) must be positive and proportional to reward, with exponential form w=exp(τ r)
- standard math Standard analytic tools (Grönwall, Jensen, Wasserstein coupling)
Cite this review
Pith. "Pith review of Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization." pith.science (2026). https://pith.science/paper/BEBOFXDU
@misc{pith2026250206061,
author = {Pith},
title = {Pith review of: Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization},
year = {2026},
howpublished = {\url{https://pith.science/paper/BEBOFXDU}},
note = {Machine review of arXiv:2502.06061}
}
read the original abstract
Recent advancements in reinforcement learning (RL) have achieved great success in fine-tuning diffusion-based generative models. However, fine-tuning continuous flow-based generative models to align with arbitrary user-defined reward functions remains challenging, particularly due to issues such as policy collapse from overoptimization and the prohibitively high computational cost of likelihoods in continuous-time flows. In this paper, we propose an easy-to-use and theoretically sound RL fine-tuning method, which we term Online Reward-Weighted Conditional Flow Matching with Wasserstein-2 Regularization (ORW-CFM-W2). Our method integrates RL into the flow matching framework to fine-tune generative models with arbitrary reward functions, without relying on gradients of rewards or filtered datasets. By introducing an online reward-weighting mechanism, our approach guides the model to prioritize high-reward regions in the data manifold. To prevent policy collapse and maintain diversity, we incorporate Wasserstein-2 (W2) distance regularization into our method and derive a tractable upper bound for it in flow matching, effectively balancing exploration and exploitation of policy optimization. We provide theoretical analyses to demonstrate the convergence properties and induced data distributions of our method, establishing connections with traditional RL algorithms featuring Kullback-Leibler (KL) regularization and offering a more comprehensive understanding of the underlying mechanisms and learning behavior of our approach. Extensive experiments on tasks including target image generation, image compression, and text-image alignment demonstrate the effectiveness of our method, where our method achieves optimal policy convergence while allowing controllable trade-offs between reward maximization and diversity preservation.
Figures
Figures from the paper (13 more)
Forward citations
Cited by 2 Pith papers
-
FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning
An online imitation-learning method uses a flow-matching teacher's class-conditional loss as a reward and a regularizer to train a simple MLP policy, beating cloning and adversarial-imitation baselines on five of six tasks.
-
Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance
Blending a base diffusion model with its RL-finetuned version at sampling time lets users dial alignment strength, with the blend weight corresponding to the KL-regularization coefficient beta/w.
Reference graph
Works this paper leans on
-
[1]
Spatial Semantic Coherence/Alignment: Our method shows superior understanding of spatial relationships (“left of”, “on top of”, “underneath”) while maintaining high image quality. This validates our theoretical analysis that ORW-CFM-W2 can effectively optimize arbitrary reward functions without compromising generation quality
-
[2]
Controlled Optimization: As shown in Figure 3 and Figure 7, online RL methods trained on self-generated data (Dong et al., 2023; Huguet et al., 2024; Black et al., 2024; Shumailov et al., 2024) normally face great challenges of policy collapse, where models converge to limited, homogeneous outputs that maximize rewards but lack diversity - a phenomenon th...
work page 2023
-
[3]
without requiring manually collected (Ouyang et al., 2022a) or filtered datasets (Rafailov et al., 2023), while effectively preventing policy collapse through our theoretically-derived W2 regularization (See App. C.4). A.1 T EXT-I MAGE ALIGNMENT WITH SPATIAL UNDERSTANDING Figure 7: General comparison of positional relationship understanding between differ...
work page 2023
-
[6]
Easy-to-Use and Stable Fine-tuning Method: Our method achieves these improvements without requiring filtered datasets (Rafailov et al., 2023), likelihood calculations (Black et al., 2024) or differentiable rewards (Domingo-Enrich et al., 2024), demonstrating the practical advantages of our reward-weighted flow matching framework. This aligns with our theo...
work page 2021
-
[7]
a train on top of a surfboard,
illustrates how our theoretical framework enables stable fine-tuning that maximizes rewards while maintaining the rich generative capabilities of flow-based models (Lipman et al., 2023; Esser et al., 2024). Table 1: Performance and Diversity comparison of different fine-tuning methods on text-image alignment using SD3. ’CLIP Score’ measures alignment with...
work page 2023
-
[8]
Middle row: Pick Score reward further demonstrates robust spatial understanding across varying train models and surfboard configurations. 3) Bottom row: Alpha Clip reward showcases consis- tent performance even with different text-image alignment rewards, validating the reward-agnostic nature/property of our approach. This comprehensive evaluation across ...
work page 2023
-
[9]
HPS-V2: Achieves precise locomotive placement while capturing realistic wave interac- tions
-
[10]
Pick Score: Maintains consistent spatial relationships across varying train models and surfboard configurations
Show all 22 references
-
[11]
on top",
Alpha Clip: Shows stable performance even with a different similarity metric framework. Diversity Preservation: Consistent with our theoretical analysis in Theorem 5, the generated images maintain natural variations in train appearances, surfboard designs, and ocean conditions...
2025
-
[14]
Given that w (x1) > 0 for all x1 ∈ Xand attains its maximum at x∗ 1, we define: ϵ (x1) = w (x1) w (x∗
(71) Proof of Lemma 1. Given that w (x1) > 0 for all x1 ∈ Xand attains its maximum at x∗ 1, we define: ϵ (x1) = w (x1) w (x∗
-
[15]
(72) which means, ϵ (x∗
-
[16]
Then, we can rewrite qN θ (x1) using ϵ(x1) as: qN θ (x1) = [w (x∗ 1)]N ϵ (x1)N q (x1) ZN (73) Then for x1 ̸= x∗ 1, we can have: ϵ(x1)N → 0 as N → ∞since ϵ(x1) < 1
= 1 , and 0 ≤ ϵ (x1) < 1 for x1 ̸= x∗ 1, since w (x1) < w(x∗ 1). Then, we can rewrite qN θ (x1) using ϵ(x1) as: qN θ (x1) = [w (x∗ 1)]N ϵ (x1)N q (x1) ZN (73) Then for x1 ̸= x∗ 1, we can have: ϵ(x1)N → 0 as N → ∞since ϵ(x1) < 1. Thus qN θ (x1) → 0, ∀x1 ̸= x∗ 1. And for x1 = x∗...
-
[17]
And we can have the normalization constant as follows: ZN = Z X w (x1)N q (x1) dx1 = [w (x∗ 1)]N Z X ϵ (x1)N q (x1) dx1
= [w(x∗ 1 )]N q(x1) ZN . And we can have the normalization constant as follows: ZN = Z X w (x1)N q (x1) dx1 = [w (x∗ 1)]N Z X ϵ (x1)N q (x1) dx1. (74) Similarly, we can obtain ZN ≈ [w (x∗ 1)]N q (x∗
-
[18]
Then, we can have the limit behavior: For x1 ̸= x∗ 1 : qN θ (x1) = [w (x∗ 1)]N ϵ (x1)N q (x1) [w (x∗ 1)]N q (x∗ 1) = ϵ (x1)N q (x1) q (x∗
as N → ∞. Then, we can have the limit behavior: For x1 ̸= x∗ 1 : qN θ (x1) = [w (x∗ 1)]N ϵ (x1)N q (x1) [w (x∗ 1)]N q (x∗ 1) = ϵ (x1)N q (x1) q (x∗
-
[19]
(75) For x1 = x∗ 1 : qN θ (x∗
→ 0. (75) For x1 = x∗ 1 : qN θ (x∗
-
[20]
= [w (x∗ 1)]N q (x∗ 1) [w (x∗ 1)]N q (x∗ 1) = 1 (76) Then, we can have: lim N →∞ qN θ (x1) = δ (x1 − x∗
-
[21]
cat") = pclip (x,
(77) According to Lemma 1, iteratively utilize the ORW-CFM loss to fine-tune the flow matching model without bounding the distance between reference model and fine-tuned model may lead us to a greedy policy over x1, which may induce the overoptimization problem (Black et al., ...
2024
-
[22]
normally rely on a well-collected or filtered offline dataset, requiring massive manual assistance and high costs, which is far from automated fine-tuning. However, traditional online fine-tuning methods are usually too complicated, with too many hyper-parameters, and it is di...
2024
-
[2014]
In this paper, we introduce two methods to handle the Overoptimization and ease the mode collapse risk in online RW-CFM algorithms
or overoptimization in fine-tuning diffusion models (Black et al., 2024). In this paper, we introduce two methods to handle the Overoptimization and ease the mode collapse risk in online RW-CFM algorithms. The first is to introduce a W2 distance bound between fine- tuned model...
2024
-
[2019]
James Kirkpatrick, Razvan Pascanu, Neil C
URL https://openreview.net/forum?id=r1lyTjAqYX. James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, a...
2016 arXiv
-
[2023]
while maintaining coherent generations provides strong empirical support for our theoretical contributions while demonstrating the practical utility of our approach in real-world applications. 22 Published as a conference paper at ICLR 2025 B B ACKGROUND B.1 R EINFORCEMENT LEA...
2024
-
[2024]
URL https://doi.org/10.48550/arXiv
doi: 10.48550/ARXIV .2405.08448. URL https://doi.org/10.48550/arXiv. 2405.08448. Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with mini...
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.