Pith. sign in

REVIEW 3 major objections 5 minor 45 references

Flow-Based Policy for Online Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One flow step is enough for competitive online RL policies

desk verdict A promising one-step flow-policy recipe with solid empirical results, but the core policy-improvement theorem is circular and should be fixed or cut before publication. read the letter →

arxiv 2506.12811 v1 pith:JTRISRMI submitted 2025-06-15 cs.LG cs.AI

Lei Lv , Yunfei Li , Yu Luo , Fuchun Sun , Tao Kong , Jiafeng Xu , Xiao Ma This is my paper · ORCID
classification cs.LGcs.AI
keywords flowmatchingreinforcementlearningpolicyoptimizationWasserstein-2distancecontinuousnormalizingexpectileregressiononlineRLdiffusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FlowRL claims that flow-based generative models can serve as policy networks in online reinforcement learning if their optimization is constrained to stay close to the best behavior policy in the replay buffer. The paper derives a constrained policy-search objective that maximizes the Q-value through a flow actor while bounding the Wasserstein-2 distance to an implicitly estimated behavior-optimal policy. On DMControl and HumanoidBench, this yields competitive returns against deterministic, Gaussian, and diffusion-based actors, with policy inference performed in a single flow step. The significance is that expressive, multimodal policies can be used online without backpropagation through time or expensive iterative sampling.

What carries the argument

The central object is a state-dependent velocity field vθ(t, s, a_t) whose ODE integration maps Gaussian noise a0 to policy actions, so the policy is a continuous normalizing flow rather than a Gaussian or a diffusion sampler. The argument runs through the Wasserstein-2 distance: Theorem 4.1 bounds W2(πθ, πβ*) by an integral of squared velocity-field differences, giving a tractable constraint that needs no density evaluation. The practical surrogate is the weighted flow-matching objective: a nonnegative weight f proportional to (Qπβ* - Qπθ)+ reweights buffer actions, so the flow field is pulled toward high-value behavior while the Q-maximization term drives exploration. Expectile regression supplies Qπβ* and Vπβ*, and a single-step midpoint Euler integration generates actions during training and evaluation.

What would settle it

Run FlowRL's weighted flow-matching objective on a low-dimensional bandit with known Qπβ* and compare the learned action distribution against the exact reweighted policy f πβ*/Z: if the empirical actions diverge from that distribution at N=1 flow step, the policy improvement guarantee does not hold in its practical regime. Equivalently, a tabular setting where the expectile estimate of Qπβ* is computed exactly, and the flow field is evaluated by line integration, would reveal whether the surrogate objective reproduces the claimed improved policy.

Watch

Extended reading notes

Core claim

The central claim is that the objective mismatch between flow matching and RL can be resolved by a Wasserstein-2-regularized policy search: instead of training the velocity field to imitate static buffer data, FlowRL maximizes the Q-value of actions generated by the flow ODE while penalizing the expected squared deviation of the field from the velocity field of high-reward buffer actions. Theorem 4.1 gives an upper bound on the W2 distance between policies in terms of velocity-field mismatch, and Theorem 4.2 states that minimizing the weighted flow-matching objective, with weights f proportional to (Qπβ* - Qπθ)+, produces an improved policy proportional to f πβ*. The behavior-optimal value Qπβ* is estimated by expectile regression, avoiding direct access to the mixture of behavior policies. Experimentally, single-step flow inference (N=1) is sufficient and competitive on high-dimensional locomotion and humanoid tasks.

Load-bearing premise

The load-bearing premise is that the expectile-regressed Qπβ* accurately tracks the best behavior policy in the buffer, and that minimizing the weighted flow-matching loss is equivalent to regressing under the reweighted policy πk+1 ∝ f πβ*, an equivalence the appendix assumes rather than proves.

Editorial extensions

If this is right

  • Flow-based RL policies can be updated with one-step inference, removing the backpropagation-through-time cost that burdens diffusion-based actors.
  • The Wasserstein-2 constraint makes high-reward buffer actions reusable for policy improvement without explicit density estimation or sampling from a mixture policy.
  • The method retains actor-critic simplicity, so it can be dropped into existing off-policy RL loops with minimal changes.
  • Ablations show the adaptive constraint improves returns, especially in high-dimensional humanoid tasks, and performance is insensitive to flow steps N=1, 5, and 10.
  • FlowRL achieves competitive or better returns than model-free baselines on DMC-hard and HumanoidBench, including the 38-dimensional dog domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Theorem 4.2's equivalence between the weighted flow-matching loss and regression under πk+1 ∝ f πβ* holds only asymptotically, deviations should be expected at small flow steps; a quantitative check of the implied action distribution at N=1 would test the practical force of the guarantee.
  • The same constraint mechanism may transfer to offline RL by replacing the online replay buffer with a static dataset, where expectile regression is already a standard tool.
  • Because the flow actor is a deterministic map on noise, its stochasticity comes entirely from the source distribution, suggesting a natural way to control exploration by widening or shaping p0(a0).
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FlowRL, an off-policy actor-critic method in which the policy is a flow-matching model (a state-dependent velocity field integrated from noise) and the policy update is formulated as a constrained search: maximize expected Q while keeping the policy close to an inferred "behavior-optimal" policy pi_beta*. Since pi_beta* and its velocity field are inaccessible, the constraint is implemented as a weighted flow-matching loss in which regression targets are taken from the replay buffer and weighted by f proportional to max(Q_pi_beta* - Q_pi_theta, 0). The paper claims a policy improvement guarantee (Theorem 4.2) and reports competitive results on DMControl and HumanoidBench with one-step flow inference, no BPTT, and no explicit exploration mechanism.

Significance. The empirical contribution is potentially useful: FlowRL's one-step flow policy avoids backpropagation through time for the policy, and the benchmark evaluation covers 15 tasks with 5 seeds, including high-dimensional DMC-hard and HumanoidBench tasks, with an ablation showing that the replay-buffer constraint helps. If these results are reproducible, the method offers a practical way to use expressive flow policies in online RL. The paper also provides a theoretical framework, but the central Theorem 4.2 is not supported by the appendix proof; the proof assumes the reweighted distribution it is supposed to derive, and the objective being minimized does not match the theorem's distribution. The work is therefore better characterized as an empirical method paper with an unproven theoretical claim, and the revision must either supply a correct derivation or clearly remove the guarantee.

major comments (3)
  1. [Section 4.3, Theorem 4.2; Appendix A.2] Theorem 4.2 is not proven by Appendix A.2. The proof begins with "Assuming the weighted policy distribution is pi_{k+1}=f pi_k / Z" and concludes "provided that our assumption holds"; this is circular because the theorem is supposed to derive that distribution. Worse, the theorem as stated in Eq. (15) writes pi_{k+1}(a|s)=f(s,a) pi_beta*(a|s)/Z(s) with Z(s)=integral_A f(s,a) pi_k(a|s) da, so the numerator and denominator are taken with respect to different measures and the right-hand side is not a normalized distribution. The loss in Eq. (13) samples the regression target a from the replay buffer D, not from pi_k; the proof in Appendix A.2 changes this by inserting pi_k(a|s) into the integrand. Consequently, the global minimizer of Eq. (13) is a reweighted buffer distribution, not f pi_beta*/Z or f pi_k/Z, and no policy improvement over pi_k follows. This directly invalidates the claim "policy improvement guarantees remain valid" in Section 4.3.
  2. [Section 4.4, Eqs. (20)-(22)] Theorem 4.2 concerns objective (13), but the algorithm optimizes the Lagrangian objective (22), which adds the term E_{s,a~D,a'~pi_theta}[Q_pi_theta(s,a')] and uses f computed from Q_pi_theta(s,a'). The proof treats f as a constant, but in the implementation f depends on the current policy value unless a stop-gradient is used, which the paper does not state. The extra Q-maximization term is also not accommodated by the theorem, so the theoretical guarantee does not cover the implemented objective.
  3. [Section 4.4, Eqs. (16)-(19)] The operationalization of Q_pi_beta* is not supported. Eq. (17) defines the Bellman target for pi_beta* using max_{a'~D} Q_pi_beta*(s',a'), but the implemented expectile regression in Eqs. (18)-(19) estimates a different object: an upper expectile of the target distribution. No argument shows that the expectile solution equals Q^{pi_beta*} or that it is a valid lower bound related to Q*. Since the weighting f in Eq. (13) is the only mechanism steering the policy toward "optimal behavior", an incorrect Q_pi_beta* would break the constraint; this gap is part of the theoretical chain and must either be proven or explicitly weakened to a heuristic.
minor comments (5)
  1. [Eqs. (12) and (24)] Theorem 4.1 is stated in the main text with the expectation over a~pi_beta*, but the proof in Appendix A.1 derives the bound with E_{a~pi_t_theta}; these are different distributions and the mismatch should be fixed.
  2. [Eq. (14) and Appendix B.1] The weighting function is defined in Eq. (14) as f proportional to max(Q_pi_beta* - Q_pi_theta, 0), but the implementation in Appendix B.1 uses f(x)=I(x)*exp(x). The exact definition used in the experiments should be stated in the main text, including whether the exponential is applied to the raw Q-difference or to a normalized one.
  3. [Section 4.1, Eq. (9)] The notation pi_theta(s,a0)=a0+integral v_theta dt is confusing because the left-hand side is a policy distribution, not a deterministic function; use a terminal-action notation such as a1=psi_theta(a0;s) and define pi_theta(.|s) as the law of a1.
  4. [Figure 3 and Table 1] The learning curves in Figure 3 do not show confidence intervals or final mean/std values; given the 5-seed setup, adding shaded regions or a score table (as in Appendix D) to the main text would make the empirical comparison easier to assess. Also, Table 1 contains typos ("ODE Slover", duplicate "Hyperparameters").
  5. [Appendix A.1, Lemma 1] The lemma statement defines Delta_t(x0)=psi_t^1(x0)-psi_t^2(x1), but the subsequent inequality uses psi_t^2(x0); the variables are inconsistent. The step described as "by Jensen's inequality" after bounding f(1) is not Jensen; it is a Cauchy-Schwarz step and should be restated.

Circularity Check

2 steps flagged · score 6.0 of 10

Theorem 4.2 is circular: its proof assumes the reweighted policy distribution it claims to derive, and the weighted objective regresses onto the replay buffer, so the policy-improvement guarantee reduces to the chosen training target by construction.

  1. self definitional [Section 4.3, Theorem 4.2; Appendix A.2]
    "Theorem 4.2 ... Minimizing the objective (13) yields an improved policy distribution: πk+1(a|s) = f(s,a)πβ∗(a|s)/Z(s), where Z(s) = ∫A f·π_k(a|s)da is the normalization factor. ... Assuming the weighted policy distribution is: πk+1(a′|s) = f(s,a)π_k(a|s)/Z(s), ... minimizing LW(θ) is equivalent to minimizing the expected loss under the new distribution πk+1(a|s), provided that our assumption holds."

    The proof begins by postulating the theorem's conclusion: it assumes the weighted policy distribution is π_{k+1}=fπ_k/Z and then shows only that, under that assumption, minimizing L_W is equivalent to minimizing under π_{k+1}. The closing phrase 'provided that our assumption holds' concedes that the claimed equivalence depends on the very distribution the theorem purports to derive. The theorem's normalization is also inconsistent: the numerator is fπ_{β*} while Z integrates fπ_k, so the right-hand side is not a normalized distribution. Thus the policy-improvement guarantee is assumed rather than established from Eq. (13).

  2. fitted input called prediction [Section 4.3, Eq. (13) and Theorem 4.2]
    "E_{a′∼πθ,t∼U(0,1), s,a∼D}[ f(Qπβ∗(s,a)−Qπθ(s,a′)) ∥vθ(s,at,t)−(a−a0)∥^2 ], f∝max(Qπβ∗−Qπθ,0)."

    In Eq. (13) the regression action a is sampled only from the replay buffer D, not from π_k or π_{β*}. Consequently the global minimizer of the weighted flow-matching loss transports noise to a terminal distribution proportional to f(s,a)D(a|s), not to f(s,a)π_{β*}(a|s)/Z(s) as Theorem 4.2 claims. The 'improved policy' π_{k+1} is therefore the reweighted buffer target chosen by construction, and no independent Bellman or advantage argument connects that minimizer to an improvement over π_k. The theorem renames a weighted buffer regression as a policy improvement.

full rationale

FlowRL's empirical evaluation is not circular: it compares against external baselines (SAC, TD3, QVPO, TD-MPC2) on DMControl and HumanoidBench, so the competitive-performance claim has independent grounding. The circularity is confined to the theoretical guarantee. Theorem 4.2 is announced as a consequence of minimizing Eq. (13), but Appendix A.2 assumes exactly the reweighted distribution π_{k+1} and then concludes equivalence only 'provided that our assumption holds'; the theorem's normalization is inconsistent (numerator fπ_{β*}, denominator integrates fπ_k). Moreover, Eq. (13) samples a from the replay buffer D, so the minimizer's terminal distribution is a reweighted buffer distribution by construction, and no policy-improvement guarantee over π_k follows from the stated equations. Since the central formal claim reduces to its own assumption and to the chosen regression target, while the empirical claim remains externally benchmarked, the appropriate score is 6 rather than 8-10.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The algorithm's behavior is governed by hand-set hyperparameters (λ=0.1, τ=0.9, N=1) and an ad hoc Q normalization. Theoretical support depends on unverified Lipschitz continuity and on a reweighting step that is assumed rather than proven. No new physical entities are introduced.

free parameters (4)
  • Lagrangian multiplier λ = 0.1
    Eq. (22) sets λ=0.1 as a constant instead of solving the constrained problem via KKT; it controls the trade-off between Q-maximization and the behavior constraint.
  • Expectile factor τ = 0.9
    Eq. (18) uses expectile regression with τ=0.9 to estimate Qπβ*; this hand-set value determines the estimate of the behavior-optimal value.
  • Flow steps N = 1
    The algorithm uses N=1 flow step in training and evaluation. The ablation shows limited effect, but this is a design choice that affects inference accuracy.
  • Q normalization offset = mean of Q
    Appendix B.1 normalizes the Q function by subtracting its mean only when computing the weighting function f; this ad hoc scaling changes the weighting magnitude.
assumptions (4)
  • domain assumption The velocity field vβ* is Lipschitz continuous in action with constant L.
    Theorem 4.1 assumes this to bound the Wasserstein-2 distance; no verification or bound on L is provided, and the algorithm never uses L.
  • domain assumption The behavior-optimal policy πβ* can be estimated via expectile regression and satisfies Q* ≥ Qπβ* ≥ Qπβ.
    Section 3.1 and Eq. (18) rely on expectile regression as an approximation of the max over behavior policies in the replay buffer.
  • ad hoc to paper Minimizing the weighted flow-matching objective (13) is equivalent to training under the reweighted policy π_{k+1}=fπ_k/Z(s) and yields an improved policy.
    This is the asserted conclusion of Theorem 4.2, but the proof in Appendix A.2 assumes the reweighted distribution and does not prove the velocity field samples from it.
  • domain assumption A one-step Euler or midpoint integration of the flow ODE accurately approximates the policy distribution.
    The algorithm uses N=1 flow step; the method assumes this discretization error is small enough for stable policy learning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Flow-Based Policy for Online Reinforcement Learning." pith.science (2026). https://pith.science/paper/JTRISRMI

@misc{pith2026250612811,
  author       = {Pith},
  title        = {Pith review of: Flow-Based Policy for Online Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTRISRMI}},
  note         = {Machine review of arXiv:2506.12811}
}
read the original abstract

We present \textbf{FlowRL}, a novel framework for online reinforcement learning that integrates flow-based policy representation with Wasserstein-2-regularized optimization. We argue that in addition to training signals, enhancing the expressiveness of the policy class is crucial for the performance gains in RL. Flow-based generative models offer such potential, excelling at capturing complex, multimodal action distributions. However, their direct application in online RL is challenging due to a fundamental objective mismatch: standard flow training optimizes for static data imitation, while RL requires value-based policy optimization through a dynamic buffer, leading to difficult optimization landscapes. FlowRL first models policies via a state-dependent velocity field, generating actions through deterministic ODE integration from noise. We derive a constrained policy search objective that jointly maximizes Q through the flow policy while bounding the Wasserstein-2 distance to a behavior-optimal policy implicitly derived from the replay buffer. This formulation effectively aligns the flow optimization with the RL objective, enabling efficient and value-aware policy learning despite the complexity of the policy class. Empirical evaluations on DMControl and Humanoidbench demonstrate that FlowRL achieves competitive performance in online reinforcement learning benchmarks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 6 canonical work pages

  1. [10]

    Online reward-weighted fine-tuning of flow matching with wasserstein regularization.arXiv preprint arXiv:2502.06061, 2025

    Jiajun Fan, Shuaike Shen, Chaoran Cheng, Yuxin Chen, Chumeng Liang, and Ge Liu. Online reward-weighted fine-tuning of flow matching with wasserstein regularization.arXiv preprint arXiv:2502.06061, 2025

    arXiv 2025

  2. [1]

    A markovian decision process.Journal of mathematics and mechanics, pages 679–684, 1957

    Richard Bellman. A markovian decision process.Journal of mathematics and mechanics, pages 679–684, 1957

    1957

  3. [2]

    A vision-languageaction flow model for general robot control.arXiv preprint arXiv:2410.24164, 2(3):5, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. A vision-languageaction flow model for general robot control.arXiv preprint arXiv:2410.24164, 2(3):5, 2024

    arXiv

  4. [3]

    Se (3)-stochastic flow matching for protein backbone generation.arXiv preprint arXiv:2310.02391, 2023

    Avishek Joey Bose, Tara Akhound-Sadegh, Guillaume Huguet, Kilian Fatras, Jarrid Rector-Brooks, Cheng-Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein, and Alexander Tong. Se (3)-stochastic flow matching for protein backbone generation.arXiv preprint arXiv:2310.02391, 2023

    arXiv

  5. [4]

    Offline reinforcement learning via high-fidelity generative behavior modeling.arXiv preprint arXiv:2209.14548, 2022

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling.arXiv preprint arXiv:2209.14548, 2022

    arXiv 2022

  6. [5]

    Neural ordinary differential equations

    Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018

    2018

  7. [6]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, page 02783649241273668, 2023

    2023

  8. [7]

    Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis.Advances in neural information processing systems, 34:8780–8794, 2021

    2021

Show all 45 references
  1. [8]

    Diffusion- based reinforcement learning via q-weighted variational policy optimization.arXiv preprint arXiv:2405.16173, 2024

    Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion- based reinforcement learning via q-weighted variational policy optimization.arXiv preprint arXiv:2405.16173, 2024

    2024 arXiv

  2. [9]

    Consistency models as a rich and efficient policy class for reinforcement learning.arXiv preprint arXiv:2309.16984, 2023

    Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning.arXiv preprint arXiv:2309.16984, 2023

    2023 arXiv

  3. [11]

    A minimalist approach to offline reinforcement learning.Advances in neural information processing systems, 34:20132–20145, 2021

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning.Advances in neural information processing systems, 34:20132–20145, 2021

    2021

  4. [12]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018

    2018

  5. [13]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. Pmlr, 2018

    2018

  6. [14]

    Td-mpc2: Scalable, robust world models for continuous control

    Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpc2: Scalable, robust world models for continuous control. arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  7. [15]

    Diffcps: Diffusion model based constrained policy search for offline reinforcement learning.arXiv preprint arXiv:2310.05333, 2023

    Longxiang He, Li Shen, Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffcps: Diffusion model based constrained policy search for offline reinforcement learning.arXiv preprint arXiv:2310.05333, 2023

    2023 arXiv

  8. [16]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  9. [17]

    Alphafold meets flow matching for generating protein ensembles

    Bowen Jing, Bonnie Berger, and Tommi Jaakkola. Alphafold meets flow matching for generating protein ensembles. arXiv preprint arXiv:2402.04845, 2024

    2024 arXiv

  10. [18]

    Efficient diffusion policies for offline reinforcement learning

    Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:67195–67212, 2023

    2023

  11. [19]

    Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning.arXiv preprint arXiv:2110.06169, 2021

    2021 arXiv

  12. [20]

    Stabilizing off-policy q-learning via bootstrapping error reduction.Advances in neural information processing systems, 32, 2019

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction.Advances in neural information processing systems, 32, 2019. 11

    2019

  13. [21]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  14. [22]

    Flow matching for generative modeling

    Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  15. [23]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    2022 arXiv

  16. [24]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. InInternational Conference on Machine Learning, pages 22825–22855. PMLR, 2023

    2023

  17. [25]

    Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl.arXiv preprint arXiv:2405.18520, 2024

    Yu Luo, Tianying Ji, Fuchun Sun, Jianwei Zhang, Huazhe Xu, and Xianyuan Zhan. Offline-boosted actor-critic: Adaptively blending optimal historical behaviors in deep off-policy rl.arXiv preprint arXiv:2405.18520, 2024

    2024 arXiv

  18. [26]

    Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning.arXiv preprint arXiv:2407.20109, 2024

    Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning.arXiv preprint arXiv:2407.20109, 2024

    2024 arXiv

  19. [27]

    Self-imitation learning

    Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. InInternational conference on machine learning, pages 3878–3887. PMLR, 2018

    2018

  20. [28]

    The difficulty of passive learning in deep reinforcement learning

    Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. Advances in Neural Information Processing Systems, 34:23283–23295, 2021

    2021

  21. [29]

    Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.09329, 2024

    Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? arXiv preprint arXiv:2406.09329, 2024

    2024 arXiv

  22. [30]

    Flow q-learning.arXiv preprint arXiv:2502.02538, 2025

    Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning.arXiv preprint arXiv:2502.02538, 2025

    2025 arXiv

  23. [31]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    1910 arXiv

  24. [32]

    Reinforcement learning by reward-weighted regression for operational space control

    Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745–750, 2007

    2007

  25. [33]

    Learning a diffusion model policy from rewards via q-score matching.arXiv preprint arXiv:2312.11752, 2023

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching.arXiv preprint arXiv:2312.11752, 2023

    2023 arXiv

  26. [34]

    Diffusion policy policy optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Ben- jamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  27. [35]

    Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation.arXiv preprint arXiv:2403.10506, 2024

    Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation.arXiv preprint arXiv:2403.10506, 2024

    2024 arXiv

  28. [36]

    Deterministic policy gradient algorithms

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. InInternational conference on machine learning, pages 387–395. Pmlr, 2014

    2014

  29. [37]

    Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations.arXiv preprint arXiv:2011.13456, 2020

    2011 arXiv

  30. [38]

    Reinforcement learning.Journal of Cognitive Neuroscience, 11(1): 126–134, 1999

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning.Journal of Cognitive Neuroscience, 11(1): 126–134, 1999

    1999

  31. [39]

    Deepmind control suite.arXiv preprint arXiv:1801.00690, 2018

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite.arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  32. [40]

    Improving and generalizing flow-based generative models with minibatch optimal transport

    Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2302.00482, 2023

    2023 arXiv

  33. [41]

    Springer, 2008

    Cédric Villani et al.Optimal transport: old and new, volume 338. Springer, 2008

    2008

  34. [42]

    Diffusion actor-critic with entropy regulator.Advances in Neural Information Processing Systems, 37:54183–54204, 2024

    Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator.Advances in Neural Information Processing Systems, 37:54183–54204, 2024. 12

    2024

  35. [43]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022

    2022 arXiv

  36. [44]

    Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    2023 arXiv

  37. [45]

    Energy-weighted flow matching for offline reinforcement learning

    Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. arXiv preprint arXiv:2503.04975, 2025. 13 Appendix A Proofs in the Main Text : RL objective: constraint policy search Upper bound for W-2 distance Implicit guide La...

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.