Pith. sign in

REVIEW 3 cited by

FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning

T0 review · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlowQ uses energy-guided flow matching to learn an offline RL policy approximating π(a|s) ∝ πβ(a|s) exp(Q(s,a)) with guidance applied during training rather than at inference.

arxiv 2505.14139 v1 pith:6XCDNUM3 submitted 2025-05-20 cs.LG cs.AIcs.RO

Marvin Alles , Nutan Chen , Patrick van der Smagt , Botond Cseke This is my paper
classification cs.LGcs.AIcs.RO
keywords flowenergy-guidedlearningguidancepoliciespolicyreinforcementtraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline reinforcement learning uses pre-collected state-action-reward data to train a policy, without letting the agent interact with the environment. A common difficulty is that the data come from a behavior policy, while we want a policy that scores highly according to a learned Q-function. FlowQ models the policy as a flow: a neural network learns a velocity field that pushes noise samples into actions, which is good at capturing multimodal action distributions. Standard flow matching maps noise to data; energy-guided flow matching instead maps noise to a distribution weighted by an energy function. If the energy is the negative Q-function, the weighted policy approximately equals the behavior policy times exp(Q), which puts more probability on high-value actions. FlowQ approximates this guided path as a Gaussian path by expanding Q to first order around the unguided mean, giving a closed-form velocity label to regress. This means the policy can be trained by simple regression, without sampling actions from the flow model during policy updates, so policy-update cost is independent of the number of ODE steps.

The authors test FlowQ on 18 D4RL datasets covering locomotion, navigation, and manipulation. They report competitive average scores, especially on antmaze and adroit, and show a timing comparison where their policy update stays flat while DiffusionQL's grows with sampling steps. They also examine different time schedules for the energy weight and a learned schedule.

There are important caveats. The derivation assumes Q is locally linear along each path, which is unlikely to hold exactly. The paper selects the energy strength λ separately for each dataset, so the headline numbers partly reflect dataset-specific tuning.

Extended reading notes

Core claim

Section 4.2 states the central objective: FlowQ learns a policy "which satisfies π(at|st) ∝ πβ(at|st) exp(Q(st, at))". The implementation claims this is achieved by regressing a velocity network to the Gaussian-path target ût derived in Section 4.1, so no actions are sampled from the flow policy for the policy update. If correct, the policy update is a regression whose cost is constant in the number of flow sampling steps. The strongest version is that FlowQ both reaches the advantage-weighted target and delivers competitive D4RL performance with this constant-time update. As written, Algorithm 1 contradicts the objective by using -∇Q in the action target, so the claim as stated is not supported.

Load-bearing premise

The load-bearing modeling premise, distinct from the policy target, is that Q can be replaced by its first-order Taylor expansion around tx1 for every t, yielding the Gaussian shifted mean αc(t,x1)=t x1-(1-t)^2 λ(t) ∇Q(tx1). Section 4.1: "we approximate the energy function E(x) using a first-order Taylor expansion around the mean of pt(xt|x1): E(xt) ≈ E(tx1)+∇E(tx1)^T (xt − tx1)". If Q is strongly curved, which is typical for learned critics, the guided path is not Gaussian, the closed-form velocity field is wrong, and the policy departs from πβ exp(Q). The related assumption that the marginal path is p_t(x_t) exp(-λ(t)E(x_t)) is also imposed rather than derived (Appendix A.1).

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method rests on three free choices: the energy scale λ (tuned per D4RL dataset and selected in Tables 3 and 4), the time schedule h(t) (chosen as t^2/(1-t) after an ablation on halfcheetah), and domain-specific critic/reward modifications for antmaze and adroit. The Gaussian-path and Taylor approximations are axioms imported into the derivation. No new physical or latent entities are postulated.

free parameters (3)
  • energy scale λ = per-dataset, e.g. 1.0 (halfcheetah-medium), 0.01 (halfcheetah-medium-expert), 2.0 (antmaze-medium-play), 4.0…
    Chosen by grid search over D4RL tasks (Tables 3 and 4) with the selected value reported in final results; it controls how strongly the policy is pulled toward high Q.
  • time schedule h(t) = t^2/(1-t)
    Selected by ablation on halfcheetah (Figure 3b); the learnable schedule is explored but not used in final experiments.
  • domain-specific critic and reward preprocessing = antmaze: critic [512,512,512,512]+GELU, CQL-style reward modification, max Q backup; adroit: standardized rewards
    Changed between domains (Appendix B) without a formal justification beyond 'antmaze environments benefit'; these choices affect the reported antmaze and adroit scores.
assumptions (4)
  • ad hoc to paper Energy-guided marginal path satisfies phat_t(x_t) ∝ p_t(x_t) exp(-λ(t)E(x_t))
    Assumed in Section 4.1 and effectively defines the intermediate probability path; Appendix A.1 does not derive it consistently (Eq. 7 is invalid).
  • ad hoc to paper First-order Taylor expansion of E around tx1 is accurate
    Used to compute the Gaussian mean shift in Section 4.1; requires small Hessian of Q on the path, unlikely for nonlinear Q functions.
  • domain assumption Gaussian conditional paths pt(x_t|x1)=N(x_t; t x1, (1-t)^2 I)
    Standard flow matching choice from Lipman et al. (2023), inherited by FlowQ; the linear velocity field follows from this assumption.
  • domain assumption Q-function is differentiable in action with available gradients and Hessian-vector products
    The velocity target includes ∇Q and ∇²Q·x1; for standard MLP critics this holds, but the paper does not discuss second-derivative cost or regularization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/6XCDNUM3

@misc{pith2026250514139,
  author       = {Pith},
  title        = {Pith review of: FlowQ: Energy-Guided Flow Policies for Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XCDNUM3}},
  note         = {Machine review of arXiv:2505.14139}
}
read the original abstract

The use of guidance to steer sampling toward desired outcomes has been widely explored within diffusion models, especially in applications such as image and trajectory generation. However, incorporating guidance during training remains relatively underexplored. In this work, we introduce energy-guided flow matching, a novel approach that enhances the training of flow models and eliminates the need for guidance at inference time. We learn a conditional velocity field corresponding to the flow policy by approximating an energy-guided probability path as a Gaussian path. Learning guided trajectories is appealing for tasks where the target distribution is defined by a combination of data and an energy function, as in reinforcement learning. Diffusion-based policies have recently attracted attention for their expressive power and ability to capture multi-modal action distributions. Typically, these policies are optimized using weighted objectives or by back-propagating gradients through actions sampled by the policy. As an alternative, we propose FlowQ, an offline reinforcement learning algorithm based on energy-guided flow matching. Our method achieves competitive performance while the policy training time is constant in the number of flow sampling steps.

Figures

Figures reproduced from arXiv: 2505.14139 by the authors.

Figure 1
Figure 1. Energy-Guided Flow Matching. Flow matching learns a flow to map from a known source distribution p0(x0) to the data distribution p1(x1). In contrast, energy-guided flow matching seeks to learn a transformation from a known source distribution p0(x0) to an energy-weighted distribution pˆ1(x1) ∝ p1(x1) exp{−λ E(x1)} instead. In the minimal example shown in the figure, the energy function E guides the flow toward gener… view at source ↗
Figure 2
Figure 2. Visualization of energy-guided flow matching. Given samples of the data distribution and an energy function, energy-guided flow matching approximates the posterior. We compare fixed energy scaling schedules t, t 2 , t 2 1−t , as well as a learnable schedule hθ(t). actions sampled from the policy. An alternative is to use likelihood-based methods, which formulate the learning objective as Lπ(θ) = Est∼D h DKL h Z −1 e… view at source ↗
Figure 3
Figure 3. In (a) we plot policy training time in seconds for 104 gradient steps over the number of sampling steps t. In (b) we plot the normalized return on halfcheetah environments for different choices of h(t). 5.4 Limitations The limitations of the method we propose are the following: (1) From a methodological point of view, one limitation is the assumption Et(·) ≈ λ(t)E(·), see Section 4.1. The approximation of the poster… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Normalized return of FlowQ for the D4RL locomotion environments using 5 seeds [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Normalized return of FlowQ for the D4RL adroit environments using 5 seeds. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Normalized return of FlowQ for the D4RL adroit environments using 5 seeds. F Results when using the best result during training evaluation method In addition to the running average at training results reported in [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A new actor-critic variant that reweights samples by TD-error and uncertainty and uses pessimistic sampled values improves continuous-control RL benchmark performance.

  2. ReBRAC-v2: The Return of the King

    cs.LG 2026-08 conditional novelty 5.0 of 10

    A fixed-recipe offline RL method combining normalizing-flow actors, categorical critics, staged training, and test-time refinement beats recent flow-based baselines by 22.5 points averaged over ten OGBench categories.

  3. ConFlow: Constraints-Guided Learning with Flow Matching for Motion Generation

    cs.RO 2026-07 conditional novelty 4.0 of 10

    Training on barrier-gradient-augmented flow targets plus a Gaussian-process source reduces collision rates in synthetic two-robot motion generation, particularly when infeasible demonstrations are included as negatives.

Reference graph

Works this paper leans on

24 extracted references · 6 canonical work pages · cited by 3 Pith papers

  1. [1]

    Figure 5: Normalized return of FlowQ for the D4RL adroit environments using 5 seeds

    Figure 4: Normalized return of FlowQ for the D4RL locomotion environments using 5 seeds. Figure 5: Normalized return of FlowQ for the D4RL adroit environments using 5 seeds. 15 Figure 6: Normalized return of FlowQ for the D4RL adroit environments using 5 seeds. F Results when using the best result during training evaluation method In addition to the runni...

    work page 2025

  2. [3]

    9 Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing-Yi Jing

    URL https://arxiv.org/abs/2403.03206. 9 Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing-Yi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning,

    arXiv

  3. [4]

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine

    URL https://arxiv.org/abs/2405.20555. Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219,

    arXiv 2004

  4. [5]

    Based on the ablation study in Figure 3b, we choose h(t) = t2 1−t for our experiments. As we find, that larger antmaze environments strongly benefit from a larger critic network, we change the critic to [512, 512, 512, 512] with GELU activation functions Hendrycks & Gimpel (2016) for the antmaze-medium and antmaze-large environments. Moreover, we train th...

    work page 2016

  5. [9]

    Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, and Jakob Foerster

    URL https://arxiv.org/abs/2006.11239. Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, and Jakob Foerster. Policy-guided diffusion,

    arXiv 2006

  6. [12]

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine

    URL https://arxiv.org/abs/2110.06169. Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. CoRR, abs/2006.04779,

    arXiv 2006

  7. [13]

    URL https://arxiv.org/abs/2006. 04779. Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling,

    work page 2006

  8. [14]

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu

    URL https://arxiv.org/abs/2210.02747. Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps,

    arXiv

Show all 24 references
  1. [16]

    Seohong Park, Qiyang Li, and Sergey Levine

    URL https: //arxiv.org/abs/1908.08681. Seohong Park, Qiyang Li, and Sergey Levine. Flow Q-Learning. ArXiv,

    1908 arXiv

  2. [18]

    org/abs/2012.11547

    URL https://arxiv. org/abs/2012.11547. Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning k modes with one stone,

    2012 arXiv

  3. [19]

    Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott E

    URL https://arxiv.org/abs/2208.06193. Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott E. Reed, Bobak Shahriari, Noah Y . Siegel, Josh Merel, Çaglar Gülçehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. CoRR, abs/2006.15134,

    2006 arXiv

  4. [20]

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma

    URL https://arxiv.org/abs/ 2006.15134. Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: model-based offline policy optimization. CoRR, abs/2005.13239,

    2006 arXiv

  5. [21]

    Shiyuan Zhang, Weitong Zhang, and Quanquan Gu

    URL https://arxiv.org/abs/2005.13239. Shiyuan Zhang, Weitong Zhang, and Quanquan Gu. Energy-weighted flow matching for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations,

    2005 arXiv

  6. [22]

    Wenxuan Zhou, Sujay Bajracharya, and David Held

    URL https://arxiv.org/abs/ 2311.13443. Wenxuan Zhou, Sujay Bajracharya, and David Held. PLAS: latent action space for offline reinforce- ment learning. CoRR, abs/2011.07213,

    2011 arXiv

  7. [23]

    URL https://arxiv.org/abs/2011.07213. 11 A Derivations A.1 Conditional Probabilty Path ˆpt(xt|x1) We derive the conditional probability path ˆpt(xt|x1) from the marginal ˆpt(xt) as follows ˆpt(xt) = Z ˆpt(xt|x1)ˆp1(x1)dx1 (4) 1 Zt pt(xt) exp(−λ(t)E(xt)) = Z ˆpt(xt|x1) 1 Z1 p1(...

    2023 arXiv

  8. [2016]

    Jonathan Ho, Ajay Jain, and Pieter Abbeel

    URL http://arxiv.org/abs/1606.08415. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models,

    arXiv

  9. [2017]

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine

    URL https://arxiv.org/abs/1412.6980. Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. CoRR, abs/2110.06169,

    arXiv

  10. [2018]

    Dan Hendrycks and Kevin Gimpel

    URL http://arxiv.org/abs/1812.02900. Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415,

    arXiv

  11. [2020]

    Scott Fujimoto and Shixiang Shane Gu

    URL https://arxiv.org/ abs/2004.07219. Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. CoRR, abs/2106.06860,

    2004 arXiv

  12. [2021]

    Scott Fujimoto, David Meger, and Doina Precup

    URL https://arxiv.org/abs/2106.06860. Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. CoRR, abs/1812.02900,

    arXiv

  13. [2022]

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu

    URL https: //arxiv.org/abs/2206.00927. Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning, 2023a. URL https://arxiv.org/abs/2304.12824. Cong Lu, Phil...

    arXiv

  14. [2023]

    Patrick Kidger and Cristian Garcia

    URL https://arxiv.org/abs/2305.20081. Patrick Kidger and Cristian Garcia. Equinox: neural networks in JAX via callable PyTrees and filtered transformations. Differentiable Programming workshop at Neural Information Processing Systems 2021,

    2021 arXiv

  15. [2024]

    URLhttps://arxiv.org/abs/2410.24164. Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin R...

    arXiv

  16. [2025]

    URL https://arxiv.org/ abs/2411.04562. Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nai...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.