Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

GoRL claims the stability–expressiveness tension in online RL dissolves by confining optimization to a tractable latent prior and delegating action synthesis to a generative decoder; it reports HopperStand returns above 870, 3x the stronges

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:57 UTC pith:33YXYH6V

load-bearing objection GoRL reports large gains for latent-space policy optimization with a generative decoder, but the decoder-refinement objective as written doesn't match the claimed high-reward consolidation, and the generative baselines look undertuned. the 3 major comments →

arxiv 2512.02581 v3 pith:33YXYH6V submitted 2025-12-02 cs.LG

Training Diffusion Policies via Prior-Mapping Co-Evolution

Chubin Zhang , Zhenglin Wan , Feng Chen , Fuchao Yang , Lang Feng , Yaxin Zhou , Xingrui Yu , Yang You
show 2 more authors
Ivor Tsang Bo An
This is my paper
classification cs.LG
keywords reinforcement learninggenerative policieslatent-space policy optimizationdiffusion policiesflow matchingtwo-timescale optimizationcontinuous controlpolicy expressiveness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

GoRL claims that the chronic trade-off in online reinforcement learning — stable optimization versus expressive action distributions — is not inherent but structural, and can be dissolved by factorization. The paper separates the policy into a tractable latent prior, which receives all policy-gradient updates, and a conditional generative decoder, which is trained separately and maps a fixed Gaussian noise source to actions. An alternating two-timescale schedule anchors the decoder to the fixed prior, so the mapping grows more expressive each round without destabilizing the gradient updates. If the claim holds, expressive generative policies can be trained online from scratch using ordinary policy-gradient machinery, and the paper reports large empirical gains, including a HopperStand return above 870, more than three times the strongest baseline.

Core claim

The central claim is that the action policy πθ,φ(a|s) = ∫ πφ(a|s,ε) πθ(ε|s) dε, with a frozen decoder, admits standard unbiased policy gradients in latent space (Lemma 3.1) and trust-region-style stability bounds (Lemma 3.2), so the decoder can be refined separately without entering the gradient loop. Refinement uses supervised flow-matching or diffusion regression from the fixed Gaussian prior ε ∼ N(0,I) to the recent on-policy actions, which the paper argues consolidates high-reward behavior and expands expressiveness rather than merely reproducing current behavior. Empirical results across six continuous-control tasks show GoRL(FM) and GoRL(Diff) consistently outperform Gaussian PPO, FPO,

What carries the argument

The load-bearing object is the latent–generative factorization together with its alternating update schedule: the encoder πθ(ε|s) is a diagonal Gaussian optimized by PPO on the latent likelihood ratio with KL regularization toward N(0,I), while the decoder gφ(s,ε), instantiated as flow matching or diffusion, is trained on recent rollouts with latent inputs sampled from the fixed prior. The fixed-prior anchor is what breaks the stagnation loop — training the decoder on the evolving latent prior would just reconstruct current behavior — and the KL regularizer plus encoder reset to the prior at each stage keeps latent inputs inside the decoder's training support, acting as a behavioral warm sta

Load-bearing premise

Decoder refinement (Eq. 4) regresses all collected transitions equally, assuming the recent on-policy buffer is dominated by high-reward behavior; if the buffer contains many low-reward actions, the decoder can consolidate a mediocre policy and the next encoder phase starts from no better behavior.

What would settle it

Filter the decoder's training buffer on HopperStand to the lowest-reward half of each stage's rollouts before refinement. If GoRL still climbs past 870, the equal-weight consolidation assumption is not load-bearing; if performance collapses toward the unimodal baseline, the assumption is confirmed as the mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Policy gradients for likelihood-free generative policies can be computed at the cost of a Gaussian likelihood: Lemma 3.1 shows the latent estimator is unbiased for the true action-policy gradient whenever the decoder is frozen.
  • Expressiveness can be added monotonically through decoder refinement stages; the agent transitions from Gaussian-like behavior to multimodal action distributions, as shown by the HopperStand density visualization.
  • The framework is algorithm- and architecture-agnostic: two decoders (flow matching and diffusion) both beat all baselines, and PPO can be replaced by any on- or off-policy latent policy optimizer.
  • Direct action-space optimization of generative policies is the source of online collapse; FPO's flow-matching surrogate degrades late in training, whereas the decoupled scheme avoids this.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The decoder update treats every transition in the recent buffer equally; an advantage-weighted variant is a natural next test and would either strengthen or refute the 'consolidates high-reward behaviors' interpretation.
  • The theory covers only encoder phases with the decoder frozen; the co-evolution cycle itself — the claim that refinement plus reset yields net improvement — is an empirical assertion without a per-cycle improvement guarantee.
  • On tasks where the optimal policy is nearly unimodal, the extra decoder overhead may not pay; the method's value should concentrate on genuinely multimodal control, which also predicts where ablation gaps will be largest.
  • The abstract's mention of high-dimensional humanoid gains is not detailed in the experimental tables here; a direct humanoid benchmark would be the cleanest stress test of the framework's generality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GoRL, a framework that factorizes a policy into a tractable latent encoder πθ(ε|s) and a conditional generative decoder gφ(s,ε), trained in alternating phases: first optimizing the latent policy with a frozen decoder via standard policy gradients, then refining the decoder on recent on-policy rollouts using a supervised generative loss with latents drawn from a fixed Gaussian prior. The authors claim this decoupling resolves the stability-expressiveness tension in online RL, and report strong empirical results on six DMControl tasks, including returns above 870 on HopperStand, with ablations showing the staged decoder refinement improves performance. The theoretical section proves two standard lemmas for the encoder phase with a frozen decoder, but does not analyze the decoder-refinement step.

Significance. If the central co-evolution claim is validated, the paper offers a simple and attractive way to combine the stability of Gaussian policies with the expressiveness of diffusion/flow-matching policies in online RL. The framework is algorithm- and architecture-agnostic, the experimental design is mostly transparent (five seeds, 180M environment steps, code link provided), and the detailed appendices document hyperparameters and proofs. The key contribution, however, rests on the decoder-refinement mechanism, which is asserted to consolidate high-reward behaviors but is not supported by the stated objective or by theory. The empirical results, while internally consistent, are partly undermined by weak baselines (FPO and DPPO collapse to near-zero on several tasks) and by the lack of control for buffer composition in the ablation. If the decoder-refinement step is reworked or its behavior is convincingly characterized, this could be a solid contribution to the online RL literature.

major comments (3)
  1. [§3.2, Eq. (4) and Appendix E.4] The decoder-refinement objective in Eq. (4) regresses uniformly over all state-action pairs in the recent rollout buffer D_rollout, with no reward or advantage weighting. The text claims this 'consolidates high-reward behaviors,' but Algorithm 1 line 17's mention of 'recent high-reward data' is not implemented in the equations or Appendix E.4, which sample uniformly from the most recent on-policy buffer. Since a 60M-step encoder phase spans the policy's entire learning trajectory, including low-reward exploration, the decoder may learn the phase-averaged action distribution rather than the end-of-phase behavior. The two lemmas in §3.4 hold only for a frozen decoder and do not address whether the decoder update preserves or improves return. This is a load-bearing gap for the claimed 'virtuous cycle' and must be resolved, either by implementing reward-weighted refinement or by empirically
  2. [§4.3.2, Fig. 6] The staged decoder-refinement ablation compares encoder performance with decoders frozen at different stages, but it does not control for the composition of the rollout buffer used at each refinement step. The monotonic improvement could in principle be due to longer total training time or to the buffer containing better actions at later stages, rather than to the decoder refinement itself. A controlled experiment that re-trains the decoder on a buffer of fixed quality (e.g., end-of-stage rollouts vs. full-stage rollouts, or with and without reward weighting) is needed to isolate the contribution of Eq. (4) to the policy improvement.
  3. [§4.1.2, Table 1, Appendix E.3] The FPO and DPPO baselines collapse to near-zero returns on several tasks (e.g., FPO on WalkerWalk: 29.00±4.32; DPPO on HopperStand: 2.14±0.81), and the reported hyperparameters are not convincingly shown to match the original methods. The text states that FPO uses the 'best-performing values reported in the FPO hyperparameter sweep,' but no sweep source is cited, and DPPO is run with 10 denoising steps and σ=0.05 without evidence that this is the recommended or best configuration. If the generative baselines are not properly tuned, the headline claim of 'over 3× the strongest baseline' may be exaggerated. The authors should either provide a grid or paper reference for each baseline's hyperparameters or show that the baselines are not significantly improved by further tuning.
minor comments (5)
  1. [§3.2, Algorithm 1 line 17] Line 17 says 'Sample a mini-batch from D (e.g., recent high-reward data).' The parenthetical is not reflected in the implementation or the loss in Eq. (4). Remove it or define explicitly how high-reward data is selected.
  2. [§4.4, Figure 7] The claim that GoRL evolves 'a clearly bimodal structure' is based on projecting actions to a single principal component at one fixed state. The choice of state and the sensitivity of the KDE bandwidth are not discussed. Please clarify how the representative state is chosen and whether the bimodality is consistent across states and seeds.
  3. [Abstract and Table 1] The abstract and introduction state GoRL 'reaches a normalized return above 870' on HopperStand. In Table 1, GoRL(Diff) reaches 874.63±38.79, but GoRL(FM) reaches only 733.66±223.76 with high variance. Specify which variant achieves the claim, or report both numbers in the abstract.
  4. [§3.4, Lemma 3.2] The divergence in the main text is left generic ('D upper-bounds total variation'), while the appendix uses DTVC. Also, the appendix states the expectation over s∼dπθ',φ while the main text writes E_{s,ε∼πθ'}. These should be made consistent and the divergence made explicit in the main text.
  5. [Appendix E.4] The stage schedule is described as '60M, 60M, 30M, and 30M' but §4.1.3 says '60M, 120M, and 150M milestones.' Clarify that these are cumulative environment steps at the end of each stage; as written it is confusing.

Circularity Check

0 steps flagged

No significant circularity: the derivation reduces to standard RL identities and external benchmarks.

full rationale

The paper's central derivation is not circular. Lemma 3.1 is the standard score-function policy-gradient theorem applied to the induced action policy π_{θ,φ}(a|s) with the decoder φ fixed; its proof derives the latent gradient from the policy-gradient theorem rather than assuming it. Lemma 3.2 is a standard trust-region/performance-difference bound (Kakade–Langford) with the data-processing inequality for a fixed map g_φ; it is a genuine theorem, not a restatement of the method's desired conclusion. The decoder refinement in Eq. (4)/Eq. (6) is a training objective on recent rollouts, not a fitted parameter later relabeled as a prediction; the reported gains are comparisons against external baselines (Gaussian PPO, FPO, DPPO) under identical budgets, so no external benchmark is being manufactured from the model's own fitted constants. Self-citations (Wan et al. 2025a,b; Yu et al. 2024) appear only in contextual/related-work sentences and are not load-bearing; there is no imported uniqueness theorem and no ansatz smuggled in via citation. The closest concern is the gap between the 'high-reward data' phrasing (Algorithm 1 line 17) and the uniform 'recent on-policy rollout buffer' used in Eq. (4) and Appendix E.4: if the buffer contains early low-reward transitions, the claim that Eq. (4) 'consolidates high-reward behaviors' is under-supported. That is an unsupported-empirical-claim / implementation-fidelity concern, not a circularity, because the training objective is not being passed off as an independent predictive result. Accordingly, the circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central claim rests on five hand-chosen hyperparameters (beta, stage budgets, decoder epochs, learning rate, warm-up length), four assumptions about the deterministic decoder and the effect of decoder refinement, and no invented physical entities. The key load-bearing and unproven element is that fixed-prior decoder refinement over all recent rollouts improves the policy.

free parameters (5)
  • KL regularization coefficient beta = 1e-3 (swept 0 to 1e-1 on CheetahRun)
    Controls how strongly the latent policy is pulled toward N(0,I); chosen after ablation on one of the evaluation tasks (Section 4.3.1). The main results depend on this choice.
  • Stage interaction budgets N_m = 60M, 60M, 30M, 30M environment steps
    Lengths of the four encoder phases; selected by hand with no stated criterion (Section 4.1.3, Appendix E.4). The learning curves and final returns depend on when decoder refinement occurs.
  • Decoder epochs per refinement K_dec = 50
    Number of supervised epochs for each decoder refinement stage; no sensitivity analysis is reported (Section 4.1.3).
  • Decoder learning rate = 3e-4
    Used for FM/diffusion decoder training; standard but not ablated (Section 4.1.3).
  • Warm-up phase length and identity-like decoder = 60M steps with identity-like decoder
    The first 60M steps use an approximate identity decoder to ensure Gaussian-like early behavior; the length and identity approximation are design choices (Section 4.1.3, Appendix E.4).
axioms (4)
  • domain assumption Deterministic decoder g_phi(s,epsilon) induces the action distribution as a pushforward of the latent policy.
    Lemma 3.1's proof and the change-of-variables step rely on a deterministic map; if the diffusion or flow decoder has internal stochasticity, the unbiased latent gradient result would need modification. Invoked in Eq. (1) and Appendix D.2.
  • standard math Performance difference lemma and TV trust-region bounds for discounted MDPs.
    Lemma 3.2 uses Kakade-Langford and TRPO-style bounds; these are standard but require the stated bounded-advantage and discount assumptions in Appendix D.3.
  • ad hoc to paper Training the decoder on all recent rollouts with fixed Gaussian latents consolidates and expands high-reward behavior.
    Eq. (4) regresses all on-policy data without reward weighting; the claim that this absorbs exploration progress and increases expressiveness is asserted, not proven (Section 3.2).
  • domain assumption KL regularization keeps the latent policy close enough to N(0,I) so the decoder receives in-distribution latents.
    Used to justify the encoder reset and regularization; the OOD failure mode is shown empirically but no quantitative guarantee is given (Section 3.2, Appendix D.4).

pith-pipeline@v1.3.0-alltime-deepseek · 22554 in / 15723 out tokens · 154302 ms · 2026-08-03T18:57:41.894029+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning (RL) faces a persistent tension: policies that are stable to optimize (e.g., Gaussians) are often too simple to represent the multimodal action distributions required for complex control. Conversely, expressive generative policies -- such as diffusion and flow matching -- can be difficult to optimize in online RL due to intractable likelihoods and gradients propagating through long sampling chains. We address this tension with a key structural principle: decoupling optimization from generation. Building on this, we introduce GoRL (Generative Online Reinforcement Learning), an algorithm-agnostic framework that trains expressive policies from scratch by confining policy optimization to a tractable latent space while delegating action synthesis to a conditional generative decoder. Viewed as prior-mapping co-evolution, each stage first improves a tractable latent prior through RL and then consolidates the resulting behavior into a more expressive prior-to-action mapping. This two-timescale schedule, anchored by fixed-prior decoder refinement, enables stable optimization while continuously expanding expressiveness. Empirically, \textsc{GoRL} consistently outperforms unimodal and generative baselines across diverse continuous-control tasks. Notably, GoRL achieves returns exceeding 870 on HopperStand, more than 3* the strongest baseline; on high-dimensional humanoid tasks, it further outperforms the strongest non-GoRL baseline by over an order of magnitude.

Figures

Figures reproduced from arXiv: 2512.02581 by Bo An, Chubin Zhang, Feng Chen, Fuchao Yang, Ivor Tsang, Lang Feng, Xingrui Yu, Yang You, Yaxin Zhou, Zhenglin Wan.

Figure 1
Figure 1. Figure 1: Illustration of the “mode-covering” problem: when the optimal policy (blue) is multimodal, a unimodal policy (red) assigns high probability to the low-reward region between modes, leading to suboptimal behavior. To address this expressiveness bottleneck, recent work has turned to generative modeling. Diffusion models (Ho et al., 2020, Dhariwal and Nichol, 2021) and flow matching (FM) (Lipman et al., 2023) … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the GoRL framework. (a) Latent optimization: The decoder 𝑔𝜑 is frozen while the encoder 𝜋𝜃 is optimized in the latent space using standard policy gradients, with a KL penalty toward 𝒩 (0, 𝐼). (b) Decoder refinement: The encoder is frozen and the decoder 𝑔𝜑 is updated via supervised learning on recent rollouts, mapping the fixed Gaussian prior over 𝜀 to actions using an expressive generative los… view at source ↗
Figure 3
Figure 3. Figure 3: Visual overview of the DMControl tasks. The benchmark covers diverse control challenges: high-speed locomotion (CheetahRun), bipedal gait control (WalkerWalk), object manipulation (FingerSpin, FingerTurnHard), and fine-grained stabilization with complex contacts (HopperStand, FishSwim). 4.1.2. Baselines We compare GoRL against representative unimodal and generative baselines: • Gaussian PPO: A standard dia… view at source ↗
Figure 4
Figure 4. Figure 4: Learning curves across six DMControl tasks. Curves are smoothed using Gaussian filtering (𝜎 = 100.0) for visual clarity. Shaded regions denote standard deviation across five seeds. 4.2. Main Results The learning curves across all six tasks are presented in [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation of latent regularization on CheetahRun. Varying the KL coefficient 𝛽 significantly affects stability [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: isolates the contribution of the decoder’s progressive evolution. We compare the encoder’s performance when paired with decoders frozen at different refinement stages: Stage 0 (Identity), Stage 1, Stage 2, and Stage 3. We observe a clear monotonic improvement: the identity decoder (Stage 0) limits the agent to Gaussian-like performance, while each subsequent stage systematically raises the asymptotic retur… view at source ↗
Figure 7
Figure 7. Figure 7: Evolution of action distributions on HopperStand. Gaussian KDE plots of policy outputs at a fixed state. Initially (60M), GoRL behaves similarly to the unimodal Gaussian baseline, ensuring stability. By 180M, GoRL evolves a clear bimodal structure to capture distinct high-reward strategies, whereas baselines (PPO, FPO) remain restricted to unimodal or unstructured distributions. likelihoods, prior work in … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VINE: Taming Generative Control Policies for Reinforcement Learning

    cs.RO 2026-07 conditional novelty 7.0

    Reconstructing a fresh noisy interpolation state at every denoising step stabilizes end-to-end value-gradient training of multi-step flow-matching policies and yields state-of-the-art offline and real-robot results.

Reference graph

Works this paper leans on

20 extracted references · 12 linked inside Pith · cited by 1 Pith paper

  1. [2]

    24 GoRL : An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies D.4. Discussion: Stability and Regularization Lemma 3.2 guarantees that as long as successive encoder policies remain close in latent space,any performance degradation of the induced policy is bounded by𝑂(𝛿). In particular, when the expected advantage term...

    2048

  2. [3]

    The PPO clipping threshold differs across methods, following standard practice for expressive policies (see below). E.3. Baselines Across all methods, actor/decoder networks use comparable MLPs (typically 4 layers, 32–64 units) with SiLU activations, and critics share a common 5-layer MLP backbone, ensuring that performance differences stem from 25 GoRL :...

    2025

  3. [7]

    Flow matching policy gradients.arXiv preprint arXiv:2507.21053,

    David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients.arXiv preprint arXiv:2507.21053,

    Pith/arXiv arXiv

  4. [9]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  5. [11]

    Fm-irl: Flow-matching for reward modeling and policy regularization in reinforcement learning.arXiv preprint arXiv:2510.09222, 2025a

    Zhenglin Wan, Jingxuan Wu, Xingrui Yu, Chubin Zhang, Mingcong Lei, Bo An, and Ivor Tsang. Fm-irl: Flow-matching for reward modeling and policy regularization in reinforcement learning.arXiv preprint arXiv:2510.09222, 2025a. Zhenglin Wan, Xingrui Yu, David Mark Bossens, Yueming Lyu, Qing Guo, Flint Xiaofeng Fan, Yew-Soon Ong, and Ivor W Tsang. Diversifying...

    Pith/arXiv arXiv

  6. [14]

    19 A.2 Diffusion policies violate all three routes

    17 GoRL : An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies Appendix A Why Generative Policies Are Hard to Update with Standard Policy Gradients 19 A.1 Three classical routes to policy gradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.2 Diffusion policies violate all three routes . . . ...

    2018

  7. [15]

    19 GoRL : An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies Route I breaks: likelihoods are intractable.Applying the likelihood-ratio estimator requireslog𝜋𝜃(𝑎|𝑠) . For diffusion policies, this corresponds to the log-density of the reverse-time SDE (or its discretization), which involves solving an SDE/ODE and accu...

    2021

  8. [17]

    We adopt the Denoising Diffusion Probabilistic Model (DDPM) formulation adapted for continuous control (Chi et al., 2023, Wang et al., 2022)

    generate data by reversing a gradual noising process. We adopt the Denoising Diffusion Probabilistic Model (DDPM) formulation adapted for continuous control (Chi et al., 2023, Wang et al., 2022). Forward Process.The forward process progressively adds Gaussian noise to an action𝑎0 over𝑇 timesteps. We define the noise-corrupted action𝑎𝑡 as: 𝑎𝑡 =√ ¯𝛼𝑡𝑎0 +√1−...

    2023

  9. [1992]

    Imitation from diverse behaviors: Wasserstein quality diversity imitation learning with single-step archive exploration.arXiv preprint arXiv:2411.06965,

    Xingrui Yu, Zhenglin Wan, David Mark Bossens, Yueming Lyu, Qing Guo, and Ivor W Tsang. Imitation from diverse behaviors: Wasserstein quality diversity imitation learning with single-step archive exploration.arXiv preprint arXiv:2411.06965,

    Pith/arXiv arXiv

  10. [1999]

    Deepmind control suite.arXiv preprint arXiv:1801.00690,

    16 GoRL : An Algorithm-Agnostic Framework for Online Reinforcement Learning with Generative Policies Yuval Tassa et al. Deepmind control suite.arXiv preprint arXiv:1801.00690,

    Pith/arXiv arXiv

  11. [2002]

    By the bounded advantage assumption|𝐴𝜃,𝜑(𝑠,𝑎)|≤𝐴 max, we have|𝐹(𝑠)|≤𝐴 max for all𝑠

    states: 𝐽(𝜋′)−𝐽(𝜋) = 1 1−𝛾 E𝑠∼𝑑𝜋′ E𝑎∼𝜋′(·|𝑠) [︁ 𝐴𝜋(𝑠,𝑎) ]︁ .(24) Applying this to our induced policies𝜋=𝜋 𝜃,𝜑 and𝜋′ =𝜋 𝜃′,𝜑 and using that𝑎=𝑔 𝜑(𝑠,𝜀) with𝜀∼𝜋 𝜃′ (·|𝑠) , we obtain 𝐽(𝜃′,𝜑)−𝐽(𝜃,𝜑) = 1 1−𝛾 E𝑠∼𝑑𝜋𝜃′,𝜑 E𝜀∼𝜋𝜃′ (·|𝑠) [︁ 𝐴𝜃,𝜑 (︁ 𝑠,𝑔𝜑(𝑠,𝜀) )︁]︁ .(25) Step 2: Bounding the Distribution Shift.Define𝐹(𝑠) =E𝜀∼𝜋𝜃′ (·|𝑠)[𝐴𝜃,𝜑(𝑠,𝑔𝜑(𝑠,𝜀))]. By the bounded adv...

    2002

  12. [2013]

    Learning multimodal behaviors from scratch with diffusion policy gradient.arXiv preprint arXiv:2406.00681,

    Zechu Li, Rickmer Krohn, Tao Chen, Anurag Ajay, Pulkit Agrawal, and Georgia Chalvatzaki. Learning multimodal behaviors from scratch with diffusion policy gradient.arXiv preprint arXiv:2406.00681,

    Pith/arXiv arXiv

  13. [2015]

    Ren, Justin Lidard, Lars L

    Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Ben- jamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization.arXiv preprint arXiv:2409.00588,

    Pith/arXiv arXiv

  14. [2017]

    Let offline RL flow: Training conservative agents in the latent space of normalizing flows.arXiv preprint arXiv:2211.11096,

    Dmitry Akimov, Vladislav Kurenkov, Alexander Nikulin, Denis Tarasov, and Sergey Kolesnikov. Let offline RL flow: Training conservative agents in the latent space of normalizing flows.arXiv preprint arXiv:2211.11096,

    Pith/arXiv arXiv

  15. [2018]

    D4RL: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219,

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning.arXiv preprint arXiv:2004.07219,

    Pith/arXiv arXiv 2004

  16. [2019]

    Differentiatinglog𝜋𝜃 adds a backward ODE solve

    𝑑 𝑑𝑡 log𝑝𝑡(𝑥𝑡|𝑠) =−tr (︂ 𝜕𝑣𝜃 𝜕𝑥 (𝑥𝑡,𝑠,𝑡) )︂ ,(11) log𝜋𝜃(𝑎|𝑠) =log𝑝 0(𝜉)− ∫︁ 1 0 tr (︂ 𝜕𝑣𝜃 𝜕𝑥 (𝑥𝑡,𝑠,𝑡) )︂ 𝑑𝑡.(12) So even one likelihood evaluation requires (i) solving the ODE for𝑥0:1 and (ii) estimating the trace integral, usually via Hutchinson probes and Jacobian–vector products. Differentiatinglog𝜋𝜃 adds a backward ODE solve. In pure FM, no likelihood...

    2018

  17. [2020]

    IDQL: Implicit Q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit Q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

    Pith/arXiv arXiv

  18. [2023]

    Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma

    arXiv:2210.02747. Lei Lv, Yunfei Li, Yu Luo, Fuchun Sun, Tao Kong, Jiafeng Xu, and Xiao Ma. Flowrl: Flow-based policy for online reinforcement learning.arXiv preprint arXiv:2506.12811,

    Pith/arXiv arXiv

  19. [2024]

    Reinflow: Fine-tuning flow matching policy with online reinforcement learning, 2025a

    Tonghe Zhang, Chao Yu, Sichang Su, and Yu Wang. Reinflow: Fine-tuning flow matching policy with online reinforcement learning, 2025a. Manuscript. Yixian Zhang, Shu’ang Yu, Tonghe Zhang, et al. SAC flow: Sample-efficient reinforcement learning of flow-based policies via velocity-reparameterized sequential modeling.arXiv preprint arXiv:2509.25756, 2025b. We...

    arXiv

  20. [2025]

    Soft diffusion actor-critic: Efficient online reinforcement learning for diffusion policy.arXiv preprint arXiv:2502.00361,

    Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Soft diffusion actor-critic: Efficient online reinforcement learning for diffusion policy.arXiv preprint arXiv:2502.00361,

    Pith/arXiv arXiv