Pith. sign in

REVIEW 3 major objections 5 minor 37 cited by

Flow Matching Policy Gradients

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Flow Policy Optimization (FPO) replaces the PPO likelihood ratio with a ratio of conditional flow-matching losses, letting diffusion-style policies train from rewards without exact likelihoods.

desk verdict FPO is a genuinely useful algorithm for RL with flow policies, backed by solid experiments, but the main text overstates the theory for the OT schedule actually used; the supplementary is more honest than the main text. read the letter →

arxiv 2507.21053 v2 pith:XTWI2FWN submitted 2025-07-28 cs.LG cs.RO

David McAllister , Songwei Ge , Brent Yi , Chung Min Kim , Ethan Weber , Hongsuk Choi , Haiwen Feng , Angjoo Kanazawa This is my paper
classification cs.LGcs.RO
keywords flowmatchingpolicygradientsPPOdiffusionreinforcementlearningcontinuouscontrolmultimodalactiondistributionsevidencelowerbound
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Flow Policy Optimization (FPO) is an on-policy reinforcement learning algorithm that trains diffusion- or flow-based generative models as action policies without ever computing exact likelihoods. The central move is to replace the PPO likelihood ratio with a ratio of conditional flow-matching losses, justified by the known identity that a weighted denoising loss equals the negative evidence lower bound up to a constant that does not depend on the policy parameters. This lets flow policies drop into standard PPO-style actor-critic training as a replacement for Gaussian policies, while treating the sampling procedure as a black box. The paper shows FPO trains diffusion policies from scratch on a GridWorld, ten MuJoCo Playground tasks, and a high-dimensional physics-based humanoid, and finds that flow policies capture multimodal action distributions and beat Gaussian policies precisely when goal conditioning is sparse. A sympathetic reader would care because the method removes the main obstacle to using expressive generative models as reinforcement learning policies: the intractability of their likelihoods.

What carries the argument

The load-bearing object is the FPO ratio, a proxy for the PPO likelihood ratio computed as the exponential of the difference between old and current conditional flow matching losses on shared noise and timestep samples. Its justification is the identity relating the weighted denoising loss $L^w_\theta$ to the negative evidence lower bound (ELBO) up to a constant independent of $\theta$, which turns a cheap mean-squared-error denoising objective into a likelihood surrogate; the ratio then decomposes into the exact likelihood ratio multiplied by an inverse KL-gap correction, so one update both raises the likelihood of high-advantage actions and tightens the bound. The supporting Monte Carlo estimator mixes a small number $N_{mc}$ of $(\tau, \epsilon)$ draws, and its key property is that the gradient direction is unbiased even though the ratio scale is overestimated, which the paper demonstrates with the log-derivative trick and Jensen's inequality.

What would settle it

Take a low-dimensional flow policy whose exact action likelihoods can be computed numerically (through the ODE change-of-variables formula with divergence estimation) and compare, across checkpoints of an FPO run, the direction of the FPO surrogate gradient with that of the exact likelihood-ratio policy gradient. If the two directions disagree systematically — or if maximizing the FPO objective lowers the true likelihood of high-advantage actions — the central claim fails. A cheaper companion check is to test the paper's loss-to-ELBO identity directly by comparing the weighted denoising loss against a numerically estimated evidence lower bound across noise levels for a fixed policy.

Watch

Extended reading notes

Core claim

The central claim is that policy optimization for a flow-based generative policy can be cast as maximizing an advantage-weighted ratio built directly from the conditional flow matching loss. The proposed estimator is $\hat{r}_{\mathrm{FPO}}(\theta) = \exp\big(\hat{L}^{\mathrm{CFM}}_{\theta_{\mathrm{old}}}(a_t; o_t) - \hat{L}^{\mathrm{CFM}}_\theta(a_t; o_t)\big)$, with both losses evaluated at the same Monte Carlo draws of flow timestep $\tau$ and noise $\epsilon$; it is a drop-in replacement for the PPO likelihood ratio inside the clipped surrogate objective. Because the weighted denoising loss equals the negative evidence lower bound up to a parameter-independent constant, this ratio is an ELBO ratio that decomposes into the true likelihood ratio times an inverse KL-gap factor, so maximizing it both raises the likelihood of high-advantage actions and tightens the approximation to the true likelihood. The paper shows the Monte Carlo gradient estimate is directionally unbiased even though the ratio's scale is upward-biased, so even one $(\tau, \epsilon)$ pair yields a usable update. Empirically, FPO trains from scratch on continuous control and humanoid tracking; the paper reports that flow policies outperform Gaussian policies under sparse goal conditioning and attributes this to the expressive, multimodal distributions that flow matching can represent.

Load-bearing premise

The load-bearing premise is that a flow model's denoising training error equals the negative of a lower bound on how likely the sampled action is under the policy, up to a constant that does not depend on the policy's parameters — a link the paper itself notes holds for noise-corrupted versions of the action rather than the clean sampled action under general flow-matching schedules — so if that link proves unfaithful in practice, the FPO ratio is not a stand-in for the true likelihood ratio.

Editorial extensions

If this is right

  • FPO slots into standard PPO-style actor-critic pipelines as a drop-in replacement: the clipping, advantage estimation (GAE or GRPO), and value-function updates stay the same, only the likelihood-ratio term changes.
  • Because the sampler is a black box at both training and inference time, an FPO-trained policy can be deployed with fast deterministic ODE samplers, higher-order integrators, or any number of integration steps — a flexibility denoising-MDP methods lack.
  • Flow policies trained by FPO represent multimodal action distributions at states with several good actions, while a Gaussian policy collapses onto a single mode.
  • In under-conditioned humanoid control, where the goal signal is only the root or root-and-hands, FPO learns viable single-stage tracking policies while Gaussian PPO largely fails.
  • Increasing the Monte Carlo sample count $N_{mc}$ improves FPO's average reward without extra environment interaction, since the extra samples only reduce the bias of the ratio estimate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the FPO ratio is a faithful likelihood proxy, the same objective could fine-tune pre-trained behavior-cloned flow policies — robot visuomotor policies, for instance — with task reward, because the objective never depends on the sampler; the paper names this direction for future work but does not test it.
  • The directionally-unbiased gradient property at $N_{mc} = 1$ suggests that variance-reduction techniques, such as control variates over noise draws or stratified sampling over flow timesteps, could shrink the ratio's upward scale bias more cheaply than adding Monte Carlo samples; the paper does not explore these.
  • The paper attributes its failed image-diffusion fine-tuning to classifier-free-guidance artifacts amplified by self-generated data, so a testable extension is that stabilizing or removing guidance during FPO updates could restore stable image alignment.
  • Because the FPO ratio carries an inverse KL-gap correction, FPO's updates deviate from exact likelihood-ratio PPO by a factor that should shrink as the policy converges; measuring that factor on a tractable flow model would show how far FPO is from true PPO at any training stage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Flow Policy Optimization (FPO), an on-policy reinforcement learning algorithm for training policies represented as flow-based generative models. FPO replaces the exact likelihood ratio used in PPO with a ratio computed from conditional flow matching losses, r_FPO(θ) = exp(LCFM,θ_old(at; ot) - LCFM,θ(at; ot)), thereby avoiding expensive likelihood or divergence estimation. The authors derive this ratio from the Kingma and Gao identity that relates a weighted denoising loss to the evidence lower bound (ELBO), and they decompose the resulting ratio into a likelihood ratio times an inverse KL-gap correction. For the uniform-weighting diffusion schedule, the connection to the clean-action ELBO is exact; for more general monotonic schedules such as the optimal-transport flow matching schedule used in the experiments, the supplementary material shows only that the loss equals a negative expected ELBO over noise-perturbed actions. The method is evaluated on a GridWorld environment, ten MuJoCo Playground continuous-control tasks, and a physics-based humanoid control task, with FPO reported to outperform Gaussian PPO and DPPO on most Playground tasks and to succeed in under-conditioned humanoid settings where Gaussian policies struggle. The paper also reports a negative result for fine-tuning image diffusion models with FPO.

Significance. If the theoretical framing is corrected, FPO is a practically significant contribution: it is simple to implement, agnostic to the choice of flow/diffusion sampler, and the empirical results suggest that flow-based policies can be trained from scratch with on-policy RL and can capture multimodal action distributions. The paper includes useful ablations (number of Monte Carlo pairs, epsilon-MSE versus velocity-MSE, clipping epsilon) and an honest discussion of limitations, including the image-generation instability. The main weakness is that the central derivation is presented for the diffusion schedule while the experiments use an optimal-transport schedule, for which only a weaker ELBO identity holds; this gap affects the theoretical status of the method as a 'drop-in replacement' for the PPO likelihood ratio. The humanoid experiments also lack statistical rigor. These issues are substantial but appear correctable within the manuscript's scope.

major comments (3)
  1. [Section 3.4, Eq. (15); Supplementary A.1.2, Eq. (21)] The main derivation uses the identity L^w_θ(a_t) = -ELBO_θ(a_t) + c, which Kingma and Gao prove for the uniform-weighting diffusion schedule. However, Algorithm 1 and the experiments use an optimal-transport flow-matching schedule (velocity target a_t - ε), and for this case the supplementary's Eq. (21) provides only L^w_θ(a_t) = -E_{p_w(τ), q(a_τ|a_t)}[ELBO_τ(a_τ^t)] + c_1. Consequently, in the experiments r_FPO is a ratio of expected ELBOs over noise-perturbed samples, not a clean-action likelihood ratio. The main-text sentence 'Similar results hold for many popular schedules, including optimal transport and variance preserving schedules' (Section 3.4) is therefore misleading, and the claim that the estimator 'recovers the exact FPO ratio in the limit' is only true for the diffusion schedule. Maximizing the OT-schedule objective need not increase π_θ(a_t|o_t), so the 'drop-in replacement' status of r_FPO is not established for the schedules actually used. The authors should either restrict the theoretical claims to the diffusion schedule or explicitly reframe FPO for general schedules as an advantage-weighted expected-ELBO objective for noise-perturbed actions, stating clearly how this deviates from the policy gradient in Eq. (2).
  2. [Section 3.4, Eqs. (18)-(20)] The argument that the one-sample ratio estimator is 'directionally unbiased' is mathematically incorrect. From Eq. (19), ∇_θ r̂_FPO = -r̂_FPO ∇_θ ℓ, so taking expectations gives E[∇_θ r̂_FPO] = -E[r̂_FPO ∇_θ ℓ], which is not equal to -∇_θ L^w_θ = ∇_θ ELBO_θ(a_t). Equation (20) proves unbiasedness only for the unweighted loss gradient, not for the gradient of the ratio estimator. If the intended claim is that r̂ ≈ 1 within a PPO trust region, it should be presented as an approximation rather than as an exact identity. This matters because the text uses the (purported) unbiasedness to justify the use of Nmc = 1.
  3. [Section 4.3, Table 2] The humanoid results, which support the headline claim that FPO enables under-conditioned control where Gaussian policies fail, are reported without any indication of the number of seeds or the variance across runs. Table 2 gives single values for success rate, alive duration, and MPJPE, and Figure 4a shows single learning curves. Without error bars or at least a seed count, it is not possible to assess whether the differences between FPO and Gaussian PPO are reliable. The authors should report multiple seeds with standard errors, or clearly state the evaluation protocol if these numbers come from a single run.
minor comments (5)
  1. [Section 3.4] The statement 'Both flow matching and diffusion models optimize the ELBO using a conditional flow matching loss' is imprecise: they optimize a weighted denoising objective, and the equivalence to an ELBO is schedule-dependent, as the paper itself shows in the supplementary.
  2. [Algorithm 1] The Require line uses the symbol ϵ for the clip parameter, but ϵ is also used for the Gaussian noise throughout the paper; this is confusing and should be consistently labeled ε_clip.
  3. [Eqs. (7)-(14)] The loss definition switches from velocity prediction in Eqs. (7)-(9) to ε-prediction in Eqs. (13)-(14). The main text should explicitly state which parameterization is used in Algorithm 1 and in the experiments, especially since Table 1 shows that 'ϵ-MSE' and 'u-MSE' give different results.
  4. [Figure 1] The figure caption and text refer to 'left/center/right' panels, but the panels are not labeled in the figure itself. The caption also mentions 'a different latent noise vector' without specifying the number of denoising steps or the conditioning state, which makes the visualization harder to interpret.
  5. [Section 3.3] Equation (10) defines ELBO_θ(a_t|o_t) = log π_θ(a_t|o_t) - D^KL_θ, which conflates the ELBO with the log-likelihood minus the KL gap; this is acceptable as a definition but should be stated more carefully to avoid implying that the ELBO equals the log-likelihood.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: FPO's loss-difference ratio is justified by an external ELBO theorem, with a schedule-scope caveat admitted in the supplementary; only a minor non-load-bearing self-citation exists.

full rationale

The FPO ratio is defined directly as an exponentiated difference of conditional flow matching losses (Eq. 6), and the paper's theoretical justification for equating this with a likelihood ratio comes from an external result by Kingma and Gao (Eq. 15, Ref. [63]), not from a self-referential definition or from the authors' own prior work. The derivation is therefore not circular in the sense of Eq. X equal to Eq. Y by construction, and no fitted parameter is renamed as a prediction. The main text states that 'similar results hold for many popular schedules, including optimal transport' immediately after presenting Eq. 15 for the diffusion schedule; however, the supplementary material (A.1.1, A.1.2) explicitly admits that for general monotonic weightings, including the OT-CFM schedule used in the experiments, the weighted loss equals only the negative expected ELBO over noise-perturbed samples (Eq. 21), not the clean-action ELBO. This is a scope mismatch and a correctness risk for the 'drop-in replacement' claim, but it is not circularity. The only self-citation is to the authors' prior humanoid work [29] in Section 4.3, used to contextualize existing sparse-goal methods; it is not load-bearing and does not support any FPO-specific claim. No uniqueness theorem is imported from the authors' own work. The empirical evaluation is benchmarked against external baselines (Gaussian PPO, DPPO), so the central claim retains independent content. Overall, the derivation is self-contained against an external theorem and external benchmarks, yielding a low circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the external ELBO-loss equivalence and the heuristic that the flow matching loss difference approximates the likelihood ratio. No new physical entities are introduced. Hyperparameters like Nmc and the clip parameter are tuned on the tasks, but they are standard algorithmic choices rather than physically fitted constants.

free parameters (2)
  • Nmc (number of Monte Carlo (tau, epsilon) samples per action) = 8 (best in experiments; 1 and 4 also tested)
    Controls bias-variance tradeoff in the ratio estimate; Table 1 shows performance varies with Nmc.
  • Clip parameter (epsilon_clip) = 0.05 (swept over 0.01, 0.05, 0.1, 0.2, 0.3)
    Trust region size for PPO-style clipping; Table 1 shows sensitivity to this value.
assumptions (4)
  • standard math Kingma & Gao's theorem: for diffusion schedules, the weighted denoising loss equals the negative ELBO plus a constant independent of model parameters (Eq. 15).
    Basis for replacing the log-likelihood ratio with the flow matching loss ratio in Section 3.4.
  • standard math For general monotonic flow schedules, the weighted loss equals the negative expected ELBO of noise-perturbed samples plus a constant (supplementary Eq. 21).
    Extends the derivation to flow matching, but the ELBO is for noised samples, weakening the interpretation.
  • domain assumption The Monte Carlo estimate of the ratio using Nmc samples is directionally unbiased in expectation (Eq. 19-20).
    Assumes the gradient of the ratio estimator preserves the direction of the true ELBO gradient.
  • domain assumption PPO clipping with a biased ratio still yields stable policy improvement.
    The method relies on clipping to control updates, but the ratio bias is not fully analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Flow Matching Policy Gradients." pith.science (2026). https://pith.science/paper/XTWI2FWN

@misc{pith2026250721053,
  author       = {Pith},
  title        = {Pith review of: Flow Matching Policy Gradients},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XTWI2FWN}},
  note         = {Machine review of arXiv:2507.21053}
}
read the original abstract

Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement learning algorithm that brings flow matching into the policy gradient framework. FPO casts policy optimization as maximizing an advantage-weighted ratio computed from the conditional flow matching loss, in a manner compatible with the popular PPO-clip framework. It sidesteps the need for exact likelihood computation while preserving the generative capabilities of flow-based models. Unlike prior approaches for diffusion-based reinforcement learning that bind training to a specific sampling method, FPO is agnostic to the choice of diffusion or flow integration at both training and inference time. We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks. We find that flow-based models can capture multimodal action distributions and achieve higher performance than Gaussian policies, particularly in under-conditioned settings.

Figures

Figures reproduced from arXiv: 2507.21053 by the authors.

Figure 1
Figure 1. Grid World. (Left) 25×25 GridWorld with green goal cells. Each arrow shows a denoised action sampled from the FPO-trained policy, conditioned on a different latent noise vector. (Center) At the saddle-point state (⋆) shown on the left, we visualize three denoising steps τ as the initial Gaussian gradually transforms into the target distribution through the learned flow, illustrated by the deformation of the coordina… view at source ↗
Figure 2
Figure 2. Comparison between FPO and Gaussian PPO [20] on DM Control Suite tasks. Results show evaluation reward mean and standard error (y-axis) over 60M environment steps (x-axis). We run 5 seeds for each task; the curve with the highest terminal evaluation reward is bolded. 0 15 30 45 60 0 200 400 600 Eval Reward BallInCup 0 15 30 45 60 0 200 400 600 800 CartpoleBalance 0 15 30 45 60 0 200 400 600 800 CheetahRun 0 15 30 45… view at source ↗
Figure 3
Figure 3. Comparison between FPO and DPPO [59] on DM Control Suite tasks. Results show evaluation reward mean and standard error (y-axis) over 60M environment steps (x-axis). We run 5 seeds for each task; the curve with the highest terminal evaluation reward is bolded. policy by visualizing the flow over its denoising steps at the saddle point. The initial Gaussian evolves into a bimodal distribution, demonstrating that the p… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Physics-based Humanoid Control. (a) The curves show that FPO performance is close to that of Gaussian-PPO when conditioning on all joints and surpasses it when goals are reduced to the root or root+hands, indicating stronger robustness to sparse conditioning. (b) In th…

Discussion (0). Sign in to comment.

Forward citations

Cited by 37 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    QGF performs test-time policy optimization for flow models in RL by guiding a behavior-cloned reference policy with value-function gradients, achieving strong results on high-dimensional offline RL benchmarks without ...

  2. Sample-Efficient Diffusion-based Reinforcement Learning with Critic Guidance

    cs.RO 2026-05 unverdicted novelty 7.0 of 10

    CGPO integrates training-free critic guidance into diffusion denoising to produce high-Q actions as regression targets, yielding SOTA results on MuJoCo locomotion and successful Franka arm grasping.

  3. Explicit Critic Guidance for Aligning Diffusion Models

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Introduces a state-aligned latent actor-critic framework that lets diffusion models act as their own timestep-conditioned value functions for trajectory-level RL post-training and inference steering.

  4. DISA: Offline Importance Sampling for Distribution-Matching LLM-RL

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    DISA decouples partition function estimation using offline importance sampling for distribution-matching LLM-RL, matching or exceeding online baselines like FlowRL on math and code benchmarks while retaining more stra...

  5. Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT

    cs.RO 2026-05 unverdicted novelty 7.0 of 10

    ConSFT prevents catastrophic forgetting in fine-tuning flow-matching VLAs by dynamically scaling gradients based on model confidence, retaining over 20% more pre-trained capability than standard SFT without prior data...

  6. Generative Actor-Critic with Soft Bridge Policies

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    SoftGAC defines a stochastic bridge from base to action latent that converts the MaxEnt objective into a tractable relative-entropy term reducible to control energy, achieving competitive returns with one-pass sampling.

  7. Genuine pair density wave order on the kagome lattice

    cond-mat.supr-con 2026-04 unverdicted novelty 7.0 of 10

    A genuine primary pair-density-wave phase emerges as a competing ground state in a two-orbital kagome Hubbard model over a wide parameter range, driven by sublattice- and orbital-polarized Fermi pockets.

  8. Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    SCORE constrains sim RL to the support of a real-data policy via flow steering, raising average success on eight dexterous tasks from 37.8% to 89.9%.

  9. Transferring Contact, Not Just Motion: Compliant Grasping Across Dexterous Hands

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    A cross-embodiment force-position interface with system-identified torque calibration enables a flow-matching policy to perform transferable compliant grasping on heterogeneous dexterous hands.

  10. DiPOD: Diffusion Policy Optimization without Drifting Apart

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    DiPOD stabilizes diffusion policy optimization by interleaving self-distillation with gradient updates via an on-policy ELBO regularizer, yielding more stable training and higher rewards than prior methods.

  11. GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    GenPO++ achieves exact Jacobian-free likelihood ratio computation for generative flow policies by embedding history states as auxiliary memory in a high-order reversible ODE solver.

  12. GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    GDSD reduces RL for dLLMs to likelihood-free self-distillation via a normalization-free logit-matching objective, outperforming ELBO methods with more stable training on LLaDA-8B and Dream-7B.

  13. Adversarial Dual On-Policy Distillation from Expressive Teacher

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    FA-OPD co-trains a flow-matching teacher and MLP student via adversarial dual on-policy distillation, improving robustness over baselines on six robot benchmarks with noisy or limited demonstrations.

  14. DEFLECT: Delay-Robust Execution via Flow-matching Likelihood-Estimated Counterfactual Tuning for VLA Policies

    cs.RO 2026-05 unverdicted novelty 6.0 of 10

    DEFLECT is an offline post-training method that improves async VLA policy success rates under high inference delays by using flow-matching likelihood ratios on counterfactual fresh/stale action pairs from a frozen ref...

  15. Video Models Can Reason with Verifiable Rewards

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    VideoRLVR uses SDE-GRPO optimization, dense decomposed rewards, and Early-Step Focus to train video diffusion models on verifiable reasoning tasks, outperforming supervised fine-tuning and other video generators on Ma...

  16. Driving Intents Amplify Planning-Oriented Reinforcement Learning

    cs.RO 2026-05 unverdicted novelty 6.0 of 10

    DIAL uses intent-conditioned CFG and multi-intent GRPO to expand and preserve diverse modes in continuous-action preference RL, lifting RFS to 9.14 and surpassing both prior best (8.5) and human demonstration (8.13).

  17. Discrete Flow Matching for Offline-to-Online Reinforcement Learning

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DRIFT enables stable offline-to-online fine-tuning of CTMC policies in discrete RL via advantage-weighted discrete flow matching, path-space regularization, and candidate-set approximation.

  18. UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation

    cs.RO 2026-05 unverdicted novelty 6.0 of 10

    UniSteer unifies human corrective actions and noise-space RL for VLA adaptation by inverting actions to noise targets, raising success rates from 20% to 90% in 66 minutes across four real-world manipulation tasks.

  19. UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation

    cs.RO 2026-05 conditional novelty 6.0 of 10

    By inverting a frozen flow-matching robot policy with fixed-point iteration, UniSteer converts human corrections into noise-space supervision and lifts real-world VLA adaptation success from 20% to 90% in about 66 min...

  20. OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    OGPO enables sample-efficient full-finetuning of generative control policies via off-policy critics and modified PPO, achieving SOTA on robot manipulation tasks while rescuing poorly initialized behavior cloning polic...

  21. OGPO: Sample Efficient Full-Finetuning of Generative Control Policies

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    OGPO is a sample-efficient off-policy method for full finetuning of generative control policies that reaches SOTA on robotic manipulation tasks and can recover from poor behavior-cloning initializations without expert data.

  22. V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    V-GRPO makes ELBO surrogates stable and efficient for online RL alignment of denoising models, delivering SOTA text-to-image performance with 2-3x speedups over MixGRPO and DiffusionNFT.

  23. Positive-Only Drifting Policy Optimization

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    PODPO is a likelihood-free generative policy optimization method for online RL that steers actions to high-return regions using only positive-advantage samples and local contrastive drifting.

  24. FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Sol-RL decouples FP4-based candidate exploration from BF16 policy optimization in diffusion RL, delivering up to 4.64x faster convergence with maintained or superior alignment performance on models like FLUX.1 and SD3.5.

  25. From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning

    cs.RO 2026-03 accept novelty 6.0 of 10

    Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...

  26. SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows

    cs.RO 2026-02 unverdicted novelty 6.0 of 10

    SERNF achieves sample-efficient real-world fine-tuning of multimodal dexterous policies by pairing exact-likelihood normalizing flow policies with action-chunked value critics.

  27. SERNF: Sample-Efficient Real-World Dexterous Policy Fine-Tuning via Action-Chunked Critics and Normalizing Flows

    cs.RO 2026-02 conditional novelty 6.0 of 10

    SERNF fine-tunes dexterous manipulation policies on real hardware by pairing normalizing-flow policies with action-chunked critics and conservative off-policy RL.

  28. Training Diffusion Policies via Prior-Mapping Co-Evolution

    cs.LG 2025-12 conditional novelty 6.0 of 10

    GoRL outperforms Gaussian and generative RL baselines on continuous control by optimizing a Gaussian latent policy while a separately trained diffusion or flow decoder maps fixed noise to actions.

  29. Reinforcement Fine-Tuning of Flow-Matching Policies for Vision-Language-Action Models

    cs.LG 2025-10 conditional novelty 6.0 of 10

    FPO fine-tunes flow-matching vision-language-action policies with a PPO-style objective that replaces intractable policy ratios with per-sample conditional flow-matching loss differences, reaching 87.2% average succes...

  30. FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning

    cs.LG 2025-10 conditional novelty 6.0 of 10

    An online imitation-learning method uses a flow-matching teacher's class-conditional loss as a reward and a regularizer to train a simple MLP policy, beating cloning and adversarial-imitation baselines on five of six tasks.

  31. RLMM-Flow: A Flow-based Mobile Manipulation Framework with Latent-Space Reinforcement Learning

    cs.RO 2026-07 conditional novelty 5.0 of 10

    Steering a frozen flow-based whole-body motion policy in latent space with a warmed-up critic and coarse-to-fine residuals improves mobile manipulation success and safety.

  32. PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration

    cs.CV 2026-07 conditional novelty 5.0 of 10

    PAVXploreRL post-trains action-conditioned world models with VJEPA-2 latent rewards and perturbed 'OOD' actions, reporting a 5.6% average gain and lowered policy-overestimation bias.

  33. NavCMPO: Critic-Guided MeanFlow Policy Optimization for Adaptive Navigation

    cs.RO 2026-07 conditional novelty 5.0 of 10

    A two-stage navigation policy using five-step MeanFlow generation, critic-guided trajectory refinement, and PPO fine-tuning reports higher success and lower latency than a matched NavDP baseline.

  34. Driving Intents Amplify Planning-Oriented Reinforcement Learning

    cs.RO 2026-05 unverdicted novelty 5.0 of 10

    DIAL expands continuous-action driving policies via intent-conditioned flow matching and multi-intent GRPO, lifting best-of-N preference scores above human demonstrations for the first time on WOD-E2E.

  35. Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT

    cs.RO 2026-05 unverdicted novelty 5.0 of 10

    ConSFT is a gradient-scaling fine-tuning objective for flow-matching VLAs that bounds parameter disruption via model-confidence weighting, yielding over 20% better capability retention than vanilla SFT on LIBERO and RoboTwin.

  36. FAIL: Flow Matching Adversarial Imitation Learning for Image Generation

    cs.CV 2026-02 conditional novelty 5.0 of 10

    Post-training of flow matching can be framed as adversarial imitation learning, and the proposed FAIL methods improve FLUX's generation quality using 13K expert images without preference pairs.

  37. ReFPO: Reflow Regularization for Flow Matching Policy Gradients

    cs.RO 2026-06 unverdicted novelty 4.0 of 10

    ReFPO adds explicit Reflow regularization to FPO, stabilizing PPO-style training and supporting high-fidelity one-step inference across GridWorld, MuJoCo, and Humanoid tasks.

Reference graph

Works this paper leans on

84 extracted references · 25 canonical work pages · cited by 32 Pith papers

  1. [1]

    Hierarchical text-conditional image generation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022

    arXiv 2022

  2. [2]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. 2022

    2022

  3. [4]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https://openai.com/research/ video-generation-models-as-world-simulators

    2024

  4. [5]

    Movie gen: A cast of media foundation models

    Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410.13720, 2024

    arXiv 2024

  5. [6]

    Veo-Team, :, Agrim Gupta, Ali Razavi, Andeep Toor, Ankush Gupta, Dumitru Erhan, Eleni Shaw, Eric Lau, Frank Belletti, Gabe Barth-Maron, Gregory Shaw, Hakan Erdogan, Hakim Sidahmed, Henna Nandwani, Hernan Moraldo, Hyunjik Kim, Irina Blok, Jeff Donahue, José Lezama, Kory Mathewson, Kurtis David, Matthieu Kim Lorrain, Marc van Zee, Medhini Narasimhan, Miaose...

    2024

  6. [7]

    Plumbley

    Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. Audioldm: Text-to-audio generation with latent diffusion models, 2023. URL https://arxiv.org/abs/2301.12503

    arXiv 2023

  7. [8]

    Diffwave: A versatile diffusion model for audio synthesis, 2021

    Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis, 2021. URL https://arxiv.org/abs/2009.09761

    arXiv 2021

  8. [10]

    Action-minimization meets generative modeling: Efficient transition path sampling with the onsager-machlup functional

    Sanjeev Raja, Martin Šípka, Michael Psenka, Tobias Kreiman, Michal Pavelka, and Aditi S Krishnapriyan. Action-minimization meets generative modeling: Efficient transition path sampling with the onsager-machlup functional. arXiv preprint arXiv:2504.18506, 2025

    arXiv 2025

Show all 84 references
  1. [11]

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025

    2025 arXiv

  2. [12]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    2024 arXiv

  3. [13]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling, 2023. URL https://arxiv.org/abs/2210.02747

    2023 arXiv

  4. [14]

    Mujoco playground

    Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. arXiv preprint arXiv:2502.08844, 2025. 11

    2025 arXiv

  5. [15]

    Sutton, David McAllester, Satinder P

    Richard S. Sutton, David McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Neural Information Processing Systems (NeurIPS) , pages 1057– 1063, 1999

    1999

  6. [16]

    Simple statistical gradient-following algorithms for connectionist reinforce- ment learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 1992

    1992

  7. [17]

    Sham M. Kakade. A natural policy gradient. InProceedings of the 14th International Conference on Neural Information Processing Systems (NeurIPS) , pages 1531–1538, 2002

    2002

  8. [18]

    Natural actor–critic

    Jan Peters and Stefan Schaal. Natural actor–critic. Neurocomputing, 71(7–9):1180–1190, 2008

    2008

  9. [19]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889–1897. PMLR, 2015

    2015

  10. [20]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  11. [21]

    Asynchronous methods for deep reinforcement learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning (ICML) , pages...

    1928

  12. [22]

    Sample efficient actor–critic with experience replay

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando de Freitas. Sample efficient actor–critic with experience replay. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NeurIPS) , pages 1061–1071, 2016

    2016

  13. [23]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  14. [24]

    Benchmarking deep reinforcement learning for continuous control

    Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329–1338. PMLR, 2016

    2016

  15. [25]

    Open rl benchmark: Comprehensive tracked experiments for reinforcement learning

    Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Viktor Makoviychuk, Denys Makoviichuk, Mohamad H Danesh, et al. Open rl benchmark: Comprehensive tracked experiments for reinforcement learning. arXiv...

    2024 arXiv

  16. [26]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 91–100. PMLR, 2...

    2022

  17. [27]

    Curiosity-driven learning of joint locomotion and manipulation tasks

    Clemens Schwarke, Victor Klemm, Matthijs van der Boon, Marko Bjelonic, and Marco Hutter. Curiosity-driven learning of joint locomotion and manipulation tasks. In Proceedings of The 7th Conference on Robot Learning , volume 229 of Proceedings of Machine Learning Research, pages...

  18. [28]

    Sym- metry considerations for learning task symmetric robot policies

    Mayank Mittal, Nikita Rudin, Victor Klemm, Arthur Allshire, and Marco Hutter. Sym- metry considerations for learning task symmetric robot policies. In 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA) , pages 7433–7439, 2024. doi: 10.1109/ICRA57147.2024.10611493

    2024

  19. [29]

    Visual imitation enables contextual humanoid control

    Arthur Allshire, Hongsuk Choi, Junyi Zhang, David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, and Angjoo Kanazawa. Visual imitation enables contextual humanoid control. arXiv preprint arXiv:2505.03729, 2025. 12

    2025 arXiv

  20. [30]

    Solving rubik’s cube with a robot hand

    Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019

    1910 arXiv

  21. [31]

    A system for general in-hand object re-orientation

    Tao Chen, Jie Xu, and Pulkit Agrawal. A system for general in-hand object re-orientation. Conference on Robot Learning, 2021

    2021

  22. [32]

    General in-hand object rotation with vision and touch

    Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. General in-hand object rotation with vision and touch. In Conference on Robot Learning, pages 2549–2564. PMLR, 2023

    2023

  23. [33]

    From simple to complex skills: The case of in-hand object reorientation

    Haozhi Qi, Brent Yi, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. From simple to complex skills: The case of in-hand object reorientation. arXiv preprint arXiv:2501.05439, 2025

    2025 arXiv

  24. [34]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 2022

    2022

  25. [35]

    Brown, Miljan Martic, Shane Legg, and Dario Amodei

    Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL https://arxiv.org/abs/ 1706.03741

    2023 arXiv

  26. [36]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  27. [37]

    Mistral-AI, :, Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Saga...

    2025 arXiv

  28. [38]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 2020

    2020

  29. [39]

    Denoising diffusion implicit models, 2022

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2022. URL https://arxiv.org/abs/2010.02502

    2022 arXiv

  30. [40]

    High- resolution image synthesis with latent diffusion models, 2022

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL https://arxiv.org/ abs/2112.10752

    2022 arXiv

  31. [41]

    Generative modeling by estimating gradients of the data distribution, 2020

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020. URL https://arxiv.org/abs/1907.05600

    2020 arXiv

  32. [42]

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022. URL https://arxiv.org/abs/2204.03458

    2022 arXiv

  33. [43]

    Make-a-video: Text-to-video generation without text-video data, 2022

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data, 2022. URL https://arxiv.org/abs/ 2209.14792

    2022 arXiv

  34. [44]

    Kingma, Ben Poole, Mohammad Norouzi, David J

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022. URL https: //arxiv.org/abs/2210.02303

    2022 arXiv

  35. [45]

    Grad- tts: A diffusion probabilistic model for text-to-speech, 2021

    Vadim Popov, Ivan V ovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech, 2021. URL https://arxiv.org/abs/ 2105.06337

    2021 arXiv

  36. [46]

    Weiss, Mohammad Norouzi, Najim Dehak, and William Chan

    Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. Wavegrad 2: Iterative refinement for text-to-speech synthesis, 2021. URL https://arxiv.org/abs/2106.09660

    2021 arXiv

  37. [47]

    π0: A vision-language-action flow model for general robot control, 2024

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, Jam...

    2024 arXiv

  38. [48]

    Gr00t n1: An open foundation model for generalist humanoid robots, 2025

    NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llont...

    2025 arXiv

  39. [49]

    The superposition of diffusion models using the itô density estimator, 2025

    Marta Skreta, Lazar Atanackovic, Avishek Joey Bose, Alexander Tong, and Kirill Neklyudov. The superposition of diffusion models using the itô density estimator, 2025. URL https: //arxiv.org/abs/2412.17762

    2025 arXiv

  40. [50]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research , 2024

    2024

  41. [51]

    Tenenbaum, Tommi S

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations , 2023

    2023

  42. [52]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991, 2022

    2022 arXiv

  43. [53]

    Aligning text-to-image models using human feedback

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023

    2023 arXiv

  44. [54]

    Training diffusion models with reinforcement learning

    Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301, 2023

    2023 arXiv

  45. [55]

    Flow-grpo: Training flow matching models via online rl

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470, 2025

    2025 arXiv

  46. [56]

    Learning a diffusion model policy from rewards via q-score matching

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752, 2023

    2023 arXiv

  47. [57]

    Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control, 2025

    Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control, 2025. URL https://arxiv.org/abs/2505.22642

    2025 arXiv

  48. [58]

    Addressing function approximation error in actor-critic methods, 2018

    Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL https://arxiv.org/abs/1802.09477

    2018 arXiv

  49. [59]

    Diffusion policy policy optimization

    Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. arXiv preprint arXiv:2409.00588, 2024

    2024 arXiv

  50. [60]

    High- dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  51. [61]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems , 35: 26565–26577, 2022

    2022

  52. [62]

    Kingma, Tim Salimans, Ben Poole, and Jonathan Ho

    Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models,

  53. [63]

    Kingma and Ruiqi Gao

    Diederik P. Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation, 2023. URL https://arxiv.org/abs/2303.00848

    2023 arXiv

  54. [64]

    Openai gym, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

    2016

  55. [65]

    Gymnasium: A standard interface for reinforcement learning environments

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024

    2024 arXiv

  56. [66]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , 2012. 15

    2012

  57. [67]

    Isaac gym: High performance gpu-based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021

    2021 arXiv

  58. [68]

    Ppo-for-beginners: A simple, well-styled ppo implementation in pytorch

    Eric Yang Yu. Ppo-for-beginners: A simple, well-styled ppo implementation in pytorch. https://github.com/ericyangyu/PPO-for-Beginners , 2020. GitHub repository

    2020

  59. [69]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  60. [70]

    dm_control: Software and tasks for continuous control

    Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020

    2020

  61. [71]

    Brax–a differentiable physics engine for large scale rigid body simulation

    C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax–a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.13281, 2021

    2021 arXiv

  62. [72]

    Adam: A method for stochastic optimization

    Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  63. [73]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023

    2023

  64. [74]

    Deepmimic: Example- guided deep reinforcement learning of physics-based character skills

    Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1–14, 2018

    2018

  65. [75]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5442–5451, 2019

    2019

  66. [76]

    Maskedmimic: Unified physics-based character control through masked motion inpainting

    Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG), 43(6):1–21, 2024

    2024

  67. [77]

    Clone: Closed-loop whole-body humanoid teleoperation for long-horizon tasks

    Yixuan Li, Yutang Lin, Jieming Cui, Tengyu Liu, Wei Liang, Yixin Zhu, and Siyuan Huang. Clone: Closed-loop whole-body humanoid teleoperation for long-horizon tasks. arXiv preprint arXiv:2506.08931, 2025

    2025 arXiv

  68. [78]

    Universal humanoid motion representations for physics-based control

    Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Universal humanoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582, 2023

    2023 arXiv

  69. [79]

    Omnigrasp: Grasping diverse objects with simulated humanoids

    Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Omnigrasp: Grasping diverse objects with simulated humanoids. In Advances in Neural Information Processing Systems, volume 37, pages 2161–2184, 2024

    2024

  70. [80]

    Ai models collapse when trained on recursively generated data

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data. Nature, 631(8022): 755–759, 2024

    2024

  71. [81]

    The curse of recursion: Training on generated data makes models forget

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023

    2023 arXiv

  72. [82]

    Self-consuming generative models go mad

    Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G Baraniuk. Self-consuming generative models go mad. International Conference on Learning Representations (ICLR), 2024. 16

    2024

  73. [83]

    Progressive distillation for fast sampling of diffusion models

    Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022

    2022 arXiv

  74. [84]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL https:// arxiv.org/abs/2207.12598. 17 Flow Matching Policy Gradients Supplementary Material In this supplementary material, we discuss the deferred proofs of technical results, elaborate on the details...

    2022 arXiv

  75. [86]

    Low CFG scales tend to encourage bluriness while high CFG scales encourage saturation and sharp geometric artifacts

    In the bottom row, we display images from a training run using a CFG scale of 2. Low CFG scales tend to encourage bluriness while high CFG scales encourage saturation and sharp geometric artifacts. Both diverge after a few hundred epochs even with tuned hyperparameters. This p...

  76. [2023]

    URL https://arxiv.org/abs/2107.00630

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.