Pith

open record

sign in

arxiv: 2607.06262 · v1 · pith:UFNJM5HA · submitted 2026-07-07 · cs.RO

Optimal Transport Q-Learning for Flow Policy Steering and Acceleration

Andreas Sochopoulos , Esmeralda S. Whitammer , Nikolaos Tsagkas , Jo\~ao Moura , Michael Gienger , Sethu Vijayakumar This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved 2026-07-08 11:28 UTCglm-5.2pith:UFNJM5HArecord.jsonopen to challenge →

Figure 1
Figure 1. Figure 1: Energy-weighted Flow Matching (EWFM) [22] and weighted-marginal COT flow matching (wCOT) flows trained to generate the tilted conditional two moons distribution from the 8 Gaussians distribution. Generation with 100 (top row) and 2 (bottom row) euler integration steps is shown. The sample… reproduced from arXiv: 2607.06262
classification cs.RO
keywords optimal transportflow matchingreinforcement learningrobot policypolicy accelerationconditional optimal transportoffline RLvision-language-action model
0
0 comments X

The pith

Optimal transport steers flow policies to high reward in 2-3 steps

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Optimal Transport Q-Learning (OTQL), a method that simultaneously fine-tunes and accelerates flow-based robot policies. The core idea is to use advantage-weighted conditional optimal transport (COT) to couple noise samples with action samples during flow matching training. By reweighting the target marginal of the OT problem using advantage values w_j = e^{λA(s_j,a_j)}, the coupling concentrates transport mass on high-value actions. Because the OT plan produces straight-line transport paths between noise and action distributions, the resulting flow can be integrated in just 2-3 neural function evaluations at inference time, rather than the 10-100 steps typical of diffusion or flow policies. The method requires only 50-60 episodes of robot interaction to adapt suboptimal policies, raising success rates from 36% to 86% on single-task policies and from 38% to 76% on a vision-language-action model, while cutting inference cost by 70%.

Core claim

The central mechanism is the wCOT-CFM coupling: at each training step, a minibatch entropic optimal transport problem is solved between noise samples and action samples, where the target marginal is reweighted by advantage weights w_j ∝ e^{λA(s_j,a_j)}. Conditions (robot states) are injected into the OT cost via augmented vectors that concatenate the data with a scaled condition vector, ensuring the transport plan respects the conditional structure. The resulting optimal coupling Γ* is then used as the pairing distribution in a standard flow matching regression loss. This simultaneously achieves two goals: the advantage weighting steers the flow toward high-value actions (RL fine-tuning),and

What carries the argument

wCOT-CFM coupling (Eq. 3-5): advantage-weighted conditional optimal transport solved on condition-augmented vectors, used as the coupling distribution for flow matching regression

If this is right

  • Robot policies that currently require 10-100 inference steps could be adapted to 2-3 step inference without distillation, enabling real-time control on commodity hardware
  • VLA models that fail zero-shot on novel tasks could be adapted with 50-60 episodes of autonomous robot experience rather than new human demonstrations
  • The same mechanism could extend to any neural ODE-based generative model where one wants to steer toward a reward-weighted target distribution while maintaining few-step sampling
  • Offline-to-online RL pipelines could use OTQL as a bridge: pre-train with offline RL on suboptimal data, then refine with short online interaction budgets

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The minibatch OT approximation quality likely degrades gracefully in the action dimensions tested (tens to low hundreds) but may become a binding constraint for very high-dimensional action spaces or long action chunks, which could limit applicability to fine-grained dexterous manipulation
  • The per-task tuning of λ and w_max suggests an adaptive scheme that adjusts these parameters based on the sharpness of the learned Q-function could reduce the need for manual hyperparameter search
  • Since the method requires white-box access to the flow model's ODE formulation, it cannot directly post-train black-box API-based policy services, which may matter as robot foundation models become deployed as cloud services
  • The straight-path property of OT couplings could interact favorably with action chunking: if chunks lie near a low-dimensional manifold, the effective OT problem dimension is lower than the nominal action dimension, potentially explaining why the method works despite the stated high-dimensionality concern

Load-bearing premise

The entire method depends on a minibatch entropic optimal transport solver adequately approximating the true conditional OT plan between the noise distribution and the advantage-weighted target distribution. The authors acknowledge this approximation degrades in higher dimensions, and the practical need for weight clamping, rejection sampling, and per-task tuning of the temperature parameter λ suggests the approximation quality varies significantly across problem settings.

What would settle it

If the minibatch OT plan poorly approximates the true COT plan between noise and the advantage-weighted target distribution — for instance in high-dimensional action spaces or when the Q-function is sharply peaked — the learned flow would follow curved or suboptimal trajectories, undermining both the acceleration claim (paths would not be straight) and the steering claim (the flow would not concentrate on high-value actions).

Figures

Figures reproduced from arXiv: 2607.06262 by Andreas Sochopoulos, Esmeralda S. Whitammer, Jo\~ao Moura, Michael Gienger, Nikolaos Tsagkas, Sethu Vijayakumar.

Figure 2
Figure 2. Figure 2: OTQL is effective in offline-online RL settings 0 2 4 6 8 0.0 0.2 0.4 0.6 0.8 1.0 Success Rate OpenMicrowave-v0 0 2 4 6 8 ArrangeBoxes-v0 Environment Steps (×1k) OTQL (ours) FQL QC EWFM [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: real-wrold tasks used for the evaluation of OTQL [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: OTQL adaptation of base flow policies on 3 tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Ablation of OTQL for 𝜆, 𝑁 and inference steps NFE Temperature 𝜆. OTQL uses importance weights to assign more mass to low energy samples in the batch while calculating the empirical COT coupling. If the temperature 𝜆 is too high then there will be only a few samples in the batch that will be assigned a significant weight leading to a transport plan Γ ∗ that is concentrated around these few points. This mean… view at source ↗
Figure 7
Figure 7. Figure 7: The first (OpenMicrowave-v0) requires the robot to open the microwave door and the second (ArrangeBoxes-v0) requires the robot to arrange the two boxes on the correct colored planes. For both tasks we collect demonstrations using a gamepad by controlling only the position of the end￾effector and the gripper. We again utilized an ensemble for the critic but not the pessimistic Bellman update. All networks u… view at source ↗
Figure 8
Figure 8. Figure 8: real-world robot setup [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
read the original abstract

Diffusion and flow policies have recently demonstrated remarkable performance in robotic applications by accurately capturing multimodal robot trajectory distributions, especially in the context of vision language action (VLA) models. However, high quality policy performance also requires fast inference and high quality demonstrations, which are often hard to get. Lack of these leads to suboptimal policy behaviors and failure under distribution shifts. In this work we address the problem of fine-tuning and accelerating suboptimal flow-based policies using the robot's experience through RL post-training. We introduce Optimal Transport Q-Learning (OTQL), a new method for finetuning flow policies using advantage weighted conditional optimal transport flow matching. OTQL can finetune and accelerate flows with an interaction budget of 50-60 episodes while avoiding computationally expensive distillation in simulation and real-world robot tasks. Our results show that OTQL post-trains flow policies using the robot's own experience, increasing average success percentage of single-task policies from 36% to 86% and of a pre-trained VLA from 38% to 76% while reducing the number of inference steps per action generation by 70%.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 8 minor

Summary. This paper introduces Optimal Transport Q-Learning (OTQL), a method for simultaneously fine-tuning and accelerating flow-based robot policies via RL post-training. The core idea is to use advantage-weighted conditional optimal transport (COT) couplings in flow matching (wCOT-CFM), where target marginals are reweighted by advantage estimates to steer the flow toward high-value actions while learning straight integration paths for fast inference. The method is evaluated on 42 simulation tasks (OGBench, MuJoCo) and 4 real-world tasks, including adaptation of a VLA (SmolVLA), showing competitive or superior performance compared to baselines while reducing inference to 2-3 neural function evaluations (NFEs) with 50-60 episodes of interaction.

Significance. The paper addresses a practically important problem: adapting and accelerating flow/diffusion policies for robotics without expensive distillation. The wCOT-CFM coupling (Eqs. 3-5) is a clean and well-motivated combination of COT flow matching with advantage-weighted target marginals, grounded in the KL-regularized RL objective. The real-world VLA adaptation results (SmolVLA, Table 2) and the breadth of simulation experiments (Table 1, 42 tasks) are notable strengths. The method is falsifiable: the ablation in Fig. 6 varies key hyperparameters, and the offline results in Table 4 show the OT coupling working without rejection sampling (N=1) on lower-dimensional tasks. The code and project page are referenced (ansocho.github.io/otql-flow), supporting reproducibility.

major comments (4)
  1. §4.3, Tables 8 and 10: All real-world experiments — the headline results of 36%→86% (single-task) and 38%→76% (VLA) — use rejection sampling with N=5 (single-task) or N=1 (VLA). The ablation in Fig. 6 varies N on a single task but never compares wCOT-CFM against independent coupling (or EWFM) at the same N. Without holding N constant and varying the coupling type, it is unclear whether the OT coupling or rejection sampling drives the real-world gains. The simulation results without rejection sampling (antmaze, humanoidmaze with N=1 in Table 4) do show the OT coupling working, but these are lower-dimensional tasks where the authors themselves acknowledge (§5) that 'minibatch OT is prone to larger errors in higher dimensions.' A controlled ablation isolating the wCOT coupling contribution from rejection sampling, at least on one higher-dimensional or real-world task, is needed to support a
  2. §3.1, Eq. (2): The entropic OT regularization strength ε is never reported in any experiment table (Tables 3, 5, 7, 9) or mentioned in the ablation (§C, Fig. 6). Since ε directly controls the sharpness of the OT plan and interacts with the advantage weighting (Eq. 5) and weight clamping (w_max), its omission makes the method not fully reproducible and prevents assessment of how sensitive the coupling quality is to this parameter. Please report ε for all experiments and ideally include it in the ablation.
  3. §4.3, Table 2: The real-world VLA evaluation uses only 30 trials per task. With success rates of 14/30 vs 24/30 (tape) and 9/30 vs 22/30 (knight), the differences are within the range of statistical noise for binomial outcomes at n=30 (e.g., 14/30 has a 95% CI of approximately [0.31, 0.62]). The claim of a 37% average improvement is central to the paper's significance. Please report confidence intervals or bootstrap estimates, and temper the claims accordingly.
  4. Tables 4, 6, 8, 10: The per-task hyperparameter tuning is extensive — λ ranges from 0.4 to 1.5, w_max from 1 to 20, N from 1 to 5 across tasks. There is no principled selection procedure described; the ablation in Fig. 6 is on a single task. This raises the question of how a practitioner would select these for a new task without tuning. Please clarify the selection procedure or discuss the sensitivity more systematically.
minor comments (8)
  1. §3.2, Algorithm 1: The target network EMA rate is denoted τ in the text but λ in Tables 3, 5, 7, 9 (e.g., 'Target network update rate (λ) 5×10⁻³'). This conflicts with λ being the advantage temperature. Please use consistent notation.
  2. Fig. 1 caption: 'Energy-weighted Flow Matching (EWFM) [22]' — the caption text is slightly unclear about which panels correspond to which energy function. Consider labeling panels directly.
  3. §4.2: 'OTQL (NFE=3, N=5) outperforms EWFM (NFE=10, N=1) and performs as well as QC (NFE=10, N=32)' — the comparison is not apples-to-apples in NFE or N. Clarify that OTQL achieves comparable performance at lower NFE but with rejection sampling, or provide a matched-NFE comparison.
  4. Table 1: The 'Average' row mixes tasks with and without action chunking/rejection sampling. Consider reporting averages separately for chunked and non-chunked tasks.
  5. §4.3: 'increasing average success rates from 36% to 86%' — please specify this is across the three single-task policies in Fig. 5.
  6. References [17] and [54] are cited as NeurIPS 2026 and [24] as arXiv:2601.14234. Please verify these are correctly attributed.
  7. Fig. 4 caption: 'real-wrold tasks' — typo, should be 'real-world'.
  8. §D.1, Eq. (10): The pessimistic target backup formula uses ρ but Tables 3 and 5 list it as 'Critic target pessimistic coefficient (ρ)'. The notation is consistent but the variable ρ is not introduced in the main text. Consider a brief mention.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee raises four major points: (1) the need for a controlled ablation isolating wCOT coupling from rejection sampling on higher-dimensional or real-world tasks; (2) the omission of the entropic OT regularization strength ε across all experiment tables and ablations; (3) the statistical significance of the VLA results given n=30 trials per task; and (4) the lack of a principled hyperparameter selection procedure. We agree with all four points and will revise the manuscript accordingly. Specifically, we will add a controlled coupling-type ablation at fixed N on a higher-dimensional task, report ε for all experiments and include it in the ablation, add confidence intervals and temper VLA claims, and provide a clearer hyperparameter selection procedure with sensitivity discussion.

read point-by-point responses

  1. Referee: §4.3, Tables 8 and 10: All real-world experiments use rejection sampling with N=5 or N=1. The ablation in Fig. 6 varies N on a single task but never compares wCOT-CFM against independent coupling (or EWFM) at the same N. Without holding N constant and varying the coupling type, it is unclear whether the OT coupling or rejection sampling drives the real-world gains. A controlled ablation isolating the wCOT coupling contribution from rejection sampling, at least on one higher-dimensional or real-world task, is needed.

    Authors: The referee is correct that the current experiments do not isolate the wCOT coupling contribution from rejection sampling on higher-dimensional tasks. The simulation results with N=1 (antmaze, humanoidmaze in Table 4) do demonstrate the OT coupling working without rejection sampling, but we agree these are lower-dimensional and that a controlled comparison at fixed N on a higher-dimensional task is needed to support the real-world claims. We will add a controlled ablation on the cube-double task (which uses action chunking and has a higher-dimensional action space) comparing wCOT-CFM against independent coupling (EWFM) at N=1 and N=5, holding all other hyperparameters constant. This will directly show the marginal contribution of the OT coupling beyond rejection sampling. We will also add this comparison on one real-world task if time permits, though the cube-double ablation already addresses the core concern about higher-dimensional tasks. revision: yes

  2. Referee: §3.1, Eq. (2): The entropic OT regularization strength ε is never reported in any experiment table or mentioned in the ablation. Since ε directly controls the sharpness of the OT plan and interacts with the advantage weighting and weight clamping, its omission makes the method not fully reproducible. Please report ε for all experiments and ideally include it in the ablation.

    Authors: The referee is correct that ε is a relevant hyperparameter that was omitted from all experiment tables and the ablation. This was an oversight. We used ε = 0.01 for all OGBench tasks and ε = 0.05 for the online MuJoCo and real-world tasks (selected based on the default values from the OT solver library). We will add ε to Tables 4, 6, 8, and 10, and include it as an additional axis in the ablation in Figure 6. We expect the sensitivity to ε to be moderate in the regime we use, but reporting it is important for reproducibility and we agree it should have been included from the start. revision: yes

  3. Referee: §4.3, Table 2: The real-world VLA evaluation uses only 30 trials per task. With success rates of 14/30 vs 24/30 and 9/30 vs 22/30, the differences are within the range of statistical noise for binomial outcomes at n=30. Please report confidence intervals or bootstrap estimates, and temper the claims accordingly.

    Authors: The referee is correct that n=30 trials per task is a limited sample size and that confidence intervals should be reported. We computed Wilson 95% confidence intervals: for tape in penholder, SmolVLA is 14/30 [0.31, 0.62] and SmolVLA-OTQL is 24/30 [0.62, 0.87]; for move knight, SmolVLA is 9/30 [0.20, 0.48] and SmolVLA-OTQL is 22/30 [0.56, 0.81]. While the intervals do not overlap between base and OTQL on either task, we agree that the claim of a '37% average improvement' should be tempered given the sample size. We will add confidence intervals to Table 2 and revise the language to describe the improvement as 'substantial but based on a limited sample of 30 trials per task,' noting that the intervals do not overlap but that larger-scale evaluation would strengthen the claim. revision: yes

  4. Referee: Tables 4, 6, 8, 10: The per-task hyperparameter tuning is extensive — λ ranges from 0.4 to 1.5, w_max from 1 to 20, N from 1 to 5 across tasks. There is no principled selection procedure described; the ablation in Fig. 6 is on a single task. This raises the question of how a practitioner would select these for a new task without tuning. Please clarify the selection procedure or discuss the sensitivity more systematically.

    Authors: The referee is correct that the current paper does not describe a principled hyperparameter selection procedure, and the ablation on a single task is insufficient to guide practitioners. We will address this in two ways. First, we will add a 'Hyperparameter Selection Guidelines' paragraph in §C describing the heuristics we used: λ is selected based on the chunk size and episode length (lower λ for longer chunks to avoid effective batch size collapse, as shown in Fig. 6), w_max is set to 10 as a default and only increased to 20 for tasks with very peaked Q-functions, and N is set to 1 by default and only increased to 5 for tasks with action chunking where the Q-function has sharp peaks. Second, we will add a sensitivity analysis showing performance variation when using a single set of hyperparameters (λ=0.75, w_max=10, N=1) across multiple tasks, to demonstrate that the method is not overly sensitive to hyperparameter choices and that the per-task tuning provides modest gains over a reasonable default. revision: yes

Circularity Check

0 steps flagged

No significant circularity; one minor self-citation for cost augmentation technique that is not load-bearing.

full rationale

The paper's derivation chain is substantially self-contained against external sources. The target distribution π̃(a|s) ∝ π(a|s)e^{λA(s,a)} is attributed to AWAC [43] and KL-regularized RL [41, 42], all external citations. The theoretical justification connecting OT couplings to CFM draws from Tong et al. [10] and Kerrigan et al. [32], both external. The cost augmentation technique (Eq. 3-4) is cited to [28] (same first author) but also supported by [32] (external) for the COT framework; it is presented as a practical design choice, not as a theorem-derived necessity. The central novel contribution—adding advantage-weighted target marginals (Eq. 5) to COT couplings for RL post-training—is not present in [28], which addresses only BC acceleration without a critic or Bellman targets. No 'prediction' or 'first-principles result' reduces to its inputs by construction: the empirical results are measurements against external benchmarks (OGBench, MuJoCo, real robot tasks), and the method is a genuine combination of independently-derived components. The one minor self-citation ([28] for cost augmentation) is not load-bearing because the same technique is justified by the external COT framework [32], and the paper's central claim (advantage-weighted COT for RL) does not depend on [28]'s results being correct.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

No new entities are invented. The method reuses existing constructs (flow policies, Q-functions, OT plans) in a new combination.

free parameters (6)
  • λ (temperature/energy scale) = 0.4–1.5 (task-dependent)
    Controls the sharpness of advantage weighting. Tuned per task in Tables 4, 6, 8, 10.
  • w_max (weight clamp) = 1–20 (task-dependent)
    Clamps advantage weights to prevent batch collapse. Tuned per task.
  • α (condition scaling) = 300 (online tasks), not specified for offline
    Controls relative influence of condition in augmented OT cost (Eq. 3-4).
  • N (rejection sampling candidates) = 1 or 5 (task-dependent)
    Number of candidate actions evaluated by Q at inference. Set to 5 for chunked tasks.
  • ε (entropic OT regularization) = not explicitly stated
    Regularization strength in the entropic OT problem (Eq. 2). Not reported in hyperparameter tables.
  • τ (target network EMA rate) = 5×10⁻³
    Standard EMA rate for critic target network.
axioms (4)
  • standard math The advantage-weighted distribution π̃(a|s) ∝ π(a|s)e^{λA(s,a)} is the closed-form solution to the KL-regularized RL problem
    Invoked in §2 (RL post-training), citing [41, 42, 43]. Standard result in entropy-regularized RL.
  • domain assumption Minibatch entropic OT approximates the conditional OT plan between noise and target distributions
    Used throughout §3.1. The quality of this approximation is load-bearing and acknowledged as limited in high dimensions (§5).
  • standard math Linear interpolant x(t) = ta + (1-t)z is a valid trajectory for flow matching
    Standard CFM assumption, cited from [9, 10].
  • domain assumption The Q-function learned via TD loss provides a meaningful advantage signal for reweighting
    The entire steering mechanism depends on Q being a reasonable value estimator. Standard RL assumption but unproven for flow policies specifically.

pith-pipeline@v1.1.0-glm · 23744 in / 3982 out tokens · 222348 ms · 2026-07-08T11:28:05.655345+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

78 extracted references · 78 canonical work pages · 20 internal anchors

  1. [1]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    PaliGemma: A versatile 3B VLM for transfer

    L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdul- mohsin, M. Tschannen, E. Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer.arXiv preprint arXiv:2407.07726, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    J. Pai, L. Achenbach, V . Montesinos, B. Forrai, O. Mees, and E. Nava. mimic-video: Video- action models for generalizable robot control beyond vlas.arXiv preprint arXiv:2512.15692, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    M. J. Kim, Y . Gao, T.-Y . Lin, Y .-C. Lin, Y . Ge, G. Lam, P. Liang, S. Song, M.-Y . Liu, C. Finn, and J. Gu. Cosmos policy: Fine-tuning video models for visuomotor control and planning. In International Conference on Learning Representations (ICLR), 2026

    work page 2026

  5. [5]

    Reuss, H

    M. Reuss, H. Zhou, M. R¨uhle, ¨O. E. Ya˘gmurlu, F. Otto, and R. Lioutikov. FLOWER: Democra- tizing generalist robot policies with efficient vision-language-flow models. InConference on Robot Learning (CoRL), 2025

    work page 2025

  6. [6]

    Black, N

    K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, b. ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke...

    work page 2025

  7. [7]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky.𝜋0: A vision-language- action flow model for general robot control. InRobotics: Science and Sys...

    work page 2024

  8. [8]

    GR00T N1: An Open Foundation Model for Generalist Humanoid Robots

    J. Bjorck, F. Casta˜neda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y . Fang, D. Fox, F. Hu, S. Huang, et al. Gr00t n1: An open foundation model for generalist humanoid robots.arXiv preprint arXiv:2503.14734, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Lipman, R

    Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. InInternational Conference on Learning Representations (ICLR), 2023

    work page 2023

  10. [10]

    A. Tong, K. Fatras, N. Malkin, G. Huguet, Y . Zhang, J. Rector-Brooks, G. Wolf, and Y . Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research (TMLR), 2024

    work page 2024

  11. [11]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems (NeurIPS), 2020

    work page 2020

  12. [12]

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. InInternational Conference on Learning Representations (ICLR), 2021

    work page 2021

  13. [13]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research (IJRR), 2023

    work page 2023

  14. [14]

    Janner, Y

    M. Janner, Y . Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. InInternational Conference on Machine Learning (ICML), 2022. 9

    work page 2022

  15. [15]

    Zhang, Y

    J. Zhang, Y . Luo, A. Anwar, S. A. Sontakke, J. J. Lim, J. Thomason, E. Biyik, and J. Zhang. Rewind: Language-guided rewards teach robot policies without new demonstrations. In Conference on Robot Learning (CoRL), 2025

    work page 2025

  16. [16]

    Wagenmaker, Y

    A. Wagenmaker, Y . Zhang, M. Nakamoto, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine. Steering your diffusion policy with latent space reinforcement learning. In Conference on Robot Learning (CoRL), 2025

    work page 2025

  17. [17]

    Du and S

    M. Du and S. Song. Dynaguide: Steering diffusion polices with active dynamic guidance. In Conference on Neural Information Processing Systems (NeurIPS), 2026

    work page 2026

  18. [18]

    Ankile, A

    L. Ankile, A. Simeonov, I. Shenfeld, M. Torne, and P. Agrawal. From imitation to refinement- residual rl for precise assembly. InInternational Conference on Robotics and Automation (ICRA). IEEE, 2025

    work page 2025

  19. [19]

    Ankile, Z

    L. Ankile, Z. Jiang, R. Duan, G. Shi, P. Abbeel, and A. Nagabandi. Residual off-policy rl for finetuning behavior cloning policies.arXiv preprint arXiv:2509.19301, 2025

    work page arXiv 2025

  20. [20]

    Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InInternational Conference on Learning Representations (ICLR), 2023

    work page 2023

  21. [21]

    F. N. Tiofack, T. L. Hellard, F. Schramm, N. Perrin-Gilbert, and J. Carpentier. Guided flow policy: Learning from high-value actions in offline reinforcement learning.arXiv preprint arXiv:2512.03973, 2025

    work page arXiv 2025

  22. [22]

    Zhang, W

    S. Zhang, W. Zhang, and Q. Gu. Energy-weighted flow matching for offline reinforcement learning. InInternational Conference on Learning Representations (ICLR), 2025

    work page 2025

  23. [23]

    Q. Li, Z. Zhou, and S. Levine. Reinforcement learning with action chunking. InConference on Neural Information Processing Systems (NeurIPS), 2026

    work page 2026

  24. [24]

    Q-learning with Adjoint Matching

    Q. Li and S. Levine. Q-learning with adjoint matching.arXiv preprint arXiv:2601.14234, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  25. [25]

    S. Park, Q. Li, and S. Levine. Flow q-learning. InInternational Conference on Machine Learning (ICML), 2025

    work page 2025

  26. [26]

    Z. Wang, Z. Li, A. Mandlekar, Z. Xu, J. Fan, Y . Narang, L. Fan, Y . Zhu, Y . Balaji, M. Zhou, et al. One-step diffusion policy: Fast visuomotor policies via diffusion distillation.arXiv preprint arXiv:2410.21257, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    Prasad, K

    A. Prasad, K. Lin, J. Wu, L. Zhou, and J. Bohg. Consistency policy: Accelerated visuomotor policies via consistency distillation. InRobotics: Science and Systems (RSS), 2024

    work page 2024

  28. [28]

    Sochopoulos, N

    A. Sochopoulos, N. Malkin, N. Tsagkas, J. Moura, M. Gienger, and S. Vijayakumar. Fast flow-based visuomotor policies via conditional optimal transport couplings. InConference on Robot Learning (CoRL), 2025

    work page 2025

  29. [29]

    Hu, qiang liu, X

    X. Hu, qiang liu, X. Liu, and B. Liu. Adaflow: Imitation learning with variance-adaptive flow-based policies. InAdvances in neural information processing systems (NeurIPS), 2024

    work page 2024

  30. [30]

    G. Lu, Z. Gao, T. Chen, W. Dai, Z. Wang, and Y . Tang. Manicm: Real-time 3d diffusion policy via consistency model for robotic manipulation.arXiv preprint arXiv:2406.01586, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    H. Ding, N. Jaquier, J. Peters, and L. Rozo. Fast and robust visuomotor riemannian flow matching policy.arXiv preprint arXiv:2412.10855, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Kerrigan, G

    G. Kerrigan, G. Migliorini, and P. Smyth. Dynamic conditional optimal transport through simulation-free flows. InAdvances in neural information processing systems (NeurIPS), 2024. 10

    work page 2024

  33. [33]

    H. K. Cheng and A. Schwing. The curse of conditions: Analyzing and improving optimal transport for conditional flow-based generation. InIEEE/CVF International Conference on Computer Vision (ICCV), 2025

    work page 2025

  34. [34]

    T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park. One-step diffusion with distribution matching distillation. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    work page 2024

  35. [35]

    Salimans and J

    T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations (ICLR), 2022

    work page 2022

  36. [36]

    Frans, D

    K. Frans, D. Hafner, S. Levine, and P. Abbeel. One step diffusion via shortcut models. In International Conference on Learning Representations (ICLR), 2025

    work page 2025

  37. [37]

    Kim, C.-H

    D. Kim, C.-H. Lai, W.-H. Liao, N. Murata, Y . Takida, T. Uesaka, Y . He, Y . Mitsufuji, and S. Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. InInternational Conference on Learning Representations (ICLR), 2024

    work page 2024

  38. [38]

    L. Yang, Z. Zhang, Z. Zhang, X. Liu, M. Xu, W. Zhang, C. Meng, S. Ermon, and B. Cui. Consistency flow matching: Defining straight flows with velocity consistency.arXiv preprint arXiv:2407.02398, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics

    M. Shukor, D. Aubakirova, F. Capuano, P. Kooijmans, S. Palma, A. Zouitine, M. Aractingi, C. Pascal, M. Russi, A. Marafioti, et al. Smolvla: A vision-language-action model for affordable and efficient robotics.arXiv preprint arXiv:2506.01844, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  40. [40]

    R. T. Chen, Y . Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations.Advances in neural information processing systems (NeurIPS), 2018

    work page 2018

  41. [41]

    Peters and S

    J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. InInternational Conference on Machine learning (ICML), 2007

    work page 2007

  42. [42]

    Peters, K

    J. Peters, K. Mulling, and Y . Altun. Relative entropy policy search. InAAAI Conference on Artificial Intelligence, 2010

    work page 2010

  43. [43]

    A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  44. [44]

    Pooladian, H

    A.-A. Pooladian, H. Ben-Hamu, C. Domingo-Enrich, B. Amos, Y . Lipman, and R. T. Chen. Multisample flow matching: Straightening flows with minibatch couplings. InInternational Conference on Machine Learning (ICML), 2023

    work page 2023

  45. [45]

    Benamou and Y

    J.-D. Benamou and Y . Brenier. A computational fluid mechanics solution to the monge- kantorovich mass transfer problem.Numerische Mathematik, 84(3):375–393, 2000

    work page 2000

  46. [46]

    $\pi^{*}_{0.6}$: a VLA That Learns From Experience

    P. Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, D. Driess, M. Equi, A. Esmail, Y . Fang, C. Finn, C. Glossop, T. Godden, I. Goryachev, L. Groom, H. Hancock, K. Hausman, G. Hussein, B. Ichter, S. Jakubczak, R. Jen, T. Jones, B. Katz, L. Ke, C. Kuchi, M. Lamb, D. LeBlanc, S. Levin...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  47. [47]

    H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. InInternational Conference on Learning Representations (ICLR), 2023

    work page 2023

  48. [48]

    S. Park, K. Frans, B. Eysenbach, and S. Levine. Ogbench: Benchmarking offline goal- conditioned rl. InInternational Conference on Learning Representations (ICLR), 2025. 11

    work page 2025

  49. [49]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems (IROS), 2012

    work page 2012

  50. [50]

    Cadene, S

    R. Cadene, S. Alibert, A. Soare, Q. Gallouedec, A. Zouitine, S. Palma, P. Kooijmans, M. Ar- actingi, M. Shukor, D. Aubakirova, M. Russi, F. Capuano, C. Pascal, J. Choghari, J. Moss, and T. Wolf. Lerobot: State-of-the-art machine learning for real-world robotics in pytorch. https://github.com/huggingface/lerobot, 2024

    work page 2024

  51. [51]

    Psenka, A

    M. Psenka, A. Escontrela, P. Abbeel, and Y . Ma. Learning a diffusion model policy from rewards via q-score matching. InInternational Conference on Machine Learning (ICML), 2024

    work page 2024

  52. [52]

    IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies

    P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  53. [53]

    What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation. InarXiv preprint arXiv:2108.03298, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  54. [54]

    Driess, J

    D. Driess, J. Springenberg, B. Ichter, L. Yu, A. Li-Bell, K. Pertsch, A. Ren, H. Walke, Q. Vuong, L. X. Shi, et al. Knowledge insulating vision-language-action models: Train fast, run fast, generalize better.Advances in Neural Information Processing Systems (NeurIPS), 2026

    work page 2026

  55. [55]

    E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations (ICRL), 2022

    work page 2022

  56. [56]

    Y . Brenier. Polar factorization and monotone rearrangement of vector-valued functions.Com- munications on pure and applied mathematics, 44(4):375–417, 1991

    work page 1991

  57. [57]

    Conditional Optimal Transport on Function Spaces

    B. Hosseini, A. W. Hsu, and A. Taghvaei. Conditional optimal transport on function spaces. arXiv preprint arXiv:2311.05672, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  58. [58]

    J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633–8646, 2022

    work page 2022

  59. [59]

    Cosmos World Foundation Model Platform for Physical AI

    N. Agarwal, A. Ali, M. Bala, Y . Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y . Chen, Y . Cui, Y . Ding, et al. Cosmos world foundation model platform for physical ai.arXiv preprint arXiv:2501.03575, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  60. [60]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. InIEEE/CVF Conference on computer vision and pattern recognition (CVPR), 2022

    work page 2022

  61. [61]

    Saharia, W

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding.Advances in neural information processing systems, 35: 36479–36494, 2022

    work page 2022

  62. [62]

    Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations

    J. Urain, A. Mandlekar, Y . Du, M. Shafiullah, D. Xu, K. Fragkiadaki, G. Chalvatzaki, and J. Pe- ters. Deep generative models in robotics: A survey on learning from multimodal demonstrations. arXiv preprint arXiv:2408.04380, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  63. [63]

    Classifier-Free Diffusion Guidance

    J. Ho and T. Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  64. [64]

    Y . Hou, Z. Liu, C. Chi, E. Cousineau, N. Kuppuswamy, S. Feng, B. Burchfiel, and S. Song. Adaptive compliance policy: Learning approximate compliance for diffusion guided control. In International Conference on Robotics and Automation (ICRA), 2025. 12

    work page 2025

  65. [65]

    H. Xue, J. Ren, W. Chen, G. Zhang, Y . Fang, G. Gu, H. Xu, and C. Lu. Reactive diffusion policy: Slow-fast visual-tactile policy learning for contact-rich manipulation.arXiv preprint arXiv:2503.02881, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  66. [66]

    Chisari, N

    E. Chisari, N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada. Learning robotic manipulation policies from point clouds with conditional flow matching.Conference on Robot Learning (CoRL), 2024

    work page 2024

  67. [67]

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations.Robotics: Science and Systems (RSS), 2024

    work page 2024

  68. [68]

    S. Li, Y . Gao, D. Sadigh, and S. Song. Unified Video Action Model. InRobotics: Science and Systems (RSS), 2025

    work page 2025

  69. [69]

    C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Cou- pling video and action diffusion for pretraining on large robotic datasets.arXiv preprint arXiv:2504.02792, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  70. [70]

    Zhang, K

    J. Zhang, K. Zheng, K. Jiang, H. Wang, I. Stoica, J. E. Gonzalez, J. Chen, and J. Zhu. Turbodiffu- sion: Accelerating video diffusion models by 100-200 times.arXiv preprint arXiv:2512.16093, 2025

    work page arXiv 2025

  71. [71]

    Z. Geng, M. Deng, X. Bai, Z. Kolter, and K. He. Mean flows for one-step generative modeling. Advances in Neural Information Processing Systems (NeurIPS), 2026

    work page 2026

  72. [72]

    Black, M

    K. Black, M. Galliker, and S. Levine. Real-time execution of action chunking flow policies. Advances in Neural Information Processing Systems (NeurIPS), 2026

    work page 2026

  73. [73]

    Z. Li, R. Krohn, T. Chen, A. Ajay, P. Agrawal, and G. Chalvatzaki. Learning multimodal behav- iors from scratch with diffusion policy gradient.Advances in Neural Information Processing Systems (NeurIPS), 2024

    work page 2024

  74. [74]

    Fujimoto and S

    S. Fujimoto and S. Gu. A minimalist approach to offline reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2021

    work page 2021

  75. [75]

    Huang, X

    X. Huang, X. Liu, E. Zhang, T. Yu, and S. Li. Offline-to-online reinforcement learning with classifier-free diffusion generation.International Conference on Machine learning (ICML), 2025

    work page 2025

  76. [76]

    Venkatraman, M

    S. Venkatraman, M. Jain, L. Scimeca, M. Kim, M. Sendera, M. Hasan, L. Rowe, S. Mittal, P. Lemos, E. Bengio, et al. Amortizing intractable inference in diffusion models for vision, language, and control.Advances in neural information processing systems (NeurIPS), 2024

    work page 2024

  77. [77]

    L. Fang, R. Liu, J. Zhang, W. Wang, and B. Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. InInternational Conference on Learning Representations (ICLR), 2025. 13 A Optimal Transport The static OT problem aims to find a coupling of minimal cost between two distributio...

    work page 2025

  78. [78]

    distills the behavior of a flow policy to a single-step guassian policy while also optimizing a𝑄 loss. In robotics, access to a suboptimal pre-trained base policy is assumed and the goal is to extract a policy that samples high value actions and actions that also have a high density under the base policy density. Wagenmaker et al. [16] learn a policy that...

    work page 2080