Pith. sign in

REVIEW 3 major objections 5 minor 69 references

VINE: Taming Generative Control Policies for Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training instability of flow policies under value-gradient RL comes from the behavior-cloning sampler, not from iterative generation; VINE fixes the sampler so full end-to-end backpropagation through ten denoising steps becomes stable.

desk verdict Sampler redesign, not iteration count, is the real fix for value-gradient BPTT on flow policies; the empirical case is strong even if the appendix theorem is idealized. read the letter →

arxiv 2607.10369 v1 pith:WNIQ6ZKL submitted 2026-07-11 cs.RO cs.AI

Rushuai Yang , Zhuo Han , Houlin Li , Hecheng Wang , Zhichao Wu , Rui Zhang , Zhaowei Zhang , Zihong Chen
show 5 more authors
Xiaohan Yan Chiming Liu Yi Chen Wei Shan Maoqing Yao
This is my paper
classification cs.ROcs.AI
keywords flowmatchinggenerativecontrolpoliciesvalue-gradientRLofflinereinforcementlearningrobotdenoisingsamplerBPTTstability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Flow-matching policies generate robot actions by iteratively refining noise, which gives them the power to model complex multimodal action distributions. When people try to improve those policies with value-gradient reinforcement learning, training usually becomes unstable, and prior work blamed the multi-step generation process itself. This paper shows the real culprit is the ordinary single-trajectory sampler that was designed for behavior cloning: under value gradients it produces brittle paths that cannot support stable backpropagation. VINE redesigns only the sampler. At every denoising step it rebuilds a fresh noisy interpolation state around the current action estimate and then applies the same velocity field, creating a stable differentiable path while remaining compatible with the original flow-matching training objective. The result is that full end-to-end value-gradient optimization through all ten steps becomes practical, expressiveness and iterative refinement are preserved, and the method outperforms strong baselines on the OGBench offline suite and on a real-robot socket-insertion task.

What carries the argument

VINE sampling: at each step k reconstruct a noisy interpolation state x̂_k = t_k â_k + (1−t_k) z_k and recover the next action estimate by â_{k+1} = x̂_k + (1−t_k) v_θ(x̂_k, t_k; s). This supplies a stable differentiable path for value-gradient BPTT while keeping every network query inside the original flow-matching training semantics.

What would settle it

Train the identical velocity network with ordinary Euler BPTT versus VINE BPTT on the same offline OGBench tasks and measure both gradient-norm trajectories and final success rates; if Euler remains as stable and successful as VINE, or if VINE’s reconstructed states produce large prediction error under the pretrained velocity field after distribution shift, the central claim fails.

Watch

Extended reading notes

Core claim

The instability of value-gradient RL on flow-matching policies does not arise from iterative generation itself, but from the vanilla single-noise Euler sampler inherited from behavior cloning. By reconstructing a new interpolation state and injecting fresh noise at every denoising step, VINE creates a stable path for backpropagation of the critic’s action gradient through the entire chain while remaining compatible with the pretrained flow-matching velocity field, thereby allowing end-to-end value-gradient optimization without sacrificing expressiveness or multi-step refinement.

Load-bearing premise

The velocity field trained under ordinary flow matching still gives a valid endpoint prediction when its inputs are the reconstructed noisy states that VINE invents rather than the usual single Euler trajectory, even after value gradients have shifted the policy away from the original data.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that training instability when applying value-gradient RL to flow-matching policies arises not from iterative generation itself, but from the single-trajectory Euler sampler inherited from behavior cloning. It proposes VINE, a sampler that at each denoising step reconstructs a fresh noisy interpolation state around the current action estimate and applies the same velocity field for endpoint prediction, thereby creating a stable path for end-to-end BPTT through all K denoising steps while remaining formally compatible with conditional flow matching. Compatibility is justified by Theorem 1 (optimal velocity yields the posterior mean for any query state) and Corollary 1. Empirically, VINE with K=10 is evaluated on 40 OGBench offline tasks (12 seeds, bootstrap CIs) and a real-world plug-insertion task under human-in-the-loop online RL, reporting aggregate gains over Gaussian, around-the-chain, outside-the-chain, and through-the-chain baselines, plus gradient-norm and iterative-refinement ablations.

Significance. If the sampler redesign is the primary source of stability, the work removes a long-standing practical barrier to using highly expressive multi-step generative policies with standard actor-critic value gradients, without sacrificing iterative refinement or requiring external guidance, distillation, or scalar reweighting. The contribution is concrete and immediately usable: only the sampler changes, pretrained flow policies can be reused, and the method is demonstrated both on a large offline suite and on a contact-rich real robot with wall-clock and intervention metrics. Strengths include a clear algorithmic statement, an explicit compatibility theorem for the optimal field, multi-seed OGBench evaluation with confidence intervals, and ablations that link gradient stability (Fig. 4) and multi-step refinement (Fig. 5) to the claimed mechanism.

major comments (3)
  1. [Appendix A.1, Theorem 1 / Corollary 1] Theorem 1 and Corollary 1 (Appendix A.1–A.3) establish that the endpoint map equals the posterior mean only for the pointwise minimizer v* of the CFM loss, and Corollary 1 further assumes a Gaussian action posterior and deterministic VINE (zk=0). Algorithm 1 and the actual experiments use a finite-capacity network that is itself updated by BPTT through the reconstructed states, with fresh noise every step and multimodal actions. Under value-driven mode shift the approximation gap can grow, so the formal justification does not yet cover the trained, non-stationary network that carries the central claim. A discussion of this gap, and ideally an empirical check that the network remains a faithful endpoint predictor after the policy has moved (e.g., reconstruction error or posterior-mean residual vs. training step), is needed to make the ‘why VINE stabilizes BPTT’ argument load-bearing rathe
  2. [Algorithm 1, lines 5–8; Eqs. (7)–(8)] Algorithm 1 GENERATE is inconsistent with the method defined by Eqs. (7)–(8). Line 5 performs an Euler-style update on an undefined/uninitialized xk (‘xk ← xk + (1/K)vθ…’) with the comment ‘Replace Euler Method’, while the subsequent lines correctly implement the reconstruction ˆxk = tk âk + (1−tk)zk and endpoint prediction. As written, the procedure is not reproducible from the pseudocode alone and conflicts with the prose claim that VINE does not follow a single Euler trajectory. The algorithm should be rewritten to match Eqs. (7)–(8) exactly (initialize â0, reconstruct ˆxk, predict âk+1, no residual Euler state).
  3. [Abstract; Table 1; §5.2] The abstract and §5.2 claim that VINE ‘consistently outperforms’ SOTA methods, yet Table 1 / Table 5 show near-floor performance on cube-double ([1,6]4) and cube-triple ([0,1]1), well below several through- and outside-the-chain baselines (e.g., DSRL, QAM, CGQL). Aggregate leadership is driven by large gains on antmaze-giant and humanoidmaze-large. The paper should qualify the consistency claim, analyze why the sampler fails or underperforms on these sparse manipulation domains, and state whether the failure is expected under the method’s assumptions.
minor comments (5)
  1. [Figure 1; §2.3] Fig. 1 caption and body use ‘value BPTT’ / ‘BPTT path’ without defining BPTT on first use in the main text; a one-line expansion would help non-RL readers.
  2. [§2.3, Eq. (6)] In §2.3 Eq. (6) the chain-rule expansion writes ∂xK/∂xk+1 without specifying how that Jacobian is obtained under the Euler discretization; a short note that it is the product of (I + (1/K)∂v/∂x) terms would clarify the contrast with VINE.
  3. [Table 2; §5.1] Table 2 reports human-intervention percentages for methods that use BC init from π0.5 versus methods trained from scratch; a brief note on whether intervention protocol and budget were matched across the two groups would strengthen the real-robot comparison.
  4. [Appendix A.4, Table 4] Hyperparameter Table 4 lists large, task-specific BC coefficients α (up to 1000). Sensitivity of VINE to α, or a default selection rule, would improve reproducibility.
  5. [§3; Algorithm 1] Minor notation: the manuscript alternates between âk, ˆak, and a for the action estimate; standardizing on one hat convention in the main text would reduce friction.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: VINE is a free sampler redesign justified by a standard CFM property; success rates are external empirical measurements, not quantities forced by definition or self-citation.

full rationale

The paper's load-bearing chain is: (i) standard conditional flow-matching trains a velocity field on noisy interpolants xt = t a + (1-t)z; (ii) the Euler ODE sampler is only one of many paths consistent with that training; (iii) VINE rebuilds a fresh interpolant at each step and applies the same endpoint map â = x + (1-t)v(x,t;s); (iv) Theorem 1 (Appendix A.1) shows that for the pointwise minimizer v* this map equals the posterior mean for any query x, so the sampler change does not require a new generative objective; (v) actor-critic BPTT is then run through the VINE chain and evaluated on OGBench and a physical insertion task. None of these steps reduces a claimed prediction to its own inputs. Theorem 1 is a short derivation from the CFM squared-error objective (standard, externally cited), not a self-definition of the reported success rates. Algorithm 1 and Eqs. (7)–(8) are design choices whose merit is measured against fixed offline datasets and real-robot trials, not against quantities fitted from the same targets. Self-citations among the authors' other arXiv notes are confined to related-work context and do not underwrite uniqueness or forbid alternatives. The skeptic concern that Theorem 1 covers only optimal v* (not finite networks under value-driven shift) is a correctness/assumption risk, not circularity: the paper does not redefine validity as whatever the trained network happens to do. Score 0 is therefore the honest finding.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim rests on standard flow-matching and actor-critic machinery plus one new sampling construction. The only free parameters that materially affect reported rankings are the per-task behavior-cloning coefficients α; everything else is either a conventional hyper-parameter or a fixed design choice (K=10). No new physical entities are postulated.

free parameters (3)
  • BC coefficient α = 10–1000 per domain (Table 4)
    Task-specific scalar that balances the Q-maximization term against the behavior-cloning penalty; values range from 10 (antmaze) to 1000 (puzzle) and are chosen by hand for each of the eight domains.
  • number of denoising steps K = 10
    Fixed to 10 for all experiments; controls both expressiveness and the length of the BPTT chain.
  • MIP time grid = uniform 0.1 increments
    Uniform schedule {0.0, 0.1, …, 0.9} used for the reconstructed interpolants; not learned.
assumptions (3)
  • domain assumption The conditional flow-matching objective trains the velocity field so that x+(1-t)v*(x,t;s) equals the posterior mean of the data action (Theorem 1).
    Taken as given from the flow-matching literature and used to justify that VINE queries remain on-distribution for a pretrained velocity network.
  • domain assumption Behavior-regularized actor-critic (Eqs. 1–2) with a twin Q-network and target network is a valid offline RL objective.
    Standard assumption shared with ReBRAC, FQL and related baselines; not re-derived.
  • ad hoc to paper Fresh Gaussian noise injected at each reconstruction step remains compatible with the linear interpolation paths used in training.
    Stated without a full variance analysis under the non-stationary distribution induced by value-gradient updates.
invented entities (1)
  • VINE sampler (per-step reconstructed interpolation state ˆx_k)
    purpose: Provides a stable, noise-injected differentiable path for value-gradient BPTT while preserving flow-matching semantics.
    Defined by Eqs. 7–8 and Algorithm 1; no independent existence outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VINE: Taming Generative Control Policies for Reinforcement Learning." pith.science (2026). https://pith.science/paper/WNIQ6ZKL

@misc{pith2026260710369,
  author       = {Pith},
  title        = {Pith review of: VINE: Taming Generative Control Policies for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNIQ6ZKL}},
  note         = {Machine review of arXiv:2607.10369}
}
read the original abstract

Flow-matching policies have emerged as an effective policy parameterization for robot learning. They iteratively generate actions from noise, enabling highly expressive modeling of complex and multimodal action distributions. However, prior works observed that scaling these policies with value-gradient reinforcement learning (RL) often leads to training instability. Existing methods attribute this instability to iterative generation and therefore avoid end-to-end value-gradient optimization by sacrificing iterative generation, high expressiveness, or value-gradient optimization. Contrary to prior belief, we show the instability does not stem from iterative generation itself, but from the vanilla sampling strategy originally designed for behavior cloning, which becomes brittle under value-gradient RL. Motivated by this insight, we propose VINE, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies. Instead of following a single flow trajectory, VINE reconstructs a new interpolation state at every denoising step, creating a stable differentiable path for value-gradient propagation while remaining compatible with the original flow-matching denoising process. As a result, VINE preserves the expressiveness and iterative generation of flow-matching without sacrificing end-to-end value-gradient optimization. Despite performing end-to-end backpropagation through all ten denoising steps, VINE achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task. Videos are available on our website: https://agibottech.github.io/vine.

Figures

Figures reproduced from arXiv: 2607.10369 by the authors.

Figure 1
Figure 1. Left: A flow policy trained by behavior cloning fits the multi-modal data distribution. Middle: Directly backpropagating the critic gradient through the denoising steps (value BPTT) destabilizes the trajectory. Right: VINE produces a stable denoising trajectory that supports value￾gradient BPTT toward a1. arXiv:2607.10369v1 [cs.RO] 11 Jul 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Toy simulation of probability paths induced by different samplers under behavior cloning and offline value-gradient fine-tuning. Left: Under behavior cloning, all samplers re￾cover the multimodal data distribution, but Euler trajectories form isolated petal-like paths, DDPM explores broadly with noisy endpoints, and VINE provides structured exploration and broader state coverage. Right: After assigning different rew… view at source ↗
Figure 3
Figure 3. Real-world online RL setting. VINE learns an policy that insert successfully from any location. SAC-Flow fails to learn a successful policy during online RL [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Sampling ablation. We compare VINE against the vanilla Euler solver for flow-matching on OGBench. VINE consistently achieves higher success rates and more stable backpropagated gradients. 1 Iteration steps: 0/10 2 Iteration steps: 0/10 4 Iteration steps: 0/10 6 Iterati…
Figure 5
Figure 5. Figure 5: Preserving the Iterative Refinement. We generate the final action using different num￾bers of denoising steps K and evaluate the resulting policies on the AntMaze task. As the number of refinement iterations increases, the generated action quality improves, leading to …
Figure 6
Figure 6. Figure 6: VINE training curves (success rate vs. training steps), maze navigation domains. Each [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: VINE training curves (success rate vs. training steps), manipulation domains. Each row [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 24 linked inside Pith

  1. [1]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. InRobotics: Science and Systems, 2023

    2023

  2. [2]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2023

    2023

  3. [3]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al.π 0: A vision-language-action flow model for general robot control.arXiv preprint arXiv:2410.24164, 2024

    arXiv 2024

  4. [4]

    Physical Intelligence, A. Amin, R. Aniceto, A. Balakrishna, K. Black, K. Conley, G. Connors, J. Darpinian, K. Dhabalia, J. DiCarlo, et al.π ∗ 0.6: a vla that learns from experience.arXiv preprint, arXiv:2511.14759, 2025. arXiv preprint arXiv:2511.14759

    arXiv 2025

  5. [5]

    K. Chen, Z. Liu, T. Zhang, Z. Guo, S. Xu, H. Lin, H. Zang, X. Li, Q. Zhang, Z. Yu, et al. πrl: Online rl fine-tuning for flow-based vision-language-action models.arXiv preprint, arXiv:2510.25889, 2025. arXiv preprint arXiv:2510.25889

    arXiv 2025

  6. [6]

    Zhang, C

    T. Zhang, C. Yu, S. Su, and Y . Wang. Reinflow: Fine-tuning flow matching pol- icy with online reinforcement learning. In D. Belgrave, C. Zhang, H. Lin, R. Pas- canu, P. Koniusz, M. Ghassemi, and N. Chen, editors,Advances in Neural Informa- tion Processing Systems, volume 38, pages 106282–106319. Curran Associates, Inc.,

  7. [7]

    URLhttps://proceedings.neurips.cc/paper_files/paper/2025/file/ 98d6f928497d9eaad9f320fb2db3040d-Paper-Conference.pdf

    2025

  8. [8]

    J. Luo, C. Xu, J. Wu, and S. Levine. Precise and dexterous robotic manipulation via human- in-the-loop reinforcement learning.Science Robotics, 10(105):eads5033, 2025

    2025

Show all 69 references
  1. [9]

    R. Yang, H. Wang, Z. Wu, C. Liu, X. Yan, X. Du, S. Yue, C. Zhang, Y . Wang, Y . Liu, L. Qi, Y . Chen, W. Shan, and M. Yao. ALOE: Action-level off-policy evaluation for vision-language- action model post-training, 2026. URLhttps://arxiv.org/abs/2602.12691

    2026 arXiv

  2. [10]

    R. Yang, Z. Feng, T. Zhang, K. Wang, C. Zhang, L. Zhao, X. Su, Y . Chen, and J. Bian. Discover, learn, and reinforce: Scaling vision-language-action pretraining with diverse RL- generated trajectories, 2025. URLhttps://arxiv.org/abs/2511.19528

    2025

  3. [11]

    R. Yang, H. Wei, R. Zhang, Z. Feng, X. Chen, T. Li, C. Zhang, L. Zhao, J. Bian, X. Su, and Y . Chen. Beyond human demonstrations: Diffusion-based reinforcement learning to generate data for VLA training, 2025. URLhttps://arxiv.org/abs/2509.19752

    2025

  4. [12]

    Zhang, S

    Y . Zhang, S. Yu, T. Zhang, M. Guang, H. Hui, K. Long, Y . Wang, C. Yu, and W. Ding. Sac flow: Sample-efficient reinforcement learning of flow-based policies via velocity-reparameterized sequential modeling, 2026. URLhttps://arxiv.org/abs/2509.25756

    2026

  5. [13]

    Li and S

    Q. Li and S. Levine. Q-learning with adjoint matching.International Conference on Learning Representations, 2026

    2026

  6. [14]

    S. Park, Q. Li, and S. Levine. Flow q-learning. In A. Singh, M. Fazel, D. Hsu, S. Lacoste- Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, editors,Proceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Ma- chine Learning R...

    2025

  7. [15]

    Psenka, A

    M. Psenka, A. Escontrela, P. Abbeel, and Y . Ma. Learning a diffusion model policy from rewards via Q-score matching. InInternational Conference on Machine Learning, 2024. 10

    2024

  8. [16]

    Dhariwal and A

    P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In M. Ran- zato, A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, editors,Advances in Neural Information Processing Systems, volume 34, pages 8780–8794. Curran As- sociates, Inc., 2021. URLhttps...

    2021

  9. [17]

    H. Xu, K. Hu, S. Sojoudi, and A. Zhang. Reinforcement learning via value gradient flow. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps: //openreview.net/forum?id=JLL4VNVhM9

    2026

  10. [18]

    Z. Liu, T. Z. Xiao, C. Domingo-Enrich, W. Liu, and D. Zhang. Value gradient guidance for flow matching alignment. InNeurIPS 2025 Workshop on Structured Probabilistic Inference & Generative Modeling, 2025. URLhttps://openreview.net/forum?id=3oCOEL13OT

    2025

  11. [19]

    Wagenmaker, M

    A. Wagenmaker, M. Nakamoto, Y . Zhang, S. Park, W. Yagoub, A. Nagabandi, A. Gupta, and S. Levine. Steering your diffusion policy with latent space reinforcement learning.Conference on Robot Learning, 2025

    2025

  12. [20]

    L. Fang, R. Liu, J. Zhang, W. Wang, and B. Jing. Diffusion actor-critic: Formulating con- strained policy iteration as diffusion noise regression for offline reinforcement learning. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  13. [21]

    P. Dong, Q. Li, D. Sadigh, and C. Finn. EXPO: Stable reinforcement learning with expressive policies.arXiv preprint arXiv:2507.07986, 2025

    2025 arXiv

  14. [22]

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage weighted regression: Simple and scalable off-policy reinforcement learning, 2020. URLhttps://openreview.net/forum? id=H1gdF34FvS

    2020

  15. [23]

    Hansen-Estruch, I

    P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. IDQL: Implicit Q- learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    2023 arXiv

  16. [24]

    P. Dong, C. Zheng, C. Finn, D. Sadigh, and B. Eysenbach. Value flows. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview. net/forum?id=2VyNYUVF2k

    2026

  17. [25]

    C. Xu, J. T. Springenberg, M. Equi, A. Amin, A. Esmail, S. Levine, and L. Ke. Rl token: Bootstrapping online rl with vision-language-action models.arXiv preprint arXiv:2604.23073, 2026

    2026 arXiv

  18. [26]

    Zhang, Z

    C. Zhang, Z. Wan, F. Chen, X. Yu, I. Tsang, and B. An. Gorl: An algorithm-agnostic framework for online reinforcement learning with generative policies.arXiv preprint arXiv:2512.02581, 2025

    2025 arXiv

  19. [27]

    Zhang, Y

    S. Zhang, Y . Lou, H. Cheng, Y . Guo, C. Fu, Y . Lyu, X. Zhang, H. Li, P. Wang, Z. Wang, et al. Force: Efficient vla reinforcement fine-tuning via value-calibrated warm-up and self- distillation.arXiv preprint arXiv:2606.26006, 2026

    2026 arXiv

  20. [28]

    Y . Wu, G. Tucker, and O. Nachum. Behavior regularized offline reinforcement learning.arXiv preprint arXiv:1911.11361, 2019

    1911 arXiv

  21. [29]

    Fujimoto and S

    S. Fujimoto and S. S. Gu. A minimalist approach to offline reinforcement learning.Advances in Neural Information Processing Systems, 2021

    2021

  22. [30]

    Tarasov, A

    D. Tarasov, A. Nikulin, D. Akimov, V . Kurenkov, and S. Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library.Advances in Neural Information Processing Sys- tems, 2023. 11

    2023

  23. [31]

    R. Yang, C. Bai, H. Guo, S. Li, B. Zhao, Z. Wang, P. Liu, and X. Li. Behavior contrastive learning for unsupervised skill discovery. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors,Proceedings of the 40th International Conference on Ma- c...

    2023

  24. [32]

    C. Bai, R. Yang, Q. Zhang, K. Xu, Y . Chen, T. Xiao, and X. Li. Constrained ensem- ble exploration for unsupervised skill discovery. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp, editors,Proceedings of the 41st Inter- national...

    2024

  25. [33]

    Lipman, R

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, and M. Nickel. Flow matching for generative mod- eling.International Conference on Learning Representations, 2023

    2023

  26. [34]

    X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.International Conference on Learning Representations, 2023

    2023

  27. [35]

    M. S. Albergo and E. Vanden-Eijnden. Building normalizing flows with stochastic interpolants. International Conference on Learning Representations, 2023

    2023

  28. [36]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 2020

    2020

  29. [37]

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations.International Conference on Learning Representations, 2021

    2021

  30. [38]

    D. P. Kingma and R. Gao. Understanding diffusion objectives as the elbo with simple data augmentation.Advances in Neural Information Processing Systems, 2024

    2024

  31. [39]

    K. Shi, J. Shi, P. Hebbar, Z. Zhao, T. Amarnath, Y . Su, S. Bahl, and D. Pathak. Flowdpg: Deterministic policy gradient on flow matching policies for real-world manipulation, 2026. URLhttps://arxiv.org/abs/2606.22303

    2026 arXiv

  32. [40]

    Silver, G

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In E. P. Xing and T. Jebara, editors,Proceedings of the 31st International Conference on Machine Learning, volume 32 ofProceedings of Machine Learning Research, ...

    2014

  33. [41]

    Fujimoto, D

    S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. In K. Chaudhuri and R. Salakhutdinov, editors,Proceedings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learning Research, pages 2...

  34. [42]

    Kumar, J

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, vol- ume 32...

    2019

  35. [43]

    Ding and C

    Z. Ding and C. Jin. Consistency models as a rich and efficient policy class for reinforce- ment learning. In B. Kim, Y . Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y . Sun, editors,International Conference on Learning Representations, volume 2024, pages 53047– 53066, 2024...

    2024

  36. [44]

    T. Chen, Z. Wang, and M. Zhou. Diffusion policies creating a trust region for offline reinforce- ment learning.Advances in Neural Information Processing Systems, 2024

    2024

  37. [45]

    Espinosa-Dice, Y

    N. Espinosa-Dice, Y . Zhang, Y . Chen, B. Guo, O. Oertell, G. Swamy, K. Brantley, and W. Sun. Scaling offline RL via efficient and expressive shortcut models.arXiv preprint arXiv:2505.22866, 2025

    2025 arXiv

  38. [46]

    T. Chen, H. Ma, N. Li, K. Wang, and B. Dai. One-step flow policy mirror descent.arXiv preprint arXiv:2507.23675, 2025

    2025

  39. [47]

    Y . Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In2nd Workshop on Dexterous Manipulation: Design, Perception and Control (RSS), 2024. URLhttps://openreview. net/forum?id=KwwJuZIBXH

    2024

  40. [48]

    Barreiros, A

    J. Barreiros, A. Beaulieu, A. Bhat, R. Cory, E. Cousineau, H. Dai, C.-H. Fang, K. Hashimoto, M. Z. Irshad, M. Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation.Science Robotics, 11(113):eaea6201, 2026. doi:10.1126/scirobotics. a...

    2026 doi

  41. [49]

    Z. Zhu, H. Zhao, H. He, Y . Zhong, S. Zhang, H. Guo, T. Chen, and W. Zhang. Diffusion models for reinforcement learning: A survey, 2024. URLhttps://arxiv.org/abs/2311.01223

    2024 arXiv

  42. [50]

    Uehara, Y

    M. Uehara, Y . Zhao, T. Biancalani, and S. Levine. Understanding reinforcement learning- based fine-tuning of diffusion models: A tutorial and review, 2024. URLhttps://arxiv. org/abs/2407.13734

    2024 arXiv

  43. [51]

    Zhang, W

    S. Zhang, W. Zhang, and Q. Gu. Energy-weighted flow matching for offline reinforcement learning. InInternational Conference on Learning Representations, 2025

    2025

  44. [52]

    B. Kang, X. Ma, C. Du, T. Pang, and S. Yan. Efficient diffusion policies for offline reinforce- ment learning. InNeural Information Processing Systems, 2023

    2023

  45. [53]

    S. Ding, K. Hu, Z. Zhang, K. Ren, W. Zhang, J. Yu, J. Wang, and Y . Shi. Diffusion-based reinforcement learning via Q-weighted variational policy optimization. InNeural Information Processing Systems, 2024

    2024

  46. [54]

    C. Lu, H. Chen, J. Chen, H. Su, C. Li, and J. Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. InInternational Confer- ence on Machine Learning, 2023

    2023

  47. [55]

    Frans, S

    K. Frans, S. Park, P. Abbeel, and S. Levine. Diffusion guidance is a controllable policy im- provement operator.arXiv preprint arXiv:2505.23458, 2025

    2025 arXiv

  48. [56]

    Domingo-Enrich, W

    C. Domingo-Enrich, W. Chen, and B. Amos. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control.arXiv preprint arXiv:2409.03698, 2025

    2025 arXiv

  49. [57]

    Uehara, Y

    M. Uehara, Y . Zhao, K. Black, E. Hajiramezanali, G. Scalia, N. L. Diamant, A. M. Tseng, T. Biancalani, and S. Levine. Fine-tuning of continuous-time diffusion models as entropy- regularized control.arXiv preprint arXiv:2402.15194, 2024

    2024 arXiv

  50. [58]

    Bergmeister, S

    A. Bergmeister, S. Jegelka, N. N ¨usken, C. Domingo-Enrich, and J. Pidstrigach. Reinforce ad- joint matching: Scaling rl post-training of diffusion and flow-matching models.arXiv preprint, 2025

    2025

  51. [59]

    Z. Guo, J. Sheng, D. D. Yao, and W. Tang. Improved techniques for fine-tuning flow models via adjoint matching: A deterministic control pipeline.arXiv preprint arXiv:2605.06583, 2026. 13

    2026 arXiv

  52. [60]

    J. Shin, D. Shin, J. Lee, J. Choi, and J. Choi. Efficient adjoint matching for fine-tuning diffusion models.arXiv preprint arXiv:2605.11480, 2026

    2026 arXiv

  53. [61]

    Ankile, A

    L. Ankile, A. Simeonov, I. Shenfeld, M. Torne, and P. Agrawal. From imitation to refinement– residual rl for precise assembly.arXiv preprint arXiv:2407.16677, 2024

    2024 arXiv

  54. [62]

    X. Yuan, T. Mu, S. Tao, Y . Fang, M. Zhang, and H. Su. Policy decorator: Model-agnostic online refinement for large policy model.arXiv preprint arXiv:2412.13630, 2024

    2024 arXiv

  55. [63]

    M. S. Mark, T. Gao, G. G. Sampaio, M. K. Srirama, A. Sharma, C. Finn, and A. Kumar. Policy-agnostic RL: Offline RL and online RL fine-tuning of any class and backbone. InRobot Learning Workshop, 2025

    2025

  56. [64]

    Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InInternational Conference on Learning Representations, 2023

    2023

  57. [65]

    L. He, L. Shen, L. Zhang, J. Tan, and X. Wang. DiffCPS: Diffusion model based constrained policy search for offline reinforcement learning.arXiv preprint arXiv:2310.05333, 2023

    2023 arXiv

  58. [66]

    Zhang, Z

    R. Zhang, Z. Luo, J. Sj ¨olund, T. B. Sch ¨on, and P. Mattsson. Entropy-regularized diffusion policy with Q-ensembles for offline reinforcement learning. InNeural Information Processing Systems, 2024

    2024

  59. [67]

    Hendrycks and K

    D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus), 2016. URLhttps:// arxiv.org/abs/1606.08415

    2016 arXiv

  60. [68]

    J. L. Ba. Layer normalization.arXiv preprint arXiv:1607.06450, 2016

    2016 arXiv

  61. [69]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations (ICLR), 2015. URLhttps://arxiv.org/abs/ 1412.6980. 14 A Appendix A.1 Compatibility with Pretrained Flow Policies A natural question is whether VINE can le...

    2015 arXiv

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.