Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

The optimal policy in a greedified KL-regularized RL problem splits into a reward-maximizing and a reward-minimizing policy, making greediness an inference-time dial.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DIPOLE decomposes a KL-regularized RL objective into a pair of sigmoid-weighted diffusion policies whose score combination (CFG-like) yields stable and controllable policy improvement.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A useful empirical trick wrapped in an overstated theorem; send it to review but push for an honest theory. the 3 major comments →

arxiv 2601.00898 v3 pith:TPZ7VQAF submitted 2025-12-31 cs.LG cs.RO

Dichotomous Diffusion Policy Optimization

Ruiming Liang , Yinan Zheng , Kexin Zheng , Tianyi Tan , Jianxiong Li , Liyuan Mao , Zhihao Wang , Guang Chen
show 4 more authors
Hangjun Ye Jingjing Liu Jinqiao Wang Xianyuan Zhan
This is my paper
classification cs.LG cs.RO
keywords diffusion policiesreinforcement learningKL-regularized RLclassifier-free guidanceweighted regressionoffline RLoffline-to-online RLvision-language-action models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the optimal policy for a greedified KL-regularized RL objective can be decomposed into two policies learned with bounded weights—one that amplifies high-return actions and one that amplifies low-return actions—and that recombining their scores with a single dial recovers the optimal policy. If true, it gives diffusion and flow policies an RL fine-tuning method that avoids the two failure modes of prior approaches: unstable gradient backpropagation through denoising and crude Gaussian likelihood approximations. The paper supports the claim with a closed-form derivation, a training scheme that is just two weighted-regression losses, and experiments on standard offline and offline-to-online RL tasks plus a billion-parameter vision-language-action driving model.

Core claim

DIPOLE's central claim is that the exponential reward weight exp(βG) in the standard KL-regularized optimal policy can be replaced by a bounded sigmoid weight σ(βG) together with a greediness factor ω, yielding the closed-form optimum π*(a|s) ∝ μ(a|s)·σ(βG(s,a))·exp(ωβG(s,a)). Because σ/(1−σ) = exp(βG), this optimum factors into [π+]^(1+ω) / [π−]^ω, where π+ is trained with weight σ(βG) and π− with weight 1−σ(βG). Sampling from π* then becomes the score-level combination ∇ log π* = (1+ω)∇ log π+ − ω∇ log π−, which for diffusion policies is implemented as ε̃ = (1+w)ε+ − wε−. The paper argues this makes learning stable because both training weights are bounded between 0 and 1, and makes greedi

What carries the argument

The central object is the dichotomous policy pair (π+, π−) defined by sigmoid-weighted regression: π+(a|s) ∝ μ(a|s)σ(βG(s,a)) and π−(a|s) ∝ μ(a|s)(1−σ(βG(s,a))). The carrying identity is π* ∝ [π+]^(1+ω)/[π−]^ω, which at the score level becomes (1+ω)∇log π+ − ω∇log π−. For diffusion and flow policies, this is implemented directly as a linear combination of noise predictions. The boundedness of the sigmoid weights prevents loss explosion, while the identity σ/(1−σ)=exp(βG) is what lets the two bounded halves recover the original exponential weighting of the optimal policy.

Load-bearing premise

The load-bearing assumption is that the state-visitation distribution can be treated as fixed when deriving the optimal policy; if a better policy changes which states are visited enough, then the derived policy is not the true optimum of the stated objective and the score combination becomes a heuristic rather than a theorem.

What would settle it

On a small MDP with known transition dynamics, compute the exact optimal policy of the greedified KL objective by dynamic programming without fixing the state distribution, then compare the action distribution that DIPOLE's two sigmoid-weighted regressions produce from rollouts of the same MDP. If the two disagree systematically—or if the method behaves differently in a bandit (where the state distribution cannot shift) versus a chain MDP—the frozen-distribution step is the point of failure.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Diffusion and flow policies can be RL-fine-tuned by running two parallel weighted-regression objectives, avoiding gradient backpropagation through the denoising chain and avoiding Gaussian likelihood approximations.
  • The greediness of the final policy becomes an inference-time dial, so a single trained pair of policies can serve multiple optimality-versus-stability trade-offs without retraining.
  • Because both regression weights are bounded between 0 and 1, the loss is not dominated by a few extreme high-return samples, which should help the method scale with more data.
  • The same recipe transfers to large models: a billion-parameter vision-language-action driving policy improved its closed-loop score after fine-tuning with this method, suggesting practical RL fine-tuning at that scale.
  • Setting ω=0 recovers the positive policy alone, and increasing ω pushes toward the exponential-weighted objective, giving a continuum from conservative behavior-regularized updates to greedy reward maximization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The resemblance to classifier-free guidance is more than an analogy: the derivation supplies an RL-grounded prescription for which two distributions to use as the guided and unguided halves, something CFG heuristics typically leave open.
  • Because the derivation fixes the state-visitation distribution, the exact guarantee is per-state; in full MDPs the method is best understood as an advantage-weighted, behavior-regularized update. A natural test is to compare against exact dynamic programming on small tabular MDPs where state-visitation shifts are computable.
  • The two-policy trick should transfer to any generative policy class with tractable score functions, including autoregressive policies, where the analogous operation would be a linear combination of log-probability gradients.
  • Since the negative policy actively learns from low-return samples, the approach may also suit settings with abundant failure demonstrations or human preference data, where negative examples are cheap and informative.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes DIPOLE, an RL fine-tuning method for diffusion/flow policies. The central idea is to replace the exponential weight in standard KL-regularized weighted regression with a sigmoid-weighted, 'greedified' objective (Eq. 5), whose purported optimal policy (Eq. 6) decomposes into a positive policy and a negative policy (Eq. 8). At inference the scores of the two policies are combined linearly (Eq. 10), analogous to classifier-free guidance, with a greediness factor ω. The method is evaluated in offline and offline-to-online settings on ExORL and OGBench, and on a 1-billion-parameter VLA driving model in the NAVSIM closed-loop benchmark.

Significance. If the theoretical claims were established, DIPOLE would offer a stable, controllable, and scalable way to fine-tune diffusion/flow policies, with a conceptually appealing connection between KL-regularized RL and classifier-free guidance. The empirical scope is broad: 39 offline RL tasks, offline-to-online results, and a large VLA autonomous-driving demonstration, all with 8 seeds in the main RL benchmarks. The bounded sigmoid weighting is a practically sensible alternative to exponential weighting. However, the central optimality theorem is not proven as stated, so the paper currently provides a heuristic algorithm with strong empirical support rather than a theoretically grounded one.

major comments (3)
  1. [Appendix B, Theorem 1 and Eq. (13)-(14)] The proof of Theorem 1 differentiates the objective in Eq. (5) with respect to π(a|s) while treating dπ(s) as a constant. But dπ(s) is itself a functional of π through the transition kernel, so ∂dπ(s')/∂π(a|s) terms are omitted. Irreducibility only ensures dπ(s)>0; it does not make these derivatives vanish. Consequently Eq. (6) is not established as the optimum of Eq. (5). A simple two-state MDP where a high-reward state is reachable only by a specific action from the start state shows the per-state formula is not the global maximizer. Since Eq. (10) and the entire dichotomous decomposition rely on Eq. (6), the central 'optimal policy' claim is unsupported. The authors should either prove the claim with correct account of dπ, or explicitly reframe the method as a per-state/per-dataset surrogate objective and remove the global optimality language.
  2. [Section 3.1, Eq. (2)-(3)] The same issue appears earlier: Eq. (3) is stated as the closed-form solution of the KL-regularized objective Eq. (2), but this is only true when the state visitation distribution is fixed. In the multi-step RL setting, maximizing E_{s∼dπ(s)}[...] means changes in π affect dπ, so the simple per-state reweighting μ(a|s)exp(βG(s,a)) is not generally the optimum. This inherited issue is load-bearing because it motivates the entire DIPOLE construction. The paper should state the fixed-state-distribution assumption explicitly or replace the claim with a statement that Eq. (3) is an AWR-style surrogate.
  3. [Section 4.2 and Appendix E.2/E.4] The NAVSIM 'navtest' variant is trained on rollouts from the navtest split and then evaluated on the same navtest split. Table 4 reports DP-VLA w/ DIPOLE navtest with a 6.5-point improvement, but this does not measure generalization to unseen scenarios; it measures test-time adaptation on the evaluation split. This is a methodological limitation that should be disclosed prominently. The sentence in the abstract claiming evaluation on 'the large-scale real-world AD benchmark NAVSIM' should be qualified, and the navtest result should not be presented on equal footing with the navtrain result.
minor comments (5)
  1. [Appendix B, Eq. (14)] The expression 'exp(ωβ α_s + γ_{s,a}/dπ(s) − 1 − log Z(s))' is ambiguous. If the intended term is (α_s + γ_{s,a})/dπ(s), parentheses should be added. This matters for reproducibility of the algebra.
  2. [Lemma 1, Section 3.1] Lemma 1 is cited from Zheng et al. (2024) without stating the conditions under which the weighted diffusion loss in Eq. (4) recovers the optimal policy. Since this lemma underlies the practical training losses, a statement of the required expressiveness and sampling assumptions would improve rigor.
  3. [Appendix D.4, Figure 5] The ablation for the greediness factor ω and temperature β is reported on only two Cheetah tasks with 3 seeds. The claim that these results show 'easy-tuning property of DIPOLE' is stronger than the evidence supports.
  4. [Appendix E.5 and Figure 2/Figure 7] The captions for Figure 2 and Figure 7 are nearly identical, and the appendix figure numbering appears duplicated. Please renumber or differentiate the captions.
  5. [Section 5, Related Work] The statement that CFGRL 'lacks theoretical backing' should be softened, since the present paper's own theoretical backing is currently heuristic; otherwise the contrast reads as overstated.

Circularity Check

1 steps flagged

Partial circularity: the dichotomous score combination is an algebraic rewrite of the designed objective, not an independent prediction; benchmarks and implementation details provide independent content.

specific steps
  1. self definitional [Section 3.2, Eqs. (6)-(10) (Theorem 1 and Dichotomous policy extraction)]
    "π⋆(a|s)∝[µ(a|s)·σ(βG(s, a))]1+ω/[µ(a|s)·(1−σ(βG(s, a)))]ω.(7) ... logπ⋆(a|s) = (1+ω)logπ+(a|s)−ωlogπ−(a|s) + logC ⇒ ∇a logπ⋆(a|s) = (1+ω)∇a logπ+(a|s)−ω∇a logπ−(a|s),(10)"

    Eq. (8) defines the positive and negative policies as exactly the two factors appearing in the algebraic rewrite of Eq. (6), using exp(βG)=σ/(1−σ). Eq. (10) is then the log-gradient of that same identity. Thus the claimed 'decomposition' and the linear score combination are not independently derived properties of the optimal policy; they are the chosen sigmoid factorization restated. The greediness parameter ω enters in the same way in the KL coefficient of Eq. (5) and in the final score weighting, so the controllable-greediness result is a consequence of how Eq. (5) was constructed rather than a first-principles prediction. The external benchmarks are the independent part of the paper.

full rationale

The main circularity-adjacent step is the derivation of Eq. (10) from Eq. (7): since π+ and π− are defined as the factors of the algebraic identity, the score combination is forced by construction. This is not a fitted-parameter prediction or an imported uniqueness theorem, and the paper is explicit that the design is 'under such a design,' but the wording presents the decomposition as a discovered property of the optimal policy. The empirical evaluations on ExORL, OGBench, and NAVSIM are external and give the method independent content. Two additional issues are noted but do not themselves constitute circularity: (i) the Appendix B proof of Theorem 1 differentiates Eq. (12) while holding dπ(s) fixed, omitting ∂dπ/∂π terms—this is a correctness/rigor risk for general MDPs, not a circularity; (ii) Lemma 1 is cited from Zheng et al. (2024), which shares authors, but it is a standard weighted-regression lemma supporting the diffusion extraction step, not the load-bearing dichotomous identity. Overall, the central claim is partially by construction but still has independent empirical and algorithmic content, so score 4.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 0 invented entities

The method introduces no new physical or ontological entities; the positive and negative policies are learned models, not independent postulates. The ledger's main burden is the number of tuned hyperparameters and the unproven MDP-optimality step in Theorem 1.

free parameters (7)
  • beta (temperature) = per-task: 0.5–4.5 (Table 6)
    Temperature in σ(βG); tuned per domain to balance greediness and stability; not derived.
  • omega (greediness factor / CFG scale) = 1, 2, 4 in ablations; likely per-task
    Controls CFG-style score combination; selected by hand; no principled rule provided.
  • shift k = -2 to 1 (Table 6)
    Shifts advantage before sigmoid; ad hoc hyperparameter introduced to improve flexibility.
  • expectile tau = 0.9–0.99
    IQL expectile for value learning; tuned per domain.
  • rejection sampling count N = 2–32
    Number of sampled actions per state for Q-guided rejection; affects results.
  • discount gamma = 0.99–0.995
    Task-specific discount factor.
  • NAVSIM group size g = 10 (navtrain), 25 (navtest)
    Number of rollout trajectories per sample used to estimate advantage in Eq. (17).
axioms (6)
  • standard math Closed-form solution to KL-regularized RL: argmax E[G] - (1/β)KL(π||μ) ∝ μ exp(βG)
    Used in Eq. (3) and as template for Theorem 1; standard but assumes fixed state distribution.
  • domain assumption Lemma 1: minimizing weighted diffusion loss with weight exp(βG) yields a diffusion policy that samples from the reweighted distribution (Zheng et al., 2024)
    Bridges the weighted regression loss to the optimal policy; load-bearing, not re-derived in this paper.
  • domain assumption All policies induce irreducible Markov chains so dπ(s)>0 everywhere
    Needed for the Lagrangian derivation to divide by dπ(s); stated in Appendix B.
  • ad hoc to paper The state-visitation distribution dπ(s) is treated as fixed when differentiating the objective
    The derivative w.r.t. π(a|s) ignores that changing π changes dπ; this is an AWR-style heuristic, not a rigorous MDP optimality condition. See Appendix B, Eq. (13).
  • standard math In diffusion/flow models, the network output εθ is a valid score estimate, so linear combinations of ε's correspond to linear combinations of log-density gradients
    Makes Eq. (10) implementable as (1+w)ε+ − wε−; standard for score-based models.
  • domain assumption PDMS-based reward in the NAVSIM non-reactive simulator is an appropriate proxy for driving quality, and the advantage estimate in Eq. (17) is valid
    The autonomous-driving results depend on this reward; not theoretically justified.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Dichotomous Diffusion Policy Optimization." pith.science (2026). https://pith.science/paper/TPZ7VQAF

@misc{pith2026260100898,
  author       = {Pith},
  title        = {Pith review of: Dichotomous Diffusion Policy Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TPZ7VQAF}},
  note         = {Machine review of arXiv:2601.00898}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Diffusion-based policies have gained growing popularity in solving a wide range of decision-making tasks due to their superior expressiveness and controllable generation during inference. However, effectively training large diffusion policies using reinforcement learning (RL) remains challenging. Existing methods either suffer from unstable training due to directly maximizing value objectives, or face computational issues due to relying on crude Gaussian likelihood approximation, which requires a large amount of sufficiently small denoising steps. In this work, we propose DIPOLE (Dichotomous diffusion Policy improvement), a novel RL algorithm designed for stable and controllable diffusion policy optimization. We begin by revisiting the KL-regularized objective in RL, which offers a desirable weighted regression objective for diffusion policy extraction, but often struggles to balance greediness and stability. We then formulate a greedified policy regularization scheme, which naturally enables decomposing the optimal policy into a pair of stably learned dichotomous policies: one aims at reward maximization, and the other focuses on reward minimization. Under such a design, optimized actions can be generated by linearly combining the scores of dichotomous policies during inference, thereby enabling flexible control over the level of greediness.Evaluations in offline and offline-to-online RL settings on ExORL and OGBench demonstrate the effectiveness of our approach. We also use DIPOLE to train a large vision-language-action (VLA) model for end-to-end autonomous driving (AD) and evaluate it on the large-scale real-world AD benchmark NAVSIM, highlighting its potential for complex real-world applications.

Figures

Figures reproduced from arXiv: 2601.00898 by Guang Chen, Hangjun Ye, Jianxiong Li, Jingjing Liu, Jinqiao Wang, Kexin Zheng, Liyuan Mao, Ruiming Liang, Tianyi Tan, Xianyuan Zhan, Yinan Zheng, Zhihao Wang.

Figure 1
Figure 1. Figure 1: Illustration of the policy weighting scheme in [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: NAVSIM Results: DP-VLA w/ DIPOLE fine-tuned model trajectory; ground truth ego trajectory; DP-VLA imitation pretrained model trajectory. Evaluation results. We present the experimental results in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ExORL environments. We experiment on 4 high-dimensional complex domains: Walker, Cheetah, Quadruped, and Jaco Arm. (a) antsoccer-arena (b) humanoidmaze (c) scene (d) cube-single (e) cube-double [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: OGBench environments. We experiment on 5 complex domains: antsoccer-arena, humanoidmaze, scene, cube-single, and cube-double. • humanoidmaze (navigation):.Controlling a 21-DoF Humanoid agent to reach a goal position in a given maze. • cube-play (manipulation): Controlling a robot arm to pick and place cube-shaped blocks in order to assemble designated target configurations. • scene-play (manipulation): Lon… view at source ↗
Figure 5
Figure 5. Figure 5: Top-left: ablation on β and k; Top-right: ablation on expectile τ ; Bottom: ablation on w and β 0 1 2 Step 1e6 0 50 100 Performance Humanoidmaze-medium 0 1 2 Step 1e6 0 50 100 Performance Antsoccer-arena 0 1 2 Step 1e6 0 50 100 Performance Cube-double 0 1 2 Step 1e6 0 50 100 Performance Scene-play [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Offline-to-online visualization. DIPOLE presents a stable fine-tuning process on OGBench. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: NAVSIM Results: DP-VLA w/ DIPOLE fine-tuned model trajectory; ground truth ego trajectory; DP-VLA imitation pretrained model trajectory. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL

    cs.AI 2026-07 conditional novelty 6.0

    PRISM trains one positive policy per reward plus one global negative policy and merges their token logits, improving multi-reward RL for LLMs with inference-time controllability.

  2. World Value Models for Robotic Manipulation

    cs.RO 2026-06 unverdicted novelty 5.0

    World Value Model (WVM) integrates world models with value estimation to achieve SOTA Value-Order Correlation on expert and suboptimal robotic data and improves downstream policy performance.

  3. CRAFT: Counterfactual-to-Interactive Reinforcement Fine-Tuning for Driving Policies

    cs.LG 2026-05 unverdicted novelty 5.0

    CRAFT is an on-policy RL fine-tuning framework that decomposes closed-loop policy gradients into a group-normalized counterfactual proxy plus residual correction from interaction events, achieving top closed-loop perf...

Reference graph

Works this paper leans on

32 extracted references · 18 linked inside Pith · cited by 3 Pith papers

  1. [1]

    URLhttps://arxiv.org/ abs/2410.24164

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky.π 0: A visi...

    Pith/arXiv arXiv

  2. [6]

    17 Published as a conference paper at ICLR 2026 We implement rejection sampling for inference time policy output. Specifically, we sampleNac- tions for a single state input and select the action that has the highest Q-value: a⋆ ≜arg max a∈{a(1),...,a(N) ∼π(s)} Q(s, a).(16) Evaluation.We report the average return on ExORL and the success rate on OGBench, f...

    2026

  3. [7]

    URLB: unsupervised reinforcement learning benchmark

    Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: unsupervised reinforcement learning benchmark. InProceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual,

    2021

  4. [8]

    Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192,

    Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback.arXiv preprint arXiv:2302.12192,

    Pith/arXiv arXiv

  5. [10]

    Hydra-mdp: End-to-end multimodal planning with multi-target hydra- distillation.arXiv preprint arXiv:2406.06978,

    Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra- distillation.arXiv preprint arXiv:2406.06978,

    Pith/arXiv arXiv

  6. [11]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in Neural Information Processing Systems, 35:5775–5787,

    11 Published as a conference paper at ICLR 2026 Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in Neural Information Processing Systems, 35:5775–5787,

    2026

  7. [12]

    Efficient online reinforcement learning for diffusion policy

    Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. InForty-second International Conference on Machine Learning, 2025a. Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion mo...

    Pith/arXiv arXiv

  8. [14]

    Ogbench: Benchmarking offline goal-conditioned rl

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. InInternational Conference on Learning Representations (ICLR), 2025a. Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. InInternational Conference on Machine Learning (ICML), 2025b. Xue Bin Peng, Aviral Kumar, Grace Zhang, and Se...

    Pith/arXiv arXiv 1910

  9. [18]

    Offline rl with realistic datasets: Heteroskedasticity and support constraints.arXiv preprint arXiv:2211.01052,

    Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, and Sergey Levine. Offline rl with realistic datasets: Heteroskedasticity and support constraints.arXiv preprint arXiv:2211.01052,

    Pith/arXiv arXiv

  10. [19]

    Score-based generative modeling through stochastic differential equations

    12 Published as a conference paper at ICLR 2026 Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. InInterna- tional Conference on Learning Representations,

    2026

  11. [20]

    Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine

    URLhttps://arxiv.org/abs/1801.00690. Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning.arXiv preprint arXiv:2506.15799,

    Pith/arXiv arXiv

  12. [21]

    Behavior regularized offline reinforcement learning

    Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361,

    Pith/arXiv arXiv 1911

  13. [22]

    Offline rl with no ood actions: In-sample learning via implicit value regularization

    Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. InThe Eleventh International Conference on Learning Representations, 2023a. Haoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, and Amy Zhang. An optimal discriminat...

    Pith/arXiv arXiv

  14. [23]

    Imagereward: Learning and evaluating human preferences for text-to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903–15935, 2023b. Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Le...

    Pith/arXiv arXiv

  15. [24]

    X-vla: Soft-prompted transformer as scalable cross- embodiment vision-language-action model.arXiv preprint arXiv:2510.10274, 2025a

    Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, et al. X-vla: Soft-prompted transformer as scalable cross- embodiment vision-language-action model.arXiv preprint arXiv:2510.10274, 2025a. Kexin Zheng, Lauriane Teyssier, Yinan Zheng, Yu Luo, and Xianyuan Zhan. Towards robust zer...

    Pith/arXiv arXiv 2026

  16. [25]

    net/forum?id=j5JvZCaDM0

    URLhttps://openreview. net/forum?id=j5JvZCaDM0. Yinan Zheng, Ruiming Liang, Kexin ZHENG, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Diffusion-based planning for au- tonomous driving with flexible guidance. InThe Thirteenth International Conference on Learning Representations, 2025c. URLht...

    2026

  17. [27]

    We utilize datasets collected by unsuper- vised RL algorithmsRND(Burda et al.,

    on the DeepMind Control Suite (Tassa et al., 2018). We utilize datasets collected by unsuper- vised RL algorithmsRND(Burda et al.,

    2018

  18. [28]

    For each environment, we use the full dataset with all transitions from each dataset

    across four domains (Walker,Jaco,Quadruped, and Cheetah). For each environment, we use the full dataset with all transitions from each dataset. •Walker(locomotion): A bipedal robot with 24-dimensional states (joint positions/velocities) and 6-dimensional actions. Test tasks includerun,stand, andwalk. Rewards combine dense objec- tives: maintaining torso h...

    2026

  19. [30]

    Comparing with other offline RL methods,DIPOLEachieves better performance after the full finetuning process

    DIPOLEpossesses steady improvement after online interaction. Comparing with other offline RL methods,DIPOLEachieves better performance after the full finetuning process. 18 Published as a conference paper at ICLR 2026 Table 6: Task-specific hyperparameters for DIPOLE. Task Categorybetaβshift factorkdiscountγexpectileτsample actionsN OGBench-humanoidmaze-m...

    2026

  20. [31]

    The visual input comprises images from Front, Front-Left, and Front-Right perspectives, while the language input consists of driving commands provided by the dataset

    as the visual-language en- coder, paired with a 475M-parameter Diffusion Transformer as the action decoder. The visual input comprises images from Front, Front-Left, and Front-Right perspectives, while the language input consists of driving commands provided by the dataset. Encoder output tokens are processed by the action decoder through a cross-attentio...

    2024

  21. [32]

    F LIMITATION& DISCUSSION& FUTUREWORK Here, we discuss the limitations, potential solutions, and promising future directions of our work

    It can be observed that while the pre-fine-tuned model exhibits off-road driving or collisions, these rule- violating and dangerous behaviors are effectively rectified after fine-tuning. F LIMITATION& DISCUSSION& FUTUREWORK Here, we discuss the limitations, potential solutions, and promising future directions of our work. While this paper presents a polic...

    2026

  22. [1998]

    In this setting, a policy is a probability distribution of actions conditioned on a state

    given byM= (S,A,P, r, γ), which comprises a state space S, an action spaceA, a state transitionS, a reward functionrand a discount factorγ. In this setting, a policy is a probability distribution of actions conditioned on a state. In addition, we assume that all policies induce an irreducible Markov Chain, with any two states reachable from each other by ...

    2026

  23. [2015]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  24. [2017]

    Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models.arXiv preprint arXiv:2402.03300,

    Pith/arXiv arXiv

  25. [2018]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573,

    Pith/arXiv arXiv

  26. [2019]

    Aligning text-to- image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739,

    Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation.arXiv preprint arXiv:2310.03739,

    Pith/arXiv arXiv

  27. [2020]

    Awac: Accelerating online rein- forcement learning with offline datasets.arXiv preprint arXiv:2006.09359,

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets.arXiv preprint arXiv:2006.09359,

    Pith/arXiv arXiv 2006

  28. [2021]

    Diffusion guidance is a controllable policy improvement operator.arXiv preprint arXiv:2505.23458,

    Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a controllable policy improvement operator.arXiv preprint arXiv:2505.23458,

    Pith/arXiv arXiv

  29. [2022]

    org/abs/2207.12598

    URLhttps://arxiv. org/abs/2207.12598. Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851,

    Pith/arXiv arXiv

  30. [2023]

    Discrete diffusion for reflective vision-language-action models in autonomous driving.arXiv preprint arXiv:2509.20109,

    Pengxiang Li, Yinan Zheng, Yue Wang, Huimin Wang, Hang Zhao, Jingjing Liu, Xianyuan Zhan, Kun Zhan, and Xianpeng Lang. Discrete diffusion for reflective vision-language-action models in autonomous driving.arXiv preprint arXiv:2509.20109,

    arXiv

  31. [2024]

    Rl with kl penalties is better viewed as bayesian inference

    Tomasz Korbak, Ethan Perez, and Christopher Buckley. Rl with kl penalties is better viewed as bayesian inference. InFindings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083–1091,

    2022

  32. [2025]

    Extreme q-learning: Maxent rl without entropy

    10 Published as a conference paper at ICLR 2026 Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. InThe Eleventh International Conference on Learning Representations,

    2026

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.