Pith. sign in

REVIEW 2 major objections 1 minor 92 references

Horizon Adaptive Offline Policy Learning via Value Stitching

T0 review · 2 major / 1 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dynamic horizon composition via auxiliary value functions and stitching policies improves offline RL on long-horizon tasks.

desk verdict VAST combines an auxiliary value function with a stitching policy for dynamic horizon composition in offline RL, which targets a real bias issue and shows benchmark gains but needs full details to confirm the mechanics. read the letter →

arxiv 2606.21136 v1 pith:TJ7N5IJR submitted 2026-06-19 cs.LG cs.RO

Kexin Zheng , Xianyuan Zhan , Xintao Yan This is my paper
classification cs.LGcs.RO
keywords offlinereinforcementlearningvaluefunctionestimationhorizonadaptationstitchinglong-horizontaskstemporaldifferencepolicyoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces VAST to replace rigid fixed-horizon value backups in offline reinforcement learning with recursive, horizon-adaptive composition. It couples value optimization to an auxiliary value function conditioned on future states and horizon lengths, learned directly from data, plus a stitching policy that picks reward-maximizing horizons and sub-goals. This produces variable-length return estimates that serve as supervision for policy learning. The approach is tested on 50 tasks from OGBench and shows gains especially where tasks involve multiple stages and extended sequences.

What carries the argument

Future state- and horizon-length-conditioned auxiliary value function paired with a stitching policy that selects optimal horizon length and sub-goal for value composition.

What would settle it

A controlled comparison on a long-horizon OGBench task where policies trained with VAST value estimates achieve lower returns than those trained with fixed-horizon n-step backups.

Watch

Extended reading notes

Core claim

VAST replaces fixed-horizon backups with recursive horizon-adaptive value composition. Its key ingredient is a future state- and horizon-length-conditioned auxiliary value function learned through direct data supervision together with a stitching policy that selects the reward-maximizing horizon length and future sub-goal, enabling direct estimation and compositional stitching of variable-length returns grounded in actionable sub-goal states.

Load-bearing premise

An auxiliary value function learned from data supervision, when paired with the stitching policy, yields accurate value estimates that can be greedily exploited without adding bias or instability.

Editorial extensions

If this is right

  • Outperforms fixed-step TD, n-step TD, Q-chunking, and generative-value baselines on long-horizon tasks.
  • Enables direct estimation of variable-length returns without online interaction.
  • Yields a greedily exploitable value-supervision signal for offline policy optimization.
  • Delivers stronger results on high-complexity multi-stage tasks within the OGBench suite.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same auxiliary-function-plus-stitching pattern could be tested as a drop-in replacement for standard value heads in other offline algorithms.
  • If the stitching policy generalizes, similar conditioning on horizon length might reduce compounding errors in model-based planning.
  • The method's reliance on sub-goal states suggests it could interact with goal-conditioned RL frameworks without extra machinery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper introduces VAST for offline policy learning in long-horizon tasks. It replaces fixed-horizon TD backups with recursive horizon-adaptive value composition, using a future state- and horizon-length-conditioned auxiliary value function learned via direct data supervision together with a stitching policy that selects the reward-maximizing horizon length and sub-goal state. The method is claimed to yield accurate, greedily exploitable value signals and to outperform fixed-step, extended-horizon, and generative-value baselines across 50 OGBench tasks, with particular gains on high-complexity long-horizon problems.

Significance. If the auxiliary function and stitching mechanism indeed deliver unbiased compositional returns without introducing instability, the approach would provide a principled way to adapt horizon length dynamically, addressing a known limitation of rigid n-step and chunking methods in multi-stage offline RL.

major comments (2)
  1. [Abstract / Method description] The central claim that direct supervision of the auxiliary value function plus the stitching policy yields an accurate, bias-free supervision signal for the main policy (without post-hoc selection effects) is load-bearing yet unsupported by any derivation or stability argument in the provided text; the weakest assumption identified in the review remains unaddressed.
  2. [Abstract] No equations, algorithm pseudocode, or proof sketch are supplied to show how the horizon-adaptive composition is implemented or why it avoids the value-estimation bias accumulation that the introduction attributes to conventional TD learning.
minor comments (1)
  1. [Abstract] The abstract states results on 50 tasks but does not name the precise baselines, metrics, or statistical tests; these details are required for reproducibility claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed review and the opportunity to clarify the technical foundations of VAST. We address the two major comments point by point below and commit to revisions that strengthen the formal justification while preserving the manuscript's core contributions.

read point-by-point responses

  1. Referee: [Abstract / Method description] The central claim that direct supervision of the auxiliary value function plus the stitching policy yields an accurate, bias-free supervision signal for the main policy (without post-hoc selection effects) is load-bearing yet unsupported by any derivation or stability argument in the provided text; the weakest assumption identified in the review remains unaddressed.

    Authors: We agree that an explicit derivation of unbiased compositional returns would strengthen the paper. The method section grounds the auxiliary value function in direct supervision from observed transitions, which replaces bootstrapped TD targets with empirical returns of variable length; the stitching policy is then trained to select the horizon-subgoal pair that maximizes the composed return under the same data distribution. This joint optimization is intended to avoid post-hoc selection bias. Nevertheless, we acknowledge that a formal stability argument addressing potential assumptions (e.g., coverage of the stitching policy) is not currently provided. In the revised manuscript we will add a derivation and brief stability analysis in an appendix. revision: yes

  2. Referee: [Abstract] No equations, algorithm pseudocode, or proof sketch are supplied to show how the horizon-adaptive composition is implemented or why it avoids the value-estimation bias accumulation that the introduction attributes to conventional TD learning.

    Authors: The abstract is deliberately concise and equation-free. The full manuscript (Section 3) contains the defining equations for the horizon-conditioned auxiliary value function, the stitching objective, and the overall training procedure, together with the algorithm. To make the bias-avoidance argument more immediately accessible, we will insert a compact pseudocode block in the main text and a short proof sketch in the appendix of the revised version. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity

full rationale

The abstract describes a method where an auxiliary value function is learned via direct data supervision and combined with a stitching policy for dynamic horizon composition. No equations, derivations, or self-citations are provided that reduce any claimed prediction or result to a fitted input or self-referential definition by construction. The central claim rests on empirical outperformance on OGBench tasks rather than a closed mathematical loop. Without load-bearing self-citations or renamings of known results in the available text, the derivation chain appears self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 1 assumptions · 1 invented entities

Abstract-only; the method rests on the domain assumption that direct supervision of a horizon-conditioned auxiliary value function yields usable sub-goal signals and that the stitching policy can be optimized without circular dependence on the main value function.

assumptions (1)
  • domain assumption An auxiliary value function conditioned on future state and horizon length can be learned through direct data supervision to provide accurate variable-length return estimates.
    Stated as the key ingredient enabling horizon-adaptive stitching.
invented entities (1)
  • horizon-adaptive stitching policy
    purpose: Selects reward-maximizing horizon length and future sub-goal to compose value estimates
    New component introduced to achieve dynamic horizon composition; no independent evidence provided in abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Horizon Adaptive Offline Policy Learning via Value Stitching." pith.science (2026). https://pith.science/paper/TJ7N5IJR

@misc{pith2026260621136,
  author       = {Pith},
  title        = {Pith review of: Horizon Adaptive Offline Policy Learning via Value Stitching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TJ7N5IJR}},
  note         = {Machine review of arXiv:2606.21136}
}
read the original abstract

Learning accurate value functions plays a decisive role for reinforcement learning (RL) agents to solve long-horizon, complex tasks. Conventional temporal-difference (TD) learning objectives suffer from value-estimation bias that accumulates over the horizon, while extended-horizon modeling methods, such as n-step TD backups and Q-chunking, adopt a rigid, fixed-horizon value-modeling recipe that is often not flexible enough to capture complex value structures in long-horizon, multi-stage tasks. In this paper, we show that enabling value updates with dynamic horizon composition can yield a strong offline policy learning scheme. Our method, Horizon Adaptive Offline Policy Learning via VAlue STitching (VAST), replaces fixed-horizon backups with recursive, horizon-adaptive value composition. Its key ingredient is to couple value optimization with a future state- and horizon-length-conditioned auxiliary value function that is learned through direct data supervision, and a stitching policy that optimally selects the reward-maximizing horizon length and future sub-goal to achieve horizon-adaptive value stitching. This design enables direct estimation and compositional "stitching" of variable-length returns grounded in actionable sub-goal states, providing an accurate and greedily exploitable value-supervision signal for offline policy optimization. Across 50 tasks on OGBench, VAST outperforms fixed-step, extended-horizon methods, and generative-value offline RL baselines, achieving strong performance particularly in high-complexity, long-horizon decision-making tasks.

Figures

Figures reproduced from arXiv: 2606.21136 by the authors.

Figure 1
Figure 1. VAST compositionally stitches cumulative returns between adaptive-lengths, actionable sub￾goals, thereby enhancing value learning optimality and stability. (Left): VAST leverages the horizon-based cumulative return (G-function) to measure the expected discounted return between two reachable states over a specific horizon, coupled with a stitching policy performing horizon-aware optimal sub-goal selection. (Right): V… view at source ↗
Figure 2
Figure 2. 4x4 grid navigation. Toy example requiring navigation from the bottom-left to the top-right. The data include sub-optimal paths only (green path and purple path); the envi￾ronment is specified by dark red, light red, and skull blocks, indicat￾ing high-, low-, and punitive-reward. The depth of blue in blocks indi￾cates the value estimates from fixed￾step/chunked value backups, and the optimal stitching value. The lim… view at source ↗
Figure 3
Figure 3. Learning curves. (Top): cube-double-play. (Middle): puzzle-4x4-play. (Bottom): cube-triple-play. upward trend in the Hard setting further highlights its scalability. To better understand its be￾havior, we visualize the ground-truth trajectory together with the future states predicted by VAST on cube-triple-play-task3 in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization on cube-triple-play-task3 (pop_from_stuck). The solid robot arm shows the current pose, while the transparent arm shows the sub-goal. 5 10 25 50 Max. Horizon 60 80 100 Success Rate CD 5 10 25 50 Max. Horizon 0 25 50 Success Rate P44 0.1 0.25 0.5 1 2 Comp.…
Figure 5
Figure 5. Figure 5: Ablations on important hyperparameter. We provide the horizon length ablation (left 1-2), the compositional factor ablation (left 3-4), and the rejection sampling strength ablation (left 5) using default tasks on domain cube-double-play and puzzle-4x4-play. on the task…
Figure 6
Figure 6. Figure 6: OGBench Basic. We experiment on 6 majorly used domains on OGBench [31] in single￾task variant, shown as (a)-(f). Each domain contains 5 tasks [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: OGBench Hard. We experiment on 4 highly challenging domains on OGBench [31] in singletask variant, shown as (a)-(d). Each domain contains 5 tasks. C Experimental Setup C.1 Environment We evaluate VAST on the OGBench task suite [31]. OGBench is a benchmark for goal-cond…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

92 extracted references · 4 canonical work pages

  1. [1]

    Learning to predict by the methods of temporal differences.Machine learning, 3(1):9–44, 1988

    Richard S Sutton. Learning to predict by the methods of temporal differences.Machine learning, 3(1):9–44, 1988

    1988

  2. [2]

    Q-learning.Machine learning, 8(3):279–292, 1992

    Christopher JCH Watkins and Peter Dayan. Q-learning.Machine learning, 8(3):279–292, 1992

    1992

  3. [3]

    Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.nature, 518(7540):529–533, 2015

    2015

  4. [4]

    Mastering the game of go with deep neural networks and tree search.nature, 529 (7587):484–489, 2016

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanc- tot, et al. Mastering the game of go with deep neural networks and tree search.nature, 529 (7587):484–489, 2016

    2016

  5. [5]

    Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge.nature, 550(7676):354–359, 2017. 10

    2017

  6. [6]

    Reinforcement learning: An introduction

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. 1(1), 1998

    1998

  7. [7]

    A survey of temporal credit assignment in deep reinforcement learning.Transac- tions on Machine Learning Research, 2024

    Eduardo Pignatelli, Johan Ferret, Matthieu Geist, Thomas Mesnard, Hado van Hasselt, and Laura Toni. A survey of temporal credit assignment in deep reinforcement learning.Transac- tions on Machine Learning Research, 2024

    2024

  8. [8]

    Optimizing agent behavior over long time scales by transporting value.Nature communications, 10(1):5223, 2019

    Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value.Nature communications, 10(1):5223, 2019

    2019

Show all 92 references
  1. [9]

    Is value learning really the main bottleneck in offline rl? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  2. [10]

    Convergence of stochastic iterative dynamic programming algorithms.Advances in neural information processing systems, 6, 1993

    Tommi Jaakkola, Michael Jordan, and Satinder Singh. Convergence of stochastic iterative dynamic programming algorithms.Advances in neural information processing systems, 6, 1993

    1993

  3. [11]

    Analysis of temporal-diffference learning with function approximation.Advances in neural information processing systems, 9, 1996

    John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation.Advances in neural information processing systems, 9, 1996

    1996

  4. [12]

    Multi-step rein- forcement learning: A unifying algorithm

    Kristopher De Asis, J Hernandez-Garcia, G Holland, and Richard Sutton. Multi-step rein- forcement learning: A unifying algorithm. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  5. [13]

    Horizon reduction makes rl scalable

    Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes rl scalable. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  6. [14]

    Td_gamma: Re-evaluating complex backups in temporal difference learning.Advances in Neural Information Processing Systems, 24, 2011

    George Konidaris, Scott Niekum, and Philip S Thomas. Td_gamma: Re-evaluating complex backups in temporal difference learning.Advances in Neural Information Processing Systems, 24, 2011

    2011

  7. [15]

    Coarse-to-fine q-network with action sequence for data- efficient reinforcement learning

    Younggyo Seo and Pieter Abbeel. Coarse-to-fine q-network with action sequence for data- efficient reinforcement learning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  8. [16]

    Chunking the critic: A transformer-based soft actor-critic with n-step returns.arXiv preprint arXiv:2503.03660, 2025

    Dong Tian, Onur Celik, and Gerhard Neumann. Chunking the critic: A transformer-based soft actor-critic with n-step returns.arXiv preprint arXiv:2503.03660, 2025

    2025 arXiv

  9. [17]

    Reinforcement learning with action chunking

    Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement learning with action chunking. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  10. [18]

    Decoupled q-chunking, 2025

    Qiyang Li, Seohong Park, and Sergey Levine. Decoupled q-chunking, 2025

    2025

  11. [19]

    A distributional perspective on reinforce- ment learning

    Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning. InInternational conference on machine learning, pages 449–458. Pmlr, 2017

    2017

  12. [20]

    floq: Training critics via flow-matching for scaling compute in value-based RL

    Bhavya Kumar Agrawalla, Michal Nauman, Khush Agrawal, and Aviral Kumar. floq: Training critics via flow-matching for scaling compute in value-based RL. InNeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists, 2025

    2025

  13. [21]

    Value flows

    Perry Dong, Chongyi Zheng, Chelsea Finn, Dorsa Sadigh, and Benjamin Eysenbach. Value flows. InInternational Conference on Learning Representations (ICLR), 2026

    2026

  14. [22]

    Temporal abstrac- tion in reinforcement learning with the successor representation.Journal of machine learning research, 24(80):1–69, 2023

    Marlos C Machado, Andre Barreto, Doina Precup, and Michael Bowling. Temporal abstrac- tion in reinforcement learning with the successor representation.Journal of machine learning research, 24(80):1–69, 2023

    2023

  15. [23]

    Sutton, Doina Precup, and Satinder Singh

    Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.Artificial Intelligence, 112(1): 181–211, 1999. ISSN 0004-3702. 11

    1999

  16. [24]

    University of Massachusetts Amherst, 2000

    Doina Precup.Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000

    2000

  17. [25]

    Learning options in reinforcement learning

    Martin Stolle and Doina Precup. Learning options in reinforcement learning. InInternational Symposium on abstraction, reformulation, and approximation, pages 212–223, 2002

    2002

  18. [26]

    The option-critic architecture

    Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. InProceedings of the AAAI Conference on Artificial Intelligence, 2017

    2017

  19. [27]

    Learning abstract options

    Matthew Riemer, Miao Liu, and Gerald Tesauro. Learning abstract options. InThe Thirty- second Annual Conference on Neural Information Processing Systems, 2018

    2018

  20. [28]

    A policy-guided imitation approach for offline reinforcement learning

    Haoran Xu, Li Jiang, Jianxiong Li, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. InAdvances in Neural Information Processing Systems, 2022

    2022

  21. [29]

    Hiql: Offline goal- conditioned rl with latent states as actions, 2023

    Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions, 2023

    2023

  22. [30]

    Data-efficient hierarchical reinforcement learning.Advances in neural information processing systems, 31, 2018

    Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning.Advances in neural information processing systems, 31, 2018

    2018

  23. [31]

    Ogbench: Bench- marking offline goal-conditioned rl

    Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Bench- marking offline goal-conditioned rl. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  24. [32]

    John Wiley & Sons, 2007

    Warren B Powell.Approximate Dynamic Programming: Solving the curses of dimensionality, volume 703. John Wiley & Sons, 2007

    2007

  25. [33]

    Bridging the gap be- tween value and policy based reinforcement learning

    Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap be- tween value and policy based reinforcement learning. InThe Thirty-first Annual Conference on Neural Information Processing Systems, 2017

    2017

  26. [34]

    Conservative q-learning for offline reinforcement learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. InProceedings of the 34th International Conference on Neural Information Processing Systems, 2020

    2020

  27. [35]

    Offline reinforcement learning with implicit q-learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. InInternational Conference on Learning Representations, 2021

    2021

  28. [36]

    Offline rl with no ood actions: In-sample learning via implicit value regularization

    Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations, 2023

    2023

  29. [37]

    Learning from delayed rewards, 1989

    Christopher John Cornish Hellaby Watkins et al. Learning from delayed rewards, 1989

    1989

  30. [38]

    Incremental multi-step q-learning

    Jing Peng and Ronald J Williams. Incremental multi-step q-learning. InMachine Learning Proceedings 1994, pages 226–232. Elsevier, 1994

    1994

  31. [39]

    Policy evalu- ation using theω-return.Advances in Neural Information Processing Systems, 28, 2015

    Philip S Thomas, Scott Niekum, Georgios Theocharous, and George Konidaris. Policy evalu- ation using theω-return.Advances in Neural Information Processing Systems, 28, 2015

    2015

  32. [40]

    Learning fine-grained bimanual manipulation with low-cost hardware.Robotics: Science and Systems, 2023

    Tony Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware.Robotics: Science and Systems, 2023

    2023

  33. [41]

    John Wiley & Sons, 2014

    Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  34. [42]

    Feudal networks for hierarchical reinforcement learn- ing

    Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learn- ing. InInternational conference on machine learning, pages 3540–3549. PMLR, 2017

    2017

  35. [43]

    Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regres- sion: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019. 12

    1910 arXiv

  36. [44]

    Extreme q-learning: Maxent rl without entropy

    Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  37. [45]

    Safe offline reinforcement learning with feasibility-guided diffusion model

    Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guided diffusion model. In The Twelfth International Conference on Learning Representations, 2024

    2024

  38. [46]

    Dichoto- mous diffusion policy optimization

    Ruiming Liang, Yinan Zheng, Kexin Zheng, Tianyi Tan, Jianxiong Li, Liyuan Mao, Zhihao Wang, Guang Chen, Hangjun Ye, Jingjing Liu, Jinqiao Wang, and Xianyuan Zhan. Dichoto- mous diffusion policy optimization. InInternational Conference on Learning Representations, 2026

    2026

  39. [47]

    Deep unsuper- vised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. pmlr, 2015

    2015

  40. [48]

    Denoising diffusion probabilistic models.Ad- vances in neural information processing systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Ad- vances in neural information processing systems, 33:6840–6851, 2020

    2020

  41. [49]

    Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  42. [50]

    Towards robust zero-shot reinforcement learning

    Kexin Zheng, Lauriane Teyssier, Yinan Zheng, Yu Luo, and Xianyuan Zhan. Towards robust zero-shot reinforcement learning. InThe Thirty-ninth Annual Conference on Neural Informa- tion Processing Systems, 2025

    2025

  43. [51]

    Efficient online reinforcement learning for diffusion policy

    Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. InInternational Conference on Machine Learning, pages 41837– 41853. PMLR, 2025

    2025

  44. [52]

    Flow q-learning

    Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. InInternational Conference on Machine Learning (ICML), 2025

    2025

  45. [53]

    Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning

    Liyuan Mao, Haoran Xu, Weinan Zhang, Xianyuan Zhan, and Amy Zhang. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. InAdvances in Neural Infor- mation Processing Systems, 2024

    2024

  46. [54]

    Tucker, and Justin Fu

    Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems.ArXiv, abs/2005.01643, 2020

    2005 arXiv

  47. [55]

    Off-policy deep reinforcement learning without exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. InInternational Conference on Machine Learning, 2018

    2018

  48. [56]

    Stabilizing off- policy q-learning via bootstrapping error reduction

    Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. InAdvances in Neural Information Pro- cessing Systems, 2019

    2019

  49. [57]

    A minimalist approach to offline reinforcement learn- ing.Advances in neural information processing systems, 34:20132–20145, 2021

    Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learn- ing.Advances in neural information processing systems, 34:20132–20145, 2021

    2021

  50. [58]

    Proto: Iterative policy regularized offline-to-online reinforcement learning, 2023

    Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-online reinforcement learning, 2023

    2023

  51. [59]

    When data geometry meets deep function: Generalizing offline reinforcement learning

    Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. In International Conference on Learning Representations, 2023

    2023

  52. [60]

    Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl

    Peng Cheng, Xianyuan Zhan, Zhihao Wu, Wenjia Zhang, Shoucheng Song, Han Wang, Youfang Lin, and Li Jiang. Look beneath the surface: Exploiting fundamental symmetry for sample-efficient offline rl. InAdvances in Neural Information Processing Systems, 2023

    2023

  53. [61]

    Offline reinforcement learn- ing with fisher divergence critic regularization

    Ilya Kostrikov, Jonathan Tompson, Rob Fergus, and Ofir Nachum. Offline reinforcement learn- ing with fisher divergence critic regularization. InProceedings of the 39th International Con- ference on Machine Learning, Proceedings of Machine Learning Research, 2021. 13

    2021

  54. [62]

    When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning

    Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming HU, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and-online reinforcement learning. InAdvances in Neural Information Processing Systems, 2022

    2022

  55. [63]

    Rorl: Robust offline reinforcement learning via conservative smoothing

    Rui Yang, Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. InAdvances in neural in- formation processing systems, 2022

    2022

  56. [64]

    Constraints penalized q-learning for safe offline reinforcement learning

    Haoran Xu, Xianyuan Zhan, and Xiangyu Zhu. Constraints penalized q-learning for safe offline reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, 2022

    2022

  57. [65]

    Offline multi-agent rein- forcement learning with implicit global-to-local value regularization

    Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent rein- forcement learning with implicit global-to-local value regularization. InAdvances in Neural Information Processing Systems, 2023

    2023

  58. [66]

    Weiss, Niru Maheswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. InProceedings of the 32nd Inter- national Conference on International Conference on Machine Learning, 2015

    2015

  59. [67]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020

    2020

  60. [68]

    Score-based generative modeling through stochastic evolution equations in hilbert spaces

    Sungbin Lim, Eunbi Yoon, Taehyun Byun, Taewon Kang, Seungwoo Kim, Kyungjae Lee, and Sungjoon Choi. Score-based generative modeling through stochastic evolution equations in hilbert spaces. InProceedings of the 37th International Conference on Neural Information Processing Syst...

    2023

  61. [69]

    Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code, 2024

    2024

  62. [70]

    Hunt, and Mingyuan Zhou

    Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  63. [71]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies, 2023

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies, 2023

    2023

  64. [72]

    Offline reinforcement learn- ing via high-fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learn- ing via high-fidelity generative behavior modeling. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  65. [73]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. InProceedings of the 40th International Conference on Machine Learning. JMLR.org, 2023

    2023

  66. [74]

    Diffusion guidance is a con- trollable policy improvement operator, 2025

    Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is a con- trollable policy improvement operator, 2025

    2025

  67. [75]

    Scaling offline rl via efficient and expressive shortcut models

    Nicolas Espinosa-Dice, Yiyi Zhang, Yiding Chen, Bradley Guo, Owen Oertell, Gokul Swamy, Kiante Brantley, and Wen Sun. Scaling offline rl via efficient and expressive shortcut models. Neural Information Processing Symposium (NeurIPS), 2025

    2025

  68. [76]

    Q-learning with adjoint matching.arXiv preprint arXiv:2601.14234, 2026

    Qiyang Li and Sergey Levine. Q-learning with adjoint matching.arXiv preprint arXiv:2601.14234, 2026

    2026 arXiv

  69. [77]

    Physical Intelligence, Ali Amin, Raichelle Aniceto, Ashwin Balakrishna, Kevin Black, Ken Conley, Grace Connors, James Darpinian, Karan Dhabalia, Jared DiCarlo, Danny Driess, Michael Equi, Adnan Esmail, Yunhao Fang, Chelsea Finn, Catherine Glossop, Thomas God- den, Ivan Goryach...

    2025

  70. [78]

    Unleashing the potential of diffusion models for end-to-end autonomous driving

    Yinan Zheng, Tianyi Tan, Bin Huang, Enguang Liu, Ruiming Liang, Jianlin Zhang, Jianwei Cui, Guang Chen, Kun Ma, Hangjun Ye, Long Chen, Ya-Qin Zhang, Xianyuan Zhan, and Jingjing Liu. Unleashing the potential of diffusion models for end-to-end autonomous driving. arXiv preprint, 2026

    2026

  71. [79]

    Stop regressing: Training value functions via classification for scalable deep RL

    Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. InPr...

    2024

  72. [80]

    Dsac: Distributional soft actor-critic for risk-sensitive reinforcement learning.Journal of Artificial Intelligence Research, 83, 2025

    Xiaoteng Ma, Junyao Chen, Li Xia, Jun Yang, Qianchuan Zhao, and Zhengyuan Zhou. Dsac: Distributional soft actor-critic for risk-sensitive reinforcement learning.Journal of Artificial Intelligence Research, 83, 2025

    2025

  73. [81]

    Q-transformer: Scalable offline reinforce- ment learning via autoregressive q-functions

    Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, et al. Q-transformer: Scalable offline reinforce- ment learning via autoregressive q-functions. InConference on Robot Learning, pages 3909–

  74. [82]

    Offline actor-critic reinforcement learn- ing scales to large models

    Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Maria Elisabeth Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Offline actor-critic reinforcement learn- ing scale...

    2024

  75. [83]

    Mixtures of experts unlock parameter scaling for deep RL

    Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL. InProceedings of the 41st Inter- national Co...

    2024

  76. [84]

    Feudal reinforcement learning.Advances in neural information processing systems, 5, 1992

    Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning.Advances in neural information processing systems, 5, 1992

    1992

  77. [85]

    Hierarchical reinforcement learning with the maxq value function de- composition.Journal of artificial intelligence research, 13:227–303, 2000

    Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function de- composition.Journal of artificial intelligence research, 13:227–303, 2000

    2000

  78. [86]

    Strategic attentive writer for learning macro-actions.Advances in neural information processing systems, 29, 2016

    Alexander Vezhnevets, V olodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions.Advances in neural information processing systems, 29, 2016

    2016

  79. [87]

    Hierarchical reinforce- ment learning: A comprehensive survey.ACM Computing Surveys (CSUR), 54(5):1–35, 2021

    Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforce- ment learning: A comprehensive survey.ACM Computing Surveys (CSUR), 54(5):1–35, 2021

    2021

  80. [88]

    Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.Ad- vances in neural information processing systems, 29, 2016

    Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.Ad- vances in neural information processing systems, 29, 2016

    2016

  81. [89]

    MuJoCo Menagerie: A collection of high-quality simulation models for MuJoCo, 2022

    Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors. MuJoCo Menagerie: A collection of high-quality simulation models for MuJoCo, 2022. URLhttp://github. com/google-deepmind/mujoco_menagerie

    2022

  82. [90]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforce- ment learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforce- ment learning. InConference on robot learning, pages 1094–1100. PMLR, 2020. 15

    2020

  83. [91]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012

    2012

  84. [92]

    i−1X t=0 γtrt + k−1X t=i γtrt s0 =s,(s k, k) # (21) =E i∼Unif{1,...,k−1},s i∼π Eπ

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Yash Katariya, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URLhttp://github...

    2018

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.