Pith. sign in

REVIEW 3 major objections 4 minor 28 references

Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A subgoal planner trained on value geometry provably converges to optimal policies in long-horizon reinforcement learning.

desk verdict A clearly written high-level subgoal proposal with a distinct training loss, but Theorem 1's convergence proof has a real gap on close start-goal pairs and needs revision before the framework can be considered proven. read the letter →

arxiv 2509.05545 v1 pith:CX3TY33U submitted 2025-09-06 cs.LG

Yang Yu This is my paper · ORCID
classification cs.LG
keywords hierarchicalreinforcementlearninggoal-conditionedtaskssubgoaldiscoveryvaluefunctiongeometryshortestpathlong-horizonhindsightexperiencereplayconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Reinforcement Learning with Anticipation (RLA), a two-level scheme for goal-conditioned tasks: a low-level policy learns to reach nearby subgoals, and a high-level anticipation model learns to propose the next subgoal on an optimal route to the final goal. The high-level training signal is geometric: once the value function is learned, the optimal value $V^*(s,g)$ equals the negative shortest-path distance, and a subgoal $\hat{s}$ lies on a shortest path from $s_0$ to $s_g$ exactly when $V^*(s_0,s_g)=V^*(s_0,\hat{s})+V^*(\hat{s},s_g)$. The paper proves that, under idealized tabular deterministic shortest-path conditions, RLA converges to the globally optimal policy, and it gives bounded sub-optimality results when the value function, subgoal selection, and low-level policy are only approximately correct. A sympathetic reader would care because this gives hierarchical RL a dense, stable training target with explicit convergence guarantees, and it suggests dramatic sample-efficiency gains from shortening the horizon.

What carries the argument

The load-bearing object is the value-geometric consistency identity of Eq. (7), $V^*(s_0,s_g)=V^*(s_0,\hat{s})+V^*(\hat{s},s_g)$, which converts the geometric condition of lying on a shortest path into a differentiable training target. The framework couples this with a low-level goal-conditioned actor-critic trained using hindsight relabeling, recursive invocation of the anticipation model for $J$ steps to control subgoal proximity, and the regularized loss $L_\psi=L_{\mathrm{detour}}+\lambda(L_{\mathrm{prog}}+L_{\mathrm{nontrivial}})$ that prevents the degenerate solutions $\hat{s}\approx s_i$ and $\hat{s}\approx s_j$. The proofs combine convergence of the low-level critic to $V^*$, the triangle-inequality equality characterization, and telescoping sums of value differences between consecutive subgoals.

What would settle it

Run RLA on a deterministic grid or line graph with unit edge costs, set $c_{\mathrm{prog}}=c_{\mathrm{nontrivial}}=1$, and train on all pairs including those at distance 2, where no intermediate state is at least 1 from both endpoints. Observing that the minimum of $L_\psi$ stays strictly above zero for those nearby pairs, and that the agent's path from such a start to such a goal is not shortest, would confirm that the missing feasibility condition is essential.

Watch

Extended reading notes

Core claim

The central claim is that a hierarchy can be made convergent by replacing the high-level policy-gradient objective with a geometric consistency loss on the learned value function. In a deterministic shortest-path GMDP, $V^*(s,g)=-d(s,g)$ satisfies the triangle inequality $V^*(s_i,s_j)\ge V^*(s_i,z)+V^*(z,s_j)$, with equality if and only if $z$ lies on a shortest path from $s_i$ to $s_j$. The anticipation model is trained to minimize $\mathrm{ReLU}(V(s_i,s_j)-V(s_i,\hat{s})-V(\hat{s},s_j))$ plus margin regularizers that rule out degenerate subgoals. Theorem 1 asserts that once the critic has converged, the converged anticipation model proposes only non-trivial waypoints on shortest paths, the low-level policy reaches them optimally, and the full agent traces a shortest path from any start state to any goal state. Theorem 2 bounds the deterministic-error cost as $C_{\mathrm{RLA}}(s_0,s_g)\le d(s_0,s_g)+M(\epsilon_\pi+3\epsilon_V+\epsilon_\psi)$, and Theorem 3 extends the bound to stochastic communicating MDPs with an additional per-step drift term.

Load-bearing premise

The proof requires that every start-goal pair used in training has a shortest-path waypoint at least one margin away from both the start and the goal, and that the low-level policy reaches each proposed subgoal exactly; if either condition fails, the global-optimality conclusion does not follow.

Editorial extensions

If this is right

  • Under the idealized conditions of Assumption 1, a fully trained RLA agent will execute a shortest path from any starting state to any goal state.
  • With bounded value error, bounded anticipation loss, and bounded low-level sub-optimality, the final path's sub-optimality is at most $M(\epsilon_\pi+3\epsilon_V+\epsilon_\psi)$ plus the optimal cost, so error accumulates only linearly in the number of high-level planning steps.
  • In stochastic communicating MDPs the same linear bound holds in expectation, with an additional per-step term $\epsilon_{\mathrm{drift}}$ for the gap between the intended subgoal and the actually reached state.
  • The paper's sample-efficiency discussion argues for roughly a $(K/L)^2$ improvement in sample complexity when a task of expected length $L$ is decomposed into sub-tasks of expected length $K$.
  • The anticipation module is modular: Theorem 3 does not depend on online learning, so the same subgoal-generation rule could be combined with a world-model planner or applied to offline data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the proof requires that every start-goal pair admit a shortest-path state at least $c_{\mathrm{prog}}$ away from the start and at least $c_{\mathrm{nontrivial}}$ away from the goal; for pairs with $d(s_i,s_j)<c_{\mathrm{prog}}+c_{\mathrm{nontrivial}}$ no such state exists, so in practice the loss will plateau above zero and the iff argument fails. A curriculum that scales the
  • Editorial inference: the analysis assumes the low-level policy reaches each subgoal exactly, but Algorithm 1 executes only $K$ fixed steps; the bounded-error theorems absorb this through $\epsilon_\pi$, but they do not track where the agent is left between subgoals. Measuring the actual distance to the intended subgoal after $K$ steps would give a sharper, testable bound.
  • Editorial inference: the same value-equality objective could serve as a task-independent evaluation metric for subgoal generators: measuring how often a learned $V$ satisfies $V(s_i,s_j)\approx V(s_i,\hat{s})+V(\hat{s},s_j)$ would directly probe planning quality across environments.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Reinforcement Learning with Anticipation (RLA), a two-level hierarchical framework for goal-conditioned long-horizon tasks. The high-level anticipation model proposes intermediate subgoals, trained to satisfy a value-geometric consistency condition, namely that the optimal value of a start-goal pair equals the sum of values through the proposed subgoal, with additional regularization margins to prevent degenerate subgoals. The low-level policy is a goal-conditioned actor-critic trained with Hindsight Experience Replay. The paper claims three theoretical results: Theorem 1 states that under idealized deterministic conditions RLA converges to a globally optimal policy and the anticipation model produces subgoals on shortest paths; Theorem 2 gives a bounded sub-optimality guarantee under bounded approximation errors; Theorem 3 extends this to stochastic communicating MDPs with an additional value-drift bound. The manuscript contains no empirical evaluation; the concluding section explicitly defers experiments to future work.

Significance. If the theoretical results were correct, the paper would offer a principled convergence guarantee for a hierarchical goal-conditioned method, and the use of value-function geometry to train a high-level planner is an appealing idea with potential practical value. The error-bound framework in Theorems 2 and 3 is transparent and would be a useful contribution if its assumptions were satisfied. However, the central result, Theorem 1, rests on an unstated feasibility condition and on a convergence claim for a neural actor-critic that is not established by the cited tabular Q-learning argument. Because the main proof is broken and the algorithm-level assumptions of Theorem 2 conflict with the implemented algorithm, the paper's central claims are not currently supported. Credit is due for explicitly recognizing the chicken-and-egg problem in joint training and for proposing a warm-up phase, but those practical mechanisms are not accompanied by evidence.

major comments (3)
  1. [Section 4.1, Eq. (8), Proof of Theorem 1] The proof of Theorem 1 asserts that the global minimum of the regularized loss L_psi is 0 and that this minimum is achieved if and only if all three ReLU terms vanish. This requires, for every start-goal pair (s_i, s_j), the existence of a state s_hat satisfying d(s_i, s_hat) >= c_prog and d(s_hat, s_j) >= c_nontrivial simultaneously with the shortest-path equality. By the triangle inequality, no state can satisfy both margin conditions when d(s_i, s_j) < c_prog + c_nontrivial. For example, in a three-state line graph A-B-C with c_prog = c_nontrivial = 1, the pair (A,B) admits only s_hat = A or s_hat = B as candidate subgoals, and the loss is at least lambda in both cases, so the minimum of Eq. (8) is not 0. Consequently, the 'if and only if' step fails and Theorem 1 does not establish convergence to a globally optimal policy for all start-goal pairs. A feasibility condition on the distance between start and goal, or a revised statement of the theorem, is required but is absent from Assumption 1.
  2. [Section 4.1, Lemma 1] Lemma 1 claims that the low-level actor-critic module trained with DDPG and HER converges to the optimal Q function and policy, with a proof that invokes the classical convergence theorem for tabular Q-learning. No argument is provided to bridge the gap between tabular Q-learning and the actual algorithm, which uses neural network function approximation, a replay buffer, target networks, deterministic policy gradient updates, and HER relabeling that changes the goal distribution during training. None of the standard conditions for tabular Q-learning convergence are verified for this neural actor-critic procedure, and the actor update is not a tabular policy improvement step. Since Lemma 1 is the basis for replacing V_omega with V* in the proof of Theorem 1 and for the bounded-error assumption in Theorem 2, this is a load-bearing gap. The paper either needs a convergence proof for the actual algorithm or must explicitly state exact convergence as an assumption rather than presenting it as a derived lemma.
  3. [Section 4.2, Theorem 2 and Algorithm 1] Theorem 2's proof assumes that the low-level policy successfully reaches each proposed subgoal, as explicitly stated in the proof ('For this analysis, we assume the low-level policy successfully reaches the subgoal'). However, Algorithm 1 (line 14) executes the low-level policy for a fixed K steps regardless of whether the subgoal is reached, and then the next high-level decision is made from the resulting state s_{k+1}, which need not equal the proposed subgoal. As a result, the sequence s_{k+1} in the telescoping argument is not the subgoal sequence, and Lemma 3's bound on d(s_k, s_{k+1}) + d(s_{k+1}, s_g) - d(s_k, s_g) cannot be applied unless s_{k+1} equals the proposed subgoal. The theorem can be repaired by assuming the low-level policy always reaches the subgoal before the K steps elapse, but that assumption is incompatible with the algorithm as written and would make M random. This mismatch undermines the bounded sub-optimality guarantee as a statement about the implemented RLA procedure.
minor comments (4)
  1. [Section 4.3, Lemma 4] In Lemma 4, the claim that equality V*(s_i, s_j) = V*(s_i, z) + V*(z, s_j) holds whenever z is an intermediate state on an optimal policy's path is not justified for stochastic MDPs; the composite-policy argument only establishes the inequality, and equality can fail even if z lies on an optimal policy from s_i to s_j because the subpath from s_i to z under that optimal policy need not be optimal for reaching z. Since Theorem 3 only uses the inequality, this does not invalidate the theorem, but the lemma statement should be corrected.
  2. [Section 4.4] The claimed sample-complexity improvement by a factor of approximately (K/L)^2 is presented as a formal result, but it is only a heuristic. The cited sample-complexity bound sigma^2/(1-beta)^2 is not derived for a hierarchical decomposition, and the variance reduction argument is qualitative. This section should be clearly labeled as intuition rather than as a theorem or a proven efficiency guarantee.
  3. [Algorithm 1] Line 29 computes V(s,g) = max_a Q_{omega'}(s,a,g), but for continuous action spaces with a neural network critic this maximum is not directly computable; standard DDPG uses the target actor to approximate the maximizing action. The paper should specify how this value estimate is obtained, especially since the anticipation loss depends on it.
  4. [General] The paper contains no experiments or simulations, and the conclusion explicitly states that empirical validation is future work. Given that the theoretical arguments have the gaps noted above, the practical claims about scalability and stability remain unsubstantiated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's theoretical claims are conditional statements about the outcome of optimizing an explicit loss, not predictions that secretly reuse their fitted inputs.

full rationale

The central derivation chain is conditional rather than circular. Theorem 1 is proved under Assumption 1 together with the claim that the anticipation loss is driven to its global minimum; once the critic has converged to V* = -d, the zero-loss conditions in Eq. (8) are exactly the shortest-path and margin conditions, so the conclusion that the converged anticipation model proposes non-trivial shortest-path waypoints unpacks the loss rather than relabeling a fitted output as an independent prediction. The low-level critic's convergence in Lemma 1 is argued from tabular Q-learning with HER and an explicit sufficient-exploration assumption, not from the anticipation model's own outputs; the acknowledged chicken-and-egg loop is addressed by a warm-up phase and by Assumption 1(4), and the bounded-error theorems are explicit error-propagation statements whose assumptions are not the conclusions. There are no self-citations used as load-bearing evidence. The paper does contain a genuine correctness gap: the proof of Theorem 1 asserts that the global minimum of Eq. (8) is 0 for all start-goal pairs even when no state satisfies both margin constraints, so the 'if and only if' step fails for pairs with d(s_i, s_j) < c_prog + c_nontrivial. That is an infeasibility/validity issue, not a circular reduction, and therefore does not raise the circularity score.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the idealized and bounded-error assumptions listed, plus an unstated feasibility condition on the regularizer margins. The margins c_prog and c_nontrivial are hand-chosen and can invalidate Theorem 1's proof when no state lies on a shortest path at the required distances from both endpoints. The theory also assumes exact subgoal reaching that the algorithm's K-step execution does not guarantee.

free parameters (5)
  • c_prog = positive margin, e.g., 1
    Used in L_prog to force subgoal distance from start; the proof of Theorem 1 assumes a subgoal with d(si, subgoal) >= c_prog exists for every pair, which is not guaranteed.
  • c_nontrivial = positive margin, e.g., 1
    Used in L_nontrivial to force subgoal distance from goal; same feasibility problem as c_prog in the proof of Theorem 1.
  • lambda = positive scalar
    Balances detour loss and regularization in Eq. (8); proof requires lambda > 0.
  • J = integer (number of recursive subgoal calls)
    Algorithm 1 line 10 recursively calls the anticipation model J times; the theoretical analysis does not model the effect of J.
  • K = integer (low-level execution steps)
    Algorithm 1 executes the low-level policy for K steps; the theory assumes the subgoal is reached exactly, independent of K.
assumptions (5)
  • domain assumption Assumption 1: finite tabular spaces, deterministic dynamics, shortest-path reward, sufficient exploration (Section 4.1).
    These are the idealizations under which Theorem 1 is stated.
  • domain assumption Assumption 2: bounded value function error, bounded anticipation loss, bounded policy sub-optimality (Section 4.2).
    These bounds are assumed to hold for Theorem 2.
  • domain assumption Assumption 3: communicating GMDP and bounded errors including value drift (Section 4.3).
    These are the stochastic-environment assumptions for Theorem 3.
  • ad hoc to paper The regularized loss L_psi has a global minimum of 0 that is achievable and reached by gradient descent (used in Proof of Theorem 1).
    The proof of Theorem 1 declares the loss minimum is 0 and assumes the training process finds it, but for short paths no state satisfies both margin constraints simultaneously.
  • domain assumption The low-level policy reaches each proposed subgoal exactly (used in Proof of Theorem 2).
    Theorem 2's proof requires exact arrival at subgoals, while the algorithm executes a fixed number K of steps which may fall short or overshoot.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks." pith.science (2026). https://pith.science/paper/CX3TY33U

@misc{pith2026250905545,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CX3TY33U}},
  note         = {Machine review of arXiv:2509.05545}
}
read the original abstract

Solving long-horizon goal-conditioned tasks remains a significant challenge in reinforcement learning (RL). Hierarchical reinforcement learning (HRL) addresses this by decomposing tasks into more manageable sub-tasks, but the automatic discovery of the hierarchy and the joint training of multi-level policies often suffer from instability and can lack theoretical guarantees. In this paper, we introduce Reinforcement Learning with Anticipation (RLA), a principled and potentially scalable framework designed to address these limitations. The RLA agent learns two synergistic models: a low-level, goal-conditioned policy that learns to reach specified subgoals, and a high-level anticipation model that functions as a planner, proposing intermediate subgoals on the optimal path to a final goal. The key feature of RLA is the training of the anticipation model, which is guided by a principle of value geometric consistency, regularized to prevent degenerate solutions. We present proofs that RLA approaches the globally optimal policy under various conditions, establishing a principled and convergent method for hierarchical planning and execution in long-horizon goal-conditioned tasks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 24 canonical work pages

  1. [1]

    Hindsight experience replay

    Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. InAdvances in Neural Infor- mation Processing Systems, volume 30, 2017

    work page 2017

  2. [2]

    Mohammad Gheshlaghi Azar, R’emi Munos, and Hilbert J. Kappen. Minimax PAC bounds on the sample complexity of reinforcement learning with a generative model.Machine Learning, 91(3):325–349, 2013

    work page 2013

  3. [3]

    Barto and Sridhar Mahadevan

    Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.Discrete Event Dynamic Systems, 13(4):341–379, 2003

    work page 2003

  4. [4]

    Bertsekas and John N

    Dimitri P. Bertsekas and John N. Tsitsiklis. An analysis of stochastic shortest path problems.Mathematics of Operations Research, 16(3):580–595, 1991

    1991

  5. [5]

    Bertsekas and John N

    Dimitri P. Bertsekas and John N. Tsitsiklis.Neuro-Dynamic Programming. Athena Scientific, 1996

    1996

  6. [6]

    Near-optimal regret bounds for stochastic shortest path

    Alon Cohen, Haim Kaplan, Yishay Mansour, and Aviv Rosenberg. Near-optimal regret bounds for stochastic shortest path. InAdvances in Neural Information Processing Systems, volume 33, 2020

    work page 2020

  7. [7]

    Improving generalization for temporal difference learning: The successor representation.Neural Computation, 5(4):613–624, 1993

    Peter Dayan. Improving generalization for temporal difference learning: The successor representation.Neural Computation, 5(4):613–624, 1993

    work page 1993

  8. [8]

    Dietterich

    Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition.Jour- nal of Artificial Intelligence Research, 13:227–303, 2000

    work page 2000

Show all 28 references
  1. [9]

    Diversity is all you need: Learning skills without a reward function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. InProceedings of the 7th International Conference on Learning Representations, 2019

    2019

  2. [10]

    Contrastive learning as goal- conditioned reinforcement learning

    Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal- conditioned reinforcement learning. InAdvances in Neural Information Processing Systems, volume 35, 2022

    2022

  3. [11]

    Gershman

    Samuel J. Gershman. The successor representation: Its computational logic and neural substrates.Journal of Neuroscience, 38(33):7193–7200, 2018. 13 Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks

    2018

  4. [12]

    Learning to reach goals via iterated supervised learning

    Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088, 2019

    1912 arXiv

  5. [13]

    Dynamical distance learning for semi-parametric control

    Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-parametric control. InProceedings of the 8th International Conference on Learning Representations, 2020

    2020

  6. [14]

    Finite-sample convergence rates for Q-learning

    Michael Kearns and Satinder Singh. Finite-sample convergence rates for Q-learning. InAdvances in Neural Information Processing Systems, volume 11, pages 996–1002, 1999

    1999

  7. [15]

    Learning multi-level hierarchies with hindsight

    Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. InProceedings of the 7th International Conference on Learning Representations, 2019

    2019

  8. [16]

    Lillicrap, Jonathan J

    Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. InProceedings of the 4th International Conference on Learning Representations, 2016

    2016

  9. [17]

    Learning stochastic shortest path with linear function approximation

    Yifei Min, Jiafan He, Tianhao Wang, and Quanquan Gu. Learning stochastic shortest path with linear function approximation. InProceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15608–15639, Baltimo...

    2022

  10. [18]

    Data-efficient hierarchical reinforcement learning

    Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. InAdvances in Neural Information Processing Systems, volume 31, 2018

    2018

  11. [19]

    Hierarchical reinforcement learning: A comprehensive survey.ACM Computing Surveys, 54(5):1–35, 2021

    Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey.ACM Computing Surveys, 54(5):1–35, 2021

    2021

  12. [20]

    Efros, and Trevor Darrell

    Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervised prediction. InProceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2778–2787. PMLR, 2017

    2017

  13. [21]

    Universal value function approximators

    Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. InPro- ceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pages 1312–1320. PMLR, 2015

    2015

  14. [22]

    Toshev, Sergey Levine, and Brian Ichter

    Dhruv Shah, Peng Xu, Yao Lu, Ted Xiao, Alexander T. Toshev, Sergey Levine, and Brian Ichter. Value function spaces: Skill-centric state abstractions for long-horizon reinforcement learning. InProceedings of the 10th International Conference on Learning Representations, 2022

    2022

  15. [23]

    Richard S. Sutton. Learning to predict by the methods of temporal differences.Machine Learning, 3(1):9–44, 1988

    1988

  16. [24]

    Sutton, Doina Precup, and Satinder Singh

    Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for tem- poral abstraction in reinforcement learning.Artificial Intelligence, 112(1–2):181–211, 1999

    1999

  17. [25]

    Morgan & Claypool Publishers, 2010

    Csaba Szepesv’ari.Algorithms for Reinforcement Learning. Morgan & Claypool Publishers, 2010

    2010

  18. [26]

    Value iteration networks

    Aviv Tamar, Yi Wu, Garrett Thomas, Sergey Levine, and Pieter Abbeel. Value iteration networks. InAdvances in Neural Information Processing Systems, volume 29, 2016

    2016

  19. [27]

    Sample complexity bounds for stochas- tic shortest path with a generative model

    Jean Tarbouriech, Matteo Pirotta, Michal Valko, and Alessandro Lazaric. Sample complexity bounds for stochas- tic shortest path with a generative model. InProceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 ofProceedings of Machine Learn...

    2021

  20. [28]

    Christopher J. C. H. Watkins and Peter Dayan. Q-learning.Machine Learning, 8(3):279–292, 1992. 14

    1992

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.