Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the frequency of rewards is not a reliable measure of task difficulty in deep reinforcement learning; the deciding factor is whether mandatory, unrewarded subgoals exist and how soon after each subgoal completion…

desk verdict The paper gives a clean formalization and controlled evidence that reward density is the wrong difficulty metric, but the universal 'no countermeasure' claim runs well ahead of the experiments. read the letter →

arxiv 2507.01470 v1 pith:M6YQAMNW submitted 2025-07-02 cs.LG cs.AI

Yannick Molinghen , Tom Lenaerts This is my paper · ORCID
classification cs.LGcs.AI
keywords zero-incentivedynamicsrewardsparsitysubgoaldiscoverymulti-agentreinforcementlearningshapingstate-spacebottlenecktemporalcreditassignmentlaserenvironment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reward sparsity, usually measured by how many transitions give nonzero reward, is not a reliable measure of how hard a reinforcement learning task is. It formalizes a structural failure mode called Zero-Incentive Dynamics (ZID): a mandatory transition, or bottleneck, that must be crossed to reach the goal but yields no reward itself. Using the Laser Learning Environment, it shows that two state-of-the-art subgoal-oriented multi-agent methods perform no better than a plain value-decomposition baseline when subgoals are unrewarded. It then shows that when the same task is reshaped with potential-based rewards placed a controlled delay d from subgoal completion, learning quality drops steadily as d grows from 0 to 4 even though reward density is identical. If correct, this means algorithm designers should focus on detecting latent subgoal structure and on placing rewards close to subgoal completion, rather than on increasing reward frequency.

What carries the argument

The load-bearing object is the State Space Bottleneck, defined as a minimum directed cut between initial states and goal states in the graph induced by the MDP, with Zero-Incentive Dynamics defined as the condition that every edge in the bottleneck yields at most the base reward. This graph-theoretic definition separates structural difficulty from reward frequency. The paper pairs it with a delayed potential-based reward-shaping scheme, in which a countdown matrix records how many steps ago each agent crossed each laser and feeds a potential function, to manipulate the subgoal-to-reward delay while preserving the optimal policy.

What would settle it

Run MASER and HAVEN on a second ZID environment with a different bottleneck topology and the same hyperparameter budget used for the Laser Learning Environment; if either method outperforms VDN, the claim that subgoal-oriented methods cannot leverage unrewarded subgoals is false. Alternatively, on the same laser map, if the exit rate with delay d = 4 equals the exit rate with d = 0 under identical reward density, the claim that subgoal-to-reward proximity controls learning quality is false.

Watch

Extended reading notes

Core claim

The central claim is that a task's difficulty under deep reinforcement learning is governed by the causal and temporal alignment of rewards with mandatory unrewarded subgoals, not by reward density. Formally, a state space bottleneck is a minimum directed cut from the initial-state set to the goal-state set in the transition graph of the Markov decision process, and the bottleneck has zero-incentive dynamics when every edge in it carries at most the base reward. The paper shows that MASER and HAVEN, two subgoal-oriented methods that identify subgoals through expected-return statistics at individual and collective levels, cannot exploit unrewarded bottlenecks in the Laser Learning Environment; they match rather than beat VDN. In a controlled reward-shaping experiment, keeping reward density constant while delaying the subgoal reward by d = 0, 1, 2, 3, or 4 steps monotonically worsens the learned exit rate, so the proximity of reward to subgoal completion is the operative variable.

Load-bearing premise

The broad conclusion rests on the assumption that the single Laser Learning Environment map is a representative zero-incentive-dynamics task and that the fixed hyperparameters used for HAVEN and MASER are fair; if either is not, the failure of these two methods would not show that subgoal-oriented algorithms in general cannot exploit unrewarded subgoals.

Editorial extensions

If this is right

  • Reward density should not be used to order tasks by difficulty or to choose benchmarks for sparse-reward RL, since increasing density can lower success rate.
  • Subgoal-oriented methods that rely on rewarded or statistically salient subgoal feedback will fail to leverage unrewarded bottlenecks; useful subgoal discovery must operate without immediate incentive.
  • Reward shaping for ZID tasks should place shaped rewards as close as possible to subgoal completion; even a few steps of delay measurably degrades the final policy.
  • ZID extends naturally to delayed incentive dynamics, so environments with human validation latency or network-delayed rewards should expect similar degradation at constant density.
  • Comparisons of RL algorithms should report structural properties such as the presence of unrewarded bottlenecks, not only the fraction of nonzero rewards.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test this, but a direct prediction is that any ZID environment with the same reward density but a different bottleneck structure will reproduce the failure of MASER and HAVEN; testing on one more map would confirm the generality.
  • The paper does not test this, but the delay experiment suggests an untested design rule: if an internal subgoal signal can be detected even a few steps late, it can still support learning, so learned potential functions or count-based subgoal detectors might be combined with reward shaping to mitigate ZID.
  • The paper does not test this, but the graph-based critique implies that single-agent environments with narrow unrewarded corridors should show the same effect, which is testable without multi-agent machinery.
  • The paper does not test this, but there should be a critical delay threshold beyond which shaped rewards are no better than no shaping at all; this threshold could be identified empirically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper challenges the view that reward density alone measures task difficulty in reinforcement learning. It proposes a graph-theoretic formalization of Zero-Incentive Dynamics (ZID) as unrewarded minimum directed S0-SG cut-sets in the transition graph of an MDP, reports a random-exploration counterexample in which removing unrewarded edges increases reward density while lowering success rate, evaluates two subgoal-oriented methods (HAVEN and MASER) against VDN/QMIX/QPLEX on a single Laser Learning Environment (LLE) map, and studies delayed potential-based reward shaping to argue that temporal proximity of reward to subgoal completion matters more than reward density. The conclusion is that ZID remains an unsolved structural challenge and that no current counter-measure exists.

Significance. If the results hold, the paper makes a useful conceptual contribution by separating reward density from reward structure, and it provides a formal definition that could anchor future work on unrewarded subgoals. The random-exploration counterexample and the reward-delay experiments are falsifiable and clearly presented. However, the breadth of the conclusions exceeds the evidence: the subgoal-method evaluation uses one self-designed map with fixed hyperparameters, there is no transfer to other ZID tasks, and the delayed-PBRS mechanism as written appears to have a technical flaw. The significance is therefore conditional on correcting these points and narrowing the claims.

major comments (4)
  1. [Section 4.2, Figure 3, Appendix B, Conclusion] The universal conclusions 'there exist no counter-measure to this day' and 'the inability of these methods to identify subgoals' are not supported by experiments on a single LLE map with one fixed hyperparameter setting (HAVEN k=3, MASER lambda=0.03, 4M steps). The paper provides no hyperparameter sensitivity analysis and no second ZID environment, so the negative result may be specific to this map or to these hyperparameters. The authors should either add transfer experiments and sensitivity analyses or explicitly restrict all such claims to the tested setup.
  2. [Appendix C, Eq. (2), Figure 4] As written, the delayed PBRS potential cannot produce a reward at the moment a laser is crossed: C is initialized to -1, the indicator condition is C_i,l <= d, and an uncrossed laser therefore satisfies this condition for every d >= 0. The potential is unchanged when C goes from -1 to 0 upon crossing, so the claimed timing of the shaped reward is not implemented by Eq. (2). Moreover, the shaped runs append the countdown features C_i to the observations, so the comparison with the 'No shaping' condition changes the observation space in addition to the reward timing. The authors should correct the potential function or provide the exact update rule, and they should clarify how the countdown features affect the conclusion that temporal proximity alone drives the improvement.
  3. [Definition 3 and Definition 4] The definition of ZID depends on 'a bottleneck', but a directed S0-SG graph can have multiple minimum cut-sets, and some of these cut-sets may contain positively rewarded edges while others do not. Without a canonical rule for selecting among minimum cuts, and without specifying whether 'minimum' means minimum cardinality or minimum total edge weight, the property 'B has ZID' is not well-defined and the same MDP could be classified differently by different readers.
  4. [Section 4.2, Figure 3 and Figure 5] The inference from 'neither HAVEN nor MASER are able to outperform VDN in this setup' to 'the inability of these methods to identify subgoals' is too strong. A failure to outperform VDN on one map with fixed hyperparameters can stem from exploration efficiency, credit assignment, or hyperparameter sensitivity rather than from an inability to detect subgoals. The Figure 5 control with PBRS does not isolate subgoal-identification ability because it also adds countdown observations and changes the reward structure. The authors should either provide direct evidence about which subgoals the methods identify or soften the claim to a performance statement.
minor comments (5)
  1. [Section 3.2, Figure 2] The sentence 'the exit rate decreases when the reward sparsity increases' contradicts the preceding calculation, since D_Mn = 3/(101-n) increases with n; the text should say 'when the reward density increases' or 'when the reward sparsity decreases'.
  2. [Appendix C, Eq. (2)] The meaning of C_i,l is ambiguous: the text says it indicates how many steps have passed since crossing, but the initial value -1 is also used in the indicator condition C_i,l <= d. Please define the update rule for C_i,l at crossing and at each subsequent step, and state how episode truncation flushes pending rewards.
  3. [Appendix B] The paper should state whether the hyperparameters for HAVEN and MASER were tuned on the LLE map or taken directly from the original papers; fixed values from StarCraft benchmarks may disadvantage these methods on a different environment.
  4. [General] There are several typographical errors, including 'it it important' in Section 2.2 and 'the exponential growth the the state space' in Appendix E; the paper would also benefit from consistent typesetting of 'HAVEN' versus 'HA VEN'.
  5. [Reproducibility] No code or data release is mentioned; releasing the LLE environment and the training harness would substantially strengthen the reproducibility of the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the formal ZID definition and the empirical benchmarks are independent, and the paper's universal negative about countermeasures is an overgeneralization rather than a definitional or fitted reduction.

full rationale

The paper's derivation chain is self-contained. ZID is formally defined in Definition 4 as a minimum directed S0-SG cut-set whose edges all have weight at most the base reward; this definition does not presuppose the empirical conclusions drawn later. The Section 3.2 counterexample to reward density as a difficulty metric is constructed by disabling unrewarded edges and measuring random-exploration exit rates; the densities are computed directly from the graph, not fitted, and the result is a controlled demonstration rather than a prediction derived from a fitted parameter. Section 4.2 evaluates MASER and HAVEN with fixed hyperparameters reported in Appendix B, and the statement that these methods 'are unable to outperform VDN in this setup' is an empirical observation. The later universal claim that 'there exist no counter-measure to this day' is broader than the single-map, fixed-hyperparameter evidence supports, but overgeneralization is a correctness or scope concern, not circularity: no parameter was fitted to force the outcome, and the authors explicitly note that their prior LLE work (Molinghen et al., 2025) 'provided no evidence' for the ZID hypothesis, so the experiments here are new evidence rather than a self-citation chain. Section 4.3 compares PBRS with varying delay d while holding reward density constant via flushed pending rewards, and the observation that smaller d learns faster is an experimental result whose input (the potential function) does not by construction equal the output (relative exit rates). The PBRS control also augments observations with countdown features, which could confound the subgoal-discovery interpretation, but that is an experimental design limitation, not circularity. No step in the paper reduces an equation or a fitted value to its own inputs, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The formal results rest on the graph abstraction of MDPs and the cut-set definition of bottlenecks; the empirical claims rest on the representativeness of LLE and standard PBRS assumptions. No free parameters are fitted, and no new physical or mechanistic entities are introduced.

assumptions (4)
  • domain assumption The transition graph G_M=(V,E,W) of a deterministic MDP with edge weights equal to rewards fully captures the structure needed to define task difficulty.
    Used in Section 2.1 and Definitions 1-4 to reduce the MDP to a directed weighted graph; stochastic transitions are not treated in the formalization.
  • ad hoc to paper A minimum directed S0-SG cut-set corresponds to the intuitive notion of a mandatory subgoal or bottleneck.
    Definition 3 identifies State Space Bottlenecks with minimum cuts without proof that min cuts match human-identified subtasks in general MDPs.
  • domain assumption Performance on the single LLE map is representative of ZID tasks in general.
    Section 4.2 draws a general conclusion about subgoal-oriented methods from one environment with one hyperparameter configuration.
  • standard math PBRS (Ng et al., 1999) preserves the optimal policy for the delayed-reward shaping used in Section 4.3.
    Invoked in Appendix C; the paper extends the potential to countdown states, which is a valid state-dependent potential if the countdowns are part of the state.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals." pith.science (2026). https://pith.science/paper/M6YQAMNW

@misc{pith2026250701470,
  author       = {Pith},
  title        = {Pith review of: Zero-Incentive Dynamics: a look at reward sparsity through the lens of unrewarded subgoals},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M6YQAMNW}},
  note         = {Machine review of arXiv:2507.01470}
}
read the original abstract

This work re-examines the commonly held assumption that the frequency of rewards is a reliable measure of task difficulty in reinforcement learning. We identify and formalize a structural challenge that undermines the effectiveness of current policy learning methods: when essential subgoals do not directly yield rewards. We characterize such settings as exhibiting zero-incentive dynamics, where transitions critical to success remain unrewarded. We show that state-of-the-art deep subgoal-based algorithms fail to leverage these dynamics and that learning performance is highly sensitive to the temporal proximity between subgoal completion and eventual reward. These findings reveal a fundamental limitation in current approaches and point to the need for mechanisms that can infer latent task structure without relying on immediate incentives.

Figures

Figures reproduced from arXiv: 2507.01470 by the authors.

Figure 1
Figure 1. Laser Learning Environment. This map has four agents and three lasers that can be [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. (Left) Average exit rate over with random exploration for 200k steps for different time [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average exit rate over the course of the training. Results are shown with 95% confidence [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (Left) Exit rate over training time step for shaped reward delay [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Exit rate over the course of the training for subgoal-oriented methods when PBRS is [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: (a) Map with a State Space Bottleneck. (b), (c) and (d) respectively show the bottleneck [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Average execution time (in seconds, logarithmic scale) to identify bottlenecks in the map [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 19 canonical work pages

  1. [1]

    Feudal multi-agent hierarchies for cooperative reinforcement learning, 2019

    Sanjeevan Ahilan and Peter Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning, 2019. URL http://arxiv.org/abs/1901.08492

    arXiv 2019

  2. [2]

    Concrete Problems in AI Safety , July 2016

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete Problems in AI Safety , July 2016. URL http://arxiv.org/abs/1606.06565. arXiv:1606.06565 [cs]

    arXiv 2016

  3. [3]

    Hindsight experience replay

    Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew , Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems, Advances in Neural Information Processing Systems 30. Neural Information Processin...

    work page 2017

  4. [4]

    Dynamic programming

    Richard Bellman. Dynamic programming. Princeton Univ. Pr, 1957. ISBN 978-0-691-07951-6

    work page 1957

  5. [5]

    Exploration by Random Network Distillation , October 2018

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by Random Network Distillation , October 2018. URL http://arxiv.org/abs/1810.12894. arXiv:1810.12894 [cs, stat]

    arXiv 2018

  6. [6]

    Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3 0 (4): 0 128--135, 1999. ISSN 1364-6613. doi:https://doi.org/10.1016/S1364-6613(99)01294-2. URL https://www.sciencedirect.com/science/article/pii/S1364661399012942

    doi:10.1016/s1364-6613(99)01294-2 1999

  7. [7]

    HiSOMA : A hierarchical multi-agent model integrating self-organizing neural networks with multi-agent deep reinforcement learning

    Minghong Geng, Shubham Pateria, Budhitama Subagdja, and Ah-Hwee Tan. HiSOMA : A hierarchical multi-agent model integrating self-organizing neural networks with multi-agent deep reinforcement learning. Expert Systems with Applications, 252: 0 124117, 2024. ISSN 0957-4174. doi:10.1016/j.eswa.2024.124117. URL https://www.sciencedirect.com/science/article/pii...

    work page arXiv 2024

  8. [8]

    MASER : Multi-agent reinforcement learning with subgoals generated from experience replay buffer

    Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. MASER : Multi-agent reinforcement learning with subgoals generated from experience replay buffer. In Proceedings of the 39th International Conference on Machine Learning, 2022

    work page 2022

Show all 33 references
  1. [9]

    Automatic discovery of subgoals in reinforcement learning using strongly connected components

    Seyed Jalal Kazemitabar and Hamid Beigy. Automatic discovery of subgoals in reinforcement learning using strongly connected components. In Mario K \"o ppen, Nikola Kasabov, and George Coghill (eds.), Advances in Neuro-Information Processing, pp.\ 829--834, Berlin, Heidelberg, ...

    2009

  2. [10]

    Exploration in deep reinforcement learning: A survey

    Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion, 85: 0 1--22, 2022. ISSN 1566-2535. doi:https://doi.org/10.1016/j.inffus.2022.03.003. URL https://www.sciencedirect.com/science/article/pii/S1566253...

    2022 doi

  3. [11]

    Lecun, L

    Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998. doi:10.1109/5.726791

    1998 doi

  4. [12]

    Automatic discovery of subgoals in reinforcement learning using diverse density

    Amy McGovern and Andrew G Barto. Automatic discovery of subgoals in reinforcement learning using diverse density. In Proceedings of the Eighteenth International Conference on Machine Learning, pp.\ 361--368, 2001

    2001

  5. [13]

    Research on Multi -agent Sparse Reward Problem

    Fanxiao Meng. Research on Multi -agent Sparse Reward Problem . Highlights in Science, Engineering and Technology, 85: 0 96--103, March 2024. ISSN 2791-0210. doi:10.54097/er0mx710. URL https://drpress.org/ojs/index.php/HSET/article/view/18307

    2024 doi

  6. [14]

    Laser learning environment: A new environment for coordination-critical multi-agent tasks

    Yannick Molinghen, Rapha \"e l Avalos, Mark Van Achter, Ann Now \'e , and Tom Lenaerts. Laser learning environment: A new environment for coordination-critical multi-agent tasks. In Frans A. Oliehoek, Manon Kok, and Sicco Verwer (eds.), Artificial Intelligence and Machine Lear...

    2025

  7. [15]

    An overview of environmental features that impact deep reinforcement learning in sparse-reward domains

    Jim Martin Catacora Ocana, Roberto Capobianco, and Daniele Nardi. An overview of environmental features that impact deep reinforcement learning in sparse-reward domains. Journal of Artificial Intelligence Research, 76: 0 1181--1218, 2023. ISSN 1076-9757. doi:10.1613/jair.1.143...

    2023 doi

  8. [16]

    Efros, and Trevor Darrell

    Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pp.\ 2778–2787. JMLR.org, 2017

    2017

  9. [17]

    Learning to Drive a Bicycle using Reinforcement Learning and Shaping

    Jette Randløv and Preben Alstrøm. Learning to Drive a Bicycle using Reinforcement Learning and Shaping . In Proceedings of the Fifteenth International Conference on Machine Learning , pp.\ 463--471, San Francisco, 1998. Morgan Kaufmann Publishers Inc

    1998

  10. [18]

    QMIX : Monotonic Value Function Factorisation for Deep Multi - Agent Reinforcement Learning

    Tabish Rashid, Mikayel Samvelyan, and Christian Schroeder. QMIX : Monotonic Value Function Factorisation for Deep Multi - Agent Reinforcement Learning . In Proceedings of Machine Learning Research , 2018. arXiv: 1803.11485v2

    2018 arXiv

  11. [19]

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi - Agent Challenge , December 2019. URL http://arxiv.org/abs/1902.04043...

    2019 arXiv

  12. [20]

    Normalized cuts and image segmentation

    Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE , 22 0 (8): 0 888--905, 2000

    2000

  13. [21]

    Wolfe, and Andrew G

    \"Ozg\"ur S im s ek, Alicia P. Wolfe, and Andrew G. Barto. Identifying useful subgoals in reinforcement learning by local graph partitioning. In Proceedings of the 22nd international conference on Machine learning - ICML '05 , pp.\ 816--823. ACM Press, 2005. ISBN 978-1-59593-1...

    2005

  14. [22]

    Leibo, Karl Tuyls, and Thore Graepel

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning based on team reward. Procee...

    2018

  15. [23]

    Faster MIL -based subgoal identification for reinforcement learning by tuning fewer hyperparameters

    Saim Sunel, Erkin Çilden, and Faruk Polat. Faster MIL -based subgoal identification for reinforcement learning by tuning fewer hyperparameters. ACM Transactions on Autonomous and Adaptive Systems , 19 0 (2): 0 1--29, 2024. ISSN 1556-4665, 1556-4703. doi:10.1145/3643852. URL ht...

    2024 doi

  16. [24]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-03924-6

    2018

  17. [25]

    Sutton, Doina Precup, and Satinder Singh

    Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi- MDPs : A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112 0 (1): 0 181--211, 1999. ISSN 00043702. doi:10.1016/S0004-3702(99)00052-1. URL https://linkinghub.els...

    1999 doi

  18. [26]

    \#exploration: A study of count-based exploration for deep reinforcement learning

    Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck , and Pieter Abbeel. \#exploration: A study of count-based exploration for deep reinforcement learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, ...

    2017

  19. [27]

    Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards

    Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Inform...

    2019

  20. [28]

    Deep reinforcement learning with double Q - Learning

    Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q - Learning . 30th AAAI Conference on Artificial Intelligence, AAAI 2016, pp.\ 2094--2100, 2016. arXiv: 1509.06461 ISBN: 9781577357605

    2016 arXiv

  21. [29]

    QPLEX : Duplex Dueling Multi - Agent Q - Learning , October 2021

    Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX : Duplex Dueling Multi - Agent Q - Learning , October 2021. URL http://arxiv.org/abs/2008.01062. arXiv:2008.01062 [cs, stat]

    2021 arXiv

  22. [30]

    Robin J. Wilson. Introduction to graph theory. Prentice Hall, Harlow Munich, 4. ed., [nachdr.] edition, 2009. ISBN 978-0-582-24993-6

    2009

  23. [31]

    HAVEN : Hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism

    Zhiwei Xu, Yunpeng Bai, Bin Zhang, Dapeng Li, and Guoliang Fan. HAVEN : Hierarchical cooperative multi-agent reinforcement learning with dual coordination mechanism. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pp.\ 11735--11743, 2023. doi:10.1...

    2023 doi

  24. [32]

    Ng, Daishi Harada, and Stuart Russell

    Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy Invariance Under Reward Transformations : Theory and Application to Reward Shaping . In Proceedings of the 16th International Conference on Machine Learning , pp.\ 278--287, San Francisco, 1999. Morgan Kaufmann Publishers...

    1999

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.