Pith. sign in

REVIEW 3 major objections 4 minor 39 references

Meta-learning how to Share Credit among Macro-Actions

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding macro-actions usually hurts an RL agent; the paper claims a meta-learned similarity penalty that shares credit among related actions reverses that and improves scores.

desk verdict Plausible incremental idea with broad empirical support, but the transfer table has a likely copy-paste error and hyperparameter reporting is thin; deserves review after fixes. read the letter →

arxiv 2506.13690 v1 pith:SBL3347K submitted 2025-06-16 cs.LG cs.AI

Ionel-Alexandru Hosu , Traian Rebedea , Razvan Pascanu This is my paper
classification cs.LGcs.AI
keywords reinforcementlearningmacro-actionscreditassignmentmeta-learningexplorationaction-spacesimilaritytransferRainbow-DQN
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why naively adding macro-actions to a reinforcement learning agent often backfires, and what to do about it. The authors argue the problem is that each macro-action is treated as atomic and independent, so the action space grows faster than the number of decisions shrinks. They propose a regularization term, MASP, that pulls the Q-values of similar actions together using a similarity matrix learned from data by meta-gradients. They report that this credit-sharing penalty turns macro-actions into a consistent improvement over Rainbow-DQN across Atari games, StreetFighter II, and MiniGrid, and that the learned similarity transfers to related games.

What carries the argument

The central object is the Macro-Action Similarity Penalty (MASP), an additive term to the TD loss that penalizes divergence between the Q-values of actions deemed similar by a learned matrix $\Sigma$: $\mathcal{L}_{\text{MASP}} = \eta\,\|\mathbf{Q}(s,\cdot;\theta)-\Sigma\,\mathbf{Q}(s,\cdot;\theta)\|_2^2$. The matrix is symmetric, non-negative, and meta-learned through a two-phase loop: an inner RL update with $\Sigma$ fixed, then an outer meta-gradient step that evaluates the updated policy on held-out data and adjusts $\Sigma$; the trace-based approximation makes the outer gradient tractable. A low-dimensional embedding of $\Sigma$ is also fed into the Q-network to help it adapt to the evolving similarity geometry.

What would settle it

Train MASP in a two-room gridworld where two macro-actions have identical effects in the first room but opposite effects in the second. If the single learned $\Sigma$ still forces their Q-values together and the agent underperforms Rainbow-DQN, the state-independence of $\Sigma$ is the culprit; a state-conditioned $\Sigma$ that separates the pair in the second room should recover the loss.

Watch

Extended reading notes

Core claim

The core claim is that the structure of the action space, not the macro-actions themselves, determines whether macro-actions help. Treating every action as independent inflates the branching factor and makes exploration harder; MASP instead encodes the insight that many actions and macro-actions lead to similar outcomes on average. The penalty $\eta\,\|\mathbf{Q}(s,\cdot;\theta)-\Sigma\,\mathbf{Q}(s,\cdot;\theta)\|_2^2$ is added to the TD loss, where $\Sigma$ is a symmetric non-negative similarity matrix meta-learned jointly with the policy. On the paper's results, this single addition lifts Rainbow-DQN's Atari scores substantially (for example Breakout from 379.5 to 884.4), keeps performance stable as the macro-action set grows from 64 to 1024 entries, tolerates heavy noise in the macro-action set, and transfers across similar games.

Load-bearing premise

The load-bearing premise is that one fixed, state-independent similarity matrix can capture the useful structure of the action space, so that pulling similar actions' Q-values together helps more than it hurts.

Editorial extensions

If this is right

  • Macro-action augmentation becomes viable at large scale: MASP holds up when the macro-action set grows to 1024 entries, a regime where the naive macro-action baseline collapses.
  • Imperfect macro-action sets are tolerable: replacing up to 75% of macro-actions with random sequences degrades MASP far less than it degrades the baseline, suggesting the learned similarity can cluster and ignore useless macros.
  • The learned similarity matrix is a reusable object: freezing $\Sigma$ from one game and training a fresh policy on a related game retains much of the benefit, which could reduce retraining cost in practice.
  • MASP reaches human-normalized scores above 100 on many Atari games where Rainbow-DQN does not, so the improvement is qualitative, not just a small margin.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the similarity-geometry explanation is correct, the same regularizer should help in other structured action spaces, such as combinatorial action heads in language models or robot control, once the quadratic cost of a full $\Sigma$ is replaced by a factored approximation.
  • The fixed, state-independent $\Sigma$ is the most obvious ceiling: a state-conditioned variant $\Sigma(s)$ could capture action semantics that change as the game progresses, and comparing the two would directly test the paper's central premise.
  • The learned $\Sigma$ could be used to seed a hard action abstraction, e.g., grouping the most similar macro-actions into options, combining soft credit sharing with the decision-count reduction that macro-actions are supposed to provide.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes MASP, a regularization term that penalizes ||Q(s,·) - Σ Q(s,·)||² with a meta-learned similarity matrix Σ, in order to share credit among macro-actions and primitive actions. The method is evaluated on top of Rainbow-DQN on 57 Atari games, StreetFighter II, and MiniGrid, reporting that MASP improves over Rainbow-DQN and over Rainbow-DQN with naive macro-actions, remains robust as the macro-action set grows and under noisy macro-actions, and transfers across related games. The authors also provide ablations, hyperparameter details, and a code URL.

Significance. If the reported results are correct, MASP is a simple and potentially useful plug-in regularizer for value-based RL with structured action spaces. It directly addresses a known failure mode of naive macro-action augmentation and offers a mechanism for learning action similarity from data. The manuscript includes broad empirical coverage, a full Atari table, ablations over macro-action count and noise, and an explicit code release. However, the formulation of the penalty and the transfer table currently contain issues that must be resolved before the empirical claims can be relied upon.

major comments (3)
  1. [3.3, Eq. (2)] The penalty as written does not implement the stated objective unless Σ is row-stochastic. The paper only constrains Σ to be symmetric and non-negative (Appendix A.1) and clips entries to [0,1] (Appendix A.2); it does not require row sums to equal 1. For a constant Q-vector c·1, the penalty equals η Σ_i c²(1 - s_i)², where s_i = Σ_j Σ_ij; if any row sum differs from 1, the minimum is at c=0, so the regularizer shrinks the scale of all Q-values rather than merely pulling similar actions together. In addition, Section 3.4's statement that setting 'Σij = 1 for all i and j' makes MASP have no effect is incorrect; the identity matrix is the no-effect case. Please row-normalize Σ (or use a graph-Laplacian penalty) and correct the corresponding text and experiments.
  2. [4, Table 5] In the Breakout row, the cell for Σ trained on Private Eye is identical to the diagonal cell for Σ trained on Breakout, both reporting 884.4 ± 74.0. Since these are independent experimental conditions, exact equality of the mean and standard error is statistically implausible. This suggests either a transcription error or an evaluation that is insensitive to which Σ is transferred; under the latter reading, the meta-learned similarity is not the active ingredient. Because transferability is claimed in the abstract, this must be corrected or explicitly explained before the transfer sub-claim can be accepted.
  3. [4, 'Atari Macro-Actions' and Table 2] The main results in Table 8 use a macro-action set of size k=32, but the ablation in Table 2 sweeps k ∈ {64, 128, 256, 512, 1024} and omits k=32. Consequently, the claimed robustness to the number of macro-actions is not demonstrated for the configuration used in the headline results, and the main result cannot be directly compared with the ablation. Please add k=32 to the sweep or explain why it is omitted.
minor comments (4)
  1. [4, Table 4] The Breakout row for P(replace)=0.25 reports '208.3.3 ± 22.6', which appears to contain a typographical error ('208.3' followed by an extra decimal point). Please verify the value.
  2. [Appendix C, Table 8] The Fishing Derby MASP score is reported as '36.9.6 ± 1.0', which contains an extra decimal point. Please correct the formatting and double-check surrounding entries for similar typos.
  3. [4, Transfer Learning and Generalization] The phrase 'remains close to or above the in-domain baseline' is ambiguous because 'in-domain baseline' could mean the Rainbow-DQN score or the diagonal MASP score. Clarify which reference is intended, since the two differ substantially in several rows.
  4. [Appendix C.3, Table 10] The MiniGrid results are reported without standard errors or the number of seeds. Given that these results support the broad claim of improvement, please add at least the number of seeds and ideally error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: MASP's similarity matrix is meta-learned against an external TD objective and its reported gains are measured by external rewards, not by construction.

full rationale

The paper's derivation chain is not circular. MASP is an additive regularizer added to the TD loss (Eq. 2), and the similarity matrix Sigma is meta-learned by differentiating a held-out TD objective through a one-step inner update (Section 3.4, Algorithm 1). The meta-objective is the standard TD error on a separate trajectory, and all headline results are external episode rewards in Atari, StreetFighter II, and MiniGrid; no fitted parameter is renamed as a prediction, and no target result is defined in terms of Sigma. The method does not rely on author self-citations: the meta-gradient framework is credited to Xu et al. and the UVFA conditioning to Schaul et al., both external prior work, and no uniqueness theorem is imported. Two correctness caveats are worth noting but do not constitute circularity: (1) the claim that the meta-objective depends on Sigma only through theta' is inaccurate, because the Q-network is also conditioned on eSigma = W_emb vec(Sigma) (Section 3.4, Algorithm 1 step 8), so a direct gradient path exists; and (2) Table 5 contains an internally implausible exact duplication of the Breakout score under two different source-Sigma conditions, and the surrounding text's qualitative transfer claim is not fully supported by the table. These are reproducibility/data-integrity concerns, not reductions of the claimed result to its inputs. The main derivation remains self-contained and empirically evaluated against external benchmarks.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The paper introduces a similarity matrix Σ as a new learned quantity, but it is a parameter of the method, not a hypothesized entity with independent evidence. The main free parameters are the regularization weight η, the meta-learning rate β, and the macro-action set size, all tuned per environment. The state-independence of Σ is a key assumption. The entropy penalty is an ad hoc stabilizer.

free parameters (4)
  • Regularization coefficient η = Swept over {0.1, 0.3, 0.5, 0.7, 1} for Atari, {0.3, 0.5} for StreetFighter, {0.05, 0.1, 0.3} for MiniGrid; reported…
    Controls the strength of the MASP penalty. The paper states it is 'swept' but does not report the chosen value per game, so the best results are selected from a hyperparameter search. This is a free parameter fitted to validation performance.
  • Meta-learning rate β = Swept over {0.001, 0.005, 0.01} for Atari, {0.001, 0.003} for StreetFighter; final per-game values not reported.
    Controls the update rate of the similarity matrix. Tuned per environment, contributing to the selection bias in reported results.
  • Macro-action set size k = k=32 for main Atari results; swept over {64, 128, 256, 512, 1024} in ablations.
    The number of macro-actions is chosen by the experimenter, and the main results use k=32 while the ablation starts at k=64. The choice of k is not grounded in a method, and the main result is not compared to the same k in the ablation.
  • Σ embedding dimension = 32 for Atari, 16 for StreetFighter, 8 for MiniGrid.
    A hyperparameter for the learned projection Wemb. Not justified beyond architecture choice.
assumptions (4)
  • domain assumption State-independent similarity matrix can capture action relationships useful for credit assignment.
    Stated in the Limitations section: 'Our choice of parametrization makes Σ independent of state, and of current policy πθ'. If action similarities vary by state, a fixed Σ may be too rigid.
  • domain assumption The meta-gradient trace-based approximation (Xu et al. 2018) remains valid when combined with Rainbow-DQN's distributional and prioritized components.
    The paper uses the meta-gradient framework of Xu et al. but does not analyze its interaction with Rainbow's components. The appendix says 'we use automatic differentiation and checkpointing', but no stability analysis is provided.
  • domain assumption The frequency-based heuristic on human trajectories yields macro-actions that preserve the MDP structure when appended to the action space.
    Macro-actions are extracted as frequent subsequences from the Atari Grand Challenge Dataset. If these subsequences are not truly useful or if their execution as atomic actions changes environment dynamics, the method may be biased. The paper tests robustness to noise but does not test the quality of the extracted macro-actions.
  • ad hoc to paper The entropy penalty on row-normalized Σ prevents degeneration without distorting the similarity structure.
    Mentioned in Appendix A.3: 'To prevent Σ from degenerating to the identity or to a rank-one matrix, we add a small entropy penalty to its row-normalized version during meta-learning.' No ablation shows the effect of this penalty.
invented entities (1)
  • Similarity matrix Σ (as a learned, state-independent parameter)
    purpose: Encodes an inductive bias that Q-values of similar actions should be close, enabling credit sharing among macro-actions.
    Σ is a learned construct with no external falsifiable handle. Its values are not independently measurable; it is a free matrix optimized on the same data as the policy. The paper shows transferability, but that is a post-hoc empirical property, not a designed prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Meta-learning how to Share Credit among Macro-Actions." pith.science (2026). https://pith.science/paper/SBL3347K

@misc{pith2026250613690,
  author       = {Pith},
  title        = {Pith review of: Meta-learning how to Share Credit among Macro-Actions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBL3347K}},
  note         = {Machine review of arXiv:2506.13690}
}
read the original abstract

One proposed mechanism to improve exploration in reinforcement learning is through the use of macro-actions. Paradoxically though, in many scenarios the naive addition of macro-actions does not lead to better exploration, but rather the opposite. It has been argued that this was caused by adding non-useful macros and multiple works have focused on mechanisms to discover effectively environment-specific useful macros. In this work, we take a slightly different perspective. We argue that the difficulty stems from the trade-offs between reducing the average number of decisions per episode versus increasing the size of the action space. Namely, one typically treats each potential macro-action as independent and atomic, hence strictly increasing the search space and making typical exploration strategies inefficient. To address this problem we propose a novel regularization term that exploits the relationship between actions and macro-actions to improve the credit assignment mechanism by reducing the effective dimension of the action space and, therefore, improving exploration. The term relies on a similarity matrix that is meta-learned jointly with learning the desired policy. We empirically validate our strategy looking at macro-actions in Atari games, and the StreetFighter II environment. Our results show significant improvements over the Rainbow-DQN baseline in all environments. Additionally, we show that the macro-action similarity is transferable to related environments. We believe this work is a small but important step towards understanding how the similarity-imposed geometry on the action space can be exploited to improve credit assignment and exploration, therefore making learning more effective.

Figures

Figures reproduced from arXiv: 2506.13690 by the authors.

Figure 1
Figure 1. Visual comparison between Rainbow-DQN, Rainbow-DQN + Macro-Actions and Macro￾Action Similarity Penalty . However it is often that even when one has access to a good proposal distribution for macro-actions, using them leads to worse performance. This is due to the fact that even if the average length of an episode might decrease, in many cases the exploration space actually becomes larger, as the number of available … view at source ↗
Figure 2
Figure 2. Visual comparison between Rainbow-DQN (dark blue), Rainbow-DQN + Macro-Actions (light blue), and Macro-Action Similarity Penalty (red). Learning the Similarity Embedding eΣ. A common and effective approach for representing the similarity matrix Σ is to learn a low-dimensional embedding eΣ jointly with the main network parameters. Instead of using the full (potentially large) matrix Σ directly as network input, we fi… view at source ↗
Figure 3
Figure 3. Sample Σ matrix from Street Fighter II experiments, illustrating the learned similarities between different macro-actions. Distinct clusters with higher values indicate groups of macro-actions that are functionally related or often co-activated. In contrast, the regions of the matrix with the lowest values and lacking visible structure correspond to primitive actions, which are entirely independent and dissimilar to… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 25 canonical work pages

  1. [1]

    Mas- tering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484–489, 2016

    2016

  2. [2]

    Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H

    Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dali...

    2019

  3. [3]

    Dota 2 with large scale deep reinforcement learning

    Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dkebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019

    arXiv 1912

  4. [4]

    Autonomous navigation of stratospheric balloons using reinforcement learning

    Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77–82, 2020

    work page 2020

  5. [5]

    Magnetic control of tokamak plasmas through deep reinforcement learning

    Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414–419, 2022

    work page 2022

  6. [6]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- bac...

    work page 2022

  7. [7]

    Hierarchical solution of markov decision processes using macro-actions

    Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas Dean, and Craig Boutilier. Hierarchical solution of markov decision processes using macro-actions. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, UAI’98, page 220–229, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X

    work page 1998

  8. [8]

    Fikes and Nils J

    Richard E. Fikes and Nils J. Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial Intelligence, 2(3):189–208, 1971. ISSN 0004-3702. doi: https://doi.org/10.1016/0004-3702(71)90010-5. URL https://www.sciencedirect.com/ science/article/pii/0004370271900105

    arXiv 1971

Show all 39 references
  1. [9]

    M. A. Hakim Newton, John Levine, Maria Fox, and Derek Long. Learning macro-actions for arbitrary planners and domains. In Proceedings of the Seventeenth International Conference on International Conference on Automated Planning and Scheduling, ICAPS’07, page 256–263. AAAI Pres...

    2007

  2. [10]

    Durugkar, Clemens Rosenbaum, Stefan Dernbach, and Sridhar Mahadevan

    Ishan P. Durugkar, Clemens Rosenbaum, Stefan Dernbach, and Sridhar Mahadevan. Deep reinforcement learning with macro-actions, 2016. URL https://arxiv.org/abs/1606. 04615

    2016

  3. [11]

    Rainbow: Combining improve- ments in deep reinforcement learning

    Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, ...

    2018

  4. [12]

    Learning macro-actions in reinforcement learning

    Jette Randlov. Learning macro-actions in reinforcement learning. Advances in Neural Informa- tion Processing Systems, 11, 1998. 11

    1998

  5. [13]

    Macro-actions in reinforcement learning: An empirical analysis

    Amy McGovern and Richard S Sutton. Macro-actions in reinforcement learning: An empirical analysis. 1998

    1998

  6. [14]

    Meta learning shared hierarchies

    Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations (ICLR), 2018

    2018

  7. [15]

    Hierarchical meta-reinforcement learning via automated macro-action discovery

    Minjae Cho and Chuangchuang Sun. Hierarchical meta-reinforcement learning via automated macro-action discovery. arXiv preprint arXiv:2412.11930, 2024

    2024 arXiv

  8. [16]

    Deep reinforcement learning for decentralized multi-robot exploration with macro actions

    Aaron Hao Tan, Federico Pizarro Bejarano, Yuhan Zhu, Richard Ren, and Goldie Nejat. Deep reinforcement learning for decentralized multi-robot exploration with macro actions. IEEE Robotics and Automation Letters, 8(1):272–279, 2022

    2022

  9. [17]

    Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability

    Yuchen Xiao. Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability. PhD thesis, Northeastern University, 2022

    2022

  10. [18]

    Unlocking new strategies: Intrinsic exploration for evolving macro and micro actions

    Sourav Panda, Aviral Srivastava, and Jonathan Dodge. Unlocking new strategies: Intrinsic exploration for evolving macro and micro actions. In Intrinsically-Motivated and Open-Ended Learning Workshop@ NeurIPS2024, 2024

    2024

  11. [19]

    Macro actions: Learning to act efficiently through reusable temporally extended actions

    Keng-Yu Chang, Hung-Yi Lee, and Chun-Yi Chen. Macro actions: Learning to act efficiently through reusable temporally extended actions. arXiv preprint arXiv:1908.01478, 2019

    1908 arXiv

  12. [20]

    Efficient black-box planning using macro-actions with focused effects

    Cameron Allen, Michael Katz, Tim Klinger, George Konidaris, Matthew Riemer, and Gerald Tesauro. Efficient black-box planning using macro-actions with focused effects. arXiv preprint arXiv:2004.13242, 2020

    2004 arXiv

  13. [21]

    Learning macro-actions for arbitrary planners and domains

    Muhammad Abdul Hakim Newton, John Levine, Maria Fox, and Derek Long. Learning macro-actions for arbitrary planners and domains. In ICAPS, volume 2007, pages 256–263, 2007

    2007

  14. [22]

    Modeling and planning with macro-actions in decentralized pomdps

    Christopher Amato, George Konidaris, Leslie P Kaelbling, and Jonathan P How. Modeling and planning with macro-actions in decentralized pomdps. Journal of Artificial Intelligence Research, 64:817–859, 2019

    2019

  15. [23]

    Magic: Learning macro-actions for online pomdp planning

    Yiyuan Lee, Panpan Cai, and David Hsu. Magic: Learning macro-actions for online pomdp planning. arXiv preprint arXiv:2011.03813, 2020

    2011 arXiv

  16. [24]

    Deep reinforcement learning based navigation with macro actions and topological maps

    Simon Hakenes and Tobias Glasmachers. Deep reinforcement learning based navigation with macro actions and topological maps. arXiv preprint arXiv:2504.18300, 2025

    2025 arXiv

  17. [25]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015

    2015

  18. [26]

    Bellemare, Will Dabney, and Rémi Munos

    Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning, 2017

    2017

  19. [27]

    Matteo Hessel, Joseph Modayil, H. V . Hasselt, T. Schaul, Georg Ostrovski, W. Dabney, Dan Hor- gan, B. Piot, Mohammad Gheshlaghi Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018

    2018

  20. [28]

    Schaul, John Quan, Ioannis Antonoglou, and D

    T. Schaul, John Quan, Ioannis Antonoglou, and D. Silver. Prioritized experience replay. CoRR, abs/1511.05952, 2016

    2016 arXiv

  21. [29]

    Deep reinforcement learning with double q-learning

    Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  22. [30]

    Dueling network architectures for deep reinforcement learning

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995–2003. PMLR, 2016

    1995

  23. [31]

    Piot, Jacob Menick, Ian Osband, A

    Meire Fortunato, Mohammad Gheshlaghi Azar, B. Piot, Jacob Menick, Ian Osband, A. Graves, Vlad Mnih, Rémi Munos, Demis Hassabis, O. Pietquin, Charles Blundell, and S. Legg. Noisy networks for exploration. ArXiv, abs/1706.10295, 2018. 12

    2018 arXiv

  24. [32]

    Meta-gradient reinforcement learning

    Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforcement learning. Advances in neural information processing systems, 31, 2018

    2018

  25. [33]

    Universal value function approxima- tors

    Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference on Machine Learning, pages 1312–1320. PMLR, 2015

    2015

  26. [34]

    The arcade learning environment: An evaluation platform for general agents

    Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013

    2013

  27. [35]

    Openai gym

    G Brockman. Openai gym. arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  28. [36]

    The atari grand challenge dataset

    Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and Bastian Leibe. The atari grand challenge dataset. arXiv preprint arXiv:1705.10998, 2017

    2017 arXiv

  29. [37]

    Gym-minigrid: Minimalistic gridworld environment for openai gym

    Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Gym-minigrid: Minimalistic gridworld environment for openai gym. https://github.com/maximecb/gym-minigrid, 2018. 13 A MASP Regularization and Implementation Details A.1 A.1 Detailed Formulation of MASP The Macro-Action S...

    2018

  30. [38]

    - Perform a standard TD update with the MASP penalty, updating θ → θ′ using Σ fixed

    Inner Update (Agent Step): - Sample a trajectory τ from the replay buffer. - Perform a standard TD update with the MASP penalty, updating θ → θ′ using Σ fixed

  31. [39]

    - Evaluate the performance of the updated θ′ using a meta-objective (the standard TD loss)

    Outer Update (Meta Step): - Sample a new trajectory τ ′. - Evaluate the performance of the updated θ′ using a meta-objective (the standard TD loss). - Compute the meta-gradient of this meta-objective w.r.t.Σ (backpropagating through the inner update step). - Update Σ with a se...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.