Pith. sign in

REVIEW 4 major objections 6 minor 32 references

The challenge of hidden gifts in multi-agent reinforcement learning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-agent grid-world task built around unobserved key-dropping defeats state-of-the-art MARL algorithms, but policy-gradient agents equipped with a new learning-aware correction term learn to cooperate reliably.

desk verdict A useful empirical finding about hidden gifts in MARL with a formal derivation that does not hold up; the task and the failure results deserve peer review, but the theory needs major revision. read the letter →

arxiv 2505.20579 v7 pith:SDW2CLQT submitted 2025-05-26 cs.LG cs.AIcs.MA

Dane Malenfant , Blake A. Richards This is my paper · ORCID
classification cs.LGcs.AIcs.MA
keywords CooperationMulti-AgentReinforcementLearningSelf-LearningAwareHiddenGiftsCreditAssignmentPolicyGradientManitokanTask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that "hidden gifts" — cooperative actions a beneficiary cannot observe — create a distinct credit-assignment failure in multi-agent reinforcement learning, and that a simple task can expose it. In the Manitokan grid world, two agents share one key, each must open their own door, and all receive a larger reward only if every door opens; because an agent cannot see the other drop the key, the gift is hidden. Against this task, every tested MARL system (COMA, MAPPO, IPPO, VDN, QMIX, QTRAN, MAVEN, SAF, and vanilla PG) converges to below-random collective success, while decentralized actor-critic PG agents given their own last action can learn it, though with high variance. The paper further derives a learning-aware correction term — a second-order term in the policy gradient proportional to the Hessian of the collective objective — that reduces variance and makes collective success more reliable, especially in a decentralized self-correction form. If these claims hold, they point to a concrete reason why standard MARL credit assignment fails: the other agent's policy is a non-stationary weight on the collective reward, and methods that ignore that non-stationarity collapse.

What carries the argument

The object that carries the argument is the Manitokan task itself together with the correction-term identity. The task is a Dec-POMDP grid world ($N=2$ agents, one key, individual doors, a terminal collective reward) designed so that the critical cooperative step — dropping the key — is unobservable to the beneficiary. The mathematical machinery is the factorization of each agent's objective into individual and collective sub-objectives, $J(\Theta_i)=J_d(\Theta_i)+J_c(\Theta_i)$, with the other agent's policy acting as a time-varying weight on the collective reward. The central identity is the learning-aware gradient correction in Eq. (7), which inserts the term $\nabla_{\Theta_j}\nabla_{\Theta_i}J_c(\Theta_i)\,\Psi(\pi^j_c,a^j,o^j)$ into the policy gradient, where $\Psi$ is the reciprocal of the terminal-policy entropy; this term is meant to compensate for the fact that the value estimate of the collective reward moves when the other agent's policy updates between episodes.

What would settle it

On a small matrix-game analogue of the two-agent task (where all quantities can be enumerated), compute the proposed correction term from Eq. (7) and compare it with a finite-difference estimate of $\nabla_{\Theta_j}\nabla_{\Theta_i}J_c$: if the two disagree as soon as the agents' policies are correlated (for instance, when both condition on a shared observation), the independence assumption in Eq. (5) fails, and the empirical benefit must be attributed to something other than the claimed Hessian.

Watch

Extended reading notes

Core claim

The paper claims that credit assignment in cooperative MARL fails in a specific way when the reward-relevant act of one agent is unobservable to the beneficiary. In the Manitokan task, the collective reward requires each agent to open its door with a single shared key, so someone must drop the key after using it; because drops are invisible to the other agent, they are "hidden gifts." The authors report that all tested MARL algorithms — COMA, MAPPO, IPPO, VDN, QMIX, QTRAN, MAVEN, SAF and vanilla PG — converge to collective success below the random-policy baseline, and that only decentralized actor-critic PG agents with an action-history input learn the task, with high variance. Their formal analysis splits the return into individual and collective sub-objectives and treats the other agent's non-stationary policy as a dynamic weight in the scalarized reward $\hat{R}_i = r^i + \pi^j(a^j_t|o^j_t) r_c$. From this they derive a correction term for the policy gradient, proportional to the Hessian of the collective objective partitioned by the other agent's policy and scaled by the reciprocal entropy $\Psi = 1/\mathbb{E}[\log \pi^j_c]$, which they claim reduces variance and improves convergence; a fully decentralized variant, "self-correction," uses only the agent's own policy and shows the most stable collective success.

Load-bearing premise

The derivation assumes that the gradient of the collective reward with respect to one agent's parameters factors into the product of the other agent's expected log-policy and an independent gradient term, which requires the two agents' policies to be statistically independent, and it divides by an entropy term that can be zero.

Editorial extensions

If this is right

  • In the basic Manitokan task, all tested MARL baselines converge to collective success below the rate of a random policy, and most suppress key-dropping almost entirely.
  • Adding explicit observations about door and key status does not rescue any tested algorithm; only adding the agent's own last action as input enables vanilla policy-gradient agents to learn the collective task, and it does not help the MARL baselines.
  • The derived correction term, and its decentralized self-correction version, reduce the variance of collective success and improve convergence relative to vanilla PG, max-entropy PG, and both preconditioned and regularized LOLA updates.
  • Negating the self-correction term makes agents compete for the key and suppresses gifting, which the authors take as evidence that the term is directly involved in learning hidden-gift behavior.
  • The correction term provably degenerates to zero when no collective reward signal exists, so it does not interfere with individual-objective learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors study only two agents; an immediate extension is to scale $N$ past two, where the delay between a gift and the collective reward grows in expectation and the variance-reduction benefit of the correction term may grow or fail.
  • Because self-correction uses only the agent's own policy, the same scalar term could be applied in mixed populations where some agents are not policy-gradient learners, or in human-AI cooperation where an opponent model is unavailable; the paper does not test this.
  • The supplementary result that a single self-correcting agent can stabilize a naive partner suggests a population-level experiment: injecting one self-correcting agent into a group of vanilla PG agents should make gifting norms spread; this is a testable prediction the paper does not make.
  • The strong correlation between the self-correction value and collective success suggests the term might also serve as a diagnostic signal for imminent collapse of cooperation, not merely as an update rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Manitokan, a two-agent grid-world Dec-POMDP in which two agents must share a single key to open individual doors and receive a collective reward only if both doors are opened; key drops by one agent for the other are unobservable “hidden gifts.” The authors report that several MARL algorithms (COMA, MAPPO, IPPO, VDN, QMIX, QTRAN, MAVEN, SAF) fail to reach the collective reward, that decentralized actor-critic policy gradient agents can succeed when given their own last action, and that a “self-correction” term derived from a learning-aware analysis of the collective-reward objective reduces variance and improves reliability. The central theoretical contribution is Theorem 1, which claims to derive a Hessian-like correction term from the collective-reward sub-objective.

Significance. The task is a clean and potentially useful probe of credit assignment under hidden cooperative actions, and the empirical comparison across many MARL baselines with 10 seeds and 32 parallel environments is substantial. The action-history result is an interesting, falsifiable behavioral finding, and the comparison with LOLA helps position the proposed correction. However, the formal derivation in Section 4 is currently unsupported, and the variance-reduction claim is weakened by an unspecified variance metric and absent error bars. If the derivation is repaired or honestly reframed as a heuristic, the empirical contribution could still be valuable.

major comments (4)
  1. [Section 4, Eq. (5)] The factorization E[log πj_c ∇Θi rc] = E[log πj_c] E[∇Θi rc] is not justified. Statistical independence of the two agents' policies does not imply that the random variables log πj_c(aj|oj) and ∇Θi rc are independent, since both are evaluated on the same joint trajectory and the collective reward depends on both policies. This is the load-bearing step for Theorem 1, and without it the derivation collapses.
  2. [Section 4, Eq. (6) and Proof P.1, Eqs. (18)-(20)] Dividing by E[log πj_c] is invalid as a derivation step: this expectation equals -H(πj_c) (the sign is missing), it can be zero for a deterministic terminal policy, and the reciprocal of an expectation is moved inside the expectation without any argument. The proof also switches between E[log πj_c] and E[log πi_c] across Eqs. (18)-(20) without explanation. At best Eq. (6) is an algebraic rearrangement of the definition of Ĵc rather than a derivation of the correction term.
  3. [Section 4, Eq. (7)] The Hessian term ∇Θi∇Θj Jc(Θj) appears without being derived. The manipulations in Eqs. (4)-(6) operate on first-order gradients of Ĵc and, even if the factorization were valid, would recover at most an estimate of ∇Θi rc. Moreover, ∇Θi rc is not defined in this model-free setting because the scalar reward is a function of the environment and of the other agent's actions, not directly of Θi; the remark in M.2 about finite differences does not supply the analytic differentiability used in the proof.
  4. [Section 3, Figs. 5 and 6] The main variance-reduction claim is not quantitatively supported as presented. The y-axis label "Variance" in Fig. 5(b) is not defined (variance across parallel environments, across seeds, or over a sliding window of episodes?), no confidence intervals or per-seed curves are shown for the learning curves, and the global collective success panels aggregate the data in an unspecified way. The paper should define the metric and provide error bars or per-seed plots.
minor comments (6)
  1. [Throughout] The task is called "Manitokan" but the text also refers to "Manitokanac" and "Manitokan"; please standardize the spelling and clarify the relation to the cited source.
  2. [Section 1, Contribution 3 and p.2] "we discoverer another learning aware gradient correction term" is a typo; please revise.
  3. [Eq. (1) and surrounding paragraph] The reward function and the scalarization with ω(t)=eπj are difficult to parse; in particular, R^i is defined as a scalar-valued function, but ri and rc are used both as scalars and as functions, and the use of the episode index e in "for agent i and episode e" is unclear.
  4. [Section 4.1, Figs. 5 and 6] The references to figures are out of order: the text says "Fig.6" before discussing "Fig.5" and the caption of Fig. 5 mentions "maximum entropy" while the text in 4.1 discusses it; please renumber and align captions.
  5. [Proof P.2] The proof states that the entropy of the terminal policy cannot be zero, which is false for deterministic policies; this needs qualification (e.g., with exploration noise) because the proof of the corollary relies on it.
  6. [Supplementary] There is no code or data availability statement; for a benchmark paper, releasing the environment and training code would materially strengthen reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 1's 'correction term' is derived by rearranging the estimator that defined it: Eq. (5) factorizes E[log pi^j r_c] and Eq. (6) divides by E[log pi^j] to recover E[grad r_c], so the claimed Hessian correction reduces to its own input by construction.

  1. self definitional [Section 4, Eqs. (4)-(6) and Theorem 1; Proof P.1 Eqs. (18)-(20)]
    "Since the collective objective is a scalar quantity, we can get a surrogate for an agent's gradient of the collective reward E[grad_j r_c] by dividing the terms by E[log pi^i_c] which is a scalar quantity... Theorem 1. ... the correction term is grad_i r_c = E[ grad_i Jhat_c / log pi^j_c ] = E[ grad_i Jhat_c Psi(pi^j) ] = grad_i Jc (6)"

    Equation (4) defines Jhat_c = E[log pi^j_c r_c] and then writes grad_i Jhat_c = E[log pi^j_c grad_i r_c]. Equation (5) factorizes this as E[log pi^j_c] E[grad_i r_c] by claiming statistical independence. Theorem 1 divides by E[log pi^j_c] (Psi is its reciprocal) to 'derive' grad_i r_c = E[grad_i Jhat_c Psi] = grad_i Jc. This is tautological: the log-policy factor inserted into the estimator is divided out, so the result recovers only the first-order quantity E[grad_i r_c] that was already the input. It does not produce the claimed Hessian term grad_i grad_j Jc in Eq. (7). Proof P.1 repeats the same factorization and division in Eqs. (18)-(20). The reciprocal equals -1/H(pi^j_c), which is undefined for a deterministic terminal policy, and placing it inside the expectation is unjustified.

full rationale

The central circularity is in Section 4. The paper starts with Jhat_c = E[log pi^j_c r_c], assumes the factorization E[log pi^j] E[grad r_c], and divides by E[log pi^j] to obtain grad r_c, which it then labels the correction term. Thus Eq. (6) does not derive an independent correction; it recovers, by construction, the quantity that was placed into the estimator. The proof P.1 repeats the same step (Eqs. (18)-(20)), replacing r_c with Q^j_c. The claimed second-order/Hessian term in Eq. (7) is not produced by this algebra. This is a genuine self-definitional reduction, not merely a questionable approximation. However, the paper's empirical claim - that adding this term, or its decentralized 'self-correction' variant, reduces variance and improves collective success in the Manitokan task - is tested against baselines and does not depend on the validity of Theorem 1. The cited learning-aware methods (LOLA, COLA) are external prior work, and the self-citations in the references are not load-bearing for the derivation or the experiments. The environment and benchmark results are self-contained. Hence the circularity is partial: the theoretical derivation is tautological, while the empirical contribution stands independently.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim rests on standard RL background (policy gradient theorem), the Dec-POMDP modeling assumption, and several ad hoc assumptions introduced in the proof of the correction term. The correction term itself is not fitted to data; its form is fixed by the (flawed) derivation.

assumptions (6)
  • standard math Policy gradient theorem and score function identity
    Used to derive the update in Section 4 and in the proof P.1.
  • domain assumption Dec-POMDP assumption with egocentric observations
    The problem is modeled as a Dec-POMDP in Section 2, with partial observations of size 3x3.
  • ad hoc to paper Successful episode assumption, collective reward was returned
    Stated as the first key assumption in the Theorem 1 sketch; restricts the correction term to successful trajectories.
  • ad hoc to paper Other agent's collective reward policy is differentiable
    Second key assumption in Theorem 1, needed to pass gradients through the other agent's policy.
  • ad hoc to paper Statistical independence of agent policies in expectation
    Used to factor E[log πj] E[∇Θi rc] in Eq. (5), which is questionable in a cooperative task with shared rewards.
  • ad hoc to paper Self-correction approximation, own policy replaces other agent's policy
    Justified by the claim that πi* = πj*, but no proof of convergence to this optimum is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The challenge of hidden gifts in multi-agent reinforcement learning." pith.science (2026). https://pith.science/paper/SDW2CLQT

@misc{pith2026250520579,
  author       = {Pith},
  title        = {Pith review of: The challenge of hidden gifts in multi-agent reinforcement learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDW2CLQT}},
  note         = {Machine review of arXiv:2505.20579}
}
read the original abstract

Sometimes we benefit from actions that others have taken even when we are unaware that they took those actions. For example, if your neighbor chooses not to take a parking spot in front of your house when you are not there, you can benefit, even without being aware that they took this action. These ``hidden gifts'' represent an interesting challenge for multi-agent reinforcement learning (MARL), since assigning credit when the beneficial actions of others are hidden is non-trivial. Here, we study the impact of hidden gifts with a simple MARL task. In this task, agents in a grid-world environment have individual doors to unlock in order to obtain individual rewards. As well, if all the agents unlock their door the group receives a larger collective reward. However, there is only one key for all of the doors, such that the collective reward can only be obtained when the agents drop the key for others after they use it. Notably, there is nothing to indicate to an agent that the other agents have dropped the key, thus this act for others is a ``hidden gift''. We show that several different state-of-the-art MARL algorithms, including MARL specific architectures, fail to learn how to obtain the collective reward in this simple task. Interestingly, we find that decentralized actor-critic policy gradient agents can succeed when we provide them with information about their own action history, but MARL agents still cannot solve the task with action history. Finally, we derive a correction term for policy gradient agents, inspired by learning aware approaches, which reduces the variance in learning and helps them to converge to collective success more reliably. These results show that credit assignment in multi-agent settings can be particularly challenging in the presence of ``hidden gifts'', and demonstrate that self learning-awareness in decentralized agents can benefit these settings.

Figures

Figures reproduced from arXiv: 2505.20579 by the authors.

Figure 1
Figure 1. The deceivingly simple steps to success in the Manitokan task. a) Agent 1 finds the key; b) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. a) Success rate for the collective reward, i.e. percentage of trials where both agents opened [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. a) Success rate when each agent receives information about whether they have opened their [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: a) Success rate when each agent receives their last action in the observation. b) Cumulative [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Collective success rate of PG agents comparing the vanilla model against one with a [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Collective success rate of PG agents comparing the vanilla model against one with a self [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: a) Policy loss of the COMA model b) Counterfactual baseline in the COMA policy update [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: a) Key drop rate (i.e. cumulative key drops) averaged across parallel episodes and runs. b) [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: a) The contribution of an agent’s reward accumulation to success weighted by their total [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: a) Comparing agents of MAPPO, IPPO, VDN and QMIX models with a randomization [PITH_FULL_IMAGE:figures/full_fig_p032_10.png]
Figure 11
Figure 11. Figure 11: a) Euclidean distance between agents averaged over parallel environments and simulations [PITH_FULL_IMAGE:figures/full_fig_p033_11.png]
Figure 12
Figure 12. Figure 12: a) Timestep the first reward an agent received. b) Timestep the first reward a policy [PITH_FULL_IMAGE:figures/full_fig_p033_12.png]
Figure 13
Figure 13. Figure 13: a) Success rate of policy gradient agents with action history comparing the normal reward [PITH_FULL_IMAGE:figures/full_fig_p035_13.png]
Figure 14
Figure 14. Figure 14: a) Success rate between policy gradient agents comparing a disassociation of the reward [PITH_FULL_IMAGE:figures/full_fig_p036_14.png]
Figure 15
Figure 15. Figure 15: a) The percentage of cumulative reward and collective success for anti-collective policy [PITH_FULL_IMAGE:figures/full_fig_p037_15.png]
Figure 16
Figure 16. Figure 16: a) The percentage of cumulative reward and collective success for anti-collective policy [PITH_FULL_IMAGE:figures/full_fig_p038_16.png]
Figure 17
Figure 17. Figure 17: a) Comparison of single agent PPO, PG and DQN agents where one agent needs to open a [PITH_FULL_IMAGE:figures/full_fig_p039_17.png]
Figure 18
Figure 18. Figure 18: a) A comparison of LOLA with a learning rate of 1 and 0.0005, Naive LOLA with a [PITH_FULL_IMAGE:figures/full_fig_p040_18.png]
Figure 19
Figure 19. Figure 19: a) A comparison of collective success rate between PG agents with last action inputs [PITH_FULL_IMAGE:figures/full_fig_p041_19.png]
Figure 20
Figure 20. Figure 20: a) A correlation analysis between the collective success rate rate of PG agents with [PITH_FULL_IMAGE:figures/full_fig_p042_20.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 18 canonical work pages

  1. [1]

    LOQA : Learning with opponent q-learning awareness

    Milad Aghajohari, Juan Agustin Duque, Tim Cooijmans, and Aaron Courville. LOQA : Learning with opponent q-learning awareness. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=FDQF6A1s6M

    work page 2024

  2. [2]

    Unifying temporal and structural credit assignment problems

    Adrian K Agogino and Kagan Tumer. Unifying temporal and structural credit assignment problems. In Autonomous agents and multi-agent systems conference, 2004

    work page 2004

  3. [3]

    Understanding the impact of entropy on policy optimization

    Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pages 151--160. PMLR, 2019

    2019

  4. [4]

    Effective choice in the prisoner's dilemma

    Robert Axelrod. Effective choice in the prisoner's dilemma. Journal of conflict resolution, 24 0 (1): 0 3--25, 1980

    work page 1980

  5. [5]

    Manitokanac

    Lawrence J Barkwell. Manitokanac. Gabriel Dumont Institute of Native Studies and Applied Research, 2015. URL https://www.metismuseum.ca/resource.php/148154

    work page 2015

  6. [6]

    The theory of dynamic programming

    Richard Bellman. The theory of dynamic programming. Bulletin of the American Mathematical Society, 60 0 (6): 0 503--515, 1954

    1954

  7. [7]

    Prisoner's dilemma; a study in conflict and cooperation

    Albert M Chammah. Prisoner's dilemma; a study in conflict and cooperation. Ann Arbor, U. of Michigan P, 1965

    work page 1965

  8. [8]

    Decision transformer: Reinforcement learning via sequence modeling

    Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 0 15084--15097, 2021

    2021

Show all 32 references
  1. [9]

    Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks

    Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Advances in Neural...

    2023

  2. [10]

    Learning phrase representations using RNN encoder -- decoder for statistical machine translation

    Kyunghyun Cho, Bart van Merri \"e nboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder -- decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelema...

    2014 doi

  3. [11]

    Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language models

    Logan Cross, Violet Xiang, Agam Bhatia, Daniel LK Yamins, and Nick Haber. Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/f...

    2025

  4. [12]

    Maximum entropy RL (provably) solves some robust RL problems

    Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=PtSAD3caaA2

    2022

  5. [13]

    Counterfactual multi-agent policy gradients

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  6. [14]

    Learning with opponent-learning awareness

    Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. arXiv preprint arXiv:1709.04326, 2017

    2017 arXiv

  7. [15]

    Structural credit assignment in neural networks using reinforcement learning

    Dhawal Gupta, Gabor Mihucz, Matthew Schlegel, James Kostas, Philip S Thomas, and Martha White. Structural credit assignment in neural networks using reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 30257--30270, 2021

    2021

  8. [16]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861--1870. Pmlr, 2018

    2018

  9. [17]

    Optimizing agent behavior over long time scales by transporting value

    Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, and Greg Wayne. Optimizing agent behavior over long time scales by transporting value. Nature communications, 10 0 (1): 0 5223, 2019

    2019

  10. [18]

    Stateful active facilitator: Coordination and environmental heterogeneity in cooperative multi-agent reinforcement learning

    Dianbo Liu, Vedant Shah, Oussama Boussif, Cristian Meo, Anirudh Goyal, Tianmin Shu, Michael Curtis Mozer, Nicolas Heess, and Yoshua Bengio. Stateful active facilitator: Coordination and environmental heterogeneity in cooperative multi-agent reinforcement learning. In ICLR, 2023

    2023

  11. [19]

    Maven: Multi-agent variational exploration

    Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019

    2019

  12. [20]

    Multi-agent cooperation through learning-aware policy gradients, 2025

    Alexander Meulemans, Seijin Kobayashi, Johannes von Oswald, Nino Scherrer, Eric Elmoznino, Blake Richards, Guillaume Lajoie, Blaise Agüera y Arcas, and João Sacramento. Multi-agent cooperation through learning-aware policy gradients, 2025. URL https://arxiv.org/abs/2410.18636

    2025 arXiv

  13. [21]

    Equilibrium points in n-person games

    John F Nash Jr. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36 0 (1): 0 48--49, 1950

    1950

  14. [22]

    When do transformers shine in rl? decoupling memory from credit assignment

    Tianwei Ni, Michel Ma, Benjamin Eysenbach, and Pierre-Luc Bacon. When do transformers shine in rl? decoupling memory from credit assignment. Advances in Neural Information Processing Systems, 36: 0 50429--50452, 2023

    2023

  15. [23]

    Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21 0 (178): 0 1--51, 2020

    2020

  16. [24]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  17. [25]

    Agent-time attention for sparse rewards multi-agent reinforcement learning

    Jennifer She, Jayesh K Gupta, and Mykel J Kochenderfer. Agent-time attention for sparse rewards multi-agent reinforcement learning. arXiv preprint arXiv:2210.17540, 2022

    2022 arXiv

  18. [26]

    Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning

    Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887--5896. PMLR, 2019

    2019

  19. [27]

    Pufferlib: Making reinforcement learning libraries and environments play nice

    Joseph Suarez. Pufferlib: Making reinforcement learning libraries and environments play nice. arXiv preprint arXiv:2406.12905, 2024

    2024 arXiv

  20. [28]

    Value-decomposition networks for cooperative multi-agent learning

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. The International Foundation for Autonom...

    2017

  21. [29]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999

    1999

  22. [30]

    Learning sequences of actions in collectives of autonomous agents

    Kagan Tumer, Adrian K Agogino, and David H Wolpert. Learning sequences of actions in collectives of autonomous agents. In Proceedings of the first international joint conference on autonomous agents and multiagent systems: Part 1, pages 378--385, 2002

    2002

  23. [31]

    Cola: consistent learning with opponent-learning awareness

    Timon Willi, Alistair Hp Letcher, Johannes Treutlein, and Jakob Foerster. Cola: consistent learning with opponent-learning awareness. In International Conference on Machine Learning, pages 23804--23831. PMLR, 2022

    2022

  24. [32]

    The surprising effectiveness of ppo in cooperative multi-agent games

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems, 35: 0 24611--24624, 2022

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.