Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that per-agent value functions trained with a TD(λ)-like target that satisfies Shapley's Efficiency, Additivity, and Symmetry axioms are Shapley-value credit allocations in dynamic n-agent ad hoc teamwork, and derives the…

desk verdict A useful empirical POAM variant, but the Shapley-value derivation has a load-bearing numerical error; worth peer review if the authors fix the math or soften the claim. read the letter →

arxiv 2506.11285 v1 pith:XJ7KGQJ7 submitted 2025-06-12 cs.MA cs.GTcs.LG

Jianhong Wang , Yang Li , Samuel Kaski , Jonathan Lawry This is my paper
classification cs.MAcs.GTcs.LG MSC 91A1291A06
keywords Shapleyvalueadhocteamworkmulti-agentreinforcementlearningcreditassignmenttemporaldifferencecooperativegametheoryopensystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to give n-agent ad hoc teamwork (NAHT), where only some agents in an open team are controlled, a principled credit-assignment theory. It models an NAHT process as a state-dependent vector space of cooperative games generated by basis games, then shows that requiring per-agent value functions to satisfy the three Shapley axioms turns a $\mathrm{TD}(\lambda)$-like return into a game-theoretic credit target. The resulting Shapley Machine learns each controlled agent's value as its Shapley value, so the reward signal each agent optimizes reflects its contribution to the ad hoc team. If the theory holds, heuristic reward shaping is replaced by an axiomatically grounded scheme, and the truncation horizon in $\mathrm{TD}(\lambda)$ should track the number of non-empty coalitions of the team.

What carries the argument

The carrying object is the state-space cooperative game space $\mathcal{G}_{\mathrm{NAHT}} = \bigoplus_{s\in S}\mathcal{G}(s)$, where each fixed state $s$ has a vector space generated by basis games $v^1_{C,s}$: cooperative games that pay one unit to every coalition containing $C$ and zero otherwise, so any subgame decomposes as $v_s = \sum_{C\subseteq \tilde{N}} k_C\, v^1_{C,s}$. Assumption 4.2 is the bridge: it identifies basis games ordered by coalition size with expected $n$-step returns $\mathbb{E}_\pi[G_{t:t+k}\,|\,s_t]$, and the geometric weights of truncated $\mathrm{TD}(\lambda)$ supply the coefficients $k_C$. Efficiency fixes each agent's reward share $R_{t,i}$; Additivity converts the target into a weighted $\lambda$-return; Symmetry is realized by giving identical agent types identical inferred teammate representations, so they receive equal value estimates.

What would settle it

In a small NAHT task (e.g., three controlled agents) after training, fix the joint policy, enumerate all non-empty coalitions, compute each agent's explicit Shapley value from the corresponding state values, and compare with the $V_i$ learned by Shapley Machine; a systematic mismatch would falsify Proposition 4.6, while a direct test of Assumption 4.2 would compare $\mathbb{E}_\pi[G_{t:t+k}\,|\,s_t]$ with the summed basis-game values for coalitions of the matching size.

Watch

Extended reading notes

Core claim

The paper's central claim is Proposition 4.6: Shapley Machine fulfills Efficiency, Additivity, and Symmetry, so it learns $V_i$ as Shapley values for dynamic scenarios. The derivation builds a state-space cooperative game model $\mathcal{G}_{\mathrm{NAHT}} = \bigoplus_{s\in S}\mathcal{G}(s)$, decomposes each state's subgame into basis games $v^1_{C,s}$ indexed by coalitions $C$, and imposes Assumption 4.2, which identifies the basis-game values ordered by coalition size with expected $n$-step returns of matching horizons. This identification turns the Shapley axioms into recursions on returns, yielding a truncated $\mathrm{TD}(\lambda)$ update whose horizon $m$ equals the number of basis games and whose per-agent reward is $R_{t,i} = R_t - \sum_{j\neq i}(V_j(s_t) - \gamma V_j(s_{t+1}))$. The paper further argues that the earlier POAM algorithm satisfies only Additivity and Symmetry, making it a Banzhaf Machine rather than a Shapley Machine, and reports MPE and SMAC experiments where Shapley Machine matches or outperforms POAM.

Load-bearing premise

The load-bearing premise is that the values of basis games ordered by coalition size line up exactly with expected $n$-step returns of matching horizons (Assumption 4.2); if that mapping fails, the learned values are a weighted TD target but not Shapley values.

Editorial extensions

If this is right

  • Per-agent critics trained by Shapley Machine are Shapley values, so policy gradients act on a credit signal that reflects each controlled agent's marginal contribution to the team's state value.
  • The TD(λ) target acquires a game-theoretic reading: its truncation horizon should equal the number of non-empty coalitions of the ad hoc team (7 for a 3-agent team, 31 for a 5-agent team), matching the paper's experiments where m=7 beats longer horizons on MPE.
  • POAM, the existing NAHT baseline, is classified as a Banzhaf Machine: it satisfies Additivity and Symmetry but not Efficiency, so it learns Banzhaf indices rather than Shapley values.
  • In the tested MPE and SMAC NAHT tasks, Shapley Machine converges faster and reaches lower critic loss than POAM, with test performance better in three scenarios and matching in the remaining two.
  • Under superadditivity the basis coefficients satisfy $k_C \geq 0$, which is consistent with the geometric distribution used to set the TTD(λ) weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Assumption 4.2 is the right bridge between game values and n-step returns, then any MARL algorithm training critics with truncated λ-returns is implicitly choosing a cooperative-game decomposition, and its solution concept can be identified by which Shapley axioms its target satisfies.
  • A direct empirical check is available: in a small NAHT task, fix the learned joint policy, enumerate all coalitions, compute explicit Shapley values from the resulting state values, and compare them with the $V_i$ learned by Shapley Machine; agreement would confirm Proposition 4.6, while systematic mismatch would localize the failure to Assumption 4.2 or the relaxed efficiency loss.
  • The framework suggests an adaptive variant that learns the basis-game weights $k'_C$ from data instead of fixing them by a geometric distribution, a direction the paper itself flags as future work and which would extend the method beyond superadditive games.
  • The paper's vector-space picture implies a geometric measure of openness bias: the angle between the restricted game vector (controlled agents only) and the full game vector quantifies how much credit assignment is distorted by ignoring uncontrolled agents.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a game-theoretic framework for n-agent ad hoc teamwork (NAHT). It represents each NAHT process as an element of a direct-sum vector space of cooperative games over states, generated by constant-valued basis games indexed by coalitions. Under Assumption 4.2, which maps these basis games to a sequence of expected n-step returns, the paper derives a truncated TD(λ)-like target, defines per-agent shaped rewards from an Efficiency condition, and proposes Shapley Machine, an algorithm claimed to learn Shapley-value credit allocations by satisfying Efficiency, Additivity, and Symmetry. Experiments on MPE and SMAC benchmarks compare Shapley Machine with POAM and include ablations and horizon-m studies.

Significance. If the central derivation were correct, the paper would offer a novel, principled link between cooperative game theory (Shapley value) and TD(λ)-style credit assignment in open multi-agent systems, and would provide a useful axiomatic design template. The paper is commendable for releasing code, for attempting to derive the algorithm from axioms rather than from an explicit Shapley formula, and for including ablation studies. However, the core theoretical claim rests on an assumption that, as stated, is numerically inconsistent with the definition of basis games, and the implementation weakens the Efficiency axiom; these issues need to be resolved before the significance claim can be accepted.

major comments (4)
  1. [Section 4.1, Assumption 4.2 and Eq. (6)] The assumption cannot hold for the basis games defined in Definition 2.1. For every nonempty C⊆Ñ, v^{vs(Ñ)}_{C,s}(Ñ)=vs(Ñ)=V(Ñ,s). Therefore the right-hand side of the assumed equality, Σ_{i=p}^q v^{vs(Ñ)}_{C_i,s}(Ñ), equals q·V(Ñ,s), where q is the number of coalitions of that size, while the left-hand side Eπ[G_{t:t+k}|s_t] is a discounted return whose expected magnitude is V(Ñ,s). For k=1 the equality would require R_t+γE[V(s_{t+1})]=qV(Ñ,s), which is generically false. Dividing by q(i) and assigning different horizons n(i) to different coalition sizes produces Eq. (6), which states that the same V(Ñ,s_t) equals different n-step returns; these equalities are mutually incompatible unless all those returns coincide. Appendix B offers only a variance heuristic, not a proof. Because Eqs. (7)–(11) and Proposition 4.6 inherit this step, the axiomatic characterization of V_i as Shapley values is not currently derived.
  2. [Proposition C.2 and Eq. (12)] The proof assumes the per-agent Bellman allocation ϕ_i(V(Ñ,s_t)) = ϕ_i(R_t) + γϕ_i(V(Ñ,s_{t+1})) without deriving it from the dynamics. This identity is precisely the n-step-return allocation whose validity is at issue; R_{t,i} is then defined by solving for ϕ_i(R_t), and Eq. (13)/(23) is imposed as a regularization to enforce the resulting decomposition. The Shapley conclusion is therefore circular with respect to Assumption 4.2: the allocated Bellman equation is assumed, not proved.
  3. [Appendix D.1.1, Eq. (22)] The implementation introduces a coefficient α∈(0,1) in R_{t,i}=R_t−αΣ_{j≠i}(V_j(s_t)−γV_j(s_{t+1})). This means Σ_i R_{t,i}≠R_t in general, so the Efficiency axiom is not exactly satisfied by the shaped rewards. Similarly, Eq. (23) is enforced only as a soft regularization term (the efficiency loss), not as a hard constraint. Proposition 4.6's statement that Shapley Machine 'fulfills Efficiency' is therefore not supported by the implemented algorithm; at most one can say the algorithm is biased toward Efficiency.
  4. [Section 5.2 and Figure 5a] The experiments show that m=20 and m=31 give similar performance in 5v6, and m=7 is best in MPE, but exact equality with the number of nonempty coalitions is not established; m=20 versus m=31 is a counterexample to the claim that the optimal horizon is exactly the number of basis games. In 8v9 and 10v11, m is set to the episode length rather than to the theoretical number of basis games, so Figure 15 demonstrates only a monotone empirical relationship, not the theory's prediction. The empirical results are suggestive but should be framed as approximate or additional evidence, not as verification of Assumption 4.2.
minor comments (4)
  1. [Appendix D.4] The phrase 'multinational distribution' should be 'multinomial distribution'.
  2. [Theorem 4.5] The theorem cites reference [18], but the statement as given ('given an additively factored immediate reward function...') is not located in that reference; please provide a precise citation or proof.
  3. [Appendix C.4] The proof of Proposition 4.6 is a single sentence and does not engage with the concrete implementation details in Appendix D; it should be expanded to show each axiom is satisfied by the actual loss functions.
  4. [Eq. (13) and Appendix D.1.1] Eq. (13) is introduced as a condition in the main text and later used as a regularization loss; the relation between the two should be stated explicitly.
Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on a standard uniqueness theorem for Shapley value, a borrowed superadditivity assumption from MARL, and two paper-specific choices: the n-step-return mapping of basis games and the geometric weighting of TTD(lambda). Multiple per-task hyperparameters (lambda, m, alpha, beta1, beta2, value loss clip) are tuned by hand and affect whether the axioms are truly satisfied.

free parameters (6)
  • lambda = 0.85 (MPE, 3sv5z, 5v6), 0.95 (8v9, 10v11)
    Trace-decay parameter setting the geometric weights on n-step returns; chosen per task by hand, acknowledged in the Conclusion as manually tuned.
  • m (TTD(lambda) horizon) = 7 (MPE, 3sv5z), 31 (5v6), 120 (8v9), 150 (10v11)
    Claimed to equal the number of non-empty coalitions (2^|N|-1), but for large tasks set to the episode length since 2^|N|-1 exceeds the episode; m is a tuned hyperparameter and the mapping is only approximate.
  • alpha = 0.01 (MPE, 3sv5z, 8v9, 10v11), 0.002 (5v6)
    Scales the subtracted teammate value differences in the shaped reward R_{t,i}; alpha less than 1 means the Efficiency axiom is not exactly satisfied, breaking Proposition 4.6's premise.
  • beta1 = 0.5
    Weight on the standard per-agent critic loss; chosen per task.
  • beta2 = 0.01 (MPE, 3sv5z, 8v9), 0.001 (5v6, 10v11)
    Weight on the regularization that enforces sum_i V_i = V(N,s); also tuned.
  • value loss clip = applied only to 8v9 and 10v11
    Post-hoc stabilization added only for large-scale scenarios; not used elsewhere because it reduces performance.
assumptions (5)
  • standard math Shapley value is the unique allocation satisfying Efficiency, Additivity and Symmetry (Theorem 2.2, Dubey 1975)
    External uniqueness theorem anchoring the claim that any V_i satisfying the three axioms is a Shapley value.
  • domain assumption The game class is restricted to superadditive games so that k_C >= 0 (Proposition 4.4, Appendix A.2)
    Borrowed from prior MARL work [11]; limits the game space so the geometric positive weights are admissible.
  • ad hoc to paper Assumption 4.2: basis games ordered by coalition size correspond to n-step returns with matching horizons
    Load-bearing mapping that turns the game-theoretic representation into a TTD(lambda) recursion; without it Eq. (6) and Eq. (11) do not follow.
  • ad hoc to paper k'_C weights are generated by a geometric distribution with parameter lambda (Section 4.2.1)
    Choosing geometric weights 'following the convention of RL' is a free normalization, not derived from the Shapley axioms.
  • domain assumption Existence of a factorization of the Dec-POMDP transition and reward such that the value decomposes across agents (Theorem 4.5 from [18])
    Used to justify the Efficiency axiom; the paper does not construct the factorization for the test environments.
invented entities (1)
  • state-space cooperative game space G_NAHT (direct sum of per-state game spaces)
    purpose: Formalize an NAHT process as a family of cooperative games indexed by state so that Shapley axioms can be imposed on state values.
    The model provides no falsifiable handle beyond the horizon-m prediction; for large-scale tasks the paper falls back to the episode length for m, so the model's quantitative prediction is not operational.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork." pith.science (2026). https://pith.science/paper/XJ7KGQJ7

@misc{pith2026250611285,
  author       = {Pith},
  title        = {Pith review of: Shapley Machine: A Game-Theoretic Framework for N-Agent Ad Hoc Teamwork},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XJ7KGQJ7}},
  note         = {Machine review of arXiv:2506.11285}
}
abstract

Open multi-agent systems are increasingly important in modeling real-world applications, such as smart grids, swarm robotics, etc. In this paper, we aim to investigate a recently proposed problem for open multi-agent systems, referred to as n-agent ad hoc teamwork (NAHT), where only a number of agents are controlled. Existing methods tend to be based on heuristic design and consequently lack theoretical rigor and ambiguous credit assignment among agents. To address these limitations, we model and solve NAHT through the lens of cooperative game theory. More specifically, we first model an open multi-agent system, characterized by its value, as an instance situated in a space of cooperative games, generated by a set of basis games. We then extend this space, along with the state space, to accommodate dynamic scenarios, thereby characterizing NAHT. Exploiting the justifiable assumption that basis game values correspond to a sequence of n-step returns with different horizons, we represent the state values for NAHT in a form similar to $\lambda$-returns. Furthermore, we derive Shapley values to allocate state values to the controlled agents, as credits for their contributions to the ad hoc team. Different from the conventional approach to shaping Shapley values in an explicit form, we shape Shapley values by fulfilling the three axioms uniquely describing them, well defined on the extended game space describing NAHT. To estimate Shapley values in dynamic scenarios, we propose a TD($\lambda$)-like algorithm. The resulting reinforcement learning (RL) algorithm is referred to as Shapley Machine. To our best knowledge, this is the first time that the concepts from cooperative game theory are directly related to RL concepts. In experiments, we demonstrate the effectiveness of Shapley Machine and verify reasonableness of our theory.

Figures

Figures reproduced from arXiv: 2506.11285 by the authors.

Figure 1
Figure 1. Example of the state-space cooperative game model, described by a spanning set including two vectors with the fixed state s. The red and blue ar￾rowed lines are two games, such as v and w, whose dif￾ference is measured by the θ. The shading area indi￾cates the spectrum of gen￾erated cooperative games for kC ≥ 0 and PkC = 1, which occupies the first quadrant. GNAHT is isomorphic to R (2|N˜|−1)×d , if we simply consid… view at source ↗
Figure 2
Figure 2. Testing evaluation during the training procedure. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Training evaluation during the training procedure. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Critic loss record across all scenarios during the training procedure. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Proof-of-concept experiments for verifying the necessity of basis games. In MPE, the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: One run of MPE to show the variation of test return, shaped reward and efficiency loss. [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]
Figure 7
Figure 7. Figure 7: One run of 5v6 to show the variation of test return, shaped reward and efficiency loss. [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: A case of 8v9 to show the instability of learning progress via the mean shaped rewards. [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: A case of 10v11 to show the instability of learning progress via the mean shaped rewards. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Comparison between stability of learning processes with and without the value loss clip in [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Entropy of policies across all scenarios during the training procedure. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Controlling the number of active basis games can be implemented by changing the [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]
Figure 13
Figure 13. Figure 13: The influence of different values of basis games given the same number of active basis [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Comparison between the mean win rates of [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: The law of empirical m values with respect to the number of agents. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 16
Figure 16. Figure 16: Ablation studies of Shapley Machine on all experimental domains. The Shapley Machine [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Robot Open Adaptive Teaming Across Unseen Environments, Partners, and Scales

    cs.RO 2026-07 conditional novelty 6.0 of 10

    HOLA uses hypergraphic games and open-ended partner sampling to learn multi-robot pursuit policies that adapt to unseen partners, environments, and team sizes, transferring zero-shot to physical drones and quadrupeds.

Reference graph

Works this paper leans on

47 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Multi-agent reinforcement learning for active voltage control on power distribution networks.Advances in Neural Information Processing Systems, 34:3271–3284, 2021

    Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C Green. Multi-agent reinforcement learning for active voltage control on power distribution networks.Advances in Neural Information Processing Systems, 34:3271–3284, 2021

    2021

  2. [2]

    Improving the efficiency and efficacy of multi-agent reinforcement learning on complex railway networks with a local-critic approach

    Yuan Zhang, Umashankar Deekshith, Jianhong Wang, and Joschka Boedecker. Improving the efficiency and efficacy of multi-agent reinforcement learning on complex railway networks with a local-critic approach. InProceedings of the International Conference on Automated Planning and Scheduling, volume 34, pages 698–706, 2024

    work page 2024

  3. [3]

    Scalable multi-agent reinforcement learning through intelligent infor- mation aggregation

    Siddharth Nayak, Kenneth Choi, Wenqi Ding, Sydney Dolan, Karthik Gopalakrishnan, and Hamsa Balakrishnan. Scalable multi-agent reinforcement learning through intelligent infor- mation aggregation. InInternational Conference on Machine Learning, pages 25817–25833. PMLR, 2023. 11

    work page 2023

  4. [4]

    Hola-drone: Hypergraphic open-ended learning for zero-shot multi-drone cooperative pursuit, 2024

    Yang Li, Dengyu Zhang, Junfan Chen, Ying Wen, Qingrui Zhang, Shaoshuai Mou, and Wei Pan. Hola-drone: Hypergraphic open-ended learning for zero-shot multi-drone cooperative pursuit, 2024

    work page 2024

  5. [5]

    MIT press Cambridge, 1998

    Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  6. [6]

    N-agent ad hoc teamwork.Advances in Neural Information Processing Systems, 37:111832– 111862, 2025

    Caroline Wang, Muhammad Arrasy Rahman, Ishan Durugkar, Elad Liebman, and Peter Stone. N-agent ad hoc teamwork.Advances in Neural Information Processing Systems, 37:111832– 111862, 2025

    work page 2025

  7. [7]

    Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020

    Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020

    arXiv 2011

  8. [8]

    Multi- agent risks from advanced ai.arXiv preprint arXiv:2502.14143, 2025

    Lewis Hammond, Alan Chan, Jesse Clifton, Jason Hoelscher-Obermaier, Akbir Khan, Euan McLean, Chandler Smith, Wolfram Barfuss, Jakob Foerster, Tomáš Gavenˇciak, et al. Multi- agent risks from advanced ai.arXiv preprint arXiv:2502.14143, 2025

    arXiv 2025

Show all 47 references
  1. [9]

    Cooperative multi-agent learning: The state of the art.Autonomous agents and multi-agent systems, 11:387–434, 2005

    Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art.Autonomous agents and multi-agent systems, 11:387–434, 2005

    2005

  2. [10]

    Learning to predict by the methods of temporal differences.Machine learning, 3:9–44, 1988

    Richard S Sutton. Learning to predict by the methods of temporal differences.Machine learning, 3:9–44, 1988

    1988

  3. [11]

    Shapley q-value: A local reward approach to solve global reward games

    Jianhong Wang, Yuan Zhang, Tae-Kyun Kim, and Yunjie Gu. Shapley q-value: A local reward approach to solve global reward games. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7285–7292, 2020

    2020

  4. [12]

    Shapley counterfactual credits for multi-agent reinforcement learning

    Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. InProceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 934–942, 2021

    2021

  5. [13]

    Multiagent model-based credit assignment for continuous control

    Dongge Han, Chris Xiaoxuan Lu, Tomasz Michalak, and Michael Wooldridge. Multiagent model-based credit assignment for continuous control. InProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 571–579, 2022

    2022

  6. [14]

    Shaq: Incorporating shapley value theory into multi-agent q-learning.Advances in Neural Information Processing Systems, 35:5941–5954, 2022

    Jianhong Wang, Yuan Zhang, Yunjie Gu, and Tae-Kyun Kim. Shaq: Incorporating shapley value theory into multi-agent q-learning.Advances in Neural Information Processing Systems, 35:5941–5954, 2022

    2022

  7. [15]

    Cooperative open-ended learning framework for zero-shot coordination

    Yang Li, Shao Zhang, Jichen Sun, Yali Du, Ying Wen, Xinbing Wang, and Wei Pan. Cooperative open-ended learning framework for zero-shot coordination. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceedings o...

    2023

  8. [16]

    On the uniqueness of the shapley value.International Journal of Game Theory, 4(3):131–139, 1975

    Pradeep Dubey. On the uniqueness of the shapley value.International Journal of Game Theory, 4(3):131–139, 1975

    1975

  9. [17]

    Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.arXiv preprint arXiv:2104.13478, 2021

    Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇckovi´c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges.arXiv preprint arXiv:2104.13478, 2021

    2021 arXiv

  10. [18]

    Springer, 2016

    Frans A Oliehoek, Christopher Amato, et al.A concise introduction to decentralized POMDPs, volume 1. Springer, 2016

    2016

  11. [19]

    Monotonic solutions of cooperative games.International Journal of Game Theory, 14(2):65–72, 1985

    H Peyton Young. Monotonic solutions of cooperative games.International Journal of Game Theory, 14(2):65–72, 1985

    1985

  12. [20]

    Markov games as a framework for multi-agent reinforcement learning

    Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157–163. Elsevier, 1994

    1994

  13. [21]

    Truncating temporal differences: On the efficient implementation of td (lambda) for reinforcement learning.Journal of Artificial Intelligence Research, 2:287–318, 1994

    Pawel Cichosz. Truncating temporal differences: On the efficient implementation of td (lambda) for reinforcement learning.Journal of Artificial Intelligence Research, 2:287–318, 1994. 12

    1994

  14. [22]

    The complexity of decentralized control of markov decision processes.Mathematics of operations research, 27(4):819–840, 2002

    Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes.Mathematics of operations research, 27(4):819–840, 2002

    2002

  15. [23]

    Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555, 2014

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling.arXiv preprint arXiv:1412.3555, 2014

    2014 arXiv

  16. [24]

    Morgan & Claypool Publishers, 2011

    Georgios Chalkiadakis, Edith Elkind, and Michael Wooldridge.Computational aspects of cooperative game theory. Morgan & Claypool Publishers, 2011

    2011

  17. [25]

    Weighted voting doesn’t work: A mathematical analysis.Rutgers L

    John F Banzhaf III. Weighted voting doesn’t work: A mathematical analysis.Rutgers L. Rev., 19:317, 1964

    1964

  18. [26]

    The convergence of td (λ) for generalλ.Machine learning, 8:341–362, 1992

    Peter Dayan. The convergence of td (λ) for generalλ.Machine learning, 8:341–362, 1992

    1992

  19. [27]

    Td (λ) converges with probability 1.Machine Learning, 14:295–301, 1994

    Peter Dayan and Terrence J Sejnowski. Td (λ) converges with probability 1.Machine Learning, 14:295–301, 1994

    1994

  20. [28]

    Analysis of temporal-diffference learning with function approximation.Advances in neural information processing systems, 9, 1996

    John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation.Advances in neural information processing systems, 9, 1996

    1996

  21. [29]

    Learning from delayed rewards

    Christopher John Cornish Hellaby Watkins et al. Learning from delayed rewards. 1989

    1989

  22. [30]

    Incremental multi-step q-learning

    Jing Peng and Ronald J Williams. Incremental multi-step q-learning. InMachine Learning Proceedings 1994, pages 226–232. Elsevier, 1994

    1994

  23. [31]

    True online td (lambda)

    Harm Seijen and Rich Sutton. True online td (lambda). InInternational Conference on Machine Learning, pages 692–700. PMLR, 2014

    2014

  24. [32]

    High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  25. [33]

    The surprising effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

    2022

  26. [34]

    On partially controlled multi-agent systems.Journal of Artificial Intelligence Research, 4:477–507, 1996

    Ronen I Brafman and Moshe Tennenholtz. On partially controlled multi-agent systems.Journal of Artificial Intelligence Research, 4:477–507, 1996

    1996

  27. [35]

    Leading ad hoc agents in joint action settings with multiple teammates

    Noa Agmon and Peter Stone. Leading ad hoc agents in joint action settings with multiple teammates. InAAMAS, pages 341–348, 2012

    2012

  28. [36]

    Modeling uncertainty in leading ad hoc teams

    Noa Agmon, Samuel Barrett, and Peter Stone. Modeling uncertainty in leading ad hoc teams. InProceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages 397–404, 2014

    2014

  29. [37]

    To teach or not to teach? decision making under uncertainty in ad hoc teams

    Peter Stone and Sarit Kraus. To teach or not to teach? decision making under uncertainty in ad hoc teams. InProceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1, pages 117–124, 2010

    2010

  30. [38]

    A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems

    Stefano V Albrecht and Subramanian Ramamoorthy. A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. InProceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pages 1155–1156, 2013

    2013

  31. [39]

    Towards open ad hoc teamwork using graph-based policy learning

    Muhammad A Rahman, Niklas Hopner, Filippos Christianos, and Stefano V Albrecht. Towards open ad hoc teamwork using graph-based policy learning. InInternational Conference on Machine Learning, pages 8776–8786. PMLR, 2021

    2021

  32. [40]

    Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann

    Luisa M. Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforcement learning via meta-learning. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé, editors,AAMAS ’21: 20th International Conference on Autonomous Ag...

    2021

  33. [41]

    Learning latent representations to influence multi-agent interaction

    Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. InConference on robot learning, pages 575–588. PMLR, 2021

    2021

  34. [42]

    Open ad hoc teamwork with cooperative game theory

    Jianhong Wang, Yang Li, Yuan Zhang, Wei Pan, and Samuel Kaski. Open ad hoc teamwork with cooperative game theory. InInternational Conference on Machine Learning, pages 50902–50930. PMLR, 2024

    2024

  35. [43]

    A value for n-person games.Annals of Mathematics Studies, 28:307–318, 1953

    LS SHAPLEY . A value for n-person games.Annals of Mathematics Studies, 28:307–318, 1953

    1953

  36. [44]

    Multiagent model-based credit assignment for continuous control.arXiv preprint arXiv:2112.13937, 2021

    Dongge Han, Chris Xiaoxuan Lu, Tomasz Michalak, and Michael Wooldridge. Multiagent model-based credit assignment for continuous control.arXiv preprint arXiv:2112.13937, 2021

    2021 arXiv

  37. [45]

    Aligning credit for multi-agent co- operation via model-based counterfactual imagination

    Jiajun Chai, Yuqian Fu, Dongbin Zhao, and Yuanheng Zhu. Aligning credit for multi-agent co- operation via model-based counterfactual imagination. InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 281–289, 2024

    2024

  38. [46]

    Efficient multi-agent communication via shapley message value

    Di Xue, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient multi-agent communication via shapley message value. InIJCAI, pages 578–584, 2022

    2022

  39. [47]

    On the complexity of the core over coalition structures

    Gianluigi Greco, Enrico Malizia, Luigi Palopoli, Francesco Scarcello, et al. On the complexity of the core over coalition structures. InIJCAI, volume 11, pages 216–221, 2011. A Additional Background A.1λ-Return and TD(λ) We now introduce an extension of return named λ-return. ...

    2011

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.