CORD: Generalizable Cooperation via Role Diversity

Deheng Ye; Jiangxing Wang; Kanefumi Matsuyama; Kefan Su; Zongqing Lu

REVIEW 4 major objections 6 minor 1 cited by

CORD: Generalizable Cooperation via Role Diversity

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CORD claims that maximizing the entropy of role assignment under a causal constraint lets a multi-agent team cooperate with unseen teammates and team sizes without predefined types.

desk verdict Decent empirical MARL generalization paper whose causal role-decomposition framing does not survive scrutiny; read it as a KL-regularized role-diversity heuristic. read the letter →

arxiv 2501.02221 v2 pith:SEDQY4GW submitted 2025-01-04 cs.AI cs.LGcs.MA

Kanefumi Matsuyama , Kefan Su , Jiangxing Wang , Deheng Ye , Zongqing Lu This is my paper · ORCID

classification cs.AIcs.LGcs.MA

keywords multi-agentreinforcementlearningrolediversitycooperationgeneralizationhierarchicalcausalinferenceintrinsicrewardentropymaximizationadhocteamwork

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes CORD, a hierarchical multi-agent reinforcement learning method that claims to make a trained team cooperate with unseen teammates and team sizes without predefined teammate types or communication. The core idea is that a high-level controller should assign roles to low-level agents by maximizing the entropy of the role distribution under a causal constraint: the role of one agent should respond to the causal influence of other agents. The paper proves (Theorem 4.4) that this constrained entropy objective decomposes into causal influence in role and role heterogeneity, and converts both into intrinsic rewards added to the environment reward. On resource-collection and StarCraft tasks, CORD reports higher rewards and win rates than hierarchical, role-based, and communication-based baselines in generalization tests. The authors conclude that role diversity with a causal constraint, rather than plain maximum-entropy roles, is what drives generalizable cooperation.

What carries the argument

The load-bearing object is the causal graph of Assumption 4.1 and the influence vector $\bar{I}^i_t$ built by attention over other agents' keys and values conditioned on their previous actions and roles. Theorem 4.4 decomposes the entropy of the prior role distribution into mutual information between a role and the influence vector plus the entropy of the posterior role distribution; Lemma A.1 rewrites that mutual information as a sum of per-agent KL divergences against the do-operator, and Lemma A.2 turns the posterior entropy into the log-determinant of the c-related matrix $A(c)$ whose entries are symmetric KL distances between role posteriors. These two pieces are realized as the intrinsic rewards $r_c$ and $r_d$, and the whole system is trained with Q-learning on the shaped reward $r = r_e + \lambda_c r_c + \lambda_d r_d$. The determinant term is what enforces non-redundant, coherent role clusters, and the KL-from-intervention term is what makes role assignment respond to teammates rather than to a uniform prior.

What would settle it

Train CORD in an environment where the true causal effect of teammate behavior on the optimal role is known and can be intervened on (e.g., a grid task where a teammate's action deterministically changes the optimal role). If the learned $r_c$ does not track the true interventional effect — for instance, if replacing $I_0$ with any fixed vector gives the same performance, or if the posterior $P(c|\bar{I},q)$ is unchanged when teammate actions are decoupled from roles — then the causal interpretation is unsupported and the method's gains come from the KL regularization and determinant term alone.

Watch

Extended reading notes

Core claim

CORD's central claim is that generalizable cooperation in cooperative MARL can be achieved by maximizing the entropy of a role-assignment distribution subject to a causal constraint, and that this constrained objective factorizes as $H(P(c|q)) = I(c; \bar{I}|q) + H(P(c|\bar{I}, q))$. The first term, causal influence in role, is the expected KL divergence between the role posterior given another agent's influence vector and the role posterior under the do-operator, and it is meant to make role assignment responsive to teammates' behavior. The second term, role heterogeneity, is the entropy of the posterior role distribution, which under a Gaussian assumption equals a constant plus $\beta \log |A(c)|$, a determinant that measures the volume occupied by role clusters and thus penalizes redundant roles. Both terms are implemented as intrinsic rewards, $r_c$ and $r_d$, and added to the environment reward with weights $\lambda_c$ and $\lambda_d$. The resulting shaped-reward objective is trained end-to-end by extending QMIX or REFIL, and at execution the low-level policies condition only on their local observation and assigned role.

Load-bearing premise

The argument hinges on treating $do(\bar{I})$ — the average causal influence of other agents — as a constant vector $I_0$, and on the claim that the mutual information between roles and influence decomposes into per-agent KL divergences under that substitution; if substituting a constant does not actually capture a causal intervention, the theoretical decomposition reduces to a KL-regularized heuristic.

Editorial extensions

If this is right

If CORD's claim holds, a single trained team can be deployed in tasks with more or fewer agents than seen in training without retraining or predefined role taxonomies.
The decomposition gives a principled objective for role-based MARL: causal influence covers inter-agent responsiveness, role heterogeneity covers diversity, and neither alone suffices.
CORD's performance advantage over COPA and ALMA suggests that learned role assignment under a causal constraint generalizes better than coach-player strategies or predefined subtask counts.
The method works without communication or pre-existing teammate models, so it applies to settings where teammates cannot be assumed known or coordinated with.
The ablation results indicate that maximizing prior role entropy alone (MaxEnt) is insufficient; the constraint is what makes entropy useful.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the causal-influence reward truly estimates a causal effect, a natural testable extension is to replace the constant-vector intervention $I_0$ with a learned baseline or the empirical mean influence and compare generalization; the paper's theory predicts the constant is adequate, but a learned baseline could reveal whether the effect is being captured or only a correlation.
The same decomposition could be applied to other role-discovery objectives, for instance in zero-shot coordination settings with a fixed team size, where the determinant term might encourage specialization that is actually harmful; the paper's SMAC results on homogeneous maps already hint at this failure mode.
The dependence on a central controller at execution is a real architectural constraint; a decentralized variant that infers roles from local observations alone would test whether the causal-influence term can be computed without global information.
The connection between the determinant of $A(c)$ and the log-volume of role clusters suggests that role diversity could be traded off continuously against team-specialization pressure simply by tuning $\lambda_d$; one could test whether the optimal $\lambda_d$ correlates with task heterogeneity across environments.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. The empirical story is real: CORD consistently beats the listed baselines on heterogeneous generalization tasks in both resource collection and SMAC, and the ablations show the intrinsic rewards are doing something. But the paper's central theoretical claim, that the max-entropy role objective decomposes into causal influence plus role heterogeneity, is not established. The proof in Appendix A has two gaps that matter. Lemma A.1 replaces the joint role posterior with a product of marginals without justification and identifies do(Ibar) with an arbitrary constant vector, so the 'causal influence in role' term is actually a KL divergence to a fixed baseline, not a causal effect. Lemma A.2 asserts A(c)=Sigma, but A(c) is built from symmetrized KL divergences and there is no argument that it is a valid covariance matrix. The first equality in Theorem 4.4 is just the definition of conditional mutual information, so the 'decomposition' is true by construction; the real work is in the two closed forms, which are imposed rather than derived.

What is genuinely new is the specific combination of entropy-constrained role assignment, the mutual-information causal-influence reward, and the determinant-based heterogeneity term, plus the generalization experiments across varying team sizes in SMAC. The paper is honest about its limitation (the need for global information during execution) and about CORD's weakness on homogeneous maps. The code is promised in the supplementary material, which is a plus if it is actually there.

Soft spots beyond the theory: the comparisons omit RODE and HSL, the closest role-based methods; the 'built-in unseen agents' are not specified in the main text, so we do not know how strong those partners are; and the hyperparameters lambda_c and lambda_d are fixed per environment, which is fine but adds extra knobs. None of those are fatal on their own. The main issue is that the causal interpretation collapses; the method is best understood as a KL-regularized role-diversity heuristic that happens to work well.

Who is this for? People working on ad hoc teamwork and role-based MARL will get value from the empirical results and the recipe. It deserves a serious referee, but the authors should either fix the derivation or explicitly reframe the contributions as heuristic intrinsic rewards. I would not cite the causal framing, but I would cite the algorithm if I needed a role-diversity baseline.

Referee Report

4 major / 6 minor

Summary. The paper proposes CORD, a hierarchical MARL algorithm in which a high-level controller assigns roles to low-level agents and low-level agents condition their policies on assigned roles. The stated objective is to maximize the entropy of the role distribution under a causal-graph constraint. Theorem 4.4 claims this objective decomposes into a causal-influence term I(c; Ibar | q), computed as a sum of KL divergences with a do-operator (Eq. 5), and a role-heterogeneity term beta log |A(c)| + C (Eq. 6). These terms are converted into intrinsic rewards r_c and r_d and combined with the environmental reward as r = r_e + lambda_c r_c + lambda_d r_d (Eq. 11). The method is evaluated on resource collection, MPE navigation, and multi-task SMAC against hierarchical, CTDE, communication-based, and role-based baselines, with tests that scale to unseen team sizes and unseen agents.

Significance. If the theoretical decomposition were valid, the paper would make a useful contribution: a role-assignment method that handles variable team sizes and unseen teammates without predefined types or communication, while remaining end-to-end trainable. The empirical evaluation is broad (12 SMAC generalization tasks plus MPE/resource collection) and includes an ablation of the intrinsic rewards, code in the supplementary material, and five seeds per setting. However, the central theoretical derivation is not currently sound, so the contribution rests on the empirical heuristic value of the intrinsic rewards rather than on the claimed causal/entropy foundation.

major comments (4)

[Section 4, Theorem 4.4 and Appendix A, Lemma A.1] The first equality in Theorem 4.4, H(P(c|q)) = I(c; Ibar | q) + H(P(c | Ibar, q)), is the definition of conditional mutual information and is therefore true by construction; all content lies in the closed forms (5) and (6). In Lemma A.1 the proof assumes the factorizations P(c|Ibar,q) = prod_i P(c_i|Ibar_i,q_i) and P(c|q) = prod_i P(c_i|q_i) without stating them as assumptions, and these do not follow from Definition 4.2 or Assumption 4.1. Without an explicit conditional-independence assumption, the sum over i in Eq. (5) is unsupported.
[Section 4, Eq. (5), Eq. (8), and Appendix A, Lemma A.1] The proof equates P(c_i|q_i) with P(c_i|do(Ibar_i),q_i) and then states that 'do(Ibar_i)' can be substituted with a constant vector. This conflates conditioning with intervention. Ibar_i is an attention output, i.e., a deterministic function of other agents' variables, not an intervention variable with a well-defined do-set; an arbitrary constant I0 in Eq. (8) is not the average of possible interventions and is not identifiable from the causal graph. Because I0 and lambda_c are hand-tuned, r_c is a KL divergence to a fixed baseline rather than an estimate of any causal effect.
[Section 4, Lemma A.2 and Eq. (6)] The proof asserts A(c) = Sigma with only a citation to Gaussian-process covariance. No argument is given that the matrix with entries exp(-(KL(P_i||P_j)+KL(P_j||P_i))) is positive semidefinite; symmetrized KL is not a metric in general, and the exponential kernel is PSD only for conditionally negative definite distances. If A(c) is not PSD, log|A(c)| is not a Gaussian differential entropy, and the derivation of the role-heterogeneity reward r_d in Eqs. (9)-(10) collapses. This is load-bearing because Eq. (6) is one of the two terms in the claimed decomposition.
[Section 5, Tables 2 and 5] The text's summary statements that CORD 'consistently surpasses baselines' and demonstrates 'superior generalization' are stronger than the tables support. In Table 5, ALMA is the best method on 2m, 8m, and 2csz, and in the 5m unseen-agent test CORD's win rate is comparable to, not clearly better than, the best baseline. Several advantages are within one standard deviation (e.g., Table 2, 2MMM: CORD 0.288 +/- 0.085 vs REFIL 0.254 +/- 0.034; Table 5, 8m: CORD 0.655 +/- 0.013 vs REFIL 0.656 +/- 0.090). Reporting significance tests or confidence intervals would make the empirical claim precise.

minor comments (6)

[Section 2] The sentence 'ROMA does not character more complex inter-agent interactions in environments' contains a typo: 'character' should be 'characterize'.
[Section 4, Eq. (1) vs Theorem 4.4] Eq. (1) maximizes H(P(c)), while Theorem 4.4 concerns H(P(c|q)); the relationship between the prior P(c) and the conditional P(c|q) is never formalized.
[Section 4, discussion after Theorem 4.4] The phrase 'do(Ibar_i) refers to the expectation of all possible interventions' is not standard do-calculus terminology; the paper should define this operation formally rather than citing Pearl (2009) for a nonstandard notion.
[Section 5 and Appendix C] The coefficients lambda_c and lambda_d are fixed by environment (0.001 in MPE/resource collection, 0.0025 in SMAC) with no sensitivity analysis, and the constant vector I0 used in Eq. (8) is never specified in the experimental details.
[Tables 1 and 3] The unseen-agent result for CORD in the 6-agent resource-collection task is reported as 166.334 +/- 2.65 in Table 1 and as 166.334 +/- 7.96 in Table 3; the standard-error reporting should be consistent.
[Impact Statements] The final 'Impact Statements' paragraph is boilerplate and adds no substantive information to the manuscript.

Circularity Check

3 steps flagged · score 6.0 of 10

Theorem 4.4's role-entropy decomposition is partly the definition of conditional mutual information, and Eq. (8) reduces the 'causal inference in role' term to a KL divergence against an arbitrary constant baseline.

self definitional [Appendix A, proof of Theorem 4.4, Eq. (4)]
"First, given the definitions of mutual information and entropy, the equation (4) is valid."

Equation (4), H(P(c|q)) = I(c; Ibar|q) + H(P(c| Ibar, q)), is exactly the definition of conditional mutual information, I(X;Y|Z) = H(X|Z) - H(X|Y,Z), with a sign rearrangement. Presenting this identity as the decomposition of the constrained role entropy contributes no content about roles or causality; the claimed decomposition is true by definition rather than by derivation.
ansatz smuggled in via citation [Section 4, 'Causal Inference in Role', Eq. (8)]
"Moreover, we substitute I 0, a constant vector, for do( ¯I i t ) to represent the intervention in causal inference theory (Pearl 2009). Therefore, the causal inference in role can be rewritten as follows, and we maximize it by taking it as an intrinsic reward,"

Equation (8) defines r_c = (1/M) Σ Σ DKL[P(c_i|Ibar_i,q_i) || P(c_i|I0,q_i)]. This is by construction a KL divergence to a fixed baseline vector I0, not an estimate of the interventional distribution P(c_i|do(Ibar_i),q_i). Since I0 is an arbitrary constant, changing it changes the intrinsic reward and the learned policy, so the term is not identifiable as a causal effect. The causal interpretation is imported from Pearl/Jaques but the implemented objective is a heuristic KL regularizer.

1 more flagged steps

self definitional [Appendix A, Lemma A.1 proof]
"DKL[P(c| ¯I, q)||P(c|q)] = ∫ P(c| ¯I, q) log QN i=1 P(ci| ¯I i, qi) QN i=1 P(ci|qi) dc"

The proof's first displayed equality inserts the factorization P(c|Ibar,q) = ∏_i P(c_i|Ibar_i,q_i) and P(c|q) = ∏_i P(c_i|q_i) into the log ratio. That conditional independence across agents is not stated in Theorem 4.4 and is not derived; consequently the per-agent sum in Eq. (5) is true by construction inside the proof rather than a consequence of the causal graph. The later replacement of P(c_i|q_i) by P(c_i|do(Ibar_i),q_i) is likewise imposed rather than derived.

full rationale

The paper's empirical evaluation is self-contained against baselines and could stand as evidence for a role-diversity heuristic. However, the theoretical result claimed as the foundation of CORD is not independently derived. Eq. (4) is the definition of conditional mutual information, so the 'decomposition' has no causal or role-content by itself. Eq. (5)/(8), the causal-inference term, is reduced by construction to DKL[... || P(c_i|I0,q_i)] with an arbitrary constant I0, making it a KL regularizer; the Pearl do-operator citation does not license this substitution. Lemma A.1 obtains the per-agent sum by assuming a factorization of the joint posterior that is absent from the theorem statement. Lemma A.2 asserts A(c)=Σ rather than proving the determinant entropy relation. These gaps mean the 'causal inference in role' and 'role heterogeneity' labels are imposed, not consequences of the constrained maximum-entropy objective. Because the central decomposition partly reduces to a definition and an arbitrary baseline, the circularity score is 6; the method may work empirically, but the claimed derivation does not support the interpretation.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The core theorem rests on an entropy identity that is standard, but the two closed-form expressions for the decomposed terms rely on ad hoc assumptions: factorization of the role posterior, replacement of do(Ibar) with a constant vector, and identifying the similarity matrix with the covariance. The final algorithm also introduces hand-tuned coefficients lambda_c and lambda_d.

free parameters (2)

lambda_c and lambda_d = 0.001 (MPE), 0.0025 (SMAC)
Weights for the causal-influence and role-heterogeneity intrinsic rewards in Eq. 11; hand-tuned per environment.
I0 (constant vector replacing do(Ibar)) = not specified
The constant vector used as the intervention baseline in Eq. 8; its value is not reported, so the causal-effect reward is underdetermined.

assumptions (5)

ad hoc to paper P(c|Ibar,q) = product over i of P(c_i|Ibar_i,q_i) and P(c|q) = product over i of P(c_i|q_i)
Used in Lemma A.1 to factorize the joint KL into a sum of per-agent KL terms; not implied by Gaussianity, which allows correlated roles.
ad hoc to paper do(Ibar) equals a constant vector (average influence)
Stated after Theorem 4.4: 'Practically, do(bar-I_i) can be substituted with a constant vector.' This is not Pearl's do-calculus and is a heuristic.
ad hoc to paper A(c) = exp(-d_ij) is a valid covariance matrix of the Gaussian role posterior
Lemma A.2 asserts A(c)=Sigma with no proof that the kernel matrix is positive semidefinite or equals the covariance.
domain assumption Causal Graph Assumption (Assumption 4.1): other agents' current actions and roles do not influence agent i's current role; only past actions and roles do
This is a modeling choice about the Dec-POMDP and role dynamics; it is plausible but not validated.
standard math Markov property of Dec-POMDP
Standard assumption in the Dec-POMDP background, Section 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CORD: Generalizable Cooperation via Role Diversity." pith.science (2026). https://pith.science/paper/SEDQY4GW

@misc{pith2026250102221,
  author       = {Pith},
  title        = {Pith review of: CORD: Generalizable Cooperation via Role Diversity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SEDQY4GW}},
  note         = {Machine review of arXiv:2501.02221}
}

read the original abstract

Cooperative multi-agent reinforcement learning (MARL) aims to develop agents that can collaborate effectively. However, most cooperative MARL methods overfit training agents, making learned policies not generalize well to unseen collaborators, which is a critical issue for real-world deployment. Some methods attempt to address the generalization problem but require prior knowledge or predefined policies of new teammates, limiting real-world applications. To this end, we propose a hierarchical MARL approach to enable generalizable cooperation via role diversity, namely CORD. CORD's high-level controller assigns roles to low-level agents by maximizing the role entropy with constraints. We show this constrained objective can be decomposed into causal influence in role that enables reasonable role assignment, and role heterogeneity that yields coherent, non-redundant role clusters. Evaluated on a variety of cooperative multi-agent tasks, CORD achieves better performance than baselines, especially in generalization tests. Ablation studies further demonstrate the efficacy of the constrained objective in generalizable cooperation.

Figures

Figures reproduced from arXiv: 2501.02221 by the authors.

**Figure 2.** Overview of CORD framework. The purple module is the high-level controller network. The blue modules represent [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Episode rewards on training and generalization to [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Win rates on training and generalization to [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Win rates on training and generalization to [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning
cs.MA 2025-11 conditional novelty 6.0 of 10

ACC-MARL trains decentralized multi-agent policies that solve many automaton-specified cooperative tasks at once, with a proof of optimality for the Markovian reformulation and value-based task assignment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages · cited by 1 Pith paper

[6]

MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning

Su, K., Zhou, S., Gan, C., Wang, X., and Lu, Z. MA2QL: A minimalist approach to fully decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2209.08244,

work page Pith review arXiv
[7]

Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning

Sukhbaatar, S., Denton, E., Szlam, A., and Fergus, R. Learn- ing goal embeddings via self-play for hierarchical reinforce- ment learning. arXiv preprint arXiv:1811.09083,

work page Pith review arXiv
[11]

First, given the definitions of mutual information and entropy, the equation (4) is valid

A Proof of Theorem 4.4 Proof. First, given the definitions of mutual information and entropy, the equation (4) is valid. We then prove that the equation (5) is valid. As shown in Lemma A.1, the mutual information between roles and influence from other agents can be estimated using the do operator (Pearl 2009). Lemma A.1. Suppose that N agents satisfy the ...

work page 2009
[13]

We use the original code of COPA (Liu et al

with the MIT license. We use the original code of COPA (Liu et al. 2021). For ALMA (Iqbal et al. 2022), we integrated the code of the resource collection environment and set the number of subtasks to one. ROMA (Wang et al. 2020b) is adapted by migrating entity-based environment code into the algorithm, enabling ROMA to accommodate tasks with varying numbe...

work page 2021
[256]

Three fully connected layers are used to encode observation, action, and entity information into three 128-dimensional vectors respectively

The controller network of CORD contains 8 layers. Three fully connected layers are used to encode observation, action, and entity information into three 128-dimensional vectors respectively. One multi-head attention layer with 4 heads takes observation embedding vectors as input and outputs the 128-dimensional hidden vector of the global information. One ...

work page
[2010]

Decentralized Policy Optimization

Su, K. and Lu, Z. Decentralized policy optimization. arXiv preprint arXiv:2211.03032,

work page Pith review arXiv
[2012]

Multi-agent re- inforcement learning with shared resources for inventory management

Ding, Y ., Feng, M., Liu, G., Jiang, W., Zhang, C., Zhao, L., Song, L., Li, H., Jin, Y ., and Bian, J. Multi-agent re- inforcement learning with shared resources for inventory management. arXiv preprint arXiv:2212.07684,

arXiv
[2017]

Neural machine translation by jointly learning to align and translate

Bahdanau, D., Cho, K., and Bengio, Y . Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473,

arXiv

Show all 13 references

[2018]

Reinforcement learning with task decomposition for cooperative multiagent systems.IEEE transactions on neural networks and learning systems, 32(5): 2054–2065,

Sun, C., Liu, W., and Dong, L. Reinforcement learning with task decomposition for cooperative multiagent systems.IEEE transactions on neural networks and learning systems, 32(5): 2054–2065,
[2020]

M., Zam- baldi, V ., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J

Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zam- baldi, V ., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous A...
[2021]

Generalization in coopera- tive multi-agent systems

Mahajan, A., Samvelyan, M., Gupta, T., Ellis, B., Sun, M., Rockt¨aschel, T., and Whiteson, S. Generalization in coopera- tive multi-agent systems. arXiv preprint arXiv:2202.00104,

arXiv
[2022]

Dwiel, Z., Candadai, M., Phielipp, M., and Bansal, A. K. Hierarchical policy learning is sensitive to goal space design. arXiv preprint arXiv:1905.01537,

1905 arXiv
[2023]

E., and Tian, Y

Zhang, T., Xu, H., Wang, X., Wu, Y ., Keutzer, K., Gonzalez, J. E., and Tian, Y . Multi-agent collaboration via reward attribution decomposition. arXiv preprint arXiv:2010.08531,

2010 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [6]

MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning

Su, K., Zhou, S., Gan, C., Wang, X., and Lu, Z. MA2QL: A minimalist approach to fully decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2209.08244,

work page Pith review arXiv

[2] [7]

Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning

Sukhbaatar, S., Denton, E., Szlam, A., and Fergus, R. Learn- ing goal embeddings via self-play for hierarchical reinforce- ment learning. arXiv preprint arXiv:1811.09083,

work page Pith review arXiv

[3] [11]

First, given the definitions of mutual information and entropy, the equation (4) is valid

A Proof of Theorem 4.4 Proof. First, given the definitions of mutual information and entropy, the equation (4) is valid. We then prove that the equation (5) is valid. As shown in Lemma A.1, the mutual information between roles and influence from other agents can be estimated using the do operator (Pearl 2009). Lemma A.1. Suppose that N agents satisfy the ...

work page 2009

[4] [13]

We use the original code of COPA (Liu et al

with the MIT license. We use the original code of COPA (Liu et al. 2021). For ALMA (Iqbal et al. 2022), we integrated the code of the resource collection environment and set the number of subtasks to one. ROMA (Wang et al. 2020b) is adapted by migrating entity-based environment code into the algorithm, enabling ROMA to accommodate tasks with varying numbe...

work page 2021

[5] [256]

Three fully connected layers are used to encode observation, action, and entity information into three 128-dimensional vectors respectively

The controller network of CORD contains 8 layers. Three fully connected layers are used to encode observation, action, and entity information into three 128-dimensional vectors respectively. One multi-head attention layer with 4 heads takes observation embedding vectors as input and outputs the 128-dimensional hidden vector of the global information. One ...

work page

[6] [2010]

Decentralized Policy Optimization

Su, K. and Lu, Z. Decentralized policy optimization. arXiv preprint arXiv:2211.03032,

work page Pith review arXiv

[7] [2012]

Multi-agent re- inforcement learning with shared resources for inventory management

Ding, Y ., Feng, M., Liu, G., Jiang, W., Zhang, C., Zhao, L., Song, L., Li, H., Jin, Y ., and Bian, J. Multi-agent re- inforcement learning with shared resources for inventory management. arXiv preprint arXiv:2212.07684,

arXiv

[8] [2017]

Neural machine translation by jointly learning to align and translate

Bahdanau, D., Cho, K., and Bengio, Y . Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473,

arXiv