Pith. sign in

REVIEW 2 major objections 5 minor 67 references

Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Belief-modelling agents beat state-of-the-art MARL.

desk verdict SMPE2 is a solid empirical MARL paper whose 'adversarial exploration' label overstates what the losses actually implement; the method still deserves a serious referee and likely acceptance after the mechanism claim is reined in. read the letter →

arxiv 2505.05262 v2 pith:ZJOEMNR2 submitted 2025-05-08 cs.LG cs.AIcs.MA

Andreas Kontogiannis , Konstantinos Papathanasiou , Yi Shen , Giorgos Stamou , Michael M. Zavlanos , George Vouros This is my paper · ORCID
classification cs.LGcs.AIcs.MA
keywords cooperativemulti-agentreinforcementlearningpartialobservabilitystatemodellingvariationalinferenceagentcount-basedintrinsicexplorationsparse-rewardactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that cooperative multi-agent reinforcement learning under partial observability can be substantially improved if each agent learns a compressed, probabilistic belief about the non-observable joint state, filters that belief to keep only information relevant to its own value, and uses it both as an extra input to its policy and as the space for intrinsic exploration. The authors claim that combining variational inference over filtered reconstruction targets with SimHash novelty rewards in belief space lets distributed agents coordinate in sparse-reward tasks without communication and without prior knowledge of what observation features mean. If true, any actor-critic MARL learner could be upgraded with the same state-modelling and exploration mechanism, with consequences for robotic swarms, warehouse logistics, and search-and-rescue.

What carries the argument

The central object is the per-agent variational encoder-decoder with agent-modelling filters w_i: each agent learns a latent belief z_i ~ q_omega_i(z_i | o_i) that is trained to reconstruct w_i·o_-i, with w_i produced by a sigmoid MLP so that non-informative features are down-weighted. A second critic V_k(hat-s) on the filtered state ties the belief to policy optimization, and the adversarial exploration uses SimHash counts of z_i to give intrinsic reward for novel belief states. Together these components make the belief informative, policy-relevant, and explorative, and they are what carries the claimed performance gains.

What would settle it

Run SMPE2 on a cooperative task engineered so that each agent's observation is independent random noise carrying no information about the other agents' observations or the underlying state; if the method still improves over the MAA2C backbone, the reconstruction loss must be learning something other than state information, while if it collapses to backbone performance the state-modelling assumption is confirmed as load-bearing. A quantitative complement is to measure the filtered reconstruction error of o_-i against a chance-level predictor under such constructed noise.

Watch

Extended reading notes

Core claim

Each agent i runs a variational encoder-decoder that maps its own observation o_i to a latent belief z_i, and is trained to reconstruct only the informative features of the other agents' observations o_-i, where per-feature importance weights w_i are learned alongside. A second critic conditioned on the filtered state makes w_i and z_i optimize the value function rather than reconstruction accuracy alone, and a SimHash counter on z_i supplies intrinsic rewards that push the agent to discover novel observations, which in turn become novel training targets for the other agents' decoders. The paper reports that SMPE2 outperforms MAA2C, COMA, MAPPO, ATM, EOI, EMC, and MASER on MPE, LBF, and RWARE tasks, including LBF tasks previously flagged as open challenges.

Load-bearing premise

The whole construction assumes that an agent's own observation contains enough statistical information to reconstruct, after filtering, the features of other agents' observations that matter for coordination; if individual observations are too weak or ambiguous, the belief z_i cannot encode useful state information and both the policy enhancement and belief-space exploration collapse.

Editorial extensions

If this is right

  • SMPE2 outperforms state-of-the-art MARL algorithms on complex fully cooperative tasks in the MPE, LBF, and RWARE benchmarks.
  • The ablation studies show that removing the agent-modelling filters, the KL term, the L2 normalization on w_i, or the second critic degrades performance, so each component of the method is load-bearing.
  • When the KL regularization is enabled, agents' belief embeddings become cohesive across agents, as shown by t-SNE visualization and by a logistic-regression separation accuracy that drops from 99.3% to 57.5%.
  • SMPE2 is backbone-flexible: using MAPPO as the actor-critic backbone in place of MAA2C still yields large gains over plain MAPPO.
  • SMPE2 manages to solve sparse-reward LBF tasks that prior work highlighted as open challenges, where other intrinsic-motivation methods fail.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The interpretability of the learned filters w_i suggests that the method could double as a debugging tool, revealing which local features each agent actually relies on for coordination.
  • The adversarial-exploration argument points toward a cooperative game-theoretic reading in which each agent's exploration deliberately creates harder training targets for the others; formalizing that as a min-max or influence objective is a natural extension the paper does not develop.
  • Because z_i is conditioned only on the current observation, the intrinsic reward emphasizes observation novelty over trajectory novelty; in tasks where high-value states are reached only through long action sequences this may under-reward gradual progress, which is a testable limitation.
  • The good performance in most experiments is obtained with shared policy parameters, so whether SMPE2's gains survive in fully heterogeneous teams with separate networks remains an open empirical question that the paper's backbone-flexibility result only partially answers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes SMPE2, a cooperative multi-agent reinforcement learning method for partially observable, communication-free settings. Each agent learns a variational encoder-decoder that maps its own observation o_i to a latent belief z_i and reconstructs filtered observations of other agents, with learned per-agent weight vectors w_i intended to suppress non-informative features. The belief z_i is fed into the agent's policy, and a count-based intrinsic reward is computed by applying SimHash to z_i to encourage exploration in belief space. SMPE2 is built on the MAA2C algorithm and is evaluated on MPE, LBF, and RWARE benchmarks against several baselines, reporting improved episodic rewards. Ablations examine the contribution of the filters, the KL regularization, the norm loss, the second critic, and the intrinsic reward, and the appendix documents hyperparameters, running times, and an analysis of the learned filters.

Significance. If the reported empirical results are reliable, the paper offers a practically useful and simple recipe for improving actor-critic MARL under partial observability: augment the policy with learned latent beliefs and use the same beliefs for count-based exploration. The method does not require communication or prior knowledge of observation semantics, and the provided code and running-time comparisons are useful. However, the advertised 'adversarial exploration' mechanism, which is central to the title and abstract, is not actually implemented in the objective: the intrinsic reward for an agent depends only on its own belief, with no coupling to other agents' reconstruction losses. The empirical gains are likely real, but the mechanistic claim is unsupported and the theoretical framing (Proposition 2.1) is a trivial equivalence.

major comments (2)
  1. [Section 3.2 and Algorithm 1] The paper's headline claim of an 'adversarial type of exploration' is not reflected in the algorithm. The intrinsic reward r_hat_i = 1/sqrt(n(SH(z_i))) depends only on agent i's own belief z_i, which is a deterministic function of its observation o_i. No term in the total loss L_SMPE (Section 3.1) involves another agent's reconstruction loss L_rec^j, and Algorithm 1 (line 16) computes the intrinsic reward per agent without any cross-agent term. The statement that novel observations 'adversarially increase the losses of other agents' reconstruction models' is a causal claim about training dynamics, but SMPE2 optimizes no objective that rewards such increases. The ablation in Figure 5 comparing belief-space hashing ('intr') with observation-space hashing ('obs rew') tests the hash domain, not adversarialness. To support the advertised mechanism, the authors should either (a) introduce an explicit coupling, e.g., an intrinsic reward that depends on other agents' reconstruction errors, or (b) provide direct evidence, e.g., measure that states discovered by one agent cause measurable increases in other agents' L_rec, and show that this correlates with performance, or (c) rewrite the abstract and Section 3.2 to describe the contribution as belief-based count-based exploration without the 'adversarial' label. As it stands, the central novelty claim is overstated, and the reported improvements are consistent with standard count-based exploration plus belief-conditioned policies.
  2. [Section 2.2, Proposition 2.1] Proposition 2.1 is a trivial equivalence: adding a latent variable z that the policy class is free to ignore cannot shrink the set of achievable value functions, and the proof correctly observes that any Dec-POMDP policy can be represented by a policy that ignores z. The subsequent interpretation, that this 'allows us to explore how agents can form meaningful beliefs' that 'enhance their own policies', is a non sequitur. Optimality of the unconstrained objective says nothing about whether the implemented heuristic—adding λ·ELBO (Eq. 2) with a fixed λ and training w_i through a separate critic loss (Eq. 6)—yields policies that are near-optimal for V_SM. As written, the connection from the theoretical objective to the loss L_encodings (Eq. 7) is heuristic. The authors should either weaken the theoretical framing (e.g., present Proposition 2.1 only as an equivalence observation and describe the ELBO as a practical auxiliary objective) or provide a formal statement with assumptions under which the auxiliary objective preserves value optimality.
minor comments (5)
  1. [Appendix D] The heading 'Missing Proof' should be renamed to 'Proof of Proposition 2.1' or simply 'Proposition 2.1 Proof'.
  2. [Appendix E.4.8] Table 1 reports the accuracy for 'zi (without wi)' as 80.8%, while the text in the same subsection says 'or high accuracy (80.3%)'; these numbers should be reconciled.
  3. [Equation (3)] The notation e_w_i (the target for the filter weights) is used but never defined in the main text; it is only implicitly introduced in Algorithm 1 as the target weight network (phi^w_i)'. Please define it near Eq. (3).
  4. [Section 3.2] Typographical issues: 'By doing do' should be 'By doing so', and in Section 4.2 'The only exception on this is' should be 'The only exception to this is'.
  5. [Abstract and Section 1] The method is referred to as both 'SMPE' and 'SMPE2'; please use 'SMPE2' consistently as the algorithm name, and reserve 'SMPE' for the no-intrinsic-reward ablation variant.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: SMPE2's state-modelling objective is a re-parameterization of the Dec-POMDP objective, and the empirical comparisons are external; the 'adversarial exploration' label is an unverified interpretation rather than a construction-level circularity.

full rationale

The derivation chain is not circular. The state-modelling objective in Eq. (1) is a re-parameterization of the Dec-POMDP objective: because z_i is sampled from p_omega(·|o_i), a distribution over the agent's own observation, the induced policy pi_bar_i(a_i|h_i) = E_z[pi_psi_i(a_i|h_i,z)] is an ordinary Dec-POMDP policy, so the outer max over omega cannot exceed V*, and any base policy is recovered by letting pi_psi_i ignore z. Proposition 2.1 is therefore an identity used for motivation, not a fitted input that later reappears as a predicted result. The ELBO, filter, and critic losses are auxiliary training objectives; no benchmark curve is produced by evaluating a fitted parameter on the quantity from which it was estimated. The one potentially misleading element is the label 'adversarial exploration': Section 3.2 defines r_hat_i = 1/sqrt(n(SH(z_i))) with no term depending on other agents' reconstruction losses, so the claimed adversarial coupling is an unsupported interpretation rather than a circular reduction. Self-citations (Papadopoulos et al., 2025; Kontogiannis & Vouros, 2023) are used for motivation and to describe benchmark difficulty; they do not carry the algorithm's correctness or the empirical comparison, which is against external baselines. Score 1 reflects the minor, non-load-bearing self-citation usage and the unverified mechanism label, not any circular derivation.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

No new physical or ontological entities are introduced. The belief z_i is a standard variational latent variable and the AM filters w_i are learnable weights. The ledger lists benchmark-specific hyperparameters and the domain assumptions behind the reconstruction and exploration objectives.

free parameters (4)
  • lambda_rec = 0.5 (LBF), 1 (MPE/RWARE)
    Weights the reconstruction loss in L_encodings; chosen per benchmark in Table 3.
  • lambda_norm = 0.1 (MPE), 1 (LBF), 0.1 (RWARE)
    Weights the filter norm regularization to prevent w_i from collapsing to zero; benchmark-specific in Table 3.
  • beta = 0.1 (LBF), 0.001 (RWARE)
    Coefficient of the intrinsic reward; set to zero for MPE where intrinsic rewards are not used.
  • latent_dim = 32 (MPE), 64 (LBF/RWARE)
    Dimensionality of the belief embedding z_i; chosen per benchmark in Table 3.
assumptions (6)
  • standard math Dec-POMDP formulation and optimal value function V* are standard definitions.
    Background for the state modelling objective in Section 2.1.
  • standard math Variational inference ELBO and the reparameterization trick justify the reconstruction loss.
    Used in Section 3.1 for the encoder-decoder.
  • domain assumption Joint observations provide sufficient evidence to discriminate between states.
    Stated in Section 3.1; needed for reconstruction of o_-i from o_i to be meaningful.
  • domain assumption Features of o_-i that are irrelevant to agent i's reward or cannot be inferred through z_i are non-informative and can be filtered.
    Defined in Section 2.2; justifies the AM filter mechanism.
  • domain assumption Count-based novelty in belief space is a good proxy for discovering novel high-value states.
    Section 3.2 adopts SimHash on z_i; no formal justification is given.
  • domain assumption Discoveries by one agent produce harder reconstruction targets that improve other agents' belief models.
    Section 3.2 labels this adversarial exploration, but no loss term depends on other agents' reconstruction losses, so this is an emergent assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration." pith.science (2026). https://pith.science/paper/ZJOEMNR2

@misc{pith2026250505262,
  author       = {Pith},
  title        = {Pith review of: Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZJOEMNR2}},
  note         = {Machine review of arXiv:2505.05262}
}
read the original abstract

Learning to cooperate in distributed partially observable environments with no communication abilities poses significant challenges for multi-agent deep reinforcement learning (MARL). This paper addresses key concerns in this domain, focusing on inferring state representations from individual agent observations and leveraging these representations to enhance agents' exploration and collaborative task execution policies. To this end, we propose a novel state modelling framework for cooperative MARL, where agents infer meaningful belief representations of the non-observable state, with respect to optimizing their own policies, while filtering redundant and less informative joint state information. Building upon this framework, we propose the MARL SMPE algorithm. In SMPE, agents enhance their own policy's discriminative abilities under partial observability, explicitly by incorporating their beliefs into the policy network, and implicitly by adopting an adversarial type of exploration policies which encourages agents to discover novel, high-value states while improving the discriminative abilities of others. Experimentally, we show that SMPE outperforms state-of-the-art MARL algorithms in complex fully cooperative tasks from the MPE, LBF, and RWARE benchmarks.

Figures

Figures reproduced from arXiv: 2505.05262 by the authors.

Figure 1
Figure 1. Overview of SMPE2 : SMPE2 is built on top of the MAA2C algorithm (Papoudakis et al., 2020). Each agent’s actor (blue box) has partial observability. Conditioned on agent’ own observation o i , it reconstructs other agents’ observations o −i using a variational encoder-decoder (ED) which infers a belief state embedding z i . To filter non-informative joint state information and learn only from informative features, t… view at source ↗
Figure 2
Figure 2. Results on the MPE benchmark [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Results on the LBF benchmark superior performance compared to the SOTA methods across all three tasks. These RWARE tasks are particularly challenging because they require: (1) effective exploration strategies to deal with sparse rewards; that is only very spe￾cific sequences of actions result to positive rewards (first load a specific shelf and then unload into an empty shelf), (2) agents to coordinate so that they … view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Results on the RWARE benchmark [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Ablation study: (up) Validating component selection: SMPE2 without intrinsic rewards (no intr), AM filters (no filters), LKL (no kl), Lnorm (no L2 norm), and with standard SimHash (Tang et al., 2017) replacing our exploration schema (obs rew). (down) Learning w i (and …
Figure 6
Figure 6. Figure 6: SMPE2 against a custom implementation of the multi-agent extension of LIAM (Papoudakis et al., 2021) and SIDE (Xu et al., 2022) [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: t-SNE visualization of the state modelling embedding z i (for the three agents; agent 1: red, agent 2: green, agent 3: blue) in an LBF task at the 45th time step; (left): SMPE2 , (right): SMPE2 (no kl) for ablation study with LKL = 0. for efficient credit assignment (Z…
Figure 8
Figure 8. Figure 8: Results on Double Speaker Listener 14 [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Spread: A cooperative navigation task This fully cooperative environment involves N agents and N landmarks, with the primary goal of enabling agents to learn how to cover all landmarks while avoiding collisions. A key challenge is balancing global and local incentives:…
Figure 10
Figure 10. Figure 10: Double Speaker-Listener: A cooperative communication task This fully cooperative environment features a Speaker and a Listener, where the Speaker is responsible for effectively communicating information about a target landmark while the Listener navigates based on the…
Figure 11
Figure 11. Figure 11: Level-based Foraging practical applications in multi-robot collaboration, resource management in supply chains, and coordination during disaster response efforts. The environment is depicted in Section 4.2. • Observation space: All agents receive triplets of the form …
Figure 12
Figure 12. Figure 12: Multi-Robot Warehouse (RWARE) agents to achieve their objectives. The insights gained from RWARE have meaningful applications in logistics and warehouse management, where multiple autonomous robots must work together to optimize inventory handling and order fulfillmen…
Figure 13
Figure 13. Figure 13: Comparison of SMPE2 to SMPE(no intr) on the three most difficult evaluated LBF tasks E.4.2. FURTHER RESULTS ON THE COMPARISON OF SMPE2 TO MLIAM In this subsection of the Appendix, we provide more results on the comparison of our SMPE2 with MLIAM in [PITH_FULL_IMAGE:f…
Figure 14
Figure 14. Figure 14: Further ablation study results on the comparison between SMPE and MLIAM on MPE and LBF 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Ablation study on why we need the second critic model: Baseline SMPE2 (no standard critic) is our method using the critic k, that is the one conditioned on w i , also for providing the advantages needed to train the actor. E.4.4. ABLATION STUDY ON λREC [PITH_FULL_IMA…
Figure 16
Figure 16. Figure 16: illustrates the importance of λrec to the contribution of the reconstruction loss to the overall objective in a Spread task. As can be clearly seen from the plot, lowering the value of λrec worsens both convergence and average episodic reward in this task. We attribut…
Figure 17
Figure 17. Figure 17: Ablation study on λnorm E.4.6. RESULTS ON LEARNING THE STATE BELIEF REPRESENTATIONS WITH RESPECT TO POLICY OPTIMIZATION In this subsection of the Appendix, we provide more results on the significance to learn the state belief representations with respect to policy opt…
Figure 18
Figure 18. Figure 18: Ablation study on learning the state belief representations with respect to policy optimization E.4.7. STUDYING THE DYNAMIC NATURE OF THE INTRINSIC REWARD In this subsection of the Appendix, we examine the dynamic nature of the intrinsic rewards proposed in SMPE2 . As…
Figure 19
Figure 19. Figure 19: Ablation study: Studying the dynamic nature of the intrinsic reward Embeddings Accuracy z i 57.5% z i (without KL) 99.3% z i (without w i ) 80.8% [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: SMPE2 [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]
Figure 22
Figure 22. Figure 22: SMPE2 is flexible: SMPE PPO against its backbone algorithm MAPPO E.4.10. FURTHER ABLATION STUDY ON THE EXPRESSIVENESS OF z i : LOSS CURVES 0.0 0.2 0.4 0.6 0.8 1.0 Steps 1e7 0 50 100 150 200 250 Self-Supervised Reconstruction Loss lbf-2s-9x9-3p-2f-coop [PITH_FULL_IMAG…
Figure 23
Figure 23. Figure 23: Self-supervised reconstruction loss on LBF 0.0 0.2 0.4 0.6 0.8 1.0 Steps 1e7 250 500 750 1000 1250 1500 1750 2000 Self-Supervised Reconstruction Loss Spread-8 [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]
Figure 25
Figure 25. Figure 25: Ablation study: Goal: The optimal policy (found by SMPE2 ) is all players to eat the right food. Agent1: Does not know where both the right food and agent3 are located, or their levels, and seeks to find them. Moreover, it seeks to find if Agent2 sees the right food. …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 54 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Multi-robot cooperation in the martha project

    Alami, R., Fleury, S., Herrb, M., Ingrand, F., and Robert, F. Multi-robot cooperation in the martha project. IEEE Robotics & Automation Magazine, 5 0 (1): 0 36--47, 1998

    work page 1998

  3. [3]

    Albrecht, S. V. and Stone, P. Reasoning about hypothetical agent behaviours and their parameters. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pp.\ 547--555, 2017

    work page 2017

  4. [4]

    Shared experience actor-critic for multi-agent reinforcement learning

    Christianos, F., Sch \"a fer, L., and Albrecht, S. Shared experience actor-critic for multi-agent reinforcement learning. Advances in neural information processing systems, 33: 0 10707--10717, 2020

    work page 2020

  5. [5]

    A., and Albrecht, S

    Christianos, F., Papoudakis, G., Rahman, M. A., and Albrecht, S. V. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning, pp.\ 1989--1998. PMLR, 2021

    1989

  6. [6]

    Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I

    Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp.\ 122--130, 2018 a

    work page 2018

  7. [7]

    Counterfactual multi-agent policy gradients

    Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018 b

    work page 2018

  8. [8]

    Greedy when sure and conservative when uncertain about the opponents

    Fu, H., Tian, Y., Yu, H., Liu, W., Wu, S., Xiong, J., Wen, Y., Li, K., Xing, J., Fu, Q., et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning, pp.\ 6829--6848. PMLR, 2022

    work page 2022

Show all 67 references
  1. [9]

    Efficient multi-agent communication via self-supervised information aggregation

    Guan, C., Chen, F., Yuan, L., Wang, C., Yin, H., Zhang, Z., and Yu, Y. Efficient multi-agent communication via self-supervised information aggregation. Advances in Neural Information Processing Systems, 35: 0 1020--1033, 2022 a

    2022

  2. [10]

    Efficient multi-agent communication via self-supervised information aggregation

    Guan, C., Chen, F., Yuan, L., Wang, C., Yin, H., Zhang, Z., and Yu, Y. Efficient multi-agent communication via self-supervised information aggregation. Advances in Neural Information Processing Systems, 35: 0 1020--1033, 2022 b

    2022

  3. [11]

    and Kahou, S

    Gupta, N. and Kahou, S. E. Cammarl: Conformal action modeling in multi agent reinforcement learning. arXiv preprint arXiv:2306.11128, 2023

    2023 arXiv

  4. [12]

    Opponent modeling in deep reinforcement learning

    He, H., Boyd-Graber, J., Kwok, K., and Daum \'e III, H. Opponent modeling in deep reinforcement learning. In International conference on machine learning, pp.\ 1804--1813. PMLR, 2016

    2016

  5. [13]

    Hernandez-Leal, P., Kartal, B., and Taylor, M. E. Agent modeling as auxiliary task for deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence and interactive digital entertainment, volume 15, pp.\ 31--37, 2019

    2019

  6. [14]

    and Foerster, J

    Hu, H. and Foerster, J. N. Simplified action decoder for deep multi-agent reinforcement learning. In International Conference on Learning Representations, 2019

    2019

  7. [15]

    Off-belief learning

    Hu, H., Lerer, A., Cui, B., Pineda, L., Brown, N., and Foerster, J. Off-belief learning. In International Conference on Machine Learning, pp.\ 4369--4379. PMLR, 2021

    2021

  8. [16]

    Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer

    Jeon, J., Kim, W., Jung, W., and Sung, Y. Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer. In International Conference on Machine Learning, pp.\ 10041--10052. PMLR, 2022

    2022

  9. [17]

    and Lu, Z

    Jiang, J. and Lu, Z. The emergence of individuality. In International Conference on Machine Learning, pp.\ 4992--5001. PMLR, 2021

    2021

  10. [18]

    Fox: Formation-aware exploration in multi-agent reinforcement learning

    Jo, Y., Lee, S., Yeom, J., and Han, S. Fox: Formation-aware exploration in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 12985--12994, 2024

    2024

  11. [19]

    Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  12. [20]

    and Vouros, G

    Kontogiannis, A. and Vouros, G. A. Inherently interpretable deep reinforcement learning through online mimicking. In International Workshop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems, pp.\ 160--179. Springer, 2023

    2023

  13. [21]

    Celebrating diversity in shared multi-agent reinforcement learning

    Li, C., Wang, T., Wu, C., Zhao, Q., Yang, J., and Zhang, C. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 3991--4002, 2021

    2021

  14. [22]

    Two heads are better than one: A simple exploration framework for efficient multi-agent reinforcement learning

    Li, J., Kuang, K., Wang, B., Li, X., Wu, F., Xiao, J., and Chen, L. Two heads are better than one: A simple exploration framework for efficient multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  15. [23]

    Marl-based distributed cache placement for wireless networks

    Lin, X., Tang, Y., Lei, X., Xia, J., Zhou, Q., Wu, H., and Fan, L. Marl-based distributed cache placement for wireless networks. IEEE Access, 7: 0 62606--62615, 2019

    2019

  16. [24]

    Imagine, initialize, and explore: An effective exploration method in multi-agent reinforcement learning

    Liu, Z., Wan, L., Yang, X., Chen, Z., Chen, X., and Lan, X. Imagine, initialize, and explore: An effective exploration method in multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 17487--17495, 2024

    2024

  17. [25]

    I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I

    Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017 a

    2017

  18. [26]

    I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I

    Lowe, R., Wu, Y. I., Tamar, A., Harb, J., Pieter Abbeel, O., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017 b

    2017

  19. [27]

    Fast peer adaptation with context-aware exploration

    Ma, L., Wang, Y., Zhong, F., Zhu, S.-C., and Wang, Y. Fast peer adaptation with context-aware exploration. In Forty-first International Conference on Machine Learning

  20. [28]

    Ma, M., Liu, J., Sokota, S., Kleiman-Weiner, M., and Foerster, J. N. Learning intuitive policies using action features. In International Conference on Machine Learning, pp.\ 23358--23372. PMLR, 2023

    2023

  21. [29]

    Modeling the interaction between agents in cooperative multi-agent reinforcement learning

    Ma, X., Yang, Y., Li, C., Lu, Y., Zhao, Q., and Yang, J. Modeling the interaction between agents in cooperative multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp.\ 853--861, 2021

    2021

  22. [30]

    Maven: Multi-agent variational exploration

    Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, S. Maven: Multi-agent variational exploration. Advances in neural information processing systems, 32, 2019

    2019

  23. [31]

    P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K

    Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp.\ 1928--1937. PMLR, 2016

    1928

  24. [32]

    and Abbeel, P

    Mordatch, I. and Abbeel, P. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  25. [33]

    R., Pires, B

    Moreno, P., Hughes, E., McKee, K. R., Pires, B. A., and Weber, T. Neural recursive belief states in multi-agent reinforcement learning. arXiv preprint arXiv:2102.02274, 2021

    2021 arXiv

  26. [34]

    Theory of mind with guilt aversion facilitates cooperative reinforcement learning

    Nguyen, D., Venkatesh, S., Nguyen, P., and Tran, T. Theory of mind with guilt aversion facilitates cooperative reinforcement learning. In Asian Conference on Machine Learning, pp.\ 33--48. PMLR, 2020

    2020

  27. [35]

    Social motivation for modelling other agents under partial observability in decentralised training

    Nguyen, D., Le, H., Do, K., Venkatesh, S., and Tran, T. Social motivation for modelling other agents under partial observability in decentralised training. In Elkind, E. (ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23...

    2023 doi

  28. [36]

    A., Amato, C., et al

    Oliehoek, F. A., Amato, C., et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016

    2016

  29. [37]

    An extended benchmarking of multi-agent reinforcement learning algorithms in complex fully cooperative tasks

    Papadopoulos, G., Kontogiannis, A., Papadopoulou, F., Poulianou, C., Koumentis, I., and Vouros, G. An extended benchmarking of multi-agent reinforcement learning algorithms in complex fully cooperative tasks. arXiv preprint arXiv:2502.04773, 2025

    2025 arXiv

  30. [38]

    and Albrecht, S

    Papoudakis, G. and Albrecht, S. V. Variational autoencoders for opponent modeling in multi-agent systems. In AAAI 2020 Workshop on Reinforcement Learning in Games, 2020

    2020

  31. [39]

    Papoudakis, G., Christianos, F., Sch \"a fer, L., and Albrecht, S. V. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. arXiv preprint arXiv:2006.07869, 2020

    2006 arXiv

  32. [40]

    Agent modelling under partial observability for deep reinforcement learning

    Papoudakis, G., Christianos, F., and Albrecht, S. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 19210--19222, 2021

    2021

  33. [41]

    Adversar: Adversarial search and rescue via multi-agent reinforcement learning

    Rahman, A., Bhattacharya, A., Ramachandran, T., Mukherjee, S., Sharma, H., Fujimoto, T., and Chatterjee, S. Adversar: Adversarial search and rescue via multi-agent reinforcement learning. In 2022 IEEE International Symposium on Technologies for Homeland Security (HST), pp.\ 1-...

    2022

  34. [42]

    Modeling others using oneself in multi-agent reinforcement learning

    Raileanu, R., Denton, E., Szlam, A., and Fergus, R. Modeling others using oneself in multi-agent reinforcement learning. In International conference on machine learning, pp.\ 4257--4266. PMLR, 2018

    2018

  35. [43]

    S., Farquhar, G., Foerster, J., and Whiteson, S

    Rashid, T., Samvelyan, M., De Witt, C. S., Farquhar, G., Foerster, J., and Whiteson, S. Monotonic value function factorisation for deep multi-agent reinforcement learning. The Journal of Machine Learning Research, 21 0 (1): 0 7234--7284, 2020

    2020

  36. [44]

    Gcs: Graph-based coordination strategy for multi-agent reinforcement learning

    Ruan, J., Du, Y., Xiong, X., Xing, D., Li, X., Meng, L., Zhang, H., Wang, J., and Xu, B. Gcs: Graph-based coordination strategy for multi-agent reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp.\ 1128--1...

    2022

  37. [45]

    Agent-centric representations for multi-agent reinforcement learning

    Shang, W., Espeholt, L., Raichuk, A., and Salimans, T. Agent-centric representations for multi-agent reinforcement learning. arXiv preprint arXiv:2104.09402, 2021

    2021 arXiv

  38. [46]

    J., Hostallero, D

    Son, K., Kim, D., Kang, W. J., Hostallero, D. E., and Yi, Y. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp.\ 5887--5896. PMLR, 2019

    2019

  39. [47]

    Value-decomposition multi-agent actor-critics

    Su, J., Adams, S., and Beling, P. Value-decomposition multi-agent actor-critics. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.\ 11352--11360, 2021

    2021

  40. [48]

    Decision-making with speculative opponent models

    Sun, J., Chen, S., Zhang, C., Ma, Y., and Zhang, J. Decision-making with speculative opponent models. IEEE Transactions on Neural Networks and Learning Systems, 2024

    2024

  41. [49]

    M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J

    Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zambaldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Confe...

    2018

  42. [50]

    and Chen, B

    Tan, S. and Chen, B. Attentional opponent modelling for multi-agent cooperation. In 2023 International Joint Conference on Neural Networks (IJCNN), pp.\ 1--9. IEEE, 2023

    2023

  43. [51]

    \# exploration: A study of count-based exploration for deep reinforcement learning

    Tang, H., Houthooft, R., Foote, D., Stooke, A., Xi Chen, O., Duan, Y., Schulman, J., DeTurck, F., and Abbeel, P. \# exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017

    2017

  44. [52]

    Valiente, R., Toghi, B., Pedarsani, R., and Fallah, Y. P. Robustness and adaptability of reinforcement learning-based cooperative autonomous driving in mixed-autonomy traffic. IEEE Open Journal of Intelligent Transportation Systems, 3: 0 397--410, 2022

    2022

  45. [53]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  46. [54]

    Multi-agent reinforcement learning is a sequence modeling problem

    Wen, M., Kuba, J., Lin, R., Zhang, W., Wen, Y., Wang, J., and Yang, Y. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35: 0 16509--16521, 2022

    2022

  47. [55]

    Asynchronous actor-critic for multi-agent reinforcement learning

    Xiao, Y., Tan, W., and Amato, C. Asynchronous actor-critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 35: 0 4385--4400, 2022

    2022

  48. [56]

    Subspace-aware exploration for sparse-reward multi-agent tasks

    Xu, P., Zhang, J., Yin, Q., Yu, C., Yang, Y., and Huang, K. Subspace-aware exploration for sparse-reward multi-agent tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 11717--11725, 2023

    2023

  49. [57]

    Side: State inference for partially observable cooperative multi-agent reinforcement learning

    Xu, Z., Bai, Y., Li, D., Zhang, B., and Fan, G. Side: State inference for partially observable cooperative multi-agent reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp.\ 1400--1408, 2022

    2022

  50. [58]

    Transformer-based working memory for multiagent reinforcement learning with action parsing

    Yang, Y., Chen, G., Wang, W., Hao, X., Hao, J., and Heng, P.-A. Transformer-based working memory for multiagent reinforcement learning with action parsing. Advances in Neural Information Processing Systems, 35: 0 34874--34886, 2022

    2022

  51. [59]

    The surprising effectiveness of ppo in cooperative multi-agent games

    Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35: 0 24611--24624, 2022

    2022

  52. [60]

    Multi-agent incentive communication via decentralized teammate modeling

    Yuan, L., Wang, J., Zhang, F., Wang, C., Zhang, Z., Yu, Y., and Zhang, C. Multi-agent incentive communication via decentralized teammate modeling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 9466--9474, 2022

    2022

  53. [61]

    Dynamic belief for decentralized multi-agent cooperative learning

    Zhai, Y., Peng, P., Su, C., and Tian, Y. Dynamic belief for decentralized multi-agent cooperative learning. In Proceedings of International Joint Conference on Artificial Intelligence, pp.\ 344--352, 2023

    2023

  54. [62]

    and Lesser, V

    Zhang, C. and Lesser, V. Coordinating multi-agent reinforcement learning with limited communication. In Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pp.\ 1101--1108, 2013

    2013

  55. [64]

    Boosting value decomposition via unit-wise attentive state representation for cooperative multi-agent reinforcement learning

    Zhao, Q., Zhu, Y., Liu, Z., Wang, Z., and Chen, C. Boosting value decomposition via unit-wise attentive state representation for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2305.07182, 2023 b

    2023 arXiv

  56. [65]

    Conditionally optimistic exploration for cooperative deep multi-agent reinforcement learning

    Zhao, X., Pan, Y., Xiao, C., Chandar, S., and Rajendran, J. Conditionally optimistic exploration for cooperative deep multi-agent reinforcement learning. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI '23. JMLR.org, 2023 c

    2023

  57. [66]

    Episodic multi-agent reinforcement learning with curiosity-driven exploration

    Zheng, L., Chen, J., Wang, J., He, J., Hu, Y., Chen, Y., Fan, C., Gao, Y., and Zhang, C. Episodic multi-agent reinforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems, 34: 0 3757--3769, 2021

    2021

  58. [67]

    A survey of multi-agent reinforcement learning with communication

    Zhu, C., Dastani, M., and Wang, S. A survey of multi-agent reinforcement learning with communication. arXiv preprint arXiv:2203.08975, 2022

    2022 arXiv

  59. [68]

    Deep interactive bayesian reinforcement learning via meta-learning

    Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., and Hofmann, K. Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864, 2021

    2021 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.