Pith. sign in

REVIEW 4 major objections 3 minor 2 cited by

An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks

T0 review · 4 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that many cooperative MARL algorithms celebrated on the SMAC and Google Research Football benchmarks, especially the exploration-based EOI, EMC, and MASER, underperform standard baselines on a broad set of fully…

desk verdict A useful benchmarking framework and a lot of honest work, but the headline claim about SoTA exploration methods is undercut by a concrete configuration error (MASER runs at batch size 1) that a careful referee should have caught. read the letter →

arxiv 2502.04773 v2 pith:FCGLEORN submitted 2025-02-07 cs.LG

George Papadopoulos , Andreas Kontogiannis , Foteini Papadopoulou , Chaido Poulianou , Ioannis Koumentis , George Vouros This is my paper · ORCID
classification cs.LG
keywords multi-agentreinforcementlearningfullycooperativetasksbenchmarkingimage-basedobservationsexplorationmethodsvaluedecompositionSMACGRF
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test whether cooperative multi-agent reinforcement learning algorithms that are regarded as state of the art on the SMAC and Google Research Football benchmarks keep that status in fully cooperative tasks that look like real-world coordination: warehouse logistics, foraging, kitchen teamwork, pressure-plate room unlocking, and image-based games. It compares 11 algorithms on eight benchmark families under a common training protocol, and reports that the exploration-oriented methods EOI, EMC, and MASER significantly underperform in most benchmarks compared to the standard methods, sometimes failing completely on sparse-reward tasks. The consistent performers are the standard baselines MAA2C, MAPPO, QPLEX, and CDS, and rankings from SMAC/GRF do not predict rankings here. The paper also releases an open-source extension named PyMARLzoo+, integrates previously unsupported benchmark families, and is the first in this line to include high-dimensional image observations and to report training times as part of the comparison.

What carries the argument

The central object is the evaluation protocol: a decentralized partially observable Markov decision process (Dec-POMDP) formulation with centralized training and decentralized execution, shared policy parameters, no inter-agent communication during execution, and a fixed training-timestep budget per benchmark, scored by the mean of the best policy's returns over five seeds. The load-bearing piece is the uniformity of this protocol across eleven algorithms and eight benchmark families, so that the comparison is about algorithmic behavior under controlled conditions, including sparse rewards, high-dimensional image observations, and varying numbers of agents. The open-source PyMARLzoo+ extension is the mechanism that makes this uniform comparison possible by bringing the benchmark families into one training harness and adding frozen image encoders for the image-observation tasks.

What would settle it

Run EOI, EMC, and MASER on the same LBF, RWARE, and Overcooked tasks after a per-task hyperparameter search (learning rate, batch size, buffer size, and exploration coefficients) under identical timestep budgets and seeds, and compare against the reported MAPPO, MAA2C, QPLEX, and CDS results; if tuned versions close or reverse the gap, the underperformance claim is falsified, and if the gap persists, it stands.

Watch

Extended reading notes

Core claim

The authors are trying to establish that the dominant evaluation protocol for cooperative MARL, built around SMAC and GRF, gives a misleading picture of algorithmic capability. Their evidence is a large comparative study in which algorithms published as state of the art on those two benchmarks, specifically EOI, EMC, and MASER, come out worse than plain baselines across most of the eight fully cooperative benchmark families, and in several sparse-reward cases fail to learn anything. The results also show value decomposition methods converging to suboptimal policies as the number of agents grows, whereas actor-critic methods such as MAA2C and MAPPO remain competitive, and they identify two tasks (Entombed Cooperative and Overcooked's Coordination Ring) that defeat every algorithm tested. The paper presents this as empirical evidence of overfitting to the established benchmarks rather than as a theoretical impossibility.

Load-bearing premise

The load-bearing premise is that the fixed default hyperparameters used for every algorithm are appropriate for the new tasks, so the observed underperformance of EOI, EMC, and MASER reflects the algorithms' capabilities rather than a bad configuration for these benchmarks.

Editorial extensions

If this is right

  • Performance hierarchies established on SMAC and GRF should not be assumed to transfer to fully cooperative, no-opponent tasks.
  • Exploration-heavy state-of-the-art methods EOI, EMC, and MASER are not reliable choices for sparse-reward fully cooperative tasks, where they often fail completely.
  • Value decomposition approaches such as QMIX and QPLEX tend to converge to suboptimal policies as agent counts grow, while actor-critic baselines like MAA2C and MAPPO scale more consistently.
  • Entombed Cooperative and Overcooked's Coordination Ring are concrete open challenges, since no tested algorithm finds an effective policy on them.
  • Reporting wall-clock training times alongside scores makes the comparison interpretable against compute budget, with off-policy methods generally taking longer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the hyperparameter question open: because EOI, EMC, and MASER use publication-default settings while baselines use the library defaults, a per-task tuning pass could narrow the gap; if it did, the claim would reduce to 'off-the-shelf configurations underperform.' This is my inference, not the paper's claim.
  • Since communication is disabled in every experiment, the failures on coordination-heavy tasks cannot separate exploration failure from coordination or credit-assignment failure; a cheap-talk channel added to the same harness would be a natural follow-up the paper does not run.
  • The Pistonball comparison of frozen ResNet18 against trainable CNNs suggests that observation encoding choice is a confound in image-based MARL evaluation; ablating encoder choice across more seeds and tasks could shift rankings within the PettingZoo family.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper presents a large empirical benchmarking study of 11 cooperative MARL algorithms (QMIX, QPLEX, MAA2C, MAPPO, HAPPO, MAT-DEC, COMA, EOI, MASER, EMC, CDS) across eight fully cooperative benchmarks, including several with image-based observations. The authors introduce PyMARLzoo+, an extension of (E)PyMARL that integrates PettingZoo, Overcooked, PressurePlate, Capture Target, and Box Pushing. The central empirical claim, stated in the abstract and Section 6.2.1, is that exploration-based methods that are state-of-the-art on SMAC/GRF (EOI, EMC, MASER) significantly underperform standard baselines on fully cooperative tasks. The paper also reports training times and compares frozen ResNet18 image encoders with trainable CNNs.

Significance. If the central claim is fully supported, the paper would make a valuable contribution: it broadens the benchmark diversity for cooperative MARL, provides an open-source framework that lowers the barrier to using alternative benchmarks, and is among the first to include image-based observations in a multi-agent benchmarking study. The inclusion of wall-clock training times and the systematic reporting of 75% confidence intervals are also useful methodological practices. However, the key underperformance conclusion is currently only partially supported because of configuration asymmetries in the comparison, missing data for one of the allegedly underperforming algorithms, and the absence of statistical significance tests. With the requested sensitivity analyses and missing results, the paper could become a solid benchmark reference.

major comments (4)
  1. [§6.1, Tables 23 and 16] MASER is trained with batch_size=1 (Table 23) while the QMIX baseline, on which MASER is built, uses batch_size=32 (Table 16). Since MASER is an off-policy algorithm learning from a replay buffer, a single episode per gradient step is an extreme configuration that could plausibly produce the near-zero returns reported in Tables 4, 5, and 8. The explanation in §6.2.2 that Q-value-driven subgoals mislead exploration is therefore confounded by this configuration mismatch. Please provide a sensitivity analysis over batch size (e.g., batch size 32) for MASER on at least the key benchmarks, or justify why batch size 1 is the appropriate transfer from the original paper.
  2. [§6.1, §6.2.1, and Table 12] The claim that EOI, EMC, and MASER 'significantly underperform' standard baselines is not supported by any statistical test. The paper uses 75% confidence intervals over five seeds and states in §6.2 that bold entries denote values not 'statistically significantly different' from the best algorithm, but no hypothesis test is described. Many reported intervals are wide and overlapping (e.g., LBF 2s-8x8-3p-2f, RWARE tiny-4ag-hard), so the bolding rule is not reproducible. Please specify the exact test (e.g., paired bootstrap, Mann-Whitney) or soften the claim to a descriptive comparison.
  3. [Tables 8 and 13] EMC has missing results for Overcooked's Asymmetric Advantages and Coordination Ring layouts (shown as '−'). Because the central underperformance conclusion refers to 'most benchmarks,' the absence of EMC data on two of the three Overcooked layouts weakens this claim in that benchmark. Please run or explicitly exclude these configurations, and report whether the dash indicates a failed run, a computational budget limit, or an omitted experiment.
  4. [§6.1] The decision to use original-paper hyperparameters without tuning is justified by 'preliminary results showing no significant differences regarding algorithms' performance across tasks,' but these preliminary results are not presented anywhere. Since the paper's central conclusion is a cross-algorithm comparison, the absence of any sensitivity study makes it hard to rule out configuration mismatch as the cause of underperformance. Please include the preliminary results or a brief sensitivity check for the three underperforming methods.
minor comments (3)
  1. [Table 7] The header 'Entompted Cooperative' is a typo for 'Entombed Cooperative'.
  2. [References] References [21] and [22] are duplicate entries for the same MASER paper (Jeon et al., 2022); one should be removed and the citations renumbered.
  3. [§6.1] The phrase 'we utilize the same number of training timesteps' is misleading because the number differs across benchmarks (10M for MPE/LBF, 40M for RWARE, 5M for PettingZoo, etc.); please rephrase to 'the same number of training timesteps across algorithms within each benchmark.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmarking results are external measurements, not derived predictions.

full rationale

This is an empirical benchmarking study with no derivation chain that reduces to its own inputs. The central claim, that several SMAC/GRF state-of-the-art algorithms underperform standard baselines on fully cooperative tasks, is a direct measurement reported in Tables 4-11, not a prediction from a fitted model. No parameter is fitted to a subset of data and then used to predict a closely related quantity; the hyperparameters are fixed per algorithm before training, and the reported returns are measured test-episode outcomes. The only self-citation (ref [25], Kontogiannis and Vouros) appears in the introduction as an example application of cooperative MARL (air traffic management) and is not load-bearing for any claim. The paper's interpretation that MASER and EMC underperform because of Q-value-driven intrinsic rewards is a post-hoc explanation, not a circular derivation. Concerns about MASER's batch_size=1 versus QMIX's batch_size=32 are experimental-design critiques that could affect validity, but they do not make the result circular, since the comparison is still an external measurement rather than an identity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new free parameters or invented entities are introduced; the hyperparameters are taken from prior papers. The main assumptions are model-theoretic (CTDE/Dec-POMDP) and methodological (hyperparameter transfer, task representativeness).

assumptions (3)
  • domain assumption The Dec-POMDP formulation with CTDE and no communication during execution is an appropriate model for all tasks.
    Section 2.2 states these assumptions; they define the problem class and exclude communication-based methods.
  • domain assumption Default hyperparameters from original papers and from EPyMARL transfer to the new tasks.
    Section 6.1: 'We used the hyperparameter settings of the newly integrated algorithms ... adhering to configurations suggested by the authors of their original papers.' This is load-bearing for the underperformance claim.
  • domain assumption The selected benchmarks are representative of fully cooperative real-world tasks.
    Section 4 motivates each task with a real-world application (e.g., warehouse, search and rescue), but no formal argument establishes representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks." pith.science (2026). https://pith.science/paper/FCGLEORN

@misc{pith2026250204773,
  author       = {Pith},
  title        = {Pith review of: An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCGLEORN}},
  note         = {Machine review of arXiv:2502.04773}
}
read the original abstract

Multi-Agent Reinforcement Learning (MARL) has recently emerged as a significant area of research. However, MARL evaluation often lacks systematic diversity, hindering a comprehensive understanding of algorithms' capabilities. In particular, cooperative MARL algorithms are predominantly evaluated on benchmarks such as SMAC and GRF, which primarily feature team game scenarios without assessing adequately various aspects of agents' capabilities required in fully cooperative real-world tasks such as multi-robot cooperation and warehouse, resource management, search and rescue, and human-AI cooperation. Moreover, MARL algorithms are mainly evaluated on low dimensional state spaces, and thus their performance on high-dimensional (e.g., image) observations is not well-studied. To fill this gap, this paper highlights the crucial need for expanding systematic evaluation across a wider array of existing benchmarks. To this end, we conduct extensive evaluation and comparisons of well-known MARL algorithms on complex fully cooperative benchmarks, including tasks with images as agents' observations. Interestingly, our analysis shows that many algorithms, hailed as state-of-the-art on SMAC and GRF, may underperform standard MARL baselines on fully cooperative benchmarks. Finally, towards more systematic and better evaluation of cooperative MARL algorithms, we have open-sourced PyMARLzoo+, an extension of the widely used (E)PyMARL libraries, which addresses an open challenge from [TBG++21], facilitating seamless integration and support with all benchmarks of PettingZoo, as well as Overcooked, PressurePlate, Capture Target and Box Pushing.

Figures

Figures reproduced from arXiv: 2502.04773 by the authors.

Figure 1
Figure 1. Aggregated Normalized episodic rewards of each benchmark. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. ResNet18 vs Trainable CNNs: MAA2C in Petting [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Indicative Python script for executing an episode in PettingZoo’s Pistonball. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (30 more)
Figure 4
Figure 4. Figure 4: Example of minimal args for PettingZoo tasks [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Example of all args for PettingZoo tasks. where: • key is the selected PettingZoo task. Options: "pistonball_v6", "cooperative_pong_v5", "entombed_cooperative_v3", "space_invaders_v2", "basketball_pong_v3", "boxing_v2", "combat_jet_v1", "combat_tank_v3", "double_dunk_v…
Figure 6
Figure 6. Figure 6: Example of minimal args for Overcooked tasks. 2All the arguments of each task can be found on the official page of PettingZoo: https://pettingzoo.farama.org/ [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Example of all args for Overcooked tasks. where: • key is the selected Overcooked scenario. Options: "cramped_room", "asymmetric_advantages", "coordination_ring", "counter_circuit", "forced_coordination". • seed has the same functionality as in PettingZoo. The default …
Figure 8
Figure 8. Figure 8: Example of minimal args for Pressure Plate environment. args = { "env": " pressureplate ", " env_args ": { "key": " pressureplate - linear -4p-v0", " seed ": 1 , " time_limit ": 500 , } } [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Example of all args for Pressure Plate environment. where: • key is the selected Pressure Plate scenario. Options: "pressureplate-linear-4p-v0", "pressureplate-linear-5p-v0", "pressureplate-linear￾6p-v0". • seed has the same functionality as in PettingZoo. The default …
Figure 10
Figure 10. Figure 10: Example of minimal args for Capture Target environment [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Example of all args for Capture Target environment. where: • key is the selected Capture Target scenario. Options: "CaptureTarget-6x6-1t-2a-v0". • seed has the same functionality as in PettingZoo. The default value is 1. • time_limit has the same functionality as in P…
Figure 12
Figure 12. Figure 12: Example of minimal args for Box pushing environment. args = { "env": " boxpushing ", " env_args ": { "key ": " BoxPushing -6x6 -2a-v0", " seed ": 1 , " time_limit ": 60 , " random_init ": True } } [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Example of all args for Box Pushing environment. where: • key is the selected Box Pushing scenario. Options: "BoxPushing-6x6-2a-v0". • seed has the same functionality as in PettingZoo. The default value is 1. • time_limit has the same functionality as in PettingZoo. T…
Figure 14
Figure 14. Figure 14: Example of minimal args for LBF, RWARE, and MPE environments. args = { "env": " gymma ", " env_args ": { "key": " mpe : SimpleSpeakerListener -v0", " seed ": 1 , " time_limit ": 500 , } } [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Example of all args for LBF, RWARE, and MPE environments. where: • key is the selected scenario of LBF, RWARE, or MPE environments. Options for LBF: "lbforaging:Foraging-4s-11x11-3p-2f-coop-v2", "lbforaging:Foraging-2s-11x11-3p-2f-coop-v2", "lbforaging:Foraging-2s-8x8…
Figure 16
Figure 16. Figure 16: Bash command to install PyMARLzoo+ package using the pip tool. [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Example Bash commands for training with different algorithms in different environment tasks. The [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Example Bash commands for training QMIX algorithm in PettingZoo and Overcooked environment tasks. [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]
Figure 19
Figure 19. Figure 19: Sample snapshot of the Spread MPE environment. [PITH_FULL_IMAGE:figures/full_fig_p016_19.png]
Figure 20
Figure 20. Figure 20: Sample snapshot of small size RWARE environment with four agents. [PITH_FULL_IMAGE:figures/full_fig_p017_20.png]
Figure 21
Figure 21. Figure 21: Sample snapshots of the three PettingZoo game environments. [PITH_FULL_IMAGE:figures/full_fig_p018_21.png]
Figure 22
Figure 22. Figure 22: Sample image-based observation of a piston in the Pistonball environment of PettingZoo with 10 agents. [PITH_FULL_IMAGE:figures/full_fig_p018_22.png]
Figure 23
Figure 23. Figure 23: Sample image-based partial observations of each agent in the Entombed Cooperative environment of PettingZoo. [PITH_FULL_IMAGE:figures/full_fig_p019_23.png]
Figure 24
Figure 24. Figure 24: Sample snapshot of the Overcooked layouts. On the left is, "Cramped room", followed by the "Asymmetric advantages" [PITH_FULL_IMAGE:figures/full_fig_p021_24.png]
Figure 25
Figure 25. Figure 25: Snapshot of the Pressureplate environment. [PITH_FULL_IMAGE:figures/full_fig_p023_25.png]
Figure 26
Figure 26. Figure 26: CaptureTarget environment. Two agents learn to capture the target simultaneously. [PITH_FULL_IMAGE:figures/full_fig_p024_26.png]
Figure 27
Figure 27. Figure 27: Snapshot of the Box Pushing environment. [PITH_FULL_IMAGE:figures/full_fig_p025_27.png]
Figure 28
Figure 28. Figure 28: Episodic rewards of all 11 algorithms in LBF tasks rendering the mean and the 75% confidence interval over 5 different [PITH_FULL_IMAGE:figures/full_fig_p025_28.png]
Figure 29
Figure 29. Figure 29: Episodic rewards of all 11 algorithms in MPE tasks rendering the mean and the 75% confidence interval over 5 [PITH_FULL_IMAGE:figures/full_fig_p026_29.png]
Figure 30
Figure 30. Figure 30: Episodic rewards of all 11 algorithms in RWARE tasks rendering the mean and the 75% confidence interval over 5 [PITH_FULL_IMAGE:figures/full_fig_p026_30.png]
Figure 31
Figure 31. Figure 31: Episodic rewards of all 11 algorithms in PettingZoo environments rendering the mean and the 75% confidence [PITH_FULL_IMAGE:figures/full_fig_p026_31.png]
Figure 32
Figure 32. Figure 32: Episodic rewards of all 11 algorithms in Overcooked environments rendering the mean and the 75% confidence [PITH_FULL_IMAGE:figures/full_fig_p027_32.png]
Figure 33
Figure 33. Figure 33: Episodic rewards of all 11 algorithms in Pressure Plate tasks rendering the mean and the 75% confidence interval [PITH_FULL_IMAGE:figures/full_fig_p027_33.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

    cs.LG 2026-06 conditional novelty 6.5 of 10

    Distributional shift in RL is classified by which POMDP generative component changes (internal agent vs external environment) and by whether the time boundary is explicit, implicit, or hybrid.

  2. Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A new transformer-based multi-agent world model with teammate prediction and prioritized replay achieves near-optimal performance on cooperative benchmarks in as few as 50,000 environment steps.

Reference graph

Works this paper leans on

82 extracted references · 64 canonical work pages · cited by 2 Pith papers

  1. [1]

    Ibrahim H Ahmed, Cillian Brewitt, Ignacio Carlucho, Filippos Christianos, Mhairi Dunion, Elliot Fosong, Samuel Garcin, Shangmin Guo, Balint Gyevnar, Trevor McInroe, et al. 2022. Deep reinforcement learning for multi-agent interaction. Ai Communications 35, 4 (2022), 357–368

    2022

  2. [2]

    Stefano V Albrecht, Filippos Christianos, and Lukas Schäfer. 2024. Multi-agent reinforcement learning: Foundations and modern approaches . MIT Press

    2024

  3. [3]

    Albrecht and Peter Stone

    Stefano V. Albrecht and Peter Stone. 2017. Reasoning about Hypothetical Agent Behaviours and their Parameters. In Proceedings of the 16th Conference on Au- tonomous Agents and MultiAgent Systems (São Paulo, Brazil) (AAMAS ’17). Inter- national Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 547–555

    work page 2017

  4. [4]

    Matteo Bettini, Amanda Prorok, and Vincent Moens. 2024. Benchmarl: Bench- marking multi-agent reinforcement learning. Journal of Machine Learning Re- search 25, 217 (2024), 1–10

    work page 2024

  5. [5]

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. OpenAI Gym. arXiv:arXiv:1606.01540

    arXiv 2016

  6. [6]

    Wolfram Burgard, Mark Moors, Dieter Fox, Reid Simmons, and Sebastian Thrun

    work page

  7. [7]

    Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems 32 (2019)

    work page 2019

  8. [8]

    Changyu Chen, Ramesha Karunasena, Thanh Nguyen, Arunesh Sinha, and Pradeep Varakantham. 2024. Generative modelling of stochastic actions with arbitrary constraints in reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

Show all 82 references
  1. [9]

    Zigeng Chen, Gongfan Fang, Xinyin Ma, and Xinchao Wang. 2024. SlimSAM: 0.1% Data Makes Segment Anything Slim. arXiv:2312.05284 [cs.CV] https: //arxiv.org/abs/2312.05284

    2024 arXiv

  2. [10]

    Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. 2021. Scaling multi-agent reinforcement learning with selective param- eter sharing. In International Conference on Machine Learning. PMLR, 1989–1998

    2021

  3. [11]

    Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. 2024. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  4. [12]

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shi- mon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence , Vol. 32

    2018

  5. [13]

    Center for Human-Compatible AI GIthub contributors. [n.d.]. Github Overcooked. https://github.com/HumanCompatibleAI/overcooked_ai. Overcooked GitHub repository

  6. [14]

    Elliot Fosong, Arrasy Rahman, Ignacio Carlucho, and Stefano V Albrecht. 2024. Learning Complex Teamwork Tasks using a Given Sub-task Decomposition. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems. 598–606

    2024

  7. [15]

    Jialu Gao, Kaizhe Hu, Guowei Xu, and Huazhe Xu. 2024. Can pre-trained text- to-image models generate visual goals for reinforcement learning? Advances in Neural Information Processing Systems 36 (2024)

    2024

  8. [16]

    Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. 2022. Towards a standardised performance evaluation protocol for cooperative marl. Advances in Neural Information Processing Systems 35 (2022), 5510–5521

    2022

  9. [17]

    Nikunj Gupta, Somjit Nath, and Samira Ebrahimi Kahou. 2023. CAMMARL: Conformal Action Modeling in Multi Agent Reinforcement Learning. arXiv preprint arXiv:2306.11128 (2023)

    2023 arXiv

  10. [18]

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition . 770–778

    2016

  11. [19]

    Joey Hong, Sergey Levine, and Anca Dragan. 2024. Learning to influence human behavior with offline reinforcement learning. Advances in Neural Information Processing Systems 36 (2024)

    2024

  12. [20]

    Siyi Hu, Yifan Zhong, Minquan Gao, Weixun Wang, Hao Dong, Zhihui Li, Xiaodan Liang, Yaodong Yang, and Xiaojun Chang. 2022. Marllib: Extending rllib for multi- agent reinforcement learning. (2022)

    2022

  13. [22]

    Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. 2022. Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer. In International Conference on Machine Learning . PMLR, 10041– 10052

    2022

  14. [23]

    Chengzhi Jiang and Zhaohan Sheng. 2009. Case-based reinforcement learning for dynamic inventory control in a multi-agent supply-chain system. Expert Systems with Applications 36, 3 (2009), 6520–6526

    2009

  15. [24]

    Jiechuan Jiang and Zongqing Lu. 2021. The emergence of individuality. In International Conference on Machine Learning . PMLR, 4992–5001

    2021

  16. [25]

    Andreas Kontogiannis and George A Vouros. 2023. Inherently Interpretable Deep Reinforcement Learning Through Online Mimicking. InInternational Work- shop on Explainable, Transparent Autonomous Agents and Multi-Agent Systems . Springer, 160–179

    2023

  17. [26]

    Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. 2021. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251 (2021)

    2021 arXiv

  18. [27]

    Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michał Zając, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. 2020. Google research football: A novel reinforcement learning environment. InProceedings of the AAAI conferen...

    2020

  19. [28]

    Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. 2021. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems 34 (2021), 3991–4002

    2021

  20. [29]

    Xihan Li, Jia Zhang, Jiang Bian, Yunhai Tong, and Tie-Yan Liu. 2019. A Coopera- tive Multi-Agent Reinforcement Learning Framework for Resource Balancing in Complex Logistics Network. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent System...

    2019

  21. [30]

    Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Gold- berg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018. RLlib: Abstractions for distributed reinforcement learning. In International conference on machine learning. PMLR, 3053–3062

    2018

  22. [31]

    Bo Liu, Qiang Liu, Peter Stone, Animesh Garg, Yuke Zhu, and Anima Anandku- mar. 2021. Coach-player multi-agent reinforcement learning for dynamic team composition. In International Conference on Machine Learning. PMLR, 6860–6870

    2021

  23. [32]

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30 (2017)

    2017

  24. [33]

    Igor Mordatch and Pieter Abbeel. 2018. Emergence of grounded compositional language in multi-agent populations. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32

    2018

  25. [34]

    Frans A Oliehoek, Christopher Amato, et al . 2016. A concise introduction to decentralized POMDPs. Vol. 1. Springer

    2016

  26. [35]

    Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. 2017. Deep decentralized multi-task multi-agent reinforcement learn- ing under partial observability. In International Conference on Machine Learning . PMLR, 2681–2690

    2017

  27. [36]

    Albrecht

    Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht

  28. [37]

    Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kami- enny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. 2021. Facmac: Fac- tored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems 34 (2021), 12208–12221

    2021

  29. [38]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  30. [39]

    Aowabin Rahman, Arnab Bhattacharya, Thiagarajan Ramachandran, Sayak Mukherjee, Himanshu Sharma, Ted Fujimoto, and Samrat Chatterjee. 2022. Ad- versar: Adversarial search and rescue via multi-agent reinforcement learning. In 2022 IEEE International Symposium on Technologies for...

    2022

  31. [40]

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Far- quhar, Jakob Foerster, and Shimon Whiteson. 2020. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21, 178 (2020), 1–51

    2020

  32. [41]

    Jingqing Ruan, Yali Du, Xuantang Xiong, Dengpeng Xing, Xiyun Li, Linghui Meng, Haifeng Zhang, Jun Wang, and Bo Xu. 2022. GCS: Graph-based coordination strategy for multi-agent reinforcement learning. arXiv preprint arXiv:2201.06257 (2022)

    2022 arXiv

  33. [42]

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al . 2024. JaxMARL: Multi-Agent RL Environments and Algorithms in JAX. In Proceedings of the 23rd Inte...

    2024

  34. [43]

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Far- quhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. 2019. The starcraft multi-agent challenge. arXiv preprint arXiv:1902.04043 (2019)

    2019 arXiv

  35. [44]

    Tom Schaul. 2015. Prioritized Experience Replay. arXiv preprint arXiv:1511.05952 (2015)

    2015 arXiv

  36. [45]

    Sven Seuken and Shlomo Zilberstein. 2007. Improved memory-bounded dynamic programming for decentralized POMDPs. In Proceedings of the Twenty-Third Con- ference on Uncertainty in Artificial Intelligence (Vancouver, BC, Canada) (UAI’07). AUAI Press, Arlington, Virginia, USA, 344–351

    2007

  37. [46]

    Rutav M Shah and Vikash Kumar. 2021. RRL: Resnet as representation for Reinforcement Learning. In International Conference on Machine Learning . PMLR, 9465–9476

    2021

  38. [47]

    Chapman Siu, Jason Traish, and Richard Yi Da Xu. 2021. Dynamic coordination graph for cooperative multi-agent reinforcement learning. In Asian Conference on Machine Learning. PMLR, 438–453

    2021

  39. [48]

    Jing Sun, Shuo Chen, Cong Zhang, Yining Ma, and Jie Zhang. 2024. Decision- Making With Speculative Opponent Models. IEEE Transactions on Neural Net- works and Learning Systems (2024)

    2024

  40. [49]

    Jordan Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Ro- drigo Perez-Vicente, et al. 2021. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Proce...

    2021

  41. [50]

    Justin K Terry, Benjamin Black, and Luis Santos. 2020. Multiplayer support for the arcade learning environment. arXiv preprint arXiv:2009.09341 (2020)

    2020 arXiv

  42. [51]

    Justin K Terry, Nathaniel Grammel, Sanghyun Son, Benjamin Black, and Aakriti Agrawal. 2020. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005.13625 (2020)

    2020 arXiv

  43. [52]

    Filippos Christianos Trevor McInroe. [n.d.]. Github pressureplate. https://github. com/uoe-agents/pressureplate/tree/main. Pressureplate GitHub repository

  44. [53]

    Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. 2020. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062 (2020)

    2020 arXiv

  45. [54]

    Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. 2020. ROMA: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning . 9876–9886

    2020

  46. [55]

    Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. 2022. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems 35 (2022), 16509– 16521

    2022

  47. [56]

    Yuchen Xiao, Joshua Hoffman, and Christopher Amato. 2019. Macro-Action- Based Deep Multi-Agent Reinforcement Learning. In 3rd Annual Conference on Robot Learning

    2019

  48. [57]

    Yuchen Xiao, Joshua Hoffman, and Christopher Amato. 2020. Macro-action-based deep multi-agent reinforcement learning. In Conference on Robot Learning. PMLR, 1146–1161

    2020

  49. [58]

    Yuchen Xiao, Joshua Hoffman, Tian Xia, and Christopher Amato. 2020. Learning multi-robot decentralized macro-action-based policies via a centralized q-net. In 2020 IEEE International conference on robotics and automation (ICRA) . IEEE, 10695–10701

    2020

  50. [59]

    Yuchen Xiao, Weihao Tan, and Christopher Amato. 2022. Asynchronous actor- critic for multi-agent reinforcement learning. Advances in Neural Information Processing Systems 35 (2022), 4385–4400

    2022

  51. [60]

    Mingyu Yang, Yaodong Yang, Zhenbo Lu, Wengang Zhou, and Houqiang Li

  52. [61]

    Yaodong Yang, Guangyong Chen, Weixun Wang, Xiaotian Hao, Jianye Hao, and Pheng-Ann Heng. 2022. Transformer-based working memory for multiagent reinforcement learning with action parsing. Advances in Neural Information Processing Systems 35 (2022), 34874–34886

    2022

  53. [62]

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems35 (2022), 24611–24624

    2022

  54. [63]

    Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu

  55. [64]

    Lulu Zheng, Jiarui Chen, Jianhao Wang, Jiamin He, Yujing Hu, Yingfeng Chen, Changjie Fan, Yang Gao, and Chongjie Zhang. 2021. Episodic multi-agent re- inforcement learning with curiosity-driven exploration. Advances in Neural Information Processing Systems 34 (2021), 3757–3769

    2021

  56. [65]

    Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32

    2018

  57. [66]

    env " :

    Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. 2024. Heterogeneous-agent reinforcement learning. Journal of Machine Learning Research 25, 1-67 (2024), 1. A ENVIRONMENT API The following Python script (Figure 3) provides an example of exec...

    2024

  58. [71]

    0. 0. 0. 0. 0. 1. 0. 0. 0. 1. 0. 1. 1. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0. ,

  59. [72]

    Foraging-4s-11x11-3p-2f-coop-v2

    0. 0. 1. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0. 1. 0. 0. 1. 0. 0. 0. 0. 0.] , dtype = float32 ) , D.2.2 Actions. The action space of this environment for each agent is comprised of four discrete possible options. For each agent i, the action space is the following: 𝐴𝑖 =(Turn left, Turn...

  60. [73]

    Noop" action signifies the agent remaining idle, while

    , 3. , 1. , -1. , -1. , 0.] , dtype = float32 ) , The three values in the array correspond to the triplet (position_x, position_y, level) of the first food item on the map, as observed by the agent. In this specific configuration, the environment consists of 2 food items and 3...

  61. [74]

    , 0. , 0. , 0. , 0. , 2. , 1. , 0. , 0. , 1. , 1. , 0. , 0. ,

  62. [75]

    , 0. , 0. , 0. , 1. , -2. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,

  63. [76]

    , 0. , 0. , 0. , 1. , 0. , 1. , 1. , 0. , 0. , 0. , 0. , 0. ,

  64. [77]

    , 0. , 1. , 0. , 0. , 0. , -2. , 2. , 0. , 0. , 0. , 0. , 0. ,

  65. [78]

    , 0. , 0. , 1. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , -1. , -1. ,

  66. [79]

    , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 1. , 0. , 1. ,

  67. [80]

    , 2. , -1. , 1. , 2.] , dtype = float32 ) , This array contains the observation space, which is of size(96), that belongs to a participating agents. All the values of the observation array also correspond to the fields explained earlier, which correspond to their features, the...

  68. [84]

    , 1. , 1. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,

  69. [86]

    , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. ,

  70. [87]

    , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 5. , 13.] , dtype = float32 ) , The observation array of each agent contains a combination of 5 flattened observation sub-arrays of dimensions𝑠𝑖𝑔ℎ𝑡×𝑠𝑖𝑔ℎ𝑡 each. Each observation sub-array represents the separate layout of a different el...

  71. [2000]

    InProceedings 2000 ICRA

    Collaborative multi-robot exploration. InProceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065) , Vol. 1. IEEE, IEEE, 476–481

    2000

  72. [2020]

    Benchmarking multi-agent deep reinforcement learning algorithms. (2020)

    2020

  73. [2021]

    In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS)

    Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS)

  74. [2024]

    Advances in Neural Information Processing Systems 36 (2024)

    Hierarchical multi-agent skill discovery. Advances in Neural Information Processing Systems 36 (2024)

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.