HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

Ankit Shah; Dongruo Zhou; Nathaniel D. Bastian; Runze Zhao; Sumit Kumar Jha

arxiv: 2606.29126 · v2 · pith:NLFPNR72new · submitted 2026-06-28 · 💻 cs.AI

HiComm: Hierarchical Communication for Multi-agent Reinforcement Learning

Runze Zhao , Dongruo Zhou , Sumit Kumar Jha , Nathaniel D. Bastian , Ankit Shah This is my paper

Pith reviewed 2026-07-02 20:52 UTC · model grok-4.3

classification 💻 cs.AI

keywords multi-agent reinforcement learninghierarchical communicationMARLcommunication protocolscooperative agentsobservation hierarchyGumbel-Softmax

0 comments

The pith

HiComm converts flat message vectors into receiver-driven retrieval from a sender's observation hierarchy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces HiComm as a plug-in module for cooperative multi-agent reinforcement learning that grounds communication in the sender's hierarchical observations rather than treating messages as unstructured vectors. The receiver initiates a query that resolves the hierarchy through three stages of selection to retrieve a specific feature slice. This design leverages natural structure in observations to reduce the amount of data exchanged while preserving or improving task performance compared to prior learned communication approaches. A reader would care because lower communication costs could scale multi-agent systems to larger teams or more constrained settings without sacrificing coordination.

Core claim

HiComm grounds messages in the sender's hierarchical observation through a receiver-driven three-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message. This converts communication from unstructured vector transmission into structured information retrieval over the sender's observation hierarchy. The mechanism is instantiated with Straight-Through Gumbel-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines. Experiments across cooperative MARL tasks with different observation structures show that HiComm matches o

What carries the argument

The receiver-driven three-stage decoding process that selects a group, then a sender, then an entity to retrieve a feature slice from the sender's observation hierarchy.

If this is right

HiComm attaches as a plug-in module to existing MARL pipelines with only lightweight shared projections.
Performance on cooperative tasks remains at or above the level of prior learned communication methods.
Communication volume drops by as much as 23 times per receiver per episode.
Straight-Through Gumbel-Softmax enables end-to-end differentiable training of the discrete selection steps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The receiver-driven query pattern could extend to other partially observable settings where one agent needs selective access to another's structured state.
Lower per-episode volume might permit larger agent populations under fixed bandwidth constraints.
The explicit hierarchy assumption invites tests on tasks whose observations contain different forms of structure, such as temporal or spatial trees.

Load-bearing premise

Observations in the target cooperative environments naturally follow a hierarchy of groups and entities that the three-stage receiver-driven process can resolve.

What would settle it

An experiment in environments whose observations lack any group-entity hierarchy in which HiComm produces no reduction in communication volume or no performance gain relative to flat-vector baselines.

read the original abstract

Cooperative multi-agent reinforcement learning (MARL) often relies on communication to mitigate partial observability, yet most existing protocols treat messages as flat dense vectors detached from the structure of the observations they summarize. This design overlooks an important source of inductive bias in many cooperative environments, where observations naturally follow a hierarchy such as groups and entities. We propose \textsc{HiComm}, a plug-in communication module that grounds messages in the sender's hierarchical observation. \textsc{HiComm} is receiver-driven: the receiver issues a query, and the hierarchy is resolved through a three-stage decoding process that first selects a group, then a sender, and then an entity within that group, returning the corresponding feature slice as the message. This converts communication from unstructured vector transmission into structured information retrieval over the sender's observation hierarchy. We instantiate this mechanism with Straight-Through Gumbel-Softmax for differentiable discrete selection and a lightweight shared projection design that attaches to standard MARL pipelines. Experiments across cooperative MARL tasks with different observation structures and coordination demands show that \textsc{HiComm} matches or outperforms representative learned communication baselines while reducing communication volume by up to $23\times$ per receiver per episode.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

HiComm's receiver-driven three-stage decoding offers a way to exploit observation hierarchies for lower communication volume in MARL, though the reported gains need full experimental backing to confirm.

read the letter

The main thing here is that HiComm replaces flat vector messages with a receiver-driven query that walks the sender's observation hierarchy in three discrete steps—group, sender, then entity—before returning a feature slice. The Gumbel-Softmax instantiation keeps it differentiable and the shared projection keeps it lightweight as a plug-in.

The paper does a clean job of naming the inductive bias it targets and showing how the mechanism converts unstructured transmission into structured retrieval. That framing is direct and the design choices look practical for attaching to existing MARL code.

The soft spots sit in the empirical side. The abstract gives no baselines, task descriptions, run counts, or ablation results, so the claim that it matches or beats prior methods while cutting volume 23× cannot be checked. The stress-test note is on point: because the receiver must send queries to drive the selection, any volume accounting that ignores query cost in both directions risks overstating the net savings. If those queries are cheap indices the gap may still be real, but the abstract does not address it.

This is for MARL researchers already working on learned communication who want to test a hierarchy-aware alternative. A reader with their own cooperative tasks that have natural group-entity structure could get value from trying the module once the full experiments are available.

I would send it to peer review. The mechanism is specific enough that referees can judge the implementation and controls directly.

Referee Report

2 major / 0 minor

Summary. The paper proposes HiComm, a plug-in communication module for cooperative MARL that converts unstructured vector messages into structured retrieval over the sender's observation hierarchy. Receivers issue queries resolved via a three-stage differentiable decoding process (group, sender, entity) using Straight-Through Gumbel-Softmax, returning feature slices. Experiments are claimed to show that HiComm matches or outperforms learned communication baselines while reducing communication volume by up to 23× per receiver per episode.

Significance. If the empirical claims hold under proper controls, HiComm supplies a concrete inductive bias for environments with natural observation hierarchies and demonstrates a receiver-driven retrieval mechanism that could improve bandwidth efficiency in cooperative MARL. The lightweight shared-projection design and compatibility with standard pipelines are practical strengths.

major comments (2)

[Abstract] Abstract: the central claim of up to 23× communication-volume reduction per receiver per episode does not specify whether the bandwidth cost of the receiver-issued queries (required to drive the three-stage group-sender-entity selection) is included in the measurement. Because the protocol is explicitly receiver-driven, any query transmission over the same channel would reduce the net savings relative to flat-vector baselines.
[Abstract] Abstract: the reported performance and volume-reduction results supply no information on baselines, number of independent runs, statistical tests, task definitions, or how communication volume is precisely defined and measured, rendering the empirical superiority claim unverifiable from the provided text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. We address each major comment below and will incorporate clarifications into a revised manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: the central claim of up to 23× communication-volume reduction per receiver per episode does not specify whether the bandwidth cost of the receiver-issued queries (required to drive the three-stage group-sender-entity selection) is included in the measurement. Because the protocol is explicitly receiver-driven, any query transmission over the same channel would reduce the net savings relative to flat-vector baselines.

Authors: We agree that the abstract does not explicitly define the scope of the volume measurement. In the protocol, receiver queries are generated from the receiver's local state and drive selection at the sender without requiring separate transmission of query vectors over the shared channel; only the resulting feature slices are transmitted. The reported 23× figure therefore measures transmitted message volume under this definition. We will revise the abstract to state this explicitly and add a corresponding sentence in Section 3. revision: yes
Referee: [Abstract] Abstract: the reported performance and volume-reduction results supply no information on baselines, number of independent runs, statistical tests, task definitions, or how communication volume is precisely defined and measured, rendering the empirical superiority claim unverifiable from the provided text.

Authors: Abstracts are necessarily concise and the full experimental protocol (baselines such as CommNet and TarMAC, five independent seeds with standard errors, task definitions, and volume measured as total transmitted feature bytes per receiver per episode) appears in Section 4 and the appendix. We nevertheless accept that the abstract would benefit from a brief clause on the evaluation setting. We will add one sentence to the abstract summarizing the experimental scope and volume definition. revision: yes

Circularity Check

0 steps flagged

No circularity: architectural proposal validated by direct empirical measurement

full rationale

The paper introduces HiComm as a receiver-driven three-stage query mechanism that converts flat-vector communication into structured retrieval over an observation hierarchy. The central performance claim (matching baselines while reducing volume up to 23×) is presented as an experimental outcome across tasks, not as a mathematical derivation or fitted parameter renamed as a prediction. No equations define the volume reduction in terms of the method itself, no self-citations supply load-bearing uniqueness theorems, and no ansatz is smuggled via prior work. The method is a plug-in architectural change whose benefits are measured externally to its own definitions, satisfying the criteria for a self-contained empirical result.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Review performed on abstract only; no explicit free parameters, additional axioms, or invented entities beyond the proposed module are detailed in the provided text.

axioms (1)

domain assumption Observations naturally follow a hierarchy such as groups and entities
Invoked in the abstract as an important source of inductive bias overlooked by prior protocols.

invented entities (1)

HiComm module with three-stage decoding no independent evidence
purpose: To convert flat communication into structured retrieval over observation hierarchy
New architectural component introduced without external falsifiable evidence supplied in the abstract.

pith-pipeline@v0.9.1-grok · 5750 in / 1217 out tokens · 30635 ms · 2026-07-02T20:52:43.331309+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Reference graph

Works this paper leans on

43 extracted references · 4 canonical work pages · 3 internal anchors

[1]

Feudal multi-agent hierarchies for cooperative reinforcement learning

S Ahilan and P Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning. InWorkshop on Structure & Priors in Reinforcement Learning (SPiRL 2019) at ICLR 2019, pages 1–11, 2019

2019
[2]

Tarmac: Targeted multi-agent communication

Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. InInternational Conference on machine learning, pages 1538–1546. PMLR, 2019

2019
[3]

Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533,

Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020

work page arXiv 2011
[4]

Multi-agent coordination via multi-level communication.Advances in Neural Information Processing Systems, 37:118513–118539, 2024

Ziluo Ding, Zeyuan Liu, Zhirui Fang, Kefan Su, Liwen Zhu, and Zongqing Lu. Multi-agent coordination via multi-level communication.Advances in Neural Information Processing Systems, 37:118513–118539, 2024

2024
[5]

Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 36:37567–37593, 2023

Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 36:37567–37593, 2023

2023
[6]

Learning to communicate with deep multi-agent reinforcement learning

Jakob N Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. InProceedings of the 30th International Conference on Neural Information Processing Systems, pages 2145–2153, 2016

2016
[7]

Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient communication via self-supervised information aggregation for online and offline multiagent reinforcement learning.IEEE Transactions on Neural Networks and Learning Systems, 36(5):9044–9056, 2024

2024
[8]

Learning multi-agent communication from graph modeling perspective

Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Qox9rO0kN0

2024
[9]

Categorical reparameterization with gumbel-softmax

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum? id=rkE3y85ee

2017
[10]

Learning attentional communication for multi-agent cooperation

Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31, 2018

2018
[11]

Graph convolutional reinforcement learning

Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/ forum?id=HkxdQkSYDB. 10

2020
[12]

Cage challenge 4: A scalable multi-agent reinforcement learning gym for autonomous cyber defence.AI Magazine, 46(3):e70021, 2025

Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Cage challenge 4: A scalable multi-agent reinforcement learning gym for autonomous cyber defence.AI Magazine, 46(3):e70021, 2025

2025
[13]

Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective

Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective. InProceedings of the AAAI Conference on Artificial Intelligence, volum...

2025
[14]

Learning to Schedule Communication in Multi-agent Reinforcement Learning

Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning.arXiv preprint arXiv:1902.01554, 2019

work page internal anchor Pith review Pith/arXiv arXiv 1902
[15]

Google research football: A novel reinforcement learning environment

Karol Kurach, Anton Raichuk, Piotr Sta ´nczyk, Michał Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 4501–4510, 2020

2020
[16]

Deep implicit coordina- tion graphs for multi-agent reinforcement learning

Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer. Deep implicit coordina- tion graphs for multi-agent reinforcement learning. InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 764–772, 2021

2021
[17]

Context-aware communication for multi-agent reinforcement learning

Xinran Li and Jun Zhang. Context-aware communication for multi-agent reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 1156–1164, 2024

2024
[18]

When2com: Multi-agent perception via communication graph grouping

Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 4106–4115, 2020

2020
[19]

Who2com: Collaborative perception via learnable handshake communication

Yen-Cheng Liu, Junjiao Tian, Chih-Yao Ma, Nathan Glaser, Chia-Wen Kuo, and Zsolt Kira. Who2com: Collaborative perception via learnable handshake communication. In2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6876–6883. IEEE, 2020

2020
[20]

Deep hierarchical com- munication graph in multi-agent reinforcement learning

Zeyang Liu, Lipeng Wan, Xue Sui, Zhuoran Chen, Kewu Sun, and Xuguang Lan. Deep hierarchical com- munication graph in multi-agent reinforcement learning. InProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 208–216, 2023

2023
[21]

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

Tommaso Marzi, Cesare Alippi, and Andrea Cini. Hierarchical message-passing policies for multi-agent reinforcement learning.arXiv preprint arXiv:2507.23604, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025
[22]

Multi-agent graph-attention communication and teaming

Yaru Niu, Rohan Paleja, and Matthew Gombolay. Multi-agent graph-attention communication and teaming. InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 964–973, 2021

2021
[23]

Learning multi-agent communication through structured attentive reasoning.Advances in Neural Information Processing Systems, 33:10088–10098, 2020

Murtaza Rangwala and Ryan Williams. Learning multi-agent communication through structured attentive reasoning.Advances in Neural Information Processing Systems, 33:10088–10098, 2020

2020
[24]

Multi-agent actor-critic with hierarchical graph attention network

Heechang Ryu, Hayong Shin, and Jinkyoo Park. Multi-agent actor-critic with hierarchical graph attention network. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7236–7243, 2020

2020
[25]

The starcraft multi-agent challenge

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. InProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186–2188, 2019

2019
[26]

Learning structured communication for multi-agent reinforcement learning.Autonomous Agents and Multi-Agent Systems, 36(2):50, 2022

Junjie Sheng, Xiangfeng Wang, Bo Jin, Junchi Yan, Wenhao Li, Tsung-Hui Chang, Jun Wang, and Hongyuan Zha. Learning structured communication for multi-agent reinforcement learning.Autonomous Agents and Multi-Agent Systems, 36(2):50, 2022

2022
[27]

Hierarchical multi-agent reinforcement learning for cyber network defense

Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Peter Chin, and Alina Oprea. Hierarchical multi-agent reinforcement learning for cyber network defense. InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pages 2747–2749, 2025

2025
[28]

Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks

Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks.arXiv preprint arXiv:1812.09755, 2018. 11

work page internal anchor Pith review Pith/arXiv arXiv 2018
[29]

Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness

Shoucheng Song, Youfang Lin, Sheng Han, Chang Yao, Hao Wu, Shuo Wang, and Kai Lv. Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23304–23312, 2025

2025
[30]

Boosting studies of multi-agent reinforcement learning on google research football environment: the past, present, and future

Y Song, H Jiang, H Zhang, Z Tian, W Zhang, and J Wang. Boosting studies of multi-agent reinforcement learning on google research football environment: the past, present, and future. InProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, volume 2024, pages 1772–1781. Association for Computing Machinery (ACM), 2024

2024
[31]

Learning multiagent communication with backprop- agation

Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backprop- agation. InProceedings of the 30th International Conference on Neural Information Processing Systems, pages 2252–2260, 2016

2016
[32]

T2mac: Targeted and trusted multi-agent communication through selective engagement and evidence- driven integration

Chuxiong Sun, Zehua Zang, Jiabao Li, Jiangmeng Li, Xiao Xu, Rui Wang, and Changwen Zheng. T2mac: Targeted and trusted multi-agent communication through selective engagement and evidence- driven integration. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15154–15163, 2024

2024
[33]

Learning nearly decomposable value functions via communication minimization

Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. InInternational Conference on Learning Representations,
[34]

URLhttps://openreview.net/forum?id=HJx-3grYDB
[35]

Context-aware sparse deep coordination graphs

Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang. Context-aware sparse deep coordination graphs. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=wQfgfb8VKTn

2022
[36]

Subgoal-based hierarchical reinforcement learning for multiagent collaboration.IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2026

Cheng Xu, Yuchen Shi, Changtian Zhang, Ran Wang, Shihong Duan, Yadong Wan, and Xiaotong Zhang. Subgoal-based hierarchical reinforcement learning for multiagent collaboration.IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2026

2026
[37]

The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

2022
[38]

Multi-agent incentive communication via decentralized teammate modeling

Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 9466–9474, 2022

2022
[39]

G-designer: Architecting multi-agent communication topologies via graph neural networks

Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks. InForty-second International Conference on Machine Learning, 2025. URL https: //openreview.net/forum?id=LpE54NUnmO

2025
[40]

Efficient communication in multi-agent reinforcement learning via variance based control

Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. InAdvances in Neural Information Processing Systems, pages 3235–3244, 2019

2019
[41]

Succinct and robust multi-agent communication with temporal message control.Advances in Neural Information Processing Systems, 33:17271–17282, 2020

Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Succinct and robust multi-agent communication with temporal message control.Advances in Neural Information Processing Systems, 33:17271–17282, 2020

2020
[42]

higher is better

Zhuohui Zhang, Bin He, Bin Cheng, and Gang Li. Bridging training and execution via dynamic directed graph-based communication in cooperative multi-agent systems. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23395–23403, 2025. 12 A Additional Details for Experiments All experiments were conducted on a single node equipp...

2025
[43]

owned by us

On the 5v5 scenarios this resolves (Ne, Na−1) to (5,4) , so the concrete obs_segs we hand to HICOMMis (1,4),(5,|F |),(4,|F |),(1, o a),(1,11),(1,5) , with |F | ∈ {8,9} and oa ∈ {4,5} chosen by race as above. The action space is a discrete head of size Ne + 6 (Ne unit targeted attacks plus no-op, stop, and the four cardinal moves), with a per step validity...

[1] [1]

Feudal multi-agent hierarchies for cooperative reinforcement learning

S Ahilan and P Dayan. Feudal multi-agent hierarchies for cooperative reinforcement learning. InWorkshop on Structure & Priors in Reinforcement Learning (SPiRL 2019) at ICLR 2019, pages 1–11, 2019

2019

[2] [2]

Tarmac: Targeted multi-agent communication

Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. InInternational Conference on machine learning, pages 1538–1546. PMLR, 2019

2019

[3] [3]

Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533,

Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020

work page arXiv 2011

[4] [4]

Multi-agent coordination via multi-level communication.Advances in Neural Information Processing Systems, 37:118513–118539, 2024

Ziluo Ding, Zeyuan Liu, Zhirui Fang, Kefan Su, Liwen Zhu, and Zongqing Lu. Multi-agent coordination via multi-level communication.Advances in Neural Information Processing Systems, 37:118513–118539, 2024

2024

[5] [5]

Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 36:37567–37593, 2023

Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 36:37567–37593, 2023

2023

[6] [6]

Learning to communicate with deep multi-agent reinforcement learning

Jakob N Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. InProceedings of the 30th International Conference on Neural Information Processing Systems, pages 2145–2153, 2016

2016

[7] [7]

Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient communication via self-supervised information aggregation for online and offline multiagent reinforcement learning.IEEE Transactions on Neural Networks and Learning Systems, 36(5):9044–9056, 2024

2024

[8] [8]

Learning multi-agent communication from graph modeling perspective

Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=Qox9rO0kN0

2024

[9] [9]

Categorical reparameterization with gumbel-softmax

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum? id=rkE3y85ee

2017

[10] [10]

Learning attentional communication for multi-agent cooperation

Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31, 2018

2018

[11] [11]

Graph convolutional reinforcement learning

Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. InInternational Conference on Learning Representations, 2020. URL https://openreview.net/ forum?id=HkxdQkSYDB. 10

2020

[12] [12]

Cage challenge 4: A scalable multi-agent reinforcement learning gym for autonomous cyber defence.AI Magazine, 46(3):e70021, 2025

Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Cage challenge 4: A scalable multi-agent reinforcement learning gym for autonomous cyber defence.AI Magazine, 46(3):e70021, 2025

2025

[13] [13]

Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective

Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective. InProceedings of the AAAI Conference on Artificial Intelligence, volum...

2025

[14] [14]

Learning to Schedule Communication in Multi-agent Reinforcement Learning

Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning.arXiv preprint arXiv:1902.01554, 2019

work page internal anchor Pith review Pith/arXiv arXiv 1902

[15] [15]

Google research football: A novel reinforcement learning environment

Karol Kurach, Anton Raichuk, Piotr Sta ´nczyk, Michał Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 4501–4510, 2020

2020

[16] [16]

Deep implicit coordina- tion graphs for multi-agent reinforcement learning

Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer. Deep implicit coordina- tion graphs for multi-agent reinforcement learning. InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 764–772, 2021

2021

[17] [17]

Context-aware communication for multi-agent reinforcement learning

Xinran Li and Jun Zhang. Context-aware communication for multi-agent reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 1156–1164, 2024

2024

[18] [18]

When2com: Multi-agent perception via communication graph grouping

Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 4106–4115, 2020

2020

[19] [19]

Who2com: Collaborative perception via learnable handshake communication

Yen-Cheng Liu, Junjiao Tian, Chih-Yao Ma, Nathan Glaser, Chia-Wen Kuo, and Zsolt Kira. Who2com: Collaborative perception via learnable handshake communication. In2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6876–6883. IEEE, 2020

2020

[20] [20]

Deep hierarchical com- munication graph in multi-agent reinforcement learning

Zeyang Liu, Lipeng Wan, Xue Sui, Zhuoran Chen, Kewu Sun, and Xuguang Lan. Deep hierarchical com- munication graph in multi-agent reinforcement learning. InProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pages 208–216, 2023

2023

[21] [21]

Hierarchical Message-Passing Policies for Multi-Agent Reinforcement Learning

Tommaso Marzi, Cesare Alippi, and Andrea Cini. Hierarchical message-passing policies for multi-agent reinforcement learning.arXiv preprint arXiv:2507.23604, 2025

work page internal anchor Pith review Pith/arXiv arXiv 2025

[22] [22]

Multi-agent graph-attention communication and teaming

Yaru Niu, Rohan Paleja, and Matthew Gombolay. Multi-agent graph-attention communication and teaming. InProceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 964–973, 2021

2021

[23] [23]

Learning multi-agent communication through structured attentive reasoning.Advances in Neural Information Processing Systems, 33:10088–10098, 2020

Murtaza Rangwala and Ryan Williams. Learning multi-agent communication through structured attentive reasoning.Advances in Neural Information Processing Systems, 33:10088–10098, 2020

2020

[24] [24]

Multi-agent actor-critic with hierarchical graph attention network

Heechang Ryu, Hayong Shin, and Jinkyoo Park. Multi-agent actor-critic with hierarchical graph attention network. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7236–7243, 2020

2020

[25] [25]

The starcraft multi-agent challenge

Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. InProceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 2186–2188, 2019

2019

[26] [26]

Learning structured communication for multi-agent reinforcement learning.Autonomous Agents and Multi-Agent Systems, 36(2):50, 2022

Junjie Sheng, Xiangfeng Wang, Bo Jin, Junchi Yan, Wenhao Li, Tsung-Hui Chang, Jun Wang, and Hongyuan Zha. Learning structured communication for multi-agent reinforcement learning.Autonomous Agents and Multi-Agent Systems, 36(2):50, 2022

2022

[27] [27]

Hierarchical multi-agent reinforcement learning for cyber network defense

Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Peter Chin, and Alina Oprea. Hierarchical multi-agent reinforcement learning for cyber network defense. InProceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, pages 2747–2749, 2025

2025

[28] [28]

Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks

Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks.arXiv preprint arXiv:1812.09755, 2018. 11

work page internal anchor Pith review Pith/arXiv arXiv 2018

[29] [29]

Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness

Shoucheng Song, Youfang Lin, Sheng Han, Chang Yao, Hao Wu, Shuo Wang, and Kai Lv. Code: Communication delay-tolerant multi-agent collaboration via dual alignment of intent and timeliness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23304–23312, 2025

2025

[30] [30]

Boosting studies of multi-agent reinforcement learning on google research football environment: the past, present, and future

Y Song, H Jiang, H Zhang, Z Tian, W Zhang, and J Wang. Boosting studies of multi-agent reinforcement learning on google research football environment: the past, present, and future. InProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, volume 2024, pages 1772–1781. Association for Computing Machinery (ACM), 2024

2024

[31] [31]

Learning multiagent communication with backprop- agation

Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus. Learning multiagent communication with backprop- agation. InProceedings of the 30th International Conference on Neural Information Processing Systems, pages 2252–2260, 2016

2016

[32] [32]

T2mac: Targeted and trusted multi-agent communication through selective engagement and evidence- driven integration

Chuxiong Sun, Zehua Zang, Jiabao Li, Jiangmeng Li, Xiao Xu, Rui Wang, and Changwen Zheng. T2mac: Targeted and trusted multi-agent communication through selective engagement and evidence- driven integration. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15154–15163, 2024

2024

[33] [33]

Learning nearly decomposable value functions via communication minimization

Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. InInternational Conference on Learning Representations,

[34] [34]

URLhttps://openreview.net/forum?id=HJx-3grYDB

[35] [35]

Context-aware sparse deep coordination graphs

Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang. Context-aware sparse deep coordination graphs. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=wQfgfb8VKTn

2022

[36] [36]

Subgoal-based hierarchical reinforcement learning for multiagent collaboration.IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2026

Cheng Xu, Yuchen Shi, Changtian Zhang, Ran Wang, Shihong Duan, Yadong Wan, and Xiaotong Zhang. Subgoal-based hierarchical reinforcement learning for multiagent collaboration.IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2026

2026

[37] [37]

The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surpris- ing effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022

2022

[38] [38]

Multi-agent incentive communication via decentralized teammate modeling

Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 9466–9474, 2022

2022

[39] [39]

G-designer: Architecting multi-agent communication topologies via graph neural networks

Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks. InForty-second International Conference on Machine Learning, 2025. URL https: //openreview.net/forum?id=LpE54NUnmO

2025

[40] [40]

Efficient communication in multi-agent reinforcement learning via variance based control

Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. InAdvances in Neural Information Processing Systems, pages 3235–3244, 2019

2019

[41] [41]

Succinct and robust multi-agent communication with temporal message control.Advances in Neural Information Processing Systems, 33:17271–17282, 2020

Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Succinct and robust multi-agent communication with temporal message control.Advances in Neural Information Processing Systems, 33:17271–17282, 2020

2020

[42] [42]

higher is better

Zhuohui Zhang, Bin He, Bin Cheng, and Gang Li. Bridging training and execution via dynamic directed graph-based communication in cooperative multi-agent systems. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23395–23403, 2025. 12 A Additional Details for Experiments All experiments were conducted on a single node equipp...

2025

[43] [43]

owned by us

On the 5v5 scenarios this resolves (Ne, Na−1) to (5,4) , so the concrete obs_segs we hand to HICOMMis (1,4),(5,|F |),(4,|F |),(1, o a),(1,11),(1,5) , with |F | ∈ {8,9} and oa ∈ {4,5} chosen by race as above. The action space is a discrete head of size Ne + 6 (Ne unit targeted attacks plus no-op, stop, and the four cardinal moves), with a per step validity...