Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Dynamic Graph Communication for Decentralised Multi-Agent Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This thesis claims that a decentralized multi-agent routing system can learn from the routing reward alone which neighbouring messages matter and which neighbours to stop messaging, and that this learned dynamic communication beats the…

desk verdict Promising and honest thesis, but the headline 9.5% gain rests on an Iteration Controller whose training mechanism is never described. read the letter →

arxiv 2501.00165 v1 pith:NALXRUBE submitted 2024-12-30 cs.MA

Ben McClusky This is my paper · ORCID
classification cs.MA
keywords Multi-AgentReinforcementLearningDecentralisedSystemsDynamicNetworksGraphAttentionMulti-RoundCommunicationNetworkPacketRoutingTargetingSparse-Reward
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Packet-routing networks in the real world lose nodes, but most multi-agent reinforcement learning routing systems assume a fixed topology. This thesis tries to establish that a decentralized system can adapt by learning communication itself: a graph attention layer decides which neighbours' messages matter when a router updates its view of the network, and a learned 'Iteration Controller' decides which neighbours should receive the router's updated state in later rounds of a multi-round message-passing loop. Both are trained end-to-end on the packet-delivery reward, and the paper reports that the combined system beats the base NetMon configuration by 9.5% in reward, 8.3% in throughput, 6.9% in delay, and 8.5% in looped packets, all while sending 6.4% fewer messages. The claim matters because selective, learned communication is the piece that would let such routing agents scale to large dynamic networks where bandwidth and topology are constrained.

What carries the argument

Two learnable components inserted into the base NetMon recurrent message-passing loop, in which each router encodes local observations, exchanges hidden states with immediate neighbours over up to four communication rounds, and updates a hidden representation using recurrent networks. The first is a single-head Graph Attention Network (GAT), which replaces simple summation as the aggregation step and computes attention coefficients over neighbouring hidden states to weight which messages shape the node's graph representation. The second, the Iteration Controller, runs a multi-head attention layer over the concatenation of the node's own hidden state and its neighbours' states, projects the output through a sigmoid to produce a per-neighbour transmit decision for the next round, and is stabilised by a communication-bias initialisation plus exploration noise added to the sigmoid input. The paper states that both components are trained end-to-end through the routing agent's deep Q-network temporal-difference loss.

What would settle it

Compute the gradient norm of the routing loss with respect to the Iteration Controller's weights during a training run: if it is zero, the binary decision (a thresholded sigmoid with no gradient estimator) is not being trained end-to-end at all. Then run the full GAT-based system with the Iteration Controller replaced by a fixed per-round transmission probability equal to the controller's measured average message count; if the reward gain persists, the targeting mechanism itself contributes nothing beyond sending fewer messages.

Watch

Extended reading notes

Core claim

The paper's central claim is that attention-based communication can be learned from reinforcement learning alone in a sparse-reward, dynamic packet-routing environment, something earlier multi-round message-passing routing work [21] had concluded required supervised signal. The argument is that the base NetMon system's decoupling of node-level graph representation learning from packet-level routing decisions makes the node model's learning task nearly stationary, so a single graph attention layer can be trained as the message aggregator and the Iteration Controller can be trained as a per-neighbour gate that stops communication in later rounds when the node already has what it needs. On 1,000 held-out test graphs with a 20% per-step node failure probability, the full system is reported to outperform the base NetMon configuration on reward, throughput, delay, and looped packets while using 6.4% less communication; in isolation the GAT component accounts for a 4.8% reward gain and the Iteration Controller for a 9.1% gain, which the paper reads as a cumulative effect.

Load-bearing premise

The entire gain attributed to learned targeting rests on the Iteration Controller actually learning from the routing reward, but the paper never explains how a gradient reaches the binary transmit-or-block decision it makes for each neighbour.

Editorial extensions

If this is right

  • Attention-based aggregation can be trained in a sparse-reward, dynamic routing environment with reinforcement learning alone, without the supervised pre-training that earlier multi-round message-passing work required.
  • A router that learns to withhold messages does not sacrifice performance: in isolation the Iteration Controller raised rewards by 9.1% while sending 5.4% fewer messages than the maximum-communication baseline in the dynamic routing environment.
  • The GAT and Iteration Controller gains are additive (4.8% and 9.1% in isolation versus 9.5% combined), so the two modules can be composed inside existing recurrent message-passing frameworks.
  • Communication can be cut most aggressively in the later rounds of multi-round messaging, where the Iteration Controller reduced traffic by 15.7% in the supervised regression task, suggesting high-round systems have the most overhead to reclaim.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the full system's 9.5% gain was never compared against a matched-overhead control that talks as much as the learned controller but does not target anyone; the paper's Matched Communication baseline was applied to the Iteration Controller in isolation only, so attribution of the combined gain to learned targeting remains open.
  • The Iteration Controller is not routing-specific: the same attention-plus-sigmoid gate could be dropped into any networked multi-agent setting with a message budget, such as sensor networks or traffic control, and tested unchanged.
  • A direct testable extension is to replace the paper's unspecified binary-decision training path with an explicit Gumbel-Softmax or REINFORCE estimator; matching or beating the reported gains would confirm end-to-end learnability, while failing to do so would indicate the current pipeline is not learning what it claims.
  • The claim of being the first reinforcement-learning-trained attention aggregation in sparse-reward dynamic routing leans on the base system's node/agent decoupling; retraining the same modules in a single shared model would isolate whether decoupling, rather than the new components, is what makes attention learnable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper extends the NetMon framework for decentralised multi-agent reinforcement learning in network packet routing to dynamic networks with node failures. It introduces two components: a Graph Attention Network (GAT) layer for message aggregation and an 'Iteration Controller' for multi-round communication targeting that decides which neighbours receive updates in each round. The central empirical claim is that the combined system improves reward by 9.5%, throughput by 8.3%, reduces delay by 6.9% and looped packets by 8.5% while using 6.4% less communication than NetMon (Section 7.2.1). Ablation studies report 4.8% reward improvement from GAT and 9.1% from the Iteration Controller. The paper also evaluates the components on a supervised shortest-path regression task and compares against DQN, DRQN, CommNet, DGN, and NetMon baselines.

Significance. If the central claims hold, the work would demonstrate a decentralised, end-to-end RL-trained communication targeting mechanism in a sparse-reward dynamic routing environment, an area where prior attention-based aggregation has often required supervised auxiliary losses or centralised training. The paper has notable strengths: it includes a matched-communication control that equalises average overhead, evaluates on 1,000 unseen graphs over 5 seeds, reports multiple routing metrics, and honestly discusses limitations such as small network size and lack of a centralised comparison. The comparison against NetMon and other MARL baselines is a useful contribution. However, the training mechanism for the Iteration Controller's binary decisions is underspecified, and the ablation arithmetic is internally inconsistent; these issues undermine the ability to attribute the headline 9.5% gain to the proposed components. The significance is therefore conditional on clarifying and, where necessary, correcting these points.

major comments (4)
  1. [Section 7.2.1 and Section 8.1] The Iteration Controller's forward pass outputs a continuous sigmoid value y, but the text states that a binary decision (transmit or not) is made for each neighbour. No threshold, sampling rule, or gradient estimator (e.g., straight-through, Gumbel-Softmax, REINFORCE, or a soft-hard switch) is described in Algorithm 8, Algorithm 9, or the training procedure in Algorithm 5. If a hard threshold is applied, the gradient of the routing loss with respect to the Iteration Controller's parameters is zero and the claimed end-to-end training via the routing DQN loss cannot occur. If instead the sigmoid is used as a soft mask, the system is differentiable but not truly binary, and the measured 'messages sent' reduction would not correspond to a discrete communication saving. This is load-bearing for research question Q4 and for the attribution of the 9.5% reward gain to learned targeting.
  2. [Section 7.2.1 and Section 8.1] The paper describes the combined 9.5% reward improvement as demonstrating 'a powerful synergy' (Section 7.2.1) and a 'cumulative effect' (Section 8.1) of the GAT (4.8%) and Iteration Controller (9.1%). Since these individual gains sum to 13.9%, the observed combined gain of 9.5% is sub-additive, not super-additive. The manuscript needs to report the interaction effect explicitly, or correct the 'synergy' characterisation. As written, the ablation decomposition is internally inconsistent.
  3. [Section 5.4 vs Section 6.4 vs Section 7.1] The GAT ablation in the Dynamic Network Packet Routing environment uses only one communication round, while the Iteration Controller ablation and the full-system evaluation use four communication rounds. The individual reward contributions (4.8% for GAT and 9.1% for the Iteration Controller) are therefore not measured under comparable conditions, and the statement in Section 7.2.1 that these components 'individually increased rewards by 4.8% and 9.1%' is misleading. This invalidates the implicit decomposition of the 9.5% combined gain and weakens the claims of 'cumulative effect' and 'powerful synergy'.
  4. [Section 7.2.1, Figure 7.3] The headline improvements over NetMon are reported without statistical significance testing or confidence intervals on the percentage differences. The error bars in Figure 7.3 appear substantial and overlapping for several metrics, so it is unclear whether the 9.5% reward gap, 8.3% throughput gap, and 6.9% delay reduction are statistically meaningful. Paired tests across the evaluation graphs or across the five seeds, or explicit confidence intervals for the relative differences, are needed to support the claim that the proposed system 'significantly outperformed' the baseline.
minor comments (4)
  1. [Section 1.2] The research questions are numbered Q1, Q2, Q4, Q3; this should be renumbered sequentially to Q1, Q2, Q3, Q4.
  2. [Algorithm 5] The pseudocode in Algorithm 5 uses the index j both for the outer batch sequence loop and for the inner timestep references; this makes the indexing ambiguous. Please clarify the notation (e.g., use j0 as the start index and t for the inner steps).
  3. [Section 6.5.1] The phrase 'Averages Messages per Node' should be 'Average Messages per Node' for grammatical consistency with similar captions.
  4. [Section 3.5] Algorithm 4 stores the tuple (ht, ht+1, mt, mt+1, st, st+1, ot, at, rt, ot+1), but the later text and Algorithm 5 do not explicitly define how ht is reset between episodes. Please state the reset conditions for node states at episode boundaries.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline results are empirical comparisons against external baselines, with no derivation step that reduces to its own inputs.

full rationale

The paper's central claims are empirical measurements, not derivations: the 9.5% reward improvement and 6.4% communication reduction are evaluated on 1,000 held-out test graphs against NetMon, an external baseline, and against DQN, DRQN, CommNet, and DGN. The GAT and Iteration Controller are trained with the routing DQN TD loss described in Algorithms 5 and 7; no equation in the paper defines the reported reward gain in terms of the model's own parameters or fitted values. The Matched Communication baseline is a deliberate experimental control that equalizes overhead, which isolates the targeting mechanism's performance effect rather than building the conclusion into the comparison. The communication-bias and noise-scale hyperparameters are tuned, but the reported overhead reduction is a measured outcome of the learned controller, not a fitted parameter renamed as a prediction. The most serious concern, that the binary transmit decision in Algorithms 8 and 9 lacks an explicit gradient estimator and may therefore not be trainable end-to-end, is a missing-mechanism or support gap rather than a circularity: it does not make any claimed result equivalent to its input by construction. The paper also relies on Weil et al. for its base NetMon architecture, but that is an independent external citation with no author overlap, so it is not a self-citation chain. Overall, no circular step meeting the quoted-evidence standard is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on several hand-chosen hyperparameters (communication bias, noise scaling, number of rounds, failure rate) and on unstated assumptions about the trainability of discrete gating and the realism of the idealized control plane. These are not derived from first principles, so the reader should treat the 9.5% improvement as specific to this environment and configuration.

free parameters (4)
  • Communication Bias = 0.5
    Grid-searched on validation to counter early-round bias; directly affects Iteration Controller gating behavior and final overhead.
  • Noise Scaling = 0.3
    Grid-searched to balance exploration in Iteration Controller training; affects stability and final performance.
  • Maximum Communication Rounds = 4 (1 in GAT ablation)
    Set to 4 in the final system to highlight Iteration Controller benefits; the GAT ablation used only 1 round due to compute limits, creating an inconsistency in the ablation story.
  • Node Failure Probability = 20% per step
    Environment design choice; directly defines the dynamic network and influences all reported results.
assumptions (4)
  • domain assumption Idealised control plane with instantaneous, unlimited, lossless communication
    Table 3.2: communication overhead is measured as message count, not real bandwidth or latency; this is load-bearing for the 6.4% overhead claim.
  • ad hoc to paper The binary communication decisions of the Iteration Controller can be trained through the routing reward
    Section 6.3 and Algorithm 9: the paper assumes gradients flow through hard gating without specifying a surrogate (Gumbel, REINFORCE, straight-through), which is essential for the claimed end-to-end RL training.
  • domain assumption Decoupling node and agent observations makes node representation learning nearly stationary
    Section 2.4.2 and 5.5.2: this NetMon assumption is required for attention to be trainable in a sparse-reward RL setting.
  • domain assumption Generated graphs and failure dynamics are representative of real dynamic networks
    Section 4.4: random placement, fixed degree D=3, 20% failure probability; this generalization is assumed, not validated against real topologies.
invented entities (1)
  • Iteration Controller
    purpose: Decides per neighbor per round whether to transmit the updated hidden state, to reduce communication overhead.
    A new neural module introduced by the paper; its effectiveness is only demonstrated in the authors' simulated environment, with no external benchmark or independent prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dynamic Graph Communication for Decentralised Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/NALXRUBE

@misc{pith2026250100165,
  author       = {Pith},
  title        = {Pith review of: Dynamic Graph Communication for Decentralised Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NALXRUBE}},
  note         = {Machine review of arXiv:2501.00165}
}
read the original abstract

This work presents a novel communication framework for decentralized multi-agent systems operating in dynamic network environments. Integrated into a multi-agent reinforcement learning system, the framework is designed to enhance decision-making by optimizing the network's collective knowledge through efficient communication. Key contributions include adapting a static network packet-routing scenario to a dynamic setting with node failures, incorporating a graph attention network layer in a recurrent message-passing framework, and introducing a multi-round communication targeting mechanism. This approach enables an attention-based aggregation mechanism to be successfully trained within a sparse-reward, dynamic network packet-routing environment using only reinforcement learning. Experimental results show improvements in routing performance, including a 9.5 percent increase in average rewards and a 6.4 percent reduction in communication overhead compared to a baseline system. The study also examines the ethical and legal implications of deploying such systems in critical infrastructure and military contexts, identifies current limitations, and suggests potential directions for future research.

Figures

Figures reproduced from arXiv: 2501.00165 by the authors.

Figure 2.1
Figure 2.1. Agent-Environment interaction within a Markov Decision Process [78]: The agent receives the current state st from the environment and takes an action at . The environment then provides a reward rt and the next state st+1. At its core, RL is the interaction between an agent and its environment. The agent observes the current state of the environment and takes an action according to its learned policy (mapping of stat… view at source ↗
Figure 2.2
Figure 2.2. Deep Q-Network (DQN) Architecture [10]: The agent utilises experience replay to store and sample experiences for training. The Q-network is trained by minimising the loss between the predicted Q-value and the target Q￾value, with periodic updates to the target network’s weights (θ → θ ′ ). 8 [PITH_FULL_IMAGE:figures/full_fig_p013_2_2.png] view at source ↗
Figure 2.3
Figure 2.3. Actor-Critic Framework: The Actor (Policy) selects actions based on the state. The Critic (Value Function) evaluates these actions using the TD error to update both the Policy and Value Function, iteratively improving performance. [78] 2.2 Multi-Agent Systems As intelligent and autonomous systems become more prevalent, effectively man￾aging interactions between multiple autonomous agents has become increasingly impo… view at source ↗
Figures from the paper (54 more)
Figure 2.4
Figure 2.4. Figure 2.4: POMDP framework: The agent updates its belief state bt based on observation ot and selects action at+1 according to its policy. The environment transitions to a new state st+1, emits an observation, and provides a reward rt . Non-Stationarity [PITH_FULL_IMAGE:figure…
Figure 2.5
Figure 2.5. Figure 2.5: A non-stationary environment in a multi-agent system, where each ”model” represents an agent. Agents’ actions at cause transitions between states S1, S2, S3, with evolving system dynamics as agents update their policies. [57] 11 [PITH_FULL_IMAGE:figures/full_fig_p01…
Figure 2.6
Figure 2.6. Figure 2.6: Categories of communication structures in Multi-Agent Systems [95]. Effective communication in a multi-agent system involves deciding what information to share, with whom, and when. There are three types of communication structures: centralised, where agents share lo…
Figure 2.7
Figure 2.7. Figure 2.7: Dec-POMPD: At time-step t, the environment is in state st and emits a joint observation ot . Each agent i receives observation oi,t, takes action ai,t, forming joint action at . The environment transitions to state st+1 and emits reward rt . [86] The most straightfor…
Figure 2.8
Figure 2.8. Figure 2.8: Network Topologies: Peer-to-Peer (blue), Hierarchical (green), Dense (red), and Sparse (yellow) The topology of the communication network is the first consideration, as it governs the network’s efficiency, scalability, and resilience by defining how agents are connec…
Figure 2.9
Figure 2.9. Figure 2.9: Mean field approximation: each agent (node) is influenced by the mean effect of its neighbours (blue region) [89]. Consensus Mechanism The second approach involves agents coordinating actions with neighbours via a consensus mechanism. Constraining information transfe…
Figure 2.10
Figure 2.10. Figure 2.10: Consensus Network [56]: A network where each node i exchanges information with its immediate neighbors to achieve consensus on a shared variable. In policy evaluation, agents minimise the Mean Square Projected Bellman Error (MSPBE) to learn the value function for a …
Figure 2.11
Figure 2.11. Figure 2.11: When2Comm Adjacency Matrix: Rows with a self-attention score of 1 are replaced with an identity row, indicating no further communication is needed. Other rows are derived from scaled, pruned, and activated matching scores. Conventionally, targeting mechanisms use a …
Figure 2.12
Figure 2.12. Figure 2.12: GAT: Computing attention coefficients using a shared mechanism across node pairs (left). Using these coefficients to compute the weighted sum of neighboring node features, updating each node’s feature representation (right) [81]. 23 [PITH_FULL_IMAGE:figures/full_fi…
Figure 2.13
Figure 2.13. Figure 2.13: Network packet routing from source to destination via intermediary routers, illustrating the selection of efficient paths based on routing protocols [3]. Routing Protocols Routing decisions rely on protocols that determine the optimal path based on network topology,…
Figure 2.14
Figure 2.14. Figure 2.14: Comparison of Traditional Network and SDN Architectures [47]. SDN’s flexibility supports networked MARL approaches by enabling direct commu￾nication between routers and switches for control decisions. Deep Q-routing with Communication (DQRC) [90] builds on DQN-routi…
Figure 2.15
Figure 2.15. Figure 2.15: NetMon Process [85]: Nodes encode and share observations to update local states which are transferred to the agent for routing decisions. 27 While not explicitly mentioned in the paper, the name ”NetMon” is used internally within the authors’ code repository. For de…
Figure 2.16
Figure 2.16. Figure 2.16: NetMon uses a recurrent message-passing model with two LSTM networks to encode and aggregate information across the network [85]. 28 [PITH_FULL_IMAGE:figures/full_fig_p033_2_16.png]
Figure 3.1
Figure 3.1. Figure 3.1: Foundational Architecture: Showing the left-to-right flow from the node model, where local observations are processed and exchanged between nodes, to the agent model, which integrates these observations and makes routing decisions. For the next communication round th…
Figure 3.2
Figure 3.2. Figure 3.2: Proposed Architecture: Flow through a GAT layer (white) for aggregation and an Iteration Controller (red) for dynamic communication control. The design exclusively focuses on optimising the performance of the node model, further improvements to the agent model, beyon…
Figure 4.1
Figure 4.1. Figure 4.1: Example generated graphs with L = 20 and D = 3, where nodes are color-coded by their betweenness centrality [PITH_FULL_IMAGE:figures/full_fig_p042_4_1.png]
Figure 4.2
Figure 4.2. Figure 4.2: Histogram showing the distribution of node betweenness centrality across the 1000 test graphs for the Dynamic Network Packet Routing Environment. The cumulative APSP (hop) distribution in [PITH_FULL_IMAGE:figures/full_fig_p043_4_2.png]
Figure 4.3
Figure 4.3. Figure 4.3: Cumulative distribution of APSP (hops) across the 1000 test graphs for the Dynamic Network Packet Routing Environment. 4.2 Observation Space Not all information required for routing decisions is necessary for learning graph representations. By removing packet-specifi…
Figure 4.4
Figure 4.4. Figure 4.4: Comparison of original (left) and 20% node failure (right) networks, with nodes colour-coded by betweenness centrality to show shifts in importance. 41 [PITH_FULL_IMAGE:figures/full_fig_p046_4_4.png]
Figure 5.1
Figure 5.1. Figure 5.1: GAT aggregation: Neighbouring nodes process local observations with RNN-A, exchange states, and then the GAT aggregates these states before passing them to RNN-B. The output from RNN-B is then shared with the agent model. The adapted distributed node state update wit…
Figure 5.2
Figure 5.2. Figure 5.2: Sudden spikes in validation loss (log scale) during the later stages of training on the Shortest Path Regression task, observed across multiple seeds. 46 [PITH_FULL_IMAGE:figures/full_fig_p051_5_2.png]
Figure 5.3
Figure 5.3. Figure 5.3: Validation loss over 100,000 iterations for the aggregation mechanisms. Logarithmic scale for the y-axis. Shaded areas show standard deviation bounds. Generalisation to Unseen Graphs [PITH_FULL_IMAGE:figures/full_fig_p053_5_3.png]
Figure 5.4
Figure 5.4. Figure 5.4: Test loss across sequence lengths for the aggregation mechanisms. Logarithmic scale for both axes. Error bars represent the standard deviation 48 [PITH_FULL_IMAGE:figures/full_fig_p053_5_4.png]
Figure 5.5
Figure 5.5. Figure 5.5: Running average (500-step) of Rewards in the Dynamic Network Packet Routing Environment. Shaded areas show standard deviation. 49 [PITH_FULL_IMAGE:figures/full_fig_p054_5_5.png]
Figure 5.6
Figure 5.6. Figure 5.6: Running average (500-step) of Looped Packets in the Dynamic Network Packet Routing Environment. Shaded areas show standard deviation. However, since rewards are primarily driven by throughput, they don’t capture the full picture. The reduction in looped packets sugge…
Figure 5.7
Figure 5.7. Figure 5.7: Routing metrics in the Dynamic Network Packet Routing environment, with each subplot having its own x-axis. Error bars represent standard deviations. GAT reduced looped packets by 11.9% compared to the next best method. This result supports the hypothesis that GAT en…
Figure 6.1
Figure 6.1. Figure 6.1: Nodes evaluate neighbours’ hidden states in each round, dynamically deciding whether to transmit (blue) or not transmit (red) The inclusion of GAT, a flexible and dynamic aggregation mechanism capable of handling a variable number of inputs, paves the way for develop…
Figure 6.2
Figure 6.2. Figure 6.2: Iteration Controller Forward Pass: Node and neighbouring hidden states are concatenated, processed through an MHA mechanism, and projected to a sigmoid binary classifier to determine whether to transmit the updated state. 54 [PITH_FULL_IMAGE:figures/full_fig_p059_6_2.png]
Figure 6.3
Figure 6.3. Figure 6.3: Average messages per node per step (out of 12) on the Shortest Path Regression task, showing the initial neglect of later communication rounds. 56 [PITH_FULL_IMAGE:figures/full_fig_p061_6_3.png]
Figure 6.4
Figure 6.4. Figure 6.4: Increasing the bias weight initialisation shifts sigmoid input, biasing the system towards outputting 1 (communicate). To address this issue, two measures were introduced. First, the bias weights of the final feed-forward layer before the sigmoid function were increa…
Figure 6.5
Figure 6.5. Figure 6.5: Average messages per node per step (out of 12) on the Shortest Path Regression task, highlighting volatility from insufficient exploration. 57 [PITH_FULL_IMAGE:figures/full_fig_p062_6_5.png]
Figure 6.6
Figure 6.6. Figure 6.6: Scatter plot of Average Messages vs. Total Validation Loss (summed over sequence lengths 2 to 32). Point labels (Noise Scaling, Communication Bias). Training Behaviour and Trends [PITH_FULL_IMAGE:figures/full_fig_p064_6_6.png]
Figure 6.7
Figure 6.7. Figure 6.7: Validation loss (log scale) per Communication Type (4 communication rounds) on Shortest Path Regression task. Shaded areas show standard deviation. 59 [PITH_FULL_IMAGE:figures/full_fig_p064_6_7.png]
Figure 6.8
Figure 6.8. Figure 6.8: Averages Messages per Node (max = 12) per Communication Type (4 communication rounds). Shaded areas show standard deviation. The advantages of the Iteration Controller become evident when analysing com￾munication overhead. Initially, the average number of messages se…
Figure 6.9
Figure 6.9. Figure 6.9: Averages Messages per Round (max = 3) for the Iteration Controller on the Shortest Path Regression task. Shaded areas show standard deviation. Generalisation to Unseen Graphs The first step in evaluating performance impact is to assess the quality of learned graph re…
Figure 6.10
Figure 6.10. Figure 6.10: Total MSE across sequence lengths (2,4,8,16,32) on the test graphs. Error bars represent the standard deviation 61 [PITH_FULL_IMAGE:figures/full_fig_p066_6_10.png]
Figure 6.11
Figure 6.11. Figure 6.11: Test MSE across sequence lengths per Communication Type (4 rounds). Logarithmic scale for both axes. Error bars represent the standard deviation. This leads to the Iteration Controller outperforming both baselines when the sequence length matches the RNN unroll dept…
Figure 6.12
Figure 6.12. Figure 6.12: Average messages per Communication Type (4 communication rounds) on the Shortest Path Regression task. Error bars represent the standard deviation [PITH_FULL_IMAGE:figures/full_fig_p068_6_12.png]
Figure 6.13
Figure 6.13. Figure 6.13: Heatmap showing the Average Messages per Node per Communication Round when using the Iteration Controller. 6.5.2 Dynamic Network Packet Routing Building on the promising results from the Shortest Path Regression task, this section evaluates the Iteration Controller’…
Figure 6.14
Figure 6.14. Figure 6.14: Running average (500-step) of Rewards in the Dynamic Network Packet Routing Environment. Shaded areas show standard deviation [PITH_FULL_IMAGE:figures/full_fig_p069_6_14.png]
Figure 6.15
Figure 6.15. Figure 6.15: Average Messages per Round (4 communication rounds) for the Iteration Controller. Shaded areas represent the standard deviation. 64 [PITH_FULL_IMAGE:figures/full_fig_p069_6_15.png]
Figure 6.16
Figure 6.16. Figure 6.16: Routing metrics in the Dynamic Network Packet Routing environment, with each subplot having its own x-axis. Error bars represent standard deviations. Despite a 5.4% reduction in Messages (overhead), the Iteration Controller outper￾formed the Maximum Communication ba…
Figure 7.1
Figure 7.1. Figure 7.1: Rewards over 1,000,000 steps in the Dynamic Network Packet Routing Environment. The shaded areas represent the standard deviation. Decoupling node and agent observations shows clear benefits as evidenced by the superior performance of both NetMon and our method. Inte…
Figure 7.2
Figure 7.2. Figure 7.2: Behaviour Network Q-values over 1,000,000 steps in the Dynamic Network Packet Routing Environment. Shaded areas represent the standard deviation These findings are supported by the Q-values of the behaviour network, which provide a stable leading indicator of perform…
Figure 7.3
Figure 7.3. Figure 7.3: Routing metrics in the Dynamic Network Packet Routing environment, with each subplot having its own x-axis. Error bars represent standard deviations. 70 [PITH_FULL_IMAGE:figures/full_fig_p075_7_3.png]
Figure 8.1
Figure 8.1. Figure 8.1: Distribution of edge lengths (delays) across the 1000 test graphs for the Dynamic Network Packet Routing Environment [PITH_FULL_IMAGE:figures/full_fig_p086_8_1.png]
Figure 8.3
Figure 8.3. Figure 8.3: Rewards by Aggregation Mechanism in the Dynamic Network Packet Routing Environment. Shaded areas show standard deviation [PITH_FULL_IMAGE:figures/full_fig_p093_8_3.png]
Figure 8.5
Figure 8.5. Figure 8.5: Running average (500-step) by Aggregation Mechanism of Q-Values in the Dynamic Routing Environment. Shaded areas show standard deviation [PITH_FULL_IMAGE:figures/full_fig_p093_8_5.png]
Figure 8.7
Figure 8.7. Figure 8.7: Validation MSE as a func￾tion of noise scaling. Lower noise scaling generally results in better performance [PITH_FULL_IMAGE:figures/full_fig_p094_8_7.png]
Figure 8.9
Figure 8.9. Figure 8.9: Rewards by Communication Type in the Dynamic Routing Environ￾ment. Shaded areas show standard deviation [PITH_FULL_IMAGE:figures/full_fig_p095_8_9.png]
Figure 8.11
Figure 8.11. Figure 8.11: Running average (500- step) of Q-Values by Communication Type in the Dynamic Routing Environ￾ment. Shaded areas show standard deviation [PITH_FULL_IMAGE:figures/full_fig_p095_8_11.png]
Figure 8.13
Figure 8.13. Figure 8.13: Running average (500-step) of Validation Loss in the Dynamic Routing Environment. Shaded areas show standard deviation. 90 [PITH_FULL_IMAGE:figures/full_fig_p095_8_13.png]
Figure 8.14
Figure 8.14. Figure 8.14: Overall System Looped Packets in the Dynamic Routing Envi￾ronment. Shaded areas show standard deviation [PITH_FULL_IMAGE:figures/full_fig_p096_8_14.png]
Figure 8.16
Figure 8.16. Figure 8.16: Overall System Validation Loss in the Dynamic Routing Environment. Shaded areas show standard deviation. 91 [PITH_FULL_IMAGE:figures/full_fig_p096_8_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Graph-Enhanced Policy Optimization in LLM Agent Training

    cs.AI 2025-10 conditional novelty 6.0 of 10

    GEPO adds graph-centrality-based intrinsic rewards, dynamic discounts, and two-level advantage shaping to group-based RL, improving LLM agent success on ALFWorld, WebShop, and a private Workbench benchmark.

Reference graph

Works this paper leans on

97 extracted references · 58 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep reinforcement learning meets graph neural networks: Exploring a routing optimization use case

    Paul Almasan, Jos ´e Su ´arez-Varela, Krzysztof Rusek, Pere Barlet-Ros, and Albert Cabellos-Aparicio. Deep reinforcement learning meets graph neural networks: Exploring a routing optimization use case. Computer Communications, 196:184–194,

  2. [2]

    Decentralized control of partially observable markov decision processes

    Christopher Amato, Girish Chowdhary , Alborz Geramifard, N Kemal ¨Ure, and Mykel J Kochenderfer. Decentralized control of partially observable markov decision processes. In 52nd IEEE Conference on Decision and Control, pages 2398–2405. IEEE, 2013. pages 18

    2013

  3. [3]

    A comprehensive guide to network routing

    Ignas Anfalovas. A comprehensive guide to network routing. https://www.ipxo.com/ blog/network-routing/, 2024. Accessed: 2024-06-04. pages 25

    2024

  4. [4]

    A markovian decision process

    Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pages 679–684, 1957. pages 7

    1957

  5. [5]

    Packet routing in dynamically changing networks: A reinforcement learning approach

    Justin Boyan and Michael Littman. Packet routing in dynamically changing networks: A reinforcement learning approach. Advances in neural information processing systems, 6, 1993. pages 26

    1993

  6. [6]

    Intelligent routing based on reinforcement learning for software-defined networking

    Daniela M Casas-Velasco, Oscar Mauricio Caicedo Rendon, and Nelson LS da Fonseca. Intelligent routing based on reinforcement learning for software-defined networking. IEEE Transactions on Network and Service Management , 18(1):870–881, 2020. pages 27

    2020

  7. [7]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014. pages 35

    arXiv 2014

  8. [8]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. pages 6

    2017

Show all 97 references
  1. [9]

    Multi-agent reinforcement learning for networked system control

    Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. arXiv preprint arXiv:2004.01339, 2020. pages 21

    2004 arXiv

  2. [10]

    A deep reinforcement learning-based multi- optimality routing scheme for dynamic iot networks

    Peizhuang Cong, Yuchao Zhang, Zheli Liu, Thar Baker, Hissam Tawfik, Wendong Wang, Ke Xu, Ruidong Li, and Fuliang Li. A deep reinforcement learning-based multi- optimality routing scheme for dynamic iot networks. Computer Networks, 192:108057,

  3. [11]

    Tarmac: Targeted multi-agent communication

    Abhishek Das, Th ´eophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on machine learning , pages 1538–1546. PMLR, 2019. pages 22, 23, 54 74 BIBLIOGRAPHY BIBLIOGRAPHY

    2019

  4. [12]

    Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020

    Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020. pages 17

    2011 arXiv

  5. [13]

    Convolutional neural networks on graphs with fast localized spectral filtering.Advances in neural information processing systems, 29, 2016

    Micha ¨el Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering.Advances in neural information processing systems, 29, 2016. pages 43

    2016

  6. [14]

    Learning individually inferred communication for multi-agent cooperation

    Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. Advances in neural information processing systems, 33:22069–22079, 2020. pages 22, 54

    2020

  7. [15]

    Learning correlated communication topology in multi-agent reinforcement learning

    Yali Du, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang. Learning correlated communication topology in multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, page...

    2021

  8. [16]

    A review of cooperation in multi-agent learning

    Yali Du, Joel Z Leibo, Usman Islam, Richard Willis, and Peter Sunehag. A review of cooperation in multi-agent learning. arXiv preprint arXiv:2312.05162, 2023. pages 15

    2023 arXiv

  9. [17]

    Learning to communicate with deep multi-agent reinforcement learning

    Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016. pages 21, 22, 24

    2016

  10. [18]

    Stabilising experience replay for deep multi-agent reinforcement learning

    Jakob Foerster, Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International conference on machine learning , pages 1146–1155. PMLR, 20...

    2017

  11. [19]

    Counterfactual multi-agent policy gradients

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. pages 15, 17, 20

    2018

  12. [20]

    Learning to communicate to solve riddles with deep distributed recurrent q-networks

    Jakob N Foerster, Yannis M Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate to solve riddles with deep distributed recurrent q-networks. arXiv preprint arXiv:1602.02672, 2016. pages 17

    2016 arXiv

  13. [21]

    Learning and generating distributed routing protocols using graph-based deep learning

    Fabien Geyer and Georg Carle. Learning and generating distributed routing protocols using graph-based deep learning. In Proceedings of the 2018 Workshop on Big Data Analytics and Machine Learning for Data Communication Networks, pages 40–45, 2018. pages 27, 28, 44, 50

    2018

  14. [22]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256. JMLR Workshop and Conference Proceedings, 2010. pages 57

    2010

  15. [23]

    Anti-symmetric dgn: a stable architecture for deep graph networks

    Alessio Gravina, Davide Bacciu, and Claudio Gallicchio. Anti-symmetric dgn: a stable architecture for deep graph networks. arXiv preprint arXiv:2210.09789, 2022. pages 43

    2022 arXiv

  16. [24]

    Model-based sparse communication in multi-agent reinforcement learning

    Shuai Han, Mehdi Dastani, and Shihan Wang. Model-based sparse communication in multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 439–447, 2023. pages 22, 54 75 BIBLIOGRAPHY BIBLIOGRAPHY

    2023

  17. [25]

    Deep recurrent q-learning for partially observable mdps

    Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. In 2015 aaai fall symposium series, 2015. pages 17, 26, 67

    2015

  18. [26]

    Routing information protocol

    Charles L Hedrick. Routing information protocol. Technical report, IETF, 1988. pages 25

    1988

  19. [27]

    A survey of learning in multiagent environments: Dealing with non-stationarity

    Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz De Cote. A survey of learning in multiagent environments: Dealing with non-stationarity . arXiv preprint arXiv:1707.09183, 2017. pages 12

    2017 arXiv

  20. [28]

    A survey and critique of multiagent deep reinforcement learning

    Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6):750–797, 2019. pages 12

    2019

  21. [29]

    Learning attentional communication for multi-agent cooperation

    Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31, 2018. pages 22, 24

    2018

  22. [30]

    Graph convolutional reinforcement learning

    Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. arXiv preprint arXiv:1810.09202, 2018. pages 21, 24, 43, 68

    2018 arXiv

  23. [31]

    Message-dropout: An efficient training method for multi-agent deep reinforcement learning

    Woojun Kim, Myungsik Cho, and Youngchul Sung. Message-dropout: An efficient training method for multi-agent deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6079–6086, 2019. pages 23

    2019

  24. [32]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016. pages 23

    2016 arXiv

  25. [33]

    Deep reinforcement learning for autonomous driving: A survey .IEEE Transactions on Intelligent Transportation Systems, 23(6):4909– 4926, 2021

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick P ´erez. Deep reinforcement learning for autonomous driving: A survey .IEEE Transactions on Intelligent Transportation Systems, 23(6):4909– 4926, 2021. pages 10

    2021

  26. [34]

    Reinforcement learning in robotics: A survey .The International Journal of Robotics Research, 32(11):1238–1274, 2013

    Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey .The International Journal of Robotics Research, 32(11):1238–1274, 2013. pages 6

    2013

  27. [35]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999. pages 10

    1999

  28. [36]

    An algorithm for distributed reinforcement learning in cooperative multi-agent systems

    Martin Lauer and Martin A Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In Proceedings of the seventeenth international conference on machine learning, pages 535–542, 2000. pages 17

    2000

  29. [37]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. pages 15

    2015 arXiv

  30. [38]

    Learning to ground multi-agent communication with autoencoders.Advances in Neural Information Processing Systems, 34:15230–15242, 2021

    Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning to ground multi-agent communication with autoencoders.Advances in Neural Information Processing Systems, 34:15230–15242, 2021. pages 21

    2021

  31. [39]

    Markov games as a framework for multi-agent reinforcement learning

    Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157–163. Elsevier, 1994. pages 13 76 BIBLIOGRAPHY BIBLIOGRAPHY

    1994

  32. [40]

    When2com: Multi- agent perception via communication graph grouping

    Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi- agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 4106–4115, 2020. pages 22, 54, 73

    2020

  33. [41]

    Multi- agent game abstraction via graph attention neural network

    Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi- agent game abstraction via graph attention neural network. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7211–7218, 2020. pages 22, 54

    2020

  34. [42]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. pages 82

    2017 arXiv

  35. [43]

    Border Gateway Protocol (BGP)

    K Lougheed and Y Rekhter. Border Gateway Protocol (BGP). RFC 1105, June 1989. URL https://www.rfc-editor.org/info/rfc1105. pages 25

    1989

  36. [44]

    Multi-agent actor-critic for mixed cooperative-competitive environments

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017. pages 15

    2017

  37. [45]

    Rectifier nonlinearities improve neural network acoustic models

    Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Atlanta, GA, 2013. pages 44

    2013

  38. [46]

    Distributed policy evaluation under multiple behavior strategies

    Sergio Valcarcel Macua, Jianshu Chen, Santiago Zazo, and Ali H Sayed. Distributed policy evaluation under multiple behavior strategies. IEEE Transactions on Automatic Control, 60(5):1260–1274, 2014. pages 20

    2014

  39. [47]

    An sdn perspective to mitigate the energy consumption of core networks–g´eant2

    Atefeh Maleki, Md Mohaimenul Hossain, Jean-Philippe Georges, Eric Rondeau, and Thierry Divoux. An sdn perspective to mitigate the energy consumption of core networks–g´eant2. In International SEEDS conference 2017, 2017. pages 26

    2017

  40. [48]

    Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams

    La ¨etitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 64–69. IEEE, 20...

    2007

  41. [49]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. pages 6, 9, 17, 26, 67

    2013 arXiv

  42. [50]

    Multi- agent deep learning for simultaneous optimization for time and energy in distributed routing system

    Dmitry Mukhutdinov, Andrey Filchenkov, Anatoly Shalyto, and Valeriy Vyatkin. Multi- agent deep learning for simultaneous optimization for time and energy in distributed routing system. Future Generation Computer Systems, 94:587–600, 2019. pages 26

    2019

  43. [51]

    The complexity of optimal small policies

    Martin Mundhenk. The complexity of optimal small policies. Mathematics of Operations Research, 25(1):118–129, 2000. pages 11

    2000

  44. [52]

    Graph convolutional value decomposition in multi-agent reinforcement learning

    Navid Naderializadeh, Fan H Hung, Sean Soleyman, and Deepak Khosla. Graph convolutional value decomposition in multi-agent reinforcement learning. arXiv preprint arXiv:2010.04740, 2020. pages 16

    2010 arXiv

  45. [53]

    John F. Nash. Non-cooperative games. The Annals of Mathematics, 54:286–295, 1950. doi: 10.2307/1969529. URL http://www.jstor.org/stable/1969529. pages 12

    1950

  46. [54]

    Magic: Multi-agent graph-attention communication

    Yaru Niu, Rohan Paleja, and Matthew Gombolay . Magic: Multi-agent graph-attention communication. In Mair2 Workshop at International Conference on Computer Vision (ICCV), 2021. pages 22, 24, 44 77 BIBLIOGRAPHY BIBLIOGRAPHY

    2021

  47. [55]

    Tree-based solution methods for multiagent pomdps with delayed communication

    Frans Oliehoek and Matthijs Spaan. Tree-based solution methods for multiagent pomdps with delayed communication. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 1415–1421, 2012. pages 18

    2012

  48. [56]

    A review of cooperative multi-agent deep reinforcement learning

    Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 53(11):13677–13722, 2023. pages 20

    2023

  49. [57]

    Reinforcement learning algorithm for non-stationary environments.Applied Intelligence, 50(11):3590– 3606, 2020

    Sindhu Padakandla, Prabuchandran KJ, and Shalabh Bhatnagar. Reinforcement learning algorithm for non-stationary environments.Applied Intelligence, 50(11):3590– 3606, 2020. pages 11

    2020

  50. [58]

    Multiagent bidirectionally-coordinated nets: Emergence of human- level coordination in learning to play starcraft combat games

    Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-coordinated nets: Emergence of human- level coordination in learning to play starcraft combat games. arXiv preprint arXiv:1703.10069, 2017. pages 23

    2017 arXiv

  51. [59]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014. pages 7

    2014

  52. [60]

    Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1– 51, 2020. pages 15

    2020

  53. [61]

    The graph neural network model

    Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks , 20(1):61–80, 2008. pages 16

    2008

  54. [62]

    Distributed online service coordination using deep reinforcement learning

    Stefan Schneider, Haydar Qarawlus, and Holger Karl. Distributed online service coordination using deep reinforcement learning. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS) , pages 539–549. IEEE, 2021. pages 37

    2021

  55. [63]

    Trust region policy optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. InInternational conference on machine learning, pages 1889–1897. PMLR, 2015. pages 17

    2015

  56. [64]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. pages 15

    2017 arXiv

  57. [65]

    Structured sequence modeling with graph convolutional recurrent networks

    Youngjoo Seo, Micha ¨el Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In Neural Information Processing: 25th International Conference, ICONIP 2018, Siem Reap, Cambodia, December 13-16, 2018, Proc...

    2018

  58. [66]

    Software-defined networking (sdn): A reference architecture and open apis

    Myung-Ki Shin, Ki-Hyuk Nam, and Hyoung-Jun Kim. Software-defined networking (sdn): A reference architecture and open apis. In 2012 International Conference on ICT Convergence (ICTC), pages 360–361. IEEE, 2012. pages 26

    2012

  59. [67]

    Open shortest path first (ospf) routing protocol simulation.ACM SIGCOMM Computer Communication Review, 23(4):53–62, 1993

    Deepinder Sidhu, Tayang Fu, Shukri Abdallah, Raj Nair, and Rob Coltun. Open shortest path first (ospf) routing protocol simulation.ACM SIGCOMM Computer Communication Review, 23(4):53–62, 1993. pages 25 78 BIBLIOGRAPHY BIBLIOGRAPHY

    1993

  60. [68]

    Mastering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489, 2...

    2016

  61. [69]

    The behavior of organisms: An experimental analysis

    Burrhus Frederic Skinner. The behavior of organisms: An experimental analysis . Appleton-Century , 1938. pages 6

    1938

  62. [70]

    Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning

    Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning , pages 5887–

  63. [71]

    Multi-agent temporal-difference learning with linear function approximation: Weak convergence under time-varying network topologies

    Milo S Stankovi and Srdjan S Stankovi. Multi-agent temporal-difference learning with linear function approximation: Weak convergence under time-varying network topologies. In 2016 American control conference (ACC) , pages 167–172. IEEE, 2016. pages 20

    2016

  64. [72]

    Phase transitions and critical phenomena , volume 7

    H Eugene Stanley . Phase transitions and critical phenomena , volume 7. Clarendon Press, Oxford, 1971. pages 19

    1971

  65. [73]

    Decentralized policy optimization

    Kefan Su and Zongqing Lu. Decentralized policy optimization. arXiv preprint arXiv:2211.03032, 2022. pages 17

    2022 arXiv

  66. [74]

    A general formulation of independent policy optimization in fully decentralized marl

    Kefan Su and Zongqing Lu. A general formulation of independent policy optimization in fully decentralized marl. Under review, 2024. pages 17

    2024

  67. [75]

    Learning multiagent communication with backpropagation

    Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. Advances in neural information processing systems , 29, 2016. pages 21, 22, 24, 68

    2016

  68. [76]

    Value-decomposition networks for cooperative multi-agent learning

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296, 2017. p...

    2017 arXiv

  69. [77]

    Learning to predict by the methods of temporal differences

    Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3:9–44, 1988. pages 8

    1988

  70. [78]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018. pages 6, 10, 12

    2018

  71. [79]

    Policy gradient methods for reinforcement learning with function approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. pages 9

    1999

  72. [80]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. pages 55

    2017

  73. [81]

    Graph attention networks

    Petar Veliˇckovi´c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017. pages 2, 23, 33

    2017 arXiv

  74. [82]

    Qplex: Duplex dueling multi-agent q-learning

    Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. arXiv preprint arXiv:2008.01062, 2020. pages 16 79 BIBLIOGRAPHY BIBLIOGRAPHY

    2008 arXiv

  75. [83]

    Ac2c: Adaptively controlled two-hop communication for multi-agent reinforcement learning

    Xuefeng Wang, Xinran Li, Jiawei Shao, and Jun Zhang. Ac2c: Adaptively controlled two-hop communication for multi-agent reinforcement learning. arXiv preprint arXiv:2302.12515, 2023. pages 22, 54

    2023 arXiv

  76. [84]

    Learning from delayed rewards

    Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. King’s College, Cambridge United Kingdom, 1989. pages 8, 17

    1989

  77. [85]

    Towards generalizability of multi-agent reinforcement learning in graphs with recurrent message passing

    Jannis Weil, Zhenghua Bao, Osama Abboud, and Tobias Meuser. Towards generalizability of multi-agent reinforcement learning in graphs with recurrent message passing. arXiv preprint arXiv:2402.05027, 2024. pages i, 2, 21, 27, 28, 30, 33, 36, 44, 58, 68, 71, 72

    2024 arXiv

  78. [86]

    Mambpo: Sample- efficient multi-robot reinforcement learning using learned world models

    Dani ¨el Willemsen, Mario Coppola, and Guido CHE de Croon. Mambpo: Sample- efficient multi-robot reinforcement learning using learned world models. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5635–5640. IEEE, 2021. pages 16

    2021

  79. [87]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning

    Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. pages 9

    1992

  80. [88]

    Distributed average consensus with least-mean-square deviation

    Lin Xiao, Stephen Boyd, and Seung-Jean Kim. Distributed average consensus with least-mean-square deviation. Journal of parallel and distributed computing , 67(1):33– 46, 2007. pages 19

    2007

  81. [89]

    Mean field multi-agent reinforcement learning

    Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. In International conference on machine learning, pages 5571–5580. PMLR, 2018. pages 19

    2018

  82. [90]

    Toward packet routing with fully distributed multiagent deep reinforcement learning

    Xinyu You, Xuanjie Li, Yuedong Xu, Hui Feng, Jin Zhao, and Huaicheng Yan. Toward packet routing with fully distributed multiagent deep reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 52(2):855–868, 2020. pages 26

    2020

  83. [91]

    The surprising effectiveness of ppo in cooperative multi-agent games

    Chao Yu, Akash Velu, Eugene Vinitsky , Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611–24624, 2022. pages 15

    2022

  84. [92]

    Asynchronous multi-agent reinforcement learning for efficient real-time multi-robot cooperative exploration

    Chao Yu, Xinyi Yang, Jiaxuan Gao, Jiayu Chen, Yunfei Li, Jijia Liu, Yunfei Xiang, Ruixin Huang, Huazhong Yang, Yi Wu, et al. Asynchronous multi-agent reinforcement learning for efficient real-time multi-robot cooperative exploration. arXiv preprint arXiv:2301.03398, 2023. pages 10

    2023 arXiv

  85. [93]

    Fully decentralized multi-agent reinforcement learning with networked agents

    Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning , pages 5872–5881. PMLR, 2018. pages 20

    2018

  86. [94]

    Decentralized multi-agent reinforcement learning with networked agents: Recent advances

    Kaiqing Zhang, Zhuoran Yang, and Tamer Bas ¸ar. Decentralized multi-agent reinforcement learning with networked agents: Recent advances. Frontiers of Information Technology & Electronic Engineering, 22(6):802–814, 2021. pages 20

    2021

  87. [95]

    Multi-agent reinforcement learning: A selective overview of theories and algorithms

    Kaiqing Zhang, Zhuoran Yang, and Tamer Bas ¸ar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pages 321–384, 2021. pages 13

    2021

  88. [96]

    High- speed ramp merging behavior decision for autonomous vehicles based on multi-agent reinforcement learning

    Xinfeng Zhang, Lin Wu, Huan Liu, Yajun Wang, Hao Li, and Bin Xu. High- speed ramp merging behavior decision for autonomous vehicles based on multi-agent reinforcement learning. IEEE Internet of Things Journal, 2023. pages 6, 16 80 Appendix A Test Graphs Figure 8.1: Distributio...

    2023

  89. [5896]

    pages 16

    PMLR, 2019. pages 16

    2019

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.