The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games
read the original abstract
Proximal Policy Optimization (PPO) is a ubiquitous on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due to the belief that PPO is significantly less sample efficient than off-policy methods in multi-agent systems. In this work, we carefully study the performance of PPO in cooperative multi-agent settings. We show that PPO-based multi-agent algorithms achieve surprisingly strong performance in four popular multi-agent testbeds: the particle-world environments, the StarCraft multi-agent challenge, Google Research Football, and the Hanabi challenge, with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. Importantly, compared to competitive off-policy methods, PPO often achieves competitive or superior results in both final returns and sample efficiency. Finally, through ablation studies, we analyze implementation and hyperparameter factors that are critical to PPO's empirical performance, and give concrete practical suggestions regarding these factors. Our results show that when using these practices, simple PPO-based methods can be a strong baseline in cooperative multi-agent reinforcement learning. Source code is released at \url{https://github.com/marlbenchmark/on-policy}.
This paper has not been read by Pith yet.
Forward citations
Cited by 9 Pith papers
-
ASALT: Adaptive State Alignment for Lateral Transfer in Multi-agent Reinforcement Learning
ASALT uses observation-level and state-level adapters to align mismatched dimensionalities into a shared embedding for transferring actors and critics in MARL, showing improved sample efficiency and reduced negative t...
-
A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch
A robust semi-Markov RL agent with MILP feasibility projection and Wasserstein ambiguity set achieves $1.22M net profit on an NYC EV simulator with zero feeder violations, outperforming heuristic and other RL baselines.
-
Heterogeneous Policy Networks for Composite Robot Team Communication and Coordination
HetNet achieves 5.84% to 707.65% performance gains and 200x bandwidth reduction over baselines in heterogeneous multi-agent robot teams via graph-attention networks and binarized messaging.
-
Shape Formation for the Cooperative Transportation of Arbitrary Objects Using Multi-Agent Reinforcement Learning
Multi-agent RL produces robot policies that form balanced supporting shapes under arbitrary objects for cooperative transport and generalize across cluttered scenes and complex geometries.
-
Enhancing the MADDPG Algorithm for Multi-Agent Learning via Action Inference and Importance Sampling
Action inference and geometric importance sampling enhance MADDPG, yielding better stability, cooperation, and exploration efficiency on the discrete Predator-Prey benchmark.
-
Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filtering
A decoupled estimator combining gated dynamics learning and recursive Kalman filtering improves robustness of pre-trained MARL policies under stale observations and message loss.
-
Coordination Architecture Shapes Continuous Demand Response Outcomes in Building Districts
In a 25-building district simulation, the hybrid MPC-SAC architecture delivered the strongest balance of load tracking accuracy (4.8% NMBE), thermal comfort (16.8% exceedance), and lowest spatial variability compared ...
-
A Distributionally Robust Reinforcement Learning Framework for Constrained Urban EV Dispatch
PD-RSAC, a distributionally robust SAC variant with GCN encoder and MILP constraint projection, reports $1.22M net profit on an NYC taxi-based EV simulator while achieving zero feeder violations, outperforming heurist...
-
$\alpha$-fair heterogeneous agent reinforcement learning
Introduces α-fair HATRPO and HAPPO algorithms that integrate α-fairness into HATRL via a weighted advantage function while claiming to preserve convergence to Nash equilibria.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.