REVIEW 2 cited by
MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Benchmarks play a central role in reinforcement learning (RL) research, yet their computational constraints often shape what is studied. Despite the motivation of lifelong learning, most continual RL papers consider only 3-10 sequential tasks, as CPU-bound environments make longer sequences impractical. Meanwhile, continual learning in cooperative multi-agent settings remains largely unexplored. To address these gaps, we introduce MEAL (Multi-agent Environments for Adaptive Learning), the first benchmark for continual multi-agent RL. By leveraging JAX and GPU acceleration, MEAL enables training on sequences of 100 tasks in a few hours on a single GPU. We find that long task sequences reveal failure modes that do not appear at smaller scales.
Forward citations
Cited by 2 Pith papers
-
Offline Multi-agent Continual Cooperation via Skill Partition and Reuse
COMAD discovers and reuses coordination skills from mixed offline MARL data via auto-encoders and density-based estimation to achieve continual learning with better transfer.
-
Stagnant Neuron: Towards Understanding the Plasticity Loss in Multi-Agent Reinforcement Learning Value Factorization Methods
KNIFE targets stagnant neurons in MARL value factorization by replacing them with a composite of frozen, re-initialized, and compensating units to restore plasticity while preserving cooperation knowledge.
Reference graph
Works this paper leans on
-
[1]
Loss of plasticity in continual deep reinforcement learning
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. InConference on lifelong learning agents, pages 620–636. PMLR, 2023
2023
-
[2]
Melting pot 2.0.arXiv preprint arXiv:2211.13746, 2022
John P Agapiou, Alexander Sasha Vezhnevets, Edgar A Duéñez-Guzmán, Jayd Matyas, Yi- ran Mao, Peter Sunehag, Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, et al. Melting pot 2.0.arXiv preprint arXiv:2211.13746, 2022
arXiv 2022
-
[3]
Memory aware synapses: Learning what (not) to forget
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. InProceedings of the European conference on computer vision (ECCV), pages 139–154, 2018
2018
-
[4]
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. Pseudo-rehearsal: Achieving deep reinforcement learning without catastrophic forgetting.Neurocomputing, 428: 291–307, 2021
work page 2021
-
[5]
Curriculum learning
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. InProceedings of the 26th annual international conference on machine learning, pages 41–48, 2009
2009
-
[6]
JAX: composable transformations of Python+NumPy programs, 2018
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/jax-ml/jax
2018
-
[7]
Ho, Thomas Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan
Micah Carroll, Rohin Shah, Mark K. Ho, Thomas Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. InAdvances in Neural Information Processing Systems (NeurIPS), vol- ume 32, 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/ file/f5b1b89d3db40d65b49f8f9e383ac5dd-Paper.pdf
work page 2019
-
[8]
Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420, 2018
Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem.arXiv preprint arXiv:1812.00420, 2018
arXiv 2018
Show all 51 references
-
[9]
Stable continual reinforcement learning via diffusion-based trajectory replay.arXiv preprint arXiv:2411.10809, 2024
Feng Chen, Fuguang Han, Cong Guan, Lei Yuan, Zhilong Zhang, Yang Yu, and Zongzhang Zhang. Stable continual reinforcement learning via diffusion-based trajectory replay.arXiv preprint arXiv:2411.10809, 2024
2024 arXiv
-
[10]
Parseval regularization for continual reinforcement learning.Advances in Neural Information Processing Systems, 37: 127937–127967, 2024
Wesley Chung, Lynn Cherif, Doina Precup, and David Meger. Parseval regularization for continual reinforcement learning.Advances in Neural Information Processing Systems, 37: 127937–127967, 2024
2024
-
[11]
Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020
Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020
2011 arXiv
-
[12]
Loss of plasticity in deep continual learning.Nature, 632 (8026):768–774, 2024
Shibhansh Dohare, J Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning.Nature, 632 (8026):768–774, 2024
2024
-
[13]
Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 36:37567–37593, 2023
Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning.Advances in Neural Information Processing Systems, 36:37567–37593,...
2023
-
[14]
Continual reinforcement learning via autoencoder-driven task and new environment recognition
Zeki Doruk Erden, Donia Gasmi, and Boi Faltings. Continual reinforcement learning via autoencoder-driven task and new environment recognition. InThe Seventeenth Workshop on Adaptive and Learning Agents
-
[15]
Counterfactual multi-agent policy gradients
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018
2018
-
[16]
Learning with opponent-learning awareness.arXiv preprint arXiv:1709.04326, 2017
Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness.arXiv preprint arXiv:1709.04326, 2017
2017 arXiv
-
[17]
Continual deep reinforcement learning with task-agnostic policy distillation.Scientific Reports, 14(1):31661, 2024
Muhammad Burhan Hafez and Kerim Erekmen. Continual deep reinforcement learning with task-agnostic policy distillation.Scientific Reports, 14(1):31661, 2024
2024
-
[18]
A survey and critique of multiagent deep reinforcement learning.Autonomous Agents and Multi-Agent Systems, 33(6):750–797, 2019
Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning.Autonomous Agents and Multi-Agent Systems, 33(6):750–797, 2019
2019
-
[19]
other-play
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. “other-play” for zero-shot coordination. InInternational Conference on Machine Learning, pages 4399–4410. PMLR, 2020
2020
-
[20]
Inequity aversion improves cooperation in intertemporal social dilemmas.Advances in neural information processing systems, 31, 2018
Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves cooperation in intertemporal social dilemmas.Advances in neural information processi...
2018
-
[21]
Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of scienc...
2017
-
[22]
Evaluating the robustness of collaborative agents.arXiv preprint arXiv:2101.05507, 2021
Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, Anca D Dragan, and Rohin Shah. Evaluating the robustness of collaborative agents.arXiv preprint arXiv:2101.05507, 2021
2021 arXiv
-
[23]
Google research football: A novel reinforcement learning environment
Karol Kurach, Anton Raichuk, Piotr Sta´nczyk, Michał Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. InProceedings of the AAAI conference...
2020
-
[24]
Multi-agent actor-critic for mixed cooperative-competitive environments.Advances in neural information processing systems, 30, 2017
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments.Advances in neural information processing systems, 30, 2017
2017
-
[25]
Packnet: Adding multiple tasks to a single network by iterative pruning
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. InProceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018
2018
-
[26]
Flatland-rl: Multi-agent reinforcement learning on trains.arXiv preprint arXiv:2012.05893, 2020
Sharada Mohanty, Erik Nygren, Florian Laurent, Manuel Schneider, Christian Scheller, Nilabha Bhattacharya, Jeremy Watson, Adrian Egli, Christian Eichenberger, Christian Baumberger, et al. Flatland-rl: Multi-agent reinforcement learning on trains.arXiv preprint arXiv:2012.05893, 2020
2012 arXiv
-
[27]
Emergence of grounded compositional language in multi- agent populations
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018
2018
-
[28]
Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020. 12
2020
-
[29]
A review of cooperative multi-agent deep reinforcement learning.arXiv preprint arXiv:1908.03963, 2019
Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning.arXiv preprint arXiv:1908.03963, 2019
1908 arXiv
-
[30]
A multi-agent reinforcement learning model of common-pool resource appropriation.Advances in neural information processing systems, 30, 2017
Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel. A multi-agent reinforcement learning model of common-pool resource appropriation.Advances in neural information processing systems, 30, 2017
2017
-
[31]
Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. InConference on Robot Learning, pages 835–853. PMLR, 2020
2020
-
[32]
Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents
Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents. InConference on Lifelong Learning Agents, pages 705–743. PMLR, 2022
2022
-
[33]
Monotonic value function factorisation for deep multi-agent reinforcement learning.Journal of Machine Learning Research, 21(178):1–51, 2020
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning.Journal of Machine Learning Research, 21(178):1–51, 2020
2020
-
[34]
Pommerman: A multi-agent playground.arXiv preprint arXiv:1809.07124, 2018
Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho, and Joan Bruna. Pommerman: A multi-agent playground.arXiv preprint arXiv:1809.07124, 2018
2018 arXiv
-
[35]
Jaxmarl: Multi-agent rl environments and algorithms in jax
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Garðar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent rl environments and algorithms in jax. InProceedings of the 23rd Internationa...
2024
-
[36]
The starcraft multi-agent challenge.arXiv preprint arXiv:1902.04043, 2019
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nan- tas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge.arXiv preprint arXiv:1902.04043, 2019
1902 arXiv
-
[37]
Continual and multi-task reinforcement learning with shared episodic memory.arXiv preprint arXiv:1905.02662, 2019
Artyom Y Sorokin and Mikhail S Burtsev. Continual and multi-task reinforcement learning with shared episodic memory.arXiv preprint arXiv:1905.02662, 2019
1905 arXiv
-
[38]
Collaborating with humans without human data.Advances in Neural Information Processing Systems, 34: 14502–14515, 2021
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data.Advances in Neural Information Processing Systems, 34: 14502–14515, 2021
2021
-
[39]
Value- decomposition networks for cooperative multi-agent learning.arXiv preprint arXiv:1706.05296, 2017
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning.arXiv preprint arXiv:1706.05296, 2017
2017 arXiv
-
[40]
Coom: a game benchmark for continual reinforcement learning.Advances in Neural Information Processing Systems, 36, 2023
Tristan Tomilin, Meng Fang, Yudi Zhang, and Mykola Pechenizkiy. Coom: a game benchmark for continual reinforcement learning.Advances in Neural Information Processing Systems, 36, 2023
2023
-
[41]
Contin- ual world: A robotic benchmark for continual reinforcement learning.Advances in Neural Information Processing Systems, 34:28496–28510, 2021
Maciej Wołczyk, Michał Zaj ˛ ac, Razvan Pascanu, Łukasz Kuci´nski, and Piotr Miło´s. Contin- ual world: A robotic benchmark for continual reinforcement learning.Advances in Neural Information Processing Systems, 34:28496–28510, 2021
2021
-
[42]
Too many cooks: Bayesian inference for coordinating multi-agent collaboration.Topics in Cognitive Science, 13(2):414–432, 2021
Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration.Topics in Cognitive Science, 13(2):414–432, 2021
2021
-
[43]
The surprising effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games.Advances in neural information processing systems, 35:24611–24624, 2022
2022
-
[44]
A survey of progress on cooperative multi-agent reinforcement learning in open environment.arXiv preprint arXiv:2312.01058, 2023
Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress on cooperative multi-agent reinforcement learning in open environment.arXiv preprint arXiv:2312.01058, 2023. 13
2023 arXiv
-
[45]
Multiagent continual coordination via progressive task contextualization.IEEE Transactions on Neural Networks and Learning Systems, 2024
Lei Yuan, Lihe Li, Ziqian Zhang, Fuxiang Zhang, Cong Guan, and Yang Yu. Multiagent continual coordination via progressive task contextualization.IEEE Transactions on Neural Networks and Learning Systems, 2024
2024
-
[46]
Continual learning through synaptic intelligence
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. InInternational conference on machine learning, pages 3987–3995. PMLR, 2017. 14 A Implementation Details A.1 Maximum Soup Delivery Calculator Let a kitchen layout L be defined by ...
2017
-
[47]
Draw size.Sample h, wand create an h×w matrix initialised toFLOORtiles, then overwrite the border withWALL
-
[48]
Place interactive tiles.For each symbol in {GOAL,POT,ONION_PILE,PLATE_PILE} choose a random multiplicity m∈ {1,2}and stamp the symbol onto m uniformly chosen floor cells
-
[49]
Place nadd additional walls on random floor cells
Inject extra walls.Let ntarget = ρ Nint and nadd = max 0, ntarget −N unpass(G) . Place nadd additional walls on random floor cells. 4.Place agents.Stampn a AGENTsymbols on random remaining floor cells
-
[50]
Validate.Run the deterministic evaluate_grid solver; if it returns True, terminate and return(G), otherwise restart
-
[51]
7.Return.Output the final grid
Cleanup.Remove any interactive elements and tiles that are unreachable from all agent positions. 7.Return.Output the final grid. Solvability criterion.The validator (Appendix A.3) checks (i) path connectivity between every agent and each interactive tile family, (ii) at least ...
Discussion (0). Continue with ORCID to comment.