Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Giving a team of agents one goal state, with no reward function, is enough to learn cooperation and win sparse-reward battles that standard MARL algorithms never win.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:43 UTC pith:SJZ5DPCH

load-bearing objection A simple recipe that works surprisingly well on sparse-reward MARL, with a table that oversells and an honest but still-unproven exploration story; worth a careful peer review. the 5 major comments →

arxiv 2509.10656 v2 pith:SJZ5DPCH submitted 2025-09-12 cs.LG cs.AI

Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration

Chirayu Nimonkar , Shlok Shah , Catherine Ji , Benjamin Eysenbach This is my paper
classification cs.LG cs.AI
keywords multi-agent reinforcement learninggoal-conditioned RLcontrastive RLself-supervised goal-reachingsparse rewardsemergent explorationStarCraft Multi-Agent Challengeindependent learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether a single commanded goal—not a hand-engineered reward function—can drive cooperation in multi-agent reinforcement learning. It combines contrastive RL's learned goal-distance critic with Independent PPO's shared-parameter, decentralized training, and shows that this combination learns to win on five StarCraft-style SMAX maps and two SMACv2 settings, reaching win rates of 0.84–1.00 where IPPO and MAPPO sit at 0.00. The method needs no explicit exploration bonus, no subgoals, and no intrinsic motivation; before a single success is observed, it still explores intermediate coordination strategies such as kiting and focus-fire. The paper interprets this as evidence that self-supervised goal-reaching is a tractable and viable problem statement for multi-agent tasks, and that factoring a single-agent problem into independent agents can speed early learning by shrinking the search space.

Core claim

The central claim is that a multi-agent task can be specified by a single goal state and solved by treating each agent as an independent contrastive RL learner, without any reward function, subgoals, or explicit exploration mechanism. Independent CRL learns a decentralized critic from the symmetric InfoNCE loss, approximating each agent's discounted probability of reaching the goal from its local observation and action, and trains a shared policy to maximize similarity between action representations and the goal representation. Empirically, on SMAX maps (3m, 2s3z, 6h_v_8z, 8m, 3s_v_5z) and SMACv2 with 5 and 10 units, this is the only method to achieve nonzero win rates in four environments a

What carries the argument

The load-bearing mechanism is the temporal contrastive critic: two encoders ϕ(o, a) and ψ(g) trained with a symmetric InfoNCE loss so that, at convergence, the negative squared distance between an agent's observation-action pair and a goal approximates that agent's discounted state-occupancy measure—its Q-function for the sparse goal-reaching reward. This critic is trained on goals achieved in the replay buffer but is used to score commanded goals that may never have been achieved, which is what the paper credits for directed exploration. Sampling pairs and goals from the same randomly chosen agent yields a 'mixed critic' averaged over agents, and the actor maximizes E[−||ϕ(o(i),a(i))−ψ(g)||

Load-bearing premise

Before agents ever reach the commanded goal, the contrastive critic must assign higher scores to actions that move toward that never-visited goal than to other actions; if representations learned from achieved goals do not transfer to out-of-distribution goals, the method's directed exploration and its sparse-reward advantage collapse.

What would settle it

In a small grid-world or continuous control task, train Independent CRL with a goal placed in a region the agent has never reached, and measure whether the critic orders actions by progress toward that goal before any accidental success. If the agent's early behavior is statistically indistinguishable from random until the first hit, the claimed emergent exploration is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A user can specify a cooperative task as one goal observation (e.g., 'all enemies defeated'), removing the need to design dense or shaped reward functions.
  • Whenever the contrastive critic transfers to never-achieved commanded goals, this method supplies an implicit exploration mechanism, so sparse-reward MARL no longer needs separate exploration bonuses or subgoal generators.
  • Independent, shared-parameter learning with a per-agent contrastive critic is sufficient for heterogeneous teams to specialize (by unit type) and coordinate, without centralized training or a joint value function.
  • Factoring a single-agent control problem into several independent goal-conditioned agents can speed initial learning and yield better early performance than the monolithic single-agent version, at the cost of a possible asymptotic bias.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the transfer mechanism is real, goal-conditioned contrastive learning may serve as a general-purpose exploration primitive for multi-agent domains beyond StarCraft, including long-horizon tasks where the goal is described only semantically.
  • The paper's success on goals like 'reduce sum of enemy health to zero' depends on the goal being computable from local observations; tasks with genuinely global goals (e.g., 'form a triangle around the target') may break the method, a limit the paper acknowledges only briefly.
  • The bias-variance trade-off observed in the Ant experiment suggests a design principle: deliberately factoring a control problem into weakly-coupled agents can be used to inject inductive bias for faster learning, and this could be tested on other continuous-control benchmarks.
  • Because the critic uses the soft (Gumbel-softmax) action probabilities rather than the discrete actions, the Q-function may be smoother and easier to optimize; contrasting this choice against the conventional hard-action critic would isolate whether this design decision contributes to the reported win rates.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Independent CRL (ICRL), a decentralized multi-agent variant of contrastive RL in which agents receive a single commanded goal (e.g., total enemy health zero in SMAC) and a sparse 0/1 reward, then learn via the symmetric InfoNCE objective. The method is evaluated on MPE Tag, SMAX maps, SMACv2, MASER comparisons, and a multi-agent MuJoCo/Brax Ant task. The central empirical claim is that ICRL is the only method to obtain nonzero reward in four of five SMAX maps and achieves the highest win rate in the fifth, while baselines using the same sparse reward fail; the paper also argues that this success is due to emergent goal-directed exploration before any success is observed. The manuscript includes code, hyperparameters, five-seed experiments, and a probability-of-improvement analysis.

Significance. If the empirical claims hold, the paper makes a useful contribution: it shows that a single goal state can replace a hand-designed reward in several cooperative MARL benchmarks, and that a relatively simple independent-learning formulation can outperform sparse-reward baselines. The strengths of the paper are its reproducible code release, the use of multiple seeds, the comparison against a purpose-built sparse-reward method (MASER), and the clean ablation in Sec. 6.3 showing that unit-type information matters. However, the paper's central mechanistic claim—that exploration is 'emergent' and directed by the never-achieved commanded goal—is not supported by the algorithm as written, and some of the headline numbers rely on non-standard evaluation and baseline configurations. These issues are fixable but need to be addressed before the paper can be accepted.

major comments (5)
  1. [Section 5, Eq. (8), Eq. (10), Algorithm 1, and Appendix E] The commanded goal g* is never used as a training goal in the critic or actor updates. In Eq. (8) the positive goals g_i are future states from the replay buffer, and in Eq. (10) the policy is trained on goals g^(i) sampled from the same buffer. Algorithm 1 uses p_g(g)=δ(g*) only for data collection. For the SMAX maps, g* is total enemy health equal to zero, so until the first win g* never appears as a positive InfoNCE sample. The paper itself concedes this in Appendix E ('ICRL does not directly use the collective goal in critic or actor sampling') and in Sec. 6.5 ('we suspect...'). Consequently, the abstract's claim that 'no explicit exploration mechanism, yet this approach explores' is not supported by the training objective; learning before the first success could be explained by the buffer's achieved-goal distribution alone. I recommend the authors either (a) test whether ψ(g*) actua
  2. [Table 1 and Appendix A] Table 1 reports the maximum result over training for each method ('maximum result ± 1σ at the timestep the maximum result is achieved'). This selection of the best timestep is known to inflate effect sizes and can present a transient peak as if it were typical performance. Because the paper's headline claim ('only method to get nonzero reward in four environments') is based on this table, the authors should report final performance at a fixed environment-step budget, or show aggregate learning curves with a clearly defined evaluation protocol (e.g., interquartile mean over the last 10% of training). The probability-of-improvement in Appendix D.2 also uses max win rate, inheriting this issue.
  3. [Section 6.2 and Appendix B] The IPPO and MAPPO baselines are run without recurrent policies, as stated in Appendix B: 'For all experiments, we test all algorithms without RNNs.' In the SMAC literature, the standard and strongest IPPO/MAPPO configurations use recurrent policies, and prior work (e.g., Yu et al., 2022) attributes much of their SMAC performance to recurrence. Comparing only feedforward baselines weakens the claim that 'prior methods never observe a single success' in these environments. The authors should either include recurrent IPPO/MAPPO baselines or explicitly qualify the comparison as feedforward-only and rephrase the corresponding claims.
  4. [Appendix E, Lemma E.3, and Eq. (10)–(12)] The lower-bound proof in Appendix E is for a modified actor objective (Eq. 11) in which the sampled goal is a function of all agents' collective observations. The actual ICRL actor objective (Eq. 10) samples agent-specific achieved goals g^(i), not collective goals. The final caveat of Appendix E acknowledges this gap ('ICRL does not directly use the collective goal...'), but the main text (Section 5) states that maximizing Eq. (10) is equivalent to maximizing a lower bound on the full contrastive actor objective. As written, the proof does not cover the algorithm that is run. The authors should either extend the analysis to the actual sampling scheme or remove the 'lower bound' claim from the main text and present Appendix E as a bound for a modified proxy.
  5. [Section 6.5] The qualitative claim that ICRL explores 'before the agents have seen the goal-state even once' is important but is not supported by any quantitative measurement of goal-state visitation in the text or figures. Since the paper's exploration narrative rests on this temporal ordering, the authors should report (at minimum) the number of episodes in which the goal was reached as a function of environment steps, or state clearly how this was determined. If the claim is based on there being zero wins before 2M steps, that should be shown explicitly.
minor comments (6)
  1. [Section 4 and Eq. (6)] The notation alternates between n agents in Section 3 and N agents in Eq. (6) and Appendix E. Please unify the symbols.
  2. [Eq. (8)] The regularization term R(ϕ, ψ) uses K but the batch is indexed by |B|; please clarify whether K = |B| and state the role of the 0.01 coefficient (it appears as 0.01 in Eq. 8 but Table 2 lists 'LogSumExp Penalty Coefficient 0.1').
  3. [Figure 6] The bar chart reports average win rates without error bars or confidence intervals, even though other figures show ±1σ. Please add variability or state the number of seeds explicitly.
  4. [Section 6.5 / Figure 8] The descriptions of specific strategies ('kiting', 'focus-fire', 'flocking') are qualitative and based on video stills. If these behaviors are claimed, provide a quantitative measure (e.g., number of distinct attack targets, distance to enemy) or refer readers to the videos with timestamps.
  5. [Appendix D.1] The uninformative-mg experiment is interesting, but the text says ICRL 'performs even better' than with the hand-specified mg. The overlap of error bars in Figure 11 makes this claim hard to verify; please report the numerical win rates or a significance test for this comparison.
  6. [Section 6.1] The text says ICRL 'matched' IPPO on the 3-agent Tag task, but Figure 3 shows nontrivial differences in learning speed. Please use consistent language (e.g., 'similar final performance but faster convergence' if that is the case).

Circularity Check

0 steps flagged

No circular derivation: the empirical results are independently benchmarked; the exploration explanation is an explicitly hedged transfer from the authors' prior single-agent work, not a construction.

full rationale

The paper's training procedure (Alg. 1, Eqs. 8 and 10) samples achieved future goals g^(i) from the replay buffer; the commanded goal g* is used only for episode collection and evaluation. No parameter is fit to the reported win rates, and the SMAX/Tag/Ant numbers are empirical comparisons against external baselines (IPPO, MAPPO, MASER), so the central empirical claim is not forced by the objective. Appendix E's lower-bound lemma is a genuine information-theoretic inequality (DPI/chain rule) from stated assumptions, not a restatement of the conclusion. The only circularity-adjacent element is Section 6.5's explanation of 'emergent exploration,' which imports the single-agent result from [29], an overlapping-author paper: the text says 'By extension to the multi-agent setting, we suspect...' and Section 7 concedes 'there is still no theoretical explanation for why these self-supervised goal-reaching algorithms exhibit emergent exploration.' Appendix E also admits 'ICRL does not directly use the collective goal in critic or actor sampling.' These are evidence gaps about an untested extrapolation, not derivation-by-construction. The empirical results stand independently, so the score is 2 for a minor, non-load-bearing self-citation in the exploration narrative.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The method imports the entire CRL machinery (symmetric InfoNCE critic, actor objective) from prior work [16, 29], plus IPPO-style independent learning [48]. The paper contributes an application and a lower-bound lemma whose assumptions do not exactly match the implemented goal-sampling scheme, which the authors concede in Appendix E. The only hand-chosen components are the goal mapping per environment and a regularization weight, neither of which is fitted to baseline results.

free parameters (2)
  • Goal mapping m_g and commanded goal g* for each environment = MPE Tag: distance to prey = 0; SMAX/SMACv2: sum of enemy health = 0; Ant: (x,y) position on disk of radius 10
    Hand-specified task encodings that define the goal-reaching problem (Section 4, Eqs. 5-6). The paper shows in Appendix D.1 that ICRL is robust to an uninformative (identity) m_g, so the main benchmark results depend on these choices only through the reported settings.
  • Symmetric InfoNCE regularization weight = 0.01 (with LogSumExp penalty coefficient 0.1)
    Fixed hyperparameter carried from single-agent CRL (Eq. 8); not tuned across environments. It balances the positive and negative contrastive terms in the critic objective.
axioms (5)
  • standard math Symmetric InfoNCE classifier converges to the conditional discounted state occupancy measure (density ratio) at optimizer optimum.
    Invoked in Section 5 (critic objective) to justify that the critic f is a valid Q-function; not proven in this paper.
  • standard math The goal-reaching reward r(o,a) = 1[m_g(o') = g] is equivalent in expectation to the likelihood form, giving equivalent optimization objectives.
    Equations 5-7; established in prior CRL work [16].
  • ad hoc to paper The overall goal g can be approximated as a function of local observations o(i): g ~ m_g(o(i)).
    Explicitly stated in Section 5: 'Importantly, we assume that the overall goal g can be approximated as a function of local observations o(i). While not true in the general MARL setting, we found that this assumption leads to strong empirical performance.'
  • ad hoc to paper Actor independence (Assumption E.1) and agent index included in observation (Assumption E.2) hold for the lower-bound proof.
    Used in Appendix E. The paper concedes the implemented algorithm samples agent-specific goals rather than the collective goal, so the bound does not directly apply to the executed method.
  • domain assumption Contrastive representations trained on achieved goals generalize to never-achieved commanded goals, enabling directed exploration.
    Transferred from single-agent CRL (Liu et al. 2024). Section 6.5 states: 'By extension to the multi-agent setting, we suspect that commanding a single goal to independent CRL agents will also enable such directed exploration.' This is the premise for the emergent-exploration claim.

pith-pipeline@v1.3.0-alltime-deepseek · 18080 in / 17337 out tokens · 186138 ms · 2026-08-04T17:43:10.406485+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration." pith.science (2026). https://pith.science/paper/SJZ5DPCH

@misc{pith2026250910656,
  author       = {Pith},
  title        = {Pith review of: Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SJZ5DPCH}},
  note         = {Machine review of arXiv:2509.10656}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

For groups of autonomous agents to achieve a particular goal, they must engage in coordination and long-horizon reasoning. Rather than relying on complex reward functions and explicit cooperation mechanisms, we ask what minimal ingredients are required for effective coordination and exploration to emerge in multi-agent settings. We investigate this question through self-supervised goal-reaching, where agents aim to maximize the likelihood of visiting a goal state rather than maximizing a reward. Despite a sparse feedback signal, we present empirical results that show self-supervised goal-reaching techniques enable agents to learn from such feedback. On MARL benchmarks, self-supervised goal-reaching outperforms alternative approaches that have access to the same sparse reward signal. Furthermore, we empirically demonstrate that multi-agent self-supervised goal-reaching approaches can be more robust than single-agent strategies. While there is no explicit exploration mechanism, this approach explores nontrivial intermediate coordination strategies in sparse settings where alternative approaches fail to achieve a single success.

Figures

Figures reproduced from arXiv: 2509.10656 by Benjamin Eysenbach, Catherine Ji, Chirayu Nimonkar, Shlok Shah.

Figure 1
Figure 1. Figure 1: In the multi-agent goal-reaching prob￾lem, a collection of agents cooperates to maximize the likelihood of visiting a certain state. In this example, four agents coordinate to control an ant￾like robot; each agent controls one leg (2 joints/leg). The goal is to coordinate so that the ant moves to a specific position (×). No rewards are given; no distance metrics are required. Reinforcement learning (RL) ha… view at source ↗
Figure 2
Figure 2. Figure 2: Multi-agent environments. MPE Tag [41], SMAX (SMAC) [41, 42], and multi-agent Ant [41]. We present all results with ±1σ error bars. The experiments (5 seeds per baseline per experiment) required 0.5- 3 hours (ICRL, IPPO, and MAPPO) and 16 hours (MASER) per seed on a Tesla V100 GPU (32 GB). All exper￾iments were run on an internal cluster. See Appendix A for a summary of all experiments and Appendix B for a… view at source ↗
Figure 3
Figure 3. Figure 3: “Tag” as a goal-reaching problem. We com￾pare ICRL (ours) to IPPO on the Multi-Particle Agent Tag FACMAC Environment, including both the 3-agent setting (Left) and the 6-agent setting (Right). We observe that our method learns faster in both settings and reaches higher asymptotic returns in the 6-agent setting. In the MPE Tag environment, predator agents must collide with (or “tag”) a faster prey. The envi… view at source ↗
Figure 4
Figure 4. Figure 4: We compare ICRL (ours) to IPPO and MAPPO on the randomized 5-agent and 10-agent SMACv2 environments. Our approach learns faster than both baselines, achieving higher asymptotic returns than IPPO. IPPO’s win rate is effectively zero. To answer this question, we evaluate the perfor￾mance of ICRL on the StarCraft Multi-Agent Challenge (SMAC), a common benchmark in prior MARL work that pits a team of units aga… view at source ↗
Figure 5
Figure 5. Figure 5: Efficient learning on the StarCraft Multi-Agent Challenge (SMAX). We compare ICRL (our method) to IPPO and MAPPO on five settings from the SMAX benchmark. On the 3m setting, our method achieves a win rate that is ∼ 3× higher than MAPPO, while the IPPO baseline has a win rate of zero. On the 2s3z, 6h v 8z, 8m, and 3s v 5z settings, only our method achieves a non-zero win rate. find that their exploration is… view at source ↗
Figure 6
Figure 6. Figure 6: Agents specialize in SMAC, using informa￾tion about their player type to make decisions. Win rates decline as unit type information is progressively removed from agent observations: baseline (self + team￾mate types), partial ablation (teammate types only), and full ablation (no type information). A statistical significance test located in Ap￾pendix D.2 shows that the probability of im￾provement for ICRL co… view at source ↗
Figure 8
Figure 8. Figure 8: How does ICRL learn to play StarCraft? We visualize the exploration strategies of (Top Row) ICRL (our method) and (Bottom Row) IPPO on the SMAX (2s3z) environment over 50 million training environment steps. We observe that the ICRL algorithm explores different coordination strategies over the course of learning. Interestingly, there is a significantly smaller drop in performance when removing an agent’s ow… view at source ↗
Figure 7
Figure 7. Figure 7: ICRL (Ours) vs. MASER on SMAC (2s3z), using sparse rewards. Our method achieves 60% win￾rate by step 5M while MASER’s win-rate is negligible. A long line of prior work suggests that hierarchi￾cal approaches–methods that decompose long￾horizon tasks into a sequence of shorter/easier problems–are crucial for solving long-horizon tasks, such as those in SMAC. Indeed, prior work [22] has found such designs hig… view at source ↗
Figure 9
Figure 9. Figure 9: Can ICRL solve continuous control tasks? We compare ICRL (ours) to IPPO for controlling a 4- legged robot: each leg’s joints are controlled by a sep￾arate agent [41]. IPPO makes no progress, perhaps be￾cause the sparse reward signal makes exploration chal￾lenging. The tasks we have investigated so far have featured straightforward control mechanisms. Many real-world tasks, however, have an orthog￾onal dime… view at source ↗
Figure 10
Figure 10. Figure 10: Is multi-agent RL harder than single￾agent RL? For controlling a 4-legged robot [41], we compare (blue) controlling each of the 4 legs with a sep￾arate agent to (green) controlling the legs with a single agent. Casting this as a multi-agent problem allows the agent to learn faster, highlighting how multi-agent prob￾lems are not always more challenging than single-agent problems. While multi-agent reinforc… view at source ↗
Figure 11
Figure 11. Figure 11: Specifying mg is not necessary for good performance. ±1σ error bars. D.2 Probability of Improvement In order to determine the statistical significance of our results, we use the probability of improvement, which is the chance that, in a randomly selected environment, a given algorithm performs better than another [2]. We compute the probability of improvement for ICRL compared to MAPPO over all SMAX envir… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation

    cs.LG 2026-05 unverdicted novelty 7.0

    CPPO is an on-policy contrastive RL method that derives advantages from contrastive Q-values for PPO optimization, outperforming prior CRL baselines in 14/18 tasks and matching or exceeding reward-based PPO in 12/18 tasks.

  2. Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

    cs.MA 2026-05 unverdicted novelty 7.0

    A quality-aware exploration method using return-conditioned sigmoid scheduling and per-agent RSQ metrics achieves top-tier returns on seven cooperative MARL benchmarks.

Reference graph

Works this paper leans on

54 extracted references · 17 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Achiam, J., Edwards, H., Amodei, D., and Abbeel, P. (2018). Variational option discovery algorithms.arXiv preprint arXiv:1807.10299

    Pith/arXiv arXiv 2018

  2. [2]

    S., Courville, A., and Bellemare, M

    Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. (2022). Deep reinforcement learning at the edge of the statistical precipice

    2022

  3. [3]

    Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. (2017). Hindsight experience replay.Advances in neural information processing systems, 30

    2017

  4. [4]

    Baldazo, D., Parras, J., and Zazo, S. (2019). Decentralized Multi-Agent Deep Reinforcement Learning in Swarms of Drones for Flood Monitoring. In2019 27th European Signal Processing Conference (EUSIPCO), pages 1–5. ISSN: 2076-1465

    2019

  5. [5]

    Bortkiewicz, M., Pałucki, W., Myers, V ., Dziarmaga, T., Arczewski, T., Kuci´nski, L., and Eysenbach, B. (2024). Accelerating Goal-Conditioned RL Algorithms and Research. arXiv:2408.11052 [cs]

    arXiv 2024

  6. [6]

    and Moore, A

    Boyan, J. and Moore, A. (1994). Generalization in reinforcement learning: Safely approximating the value function.Advances in neural information processing systems, 7

    1994

  7. [7]

    Cao, Y ., Yu, W., Ren, W., and Chen, G. (2012). An Overview of Recent Progress in the Study of Distributed Multi-agent Coordination. arXiv:1207.3231 [math]

    Pith/arXiv arXiv 2012

  8. [8]

    Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. (2021). Decision transformer: Reinforcement learning via sequence modeling.Advances in neural information processing systems, 34:15084–15097

    2021

  9. [9]

    and Boutilier, C

    Claus, C. and Boutilier, C. (1998). The dynamics of reinforcement learning in cooperative multiagent systems. InProceedings of the Fifteenth National/Tenth Conference on Artificial Intelligence/Innovative Applications of Artificial Intelligence, AAAI ’98/IAAI ’98, page 746–752, USA. American Association for Artificial Intelligence

    1998

  10. [10]

    and Hinton, G

    Dayan, P. and Hinton, G. E. (1992). Feudal reinforcement learning. In Hanson, S., Cowan, J., and Giles, C., editors,Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufmann

    1992

  11. [11]

    Dietterich, T. G. (1998). The maxq method for hierarchical reinforcement learning. InProceedings of the Fifteenth International Conference on Machine Learning, ICML ’98, page 118–126, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc

    1998

  12. [12]

    Ding, Y ., Florensa, C., Abbeel, P., and Phielipp, M. (2019). Goal-conditioned imitation learning.Advances in neural information processing systems, 32

    2019

  13. [13]

    Dulac-Arnold, G., Mankowitz, D., and Hester, T. (2019). Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901

    Pith/arXiv arXiv 2019

  14. [14]

    Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. (2019). Diversity is all you need: Learning skills without a reward function. InInternational Conference on Learning Representations

    2019

  15. [15]

    Eysenbach, B., Salakhutdinov, R., and Levine, S. (2021). C-learning: Learning to achieve goals via recursive classification. InInternational Conference on Learning Representations

    2021

  16. [16]

    Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. (2022). Contrastive learning as goal- conditioned reinforcement learning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors,Advances in Neural Information Processing Systems

    2022

  17. [17]

    Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2017). Counterfactual Multi-Agent Policy Gradients. arXiv:1705.08926 [cs]

    Pith/arXiv arXiv 2017

  18. [18]

    M., Eysenbach, B., and Levine, S

    Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C. M., Eysenbach, B., and Levine, S. (2021). Learning to reach goals via iterated supervised learning. InInternational Conference on Learning Representations

    2021

  19. [19]

    J., and Dragan, A

    Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. (2017). Inverse reward design. Advances in neural information processing systems, 30. 12

    2017

  20. [20]

    and Ermon, S

    Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning.Advances in neural information processing systems, 29

    2016

  21. [21]

    Jang, E., Gu, S., and Poole, B. (2016). Categorical reparameterization with gumbel-softmax.arXiv preprint arXiv:1611.01144

    Pith/arXiv arXiv 2016

  22. [22]

    Jeon, J., Kim, W., Jung, W., and Sung, Y . (2022). Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer. InInternational conference on machine learning, pages 10041–10052. PMLR

    2022

  23. [23]

    Jiang, Y ., Shao, J., He, S., Zhang, H., and Ji, X. (2022). Spd: Synergy pattern diversifying oriented unsupervised multi-agent reinforcement learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors,Advances in Neural Information Processing Systems, volume 35, pages 20661–20674. Curran Associates, Inc

    2022

  24. [24]

    Jo, Y ., Lee, S., Yeom, J., and Han, S. (2024). Fox: formation-aware exploration in multi-agent reinforcement learning. InProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence. AAAI Press

    2024

  25. [25]

    Kaelbling, L. P. (1993). Learning to achieve goals. InIJCAI, volume 2, pages 1094–8. Citeseer

    1993

  26. [26]

    and Friedman, N

    Koller, D. and Friedman, N. (2009).Probabilistic Graphical Models - Principles and Techniques.MIT Press

    2009

  27. [27]

    S., and Held, D

    Lin, X., Baweja, H. S., and Held, D. (2019). Reinforcement learning without ground-truth state.arXiv preprint arXiv:1905.07866

    Pith/arXiv arXiv 2019

  28. [28]

    Liu, B., Pu, Z., Pan, Y ., Yi, J., Liang, Y ., and Zhang, D. (2023). Lazy agents: A new perspective on solving sparse reward problem in multi-agent reinforcement learning. InProceedings of the 40th International Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 21937– 21950. PMLR

    2023

  29. [29]

    Liu, G., Tang, M., and Eysenbach, B. (2024). A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals. arXiv:2408.05804 [cs]

    Pith/arXiv arXiv 2024

  30. [30]

    A., and Schwing, A

    Liu, I.-J., Jain, U., Yeh, R. A., and Schwing, A. G. (2021). Cooperative exploration for multi-agent deep reinforcement learning.ArXiv, abs/2107.11444

    Pith/arXiv arXiv 2021

  31. [31]

    J., Mnih, A., and Teh, Y

    Maddison, C. J., Mnih, A., and Teh, Y . W. (2016). The concrete distribution: A continuous relaxation of discrete random variables.arXiv preprint arXiv:1611.00712

    Pith/arXiv arXiv 2016

  32. [32]

    Mahajan, A., Rashid, T., Samvelyan, M., and Whiteson, S. (2019). Maven: Multi-agent variational exploration. volume 32

    2019

  33. [33]

    Matari´c, M. J. (1997). Reinforcement learning in the multi-robot domain.Autonomous Robots, 4(1):73–83

    1997

  34. [34]

    and Moon, I.-C

    Na, H. and Moon, I.-C. (2024). Lagma: Latent goal-guided multi-agent reinforcement learning. In International Conference on Machine Learning, pages 37122–37140. PMLR

    2024

  35. [35]

    C., and Simon, H

    Newell, A., Shaw, J. C., and Simon, H. A. (1959). Report on a general problem solving program. InIFIP congress, volume 256, page 64. Pittsburgh, PA

    1959

  36. [36]

    A., Amato, C., et al

    Oliehoek, F. A., Amato, C., et al. (2016).A concise introduction to decentralized POMDPs, volume 1. Springer

    2016

  37. [37]

    Peng, B., Rashid, T., Witt, C. A. S. d., Kamienny, P.-A., Torr, P. H. S., B ¨ohmer, W., and Whiteson, S. (2021). FACMAC: Factored Multi-Agent Centralised Policy Gradients. arXiv:2003.06709 [cs]

    Pith/arXiv arXiv 2021

  38. [38]

    Puterman, M. L. (1994).Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition

    1994

  39. [39]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR

    2021

  40. [40]

    Rashid, T., Samvelyan, M., Witt, C. S. d., Farquhar, G., Foerster, J., and Whiteson, S. (2018). QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning. arXiv:1803.11485 [cs]. 13

    Pith/arXiv arXiv 2018

  41. [41]

    Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G., Willi, T., Hammond, R., Khan, A., Witt, C. S. d., Souly, A., Bandyopadhyay, S., Samvelyan, M., Jiang, M., Lange, R. T., Whiteson, S., Lacerda, B., Hawes, N., Rocktaschel, T., Lu, C., and Foerster, J. N. (2024). JaxMARL: Multi-Agent RL Environments and Algorithms in JAX. arXiv:2311...

    Pith/arXiv arXiv 2024

  42. [42]

    Samvelyan, M., Rashid, T., Witt, C. S. d., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C.-M., Torr, P. H. S., Foerster, J., and Whiteson, S. (2019). The StarCraft Multi-Agent Challenge. arXiv:1902.04043 [cs]

    Pith/arXiv arXiv 2019

  43. [43]

    Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T. (2016). Loss is its own reward: Self-supervision for reinforcement learning.arXiv preprint arXiv:1612.07307

    Pith/arXiv arXiv 2016

  44. [44]

    Sun, H., Li, Z., Liu, X., Zhou, B., and Lin, D. (2019). Policy continuation with hindsight inverse dynamics. Advances in Neural Information Processing Systems, 32

    2019

  45. [45]

    Sutton, R. S. (1995). Generalization in reinforcement learning: Successful examples using sparse coarse coding.Advances in neural information processing systems, 8

    1995

  46. [46]

    Tan, M. (1993). Multi-agent reinforcement learning: independent versus cooperative agents. InProceedings of the Tenth International Conference on International Conference on Machine Learning, ICML’93, pages 330–337, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc

    1993

  47. [47]

    Touati, A., Rapin, J., and Ollivier, Y . (2022). Does zero-shot reinforcement learning exist?arXiv preprint arXiv:2209.14935

    Pith/arXiv arXiv 2022

  48. [48]

    Witt, C. S. d., Gupta, T., Makoviichuk, D., Makoviychuk, V ., Torr, P. H. S., Sun, M., and Whiteson, S. (2020). Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? arXiv:2011.09533 [cs]

    Pith/arXiv arXiv 2020

  49. [49]

    Xu, P., Zhang, J., and Huang, K. (2023a). Exploration via joint policy diversity for sparse-reward multi- agent tasks. In Elkind, E., editor,Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, pages 326–334. International Joint Conferences on Artificial Intelligence Organization. Main Track

  50. [50]

    Xu, P., Zhang, J., and Huang, K. (2024). Population-based diverse exploration for sparse-reward multi- agent tasks. In Larson, K., editor,Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 283–291. International Joint Conferences on Artificial Intelligence Organization. Main Track

    2024

  51. [51]

    Xu, P., Zhang, J., Yin, Q., Yu, C., Yang, Y ., and Huang, K. (2023b). Subspace-aware exploration for sparse-reward multi-agent tasks.Proceedings of the AAAI Conference on Artificial Intelligence, 37(10):11717– 11725

  52. [52]

    Yang, M., Yang, Y ., Lu, Z., Zhou, W., and Li, H. (2023). Hierarchical multi-agent skill discovery. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors,Advances in Neural Information Processing Systems, volume 36, pages 61759–61776. Curran Associates, Inc

    2023

  53. [53]

    Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y ., Bayen, A., and Wu, Y . (2022). The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. arXiv:2103.01955 [cs]. 14 A Main Experimental Results Restatement(Experimental Setup).All experiments use sparse 0/1 rewards: +1 when in the goal state, 0 otherwise. For SMAX environments, the goal is to...

    Pith/arXiv arXiv 2022

  54. [54]

    ICRL outperforms MASER in the 0/1 reward setting (Figure 7)

    without further signal from expert domain knowledge. ICRL outperforms MASER in the 0/1 reward setting (Figure 7). We believe at least one of the reasons for ICRL’s high relative performance on such sparse-signal environments is the method’s ability to explore effectively. This emergent exploration is discussed further in Section 6.5. B Experimental Detail...