Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Expected Return Symmetries

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper defines expected return symmetries—transformations preserving optimal-policy returns—and shows they can be learned and used to improve zero-shot coordination beyond environment symmetries.

desk verdict A genuinely new symmetry class for ZSC with strong empirical results, but the theoretical guarantee doesn't cover the policies that OPΦER actually produces. read the letter →

arxiv 2502.01711 v3 pith:S6DUJ54Q submitted 2025-02-03 cs.MA

Darius Muglich , Johannes Forkel , Elise van der Pol , Jakob Foerster This is my paper · ORCID
classification cs.MA
keywords expectedreturnsymmetrieszero-shotcoordinationother-playDec-POMDPsymmetrydiscoverymulti-agentreinforcementlearningBoltzmannexplorationHanabi
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper defines expected return symmetries: transformations of agents' actions and observations that map every self-play-optimal policy with Boltzmann (softmax) exploration to another policy with the same expected return. These transformations form a group, and the usual Dec-POMDP (environment) symmetries are a subgroup, so the new group captures coordination-equivalent policies that no environment relabeling can express—for example, two opposite but equally good communication conventions in a game whose rewards break every naive symmetry. The paper shows how to discover approximate members of this group from ordinary agent–environment interaction, with no access to transition or reward functions or ground-truth symmetries, and then use them in the other-play objective so independently trained agents become compatible. Across the iterated lever game, a cat/dog communication game, Overcooked V2, and Hanabi, agents trained under expected return symmetries achieve better zero-shot coordination than agents trained under environment symmetries, with Hanabi within-population cross-play rising from 8.61 to 21.64 at roughly unchanged self-play scores.

What carries the argument

The central object is the expected return symmetry group $\Phi_{\mathrm{ER}}$, the set of transformations of actions and observations that carry every self-play-optimal Boltzmann-exploratory policy to another such policy. Boltzmann-exploratory policies here are softmax policies over local action values at temperature $\alpha$, a choice that ensures suboptimal actions are distinguished by their quality rather than lumped together. The argument runs through the other-play objective $\mathrm{OP}_\Phi(\pi) = \mathbb{E}_{\tilde\pi \in [\pi]}[\mathrm{XP}(\pi, \tilde\pi)]$, where $[\pi]$ is the equivalence class under $\Phi$; using $\Phi_{\mathrm{ER}}$ makes this objective average over diverse optimal conventions. The discovery machinery is gradient-based: with a fixed pool $\Pi'$ of approximately optimal policies, the paper learns $\phi_\theta$ by maximizing $J(\phi_\theta(\pi))$ (Equation 9) or cross-play between policy pairs (Equation 11), searching over action transpositions and a learned observation network, and Equation 10 adds penalties that push the learned maps toward compositionality and invertibility.

What would settle it

Train a held-out set of self-play-optimal Boltzmann-exploratory policies and apply the learned transformations to them: if the expected return of $\phi(\pi)$ is systematically below $J(\pi)$, or if replacing the learned transformations with random policy permutations matched for diversity gives the same cross-play improvement, then the transformations are not expected return symmetries and the coordination gain is not due to the group.

Watch

Extended reading notes

Core claim

The central claim is that the right symmetry group for zero-shot coordination is not the group of relabelings that leave the environment dynamics and rewards invariant, but the larger group $\Phi_{\mathrm{ER}}$ of transformations that preserve expected return on the set $\Pi^\alpha_*$ of self-play-optimal Boltzmann-exploratory policies. Formally, $\Phi_{\mathrm{ER}} = \{\phi \in \Psi : \phi(\pi) \in \Pi^\alpha_* \text{ for all } \pi \in \Pi^\alpha_*\}$, which the paper proves is a group under composition and contains the Dec-POMDP symmetries $\Phi_{\mathrm{MDP}}$ as a subgroup. Because $\Phi_{\mathrm{ER}}$ puts more, and more diverse, optimal policies into the same equivalence class, other-play over $\Phi_{\mathrm{ER}}$ trains each agent to be compatible with partners that share its optimality but break conventions differently, which is exactly the failure mode of mutually incompatible symmetry breaking. The paper further claims that approximate elements of $\Phi_{\mathrm{ER}}$ can be learned from interaction alone by maximizing the expected return of transformed fixed optimal policies, with regularization toward compositionality and invertibility, and that this learned group is what delivers the improved zero-shot coordination in the experiments.

Load-bearing premise

The method assumes the transformations trained on a finite pool of good policies really are symmetries of the whole set of optimal policies, and that the approximate invertibility penalty makes them true bijections; if either fails, the group-theoretic coordination guarantees lapse.

Editorial extensions

If this is right

  • Other-play with a symmetry group that preserves optimal policies, rather than environment structure, generalizes beyond label relabelings, so independently trained agents can coordinate on any of several equivalent but differently broken optimal conventions.
  • Because $\Phi_{\mathrm{MDP}} \subseteq \Phi_{\mathrm{ER}}$, all previous gains from environment-symmetry training are retained, and additional gains appear in games with no non-trivial Dec-POMDP symmetries, such as the cat/dog game.
  • Symmetry-based coordination no longer requires privileged knowledge of the environment's transition, reward, or symmetry structure; a model-free agent can discover approximate expected return symmetries from its own interactions.
  • A small set of learned transformations can be enough: in Hanabi, 11 learned expected return symmetries produced a within-population cross-play score of 21.64, versus 8.61 for agents using all 120 Dec-POMDP symmetries, while self-play scores stayed near 23.7–24.0.
  • The approach works in settings where off-belief learning and cognitive-hierarchy methods fail, including the iterated lever game, the cat/dog game, and Overcooked V2.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If outcome-based symmetry is the right abstraction, the same recipe could be carried to single-agent reinforcement learning: transformations that preserve optimal value would justify data augmentation without knowing the dynamics, and one could test whether policy-space symmetrization improves generalization to unseen tasks.
  • The paper fixes action transformations to transpositions and learns observation transformations; a natural extension is to learn joint transformations over continuous or structured action and observation spaces, where bijections are harder, and to check whether the cross-play gains persist.
  • A direct test of the mechanism would compare other-play under learned expected return symmetries against other-play under an equal number of random transformations chosen for the same policy diversity; if the random baseline matches the cross-play gains, the gains come from diversity rather than from the group structure.
  • The regularized learning objective enforces invertibility only softly, so a stricter parameterization with exact bijectivity could turn the approximate group-theoretic guarantees into quantitative bounds on cross-play degradation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces expected return symmetries (ΦER), defined as the transformations in Ψ that preserve the set of self-play-optimal Boltzmann-exploratory policies, and argues that this group contains Dec-POMDP symmetries as a subgroup. The authors propose three practical objectives and associated algorithms for learning approximate ER symmetries from agent-environment interaction, and evaluate the resulting other-play variants in four environments. They report that agents trained with OP over learned ER symmetries achieve substantially better zero-shot coordination than self-play or OP over Dec-POMDP symmetries, e.g., Hanabi cross-play of 21.64 versus 8.61 for OP with ΦMDP. The central claims are that ΦER is a broader and learnable symmetry class and that it improves ZSC without privileged environment information.

Significance. If the central claims hold, the paper makes a useful contribution: it identifies a broader symmetry class than Dec-POMDP symmetries, gives a model-free discovery procedure, and provides evidence of practical ZSC gains in several environments. The group-theoretic lemmas in Appendix B are clean, and the main experiments evaluate learned transformations on independently retrained other-play policies, so the reported cross-play improvements are not merely an artifact of the symmetry-fitting procedure itself. The strengths of the paper are its clear formal setup, the substantial empirical gains, and the reproducible code links for several environments. The main weaknesses are that the formal guarantees do not apply to the policies actually produced by the OPΦER learning rule, and that the learned transformations are not certified to be group elements.

major comments (3)
  1. [§3.2 (Definition 4) and §3.1 (Eq. 7)] Definition 4 constrains transformations only on the set Πα* of self-play-optimal Boltzmann-exploratory policies, but the OP objective in Eq. (4) is maximized over the full policy space Π, and OPΦER-optimal policies are generally not in Πα*. The cat/dog example makes this explicit: the OPΦER-optimal policy is the grounded policy, which is deliberately suboptimal in self-play. For such π ∉ Πα*, the definition of ΦER imposes no constraint on φ(π), so the invariance proofs in Appendix B do not apply to the policies selected by OPΦER. Consequently, the derivation leading to Eq. (7), and the claim that satisfying Items 1 and 2 makes OPΦ equal to expected XP among OPΦ-optimal policies, is incomplete: it requires [π] = ΠΦ* for π ∈ ΠΦ*, which is not established. The empirical check in §4.4, OPΦER(φ(π)) ≈ OPΦER(π), only shows that the orbit of π under the learned transformations has similar OP values; it does not show that the equivalence class of an OPΦER-optimal policy coincides with the set of all OPΦER-optimal policies. This is load-bearing because the theoretical link between the ER-symmetry group and zero-shot coordination depends on it.
  2. [§3.3 (Eq. 10) and Appendix D] The learned observation transformations are not shown to be bijections on O, or even maps from O to O. In Eq. (10), the invertibility term E_o d(o, φ_θ^2(o))^2 is computed over a continuous extension of O, and the text notes that this extension is needed to avoid a locally constant gradient. The parameterization is a feed-forward neural network with ReLU output in the large-scale experiments, so after training there is no guarantee that φ_O,θ is invertible or that it maps observations to observations. Since membership in ΦER requires an element of Ψ, which consists of bijective maps on S, A, and O, the learned transformations are not certified to lie in ΦER. The group-theoretic guarantees of Appendix B therefore do not apply to the symmetries used in the OP objective. Appendix D gives approximate checks of compositionality and invertibility, but these are measured on a finite set of unseen policies and do not provide a certificate for the actual training procedure.
  3. [§4.4, Eq. (9), and Algorithms 1–3] The learned symmetries are selected by maximizing expected return on a finite pool Π′ of approximately optimal policies, e.g., k = 6 self-play seeds in Hanabi. This creates a selection bias: a transformation that happens to preserve returns on Π′ may not preserve returns on other self-play-optimal policies or on the policies encountered by OP. The paper acknowledges that the broader Π′ is, the more representative it is, but it does not provide any bound or statistical test connecting the finite-pool objective to preservation on Πα*. The practical claim that the method 'learns' ΦER is therefore only an approximation, and the experimental gains could partly come from generic policy diversity rather than from membership in ΦER. This concern is not fatal to the empirical results, but it should be stated more carefully as an approximation with no formal guarantee.
minor comments (4)
  1. [§2.3, Eq. (3)] The observation invariance condition contains a typographical error: 'U(ϕ(o_{t+1}) |, ϕ(s_{t+1}), ϕ(a_t))' has an extra comma after the conditioning bar.
  2. [§3.3, Eqs. (8)–(9)] The text calls Eq. (9) an 'equivalent objective' to Eq. (8), but this equivalence holds only approximately because the policies in Π′ are approximately self-play-optimal; the paper should state this as an approximation rather than an equivalence.
  3. [§3.3, Algorithms 1 and 2] The pseudocode in Algorithms 1 and 2 loops over all tuples of local action transpositions, but the Hanabi experiments randomly sample 64 transpositions; the algorithms should indicate that random subsampling is part of the described procedure.
  4. [§4.4, Symmetrizer definition] The symmetrizer S is defined in the experiment section but not in the background; since it plays a role in the reported cross-play results, a brief formal definition in Section 2 would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ER-symmetry discovery fit and the reported zero-shot coordination results are evaluated on disjoint policy sets, so the headline cross-play numbers are not forced by the fitting procedure.

full rationale

The paper defines Phi_ER (Definition 4) as transformations preserving the set Pi_alpha* of self-play-optimal Boltzmann-exploratory policies. While this definition is tailored to self-play-optimal policies, the reported zero-shot coordination improvements are not obtained by re-evaluating the same policies used to fit the symmetries: agents first discover transformations on a fixed pool Pi' of SP-optimal policies (Eqs. 8-11, Algorithms 1-3), then train new OP_Phi_ER policies via Eq. 4, and finally cross-play among these independently retrained policies is measured (Sections 4.1-4.4). The OP_Phi_ER values in Section 4.4 (23.59 vs 23.34) are a post-hoc check on OP-optimal policies, not quantities that enter the symmetry-selection objective. The paper's own limitation statement (Section 6) notes dependence on the policy type used for discovery and on the bijective action/observation parameterization; these are scope and correctness constraints, not evidence that the conclusion is equivalent to its inputs. The main theoretical gap identified by a skeptical reader, namely that Definition 4 restricts Phi_ER to Pi_alpha* while OP is optimized over all policies, so the invariance guarantees need not transfer to OP_Phi_ER-optimal policies, is a validity concern about the strength of the theoretical link, not a circularity: the empirical cross-play results are not encoded in the definition or in the fitting objective. Self-citations, such as the use of a symmetrizer from Muglich et al. 2022a, are methodological prior work and are not load-bearing for the core claim. Thus no circular step can be exhibited from the paper's equations.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The ledger shows the central idea is cheap in invented entities and relies mainly on approximation assumptions: Boltzmann representatives, finite policy pools, and treatable-as-bijective neural maps. There are no new physical or ontological entities; expected return symmetries are a mathematical construction defined from existing Dec-POMDP quantities.

free parameters (5)
  • Boltzmann temperature alpha = 1 (three-lever, Hanabi), 1/2.667 (cat/dog), 1.1 (Overcooked V2)
    Hand-tuned per environment in Appendix A. The definition of Pi_alpha and therefore Phi_ER depends on alpha; different alpha changes which transformations preserve optimality.
  • Regularization weights lambda_1, lambda_2 = lambda_1 = 0.65, lambda_2 = 2.5e-9 (Hanabi)
    Hand-chosen in Equation 10 to balance compositionality and invertibility against return preservation; no ablation is shown.
  • Pool sizes k, l, m = k = 20/10/12/6, l = 6/3/16/11, m = 3/3/2/3 across environments
    Number of self-play policies used to discover symmetries, number of selected symmetries, and number of other-play policies trained per agent; these are chosen without systematic ablation.
  • Sampled action transpositions = 64 of 190 in Hanabi; exhaustive in smaller games
    Algorithm 2 undersamples the action transposition space. The paper acknowledges this in Section 4.4 but does not characterize the effect on the approximation of Phi_ER.
  • phi_O,theta initialization and activation = Identity-initialized linear layers with ReLU output
    Appendix A states this initialization is necessary because symmetry discovery is highly initialization sensitive; ReLU breaks guaranteed bijectivity.
assumptions (6)
  • standard math Standard Dec-POMDP model and Bellman equations define J, Q, and value functions.
    Used throughout Section 2 and Appendix B without proof.
  • domain assumption Boltzmann-exploratory policies Pi_alpha are the right representative class of self-play optimal policies for defining expected return symmetries.
    Definition 4 and Appendix C argue this is necessary to avoid spurious symmetries; it restricts the group to policies whose Q-values follow softmax at all action-observation histories.
  • domain assumption Actor-critic policies with entropy regularization converge to Boltzmann policies with logits proportional to Q-values.
    Appendix B.2 proves this only under the assumption that Q-functions are fixed and converged; in practice PPO and IPPO update Q and policy simultaneously, so Pi_alpha membership is approximate.
  • domain assumption The finite pool Pi-prime is representative of Pi_alpha*.
    Equations 8 through 11 replace expectations over Pi_alpha* with a fixed pool of k policies; the paper notes broader pools are less overfit but does not quantify representativeness.
  • ad hoc to paper Learned phi_O,theta can be treated as bijections on observation space.
    Section 3.3 and Algorithm 2 enforce approximate invertibility via regularization, but the neural parameterization with ReLU is not bijective in general; the group guarantees in Appendix B apply only to exact Phi_ER.
  • standard math The group Psi of bijective relabelings is finite, so every element has finite order.
    Used in Appendix B.1 to show inverses exist via powers; true when S, A, and O are finite.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expected Return Symmetries." pith.science (2026). https://pith.science/paper/S6DUJ54Q

@misc{pith2026250201711,
  author       = {Pith},
  title        = {Pith review of: Expected Return Symmetries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6DUJ54Q}},
  note         = {Machine review of arXiv:2502.01711}
}
read the original abstract

Symmetry is an important inductive bias that can improve model robustness and generalization across many deep learning domains. In multi-agent settings, a priori known symmetries have been shown to address a fundamental coordination failure mode known as mutually incompatible symmetry breaking; e.g. in a game where two independent agents can choose to move "left'' or "right'', and where a reward of +1 or -1 is received when the agents choose the same action or different actions, respectively. However, the efficient and automatic discovery of environment symmetries, in particular for decentralized partially observable Markov decision processes, remains an open problem. Furthermore, environmental symmetry breaking constitutes only one type of coordination failure, which motivates the search for a more accessible and broader symmetry class. In this paper, we introduce such a broader group of previously unexplored symmetries, which we call expected return symmetries, which contains environment symmetries as a subgroup. We show that agents trained to be compatible under the group of expected return symmetries achieve better zero-shot coordination results than those using environment symmetries. As an additional benefit, our method makes minimal a priori assumptions about the structure of their environment and does not require access to ground truth symmetries.

Figures

Figures reproduced from arXiv: 2502.01711 by the authors.

Figure 1
Figure 1. Mutually incompatible symmetry breaking between chess players is shown in the left side panels (Chess.com, 2021). Let π1, π2 represent the joint policies under which both play￾ers choose handshake, fist bump, respectively, and let the reward be ±1, depending on whether they match or not. The self-play score of both joint policies is 1, but the cross-play score between them is −1. Thus, policies incompatibly break th… view at source ↗
Figure 2
Figure 2. Cross-play score distribution of the IPPO self-play baseline population and the ER symmetry agent popula￾tion in Overcooked V2. The baseline population achieves mean SP scores of 162.33 ± 0.14, and the ER symmetry popu￾lation achieves mean SP scores of 27.81 ± 0.3. for the OPΦ ER -optimal population. The ER symmetry population thus significantly reduces the SP￾XP gap, leading to more consistent coordination. Overcoo… view at source ↗
Figure 3
Figure 3. Conditional action matrices of OPΦ MDP -optimal and OPΦ ER -optimal policies; i.e., P(a i t | a j t−1 ). We select the agent from both respective populations achieving the highest cross-play scores. We can see the OPΦ ER -optimal policy more consistently uses a rank hint to signal playing the fifth card, whereas the OPΦ MDP -optimal policy uses a similar convention but less consistently. G HANABI Hanabi is a coopera… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. High entropy leads to symmetry-equivariant policies in Dec-POMDPs

    cs.LG 2025-11 reject novelty 6.0 of 10

    Standard PPO with higher entropy and a tuned GAE reaches state-of-the-art cross-play between Hanabi seeds, but the stated proof that entropy forces symmetry-compatible policies is missing.

  2. Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details

    cs.AI 2026-08 conditional novelty 5.0 of 10

    For Other-Play in Yokai, agents trained with different implementation details coordinate across implementations about as well as across seeds, supporting inter-seed cross-play as a proxy for cross-implementation evaluation.

Reference graph

Works this paper leans on

44 extracted references · 35 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Addressing Imperfect Symmetry: a Novel Symmetry-Learning Actor-Critic Extension

    Miguel Abreu, Luis Paulo Reis, and Nuno Lau. Addressing Imperfect Symmetry: a Novel Symmetry-Learning Actor-Critic Extension . arXiv preprint arXiv:2309.02711, 2023

    arXiv 2023

  3. [3]

    The Hanabi Challenge: A New Frontier for AI Research

    Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The Hanabi Challenge: A New Frontier for AI Research . Artificial Intelligence, 280: 0 103216, 2020

    work page 2020

  4. [4]

    Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

    Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veli c kovi \'c . Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges . arXiv preprint arXiv:2104.13478, 2021

    arXiv 2021

  5. [5]

    A Cognitive Hierarchy Model of Games

    Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. A Cognitive Hierarchy Model of Games . The Quarterly Journal of Economics, 119 0 (3): 0 861--898, 2004

    work page 2004

  6. [6]

    On the Utility of Learning about Humans for Human-AI Coordination

    Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the Utility of Learning about Humans for Human-AI Coordination . Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  7. [7]

    The Worst And The Best Chess Handshakes Of World Cup 2021

    Chess.com. The Worst And The Best Chess Handshakes Of World Cup 2021 . https://www.youtube.com/watch?v=6fS7bDyNYHI, 2021. Accessed: 2024-09-30

    work page 2021

  8. [8]

    Gauge Equivariant Convolutional Networks and the Icosahedral CNN

    Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge Equivariant Convolutional Networks and the Icosahedral CNN . In International Conference on Machine learning, pp.\ 1321--1330. PMLR, 2019

    work page 2019

Show all 44 references
  1. [9]

    K-level Reasoning for Zero-Shot Coordination in Hanabi

    Brandon Cui, Hengyuan Hu, Luis Pineda, and Jakob Foerster. K-level Reasoning for Zero-Shot Coordination in Hanabi . Advances in Neural Information Processing Systems, 34: 0 8215--8228, 2021

    2021

  2. [10]

    Adversarial Diversity in Hanabi

    Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David J Wu, and Jakob Foerster. Adversarial Diversity in Hanabi . In International Conference on Learning Representations, 2023

    2023

  3. [11]

    Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data

    Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data . In International Conference on Machine Learning, pp.\ 3165--3176. PMLR, 2020

    2020

  4. [12]

    OvercookedV2: Rethinking Overcooked for Zero-Shot Coordination

    Tobias Gessler, Tin Dizdarevic, Ani Calinescu, Benjamin Ellis, Andrei Lupu, and Jakob Foerster. OvercookedV2: Rethinking Overcooked for Zero-Shot Coordination . In International Conference on Learning Representations, 2025

    2025

  5. [13]

    Proper Value Equivalence

    Christopher Grimm, Andr \'e Barreto, Greg Farquhar, David Silver, and Satinder Singh. Proper Value Equivalence . Advances in Neural Information Processing Systems, 34: 0 7773--7786, 2021

    2021

  6. [14]

    Other-Play

    Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-Play" for Zero-Shot Coordination . In International Conference on Machine Learning, pp.\ 4399--4410. PMLR, 2020

    2020

  7. [15]

    Off-Belief Learning

    Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. Off-Belief Learning . In International Conference on Machine Learning, pp.\ 4369--4379. PMLR, 2021

    2021

  8. [16]

    AI Alignment: A Comprehensive Survey

    Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. AI Alignment: A Comprehensive Survey . arXiv preprint arXiv:2310.19852, 2023

    2023 arXiv

  9. [17]

    Open-source AI Assistant for Cooperative Multi-agent Systems for Lunar Prospecting Missions

    Zahi M Kakish, F Rodr \ A guez-Lera, D Bischel, A Mosquera, R Boumghar, S Kaczmarek, T Seabrook, P Metzger, and JL Galanche. Open-source AI Assistant for Cooperative Multi-agent Systems for Lunar Prospecting Missions . In 8th European Conference for Aeronautics and Space Scien...

    2019

  10. [18]

    ImageNet Classification with Deep Convolutional Neural Networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks . Communications of the ACM, 60 0 (6): 0 84--90, 2017

    2017

  11. [19]

    Towards a Unified Theory of State Abstraction for MDPs

    Lihong Li, Thomas J Walsh, and Michael L Littman. Towards a Unified Theory of State Abstraction for MDPs . AI&M, 1 0 (2): 0 3, 2006

    2006

  12. [20]

    Trajectory Diversity for Zero-Shot Coordination

    Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory Diversity for Zero-Shot Coordination . In International Conference on Machine Learning, pp.\ 7204--7213. PMLR, 2021

    2021

  13. [21]

    Coordination of Autonomous Vehicles: Taxonomy and Survey

    Stefano Mariani, Giacomo Cabri, and Franco Zambonelli. Coordination of Autonomous Vehicles: Taxonomy and Survey . ACM Computing Surveys (CSUR), 54 0 (1): 0 1--33, 2021

    2021

  14. [22]

    Equivariant Networks for Zero-Shot Coordination

    Darius Muglich, Christian Schroeder de Witt, Elise van der Pol, Shimon Whiteson, and Jakob Foerster. Equivariant Networks for Zero-Shot Coordination . Advances in Neural Information Processing Systems, 35: 0 6410--6423, 2022 a

    2022

  15. [23]

    Generalized Beliefs for Cooperative AI

    Darius Muglich, Luisa M Zintgraf, Christian A Schroeder De Witt, Shimon Whiteson, and Jakob Foerster. Generalized Beliefs for Cooperative AI . In International Conference on Machine Learning, pp.\ 16062--16082. PMLR, 2022 b

    2022

  16. [24]

    On the hardness of finding symmetries in Markov decision processes

    Shravan Matthur Narayanamurthy and Balaraman Ravindran. On the hardness of finding symmetries in Markov decision processes . In International Conference on Machine learning, pp.\ 688--695, 2008

    2008

  17. [25]

    Continuous Coordination As a Realistic Scenario for Lifelong Learning

    Hadi Nekoei, Akilesh Badrinaaraayanan, Aaron Courville, and Sarath Chandar. Continuous Coordination As a Realistic Scenario for Lifelong Learning . In International Conference on Machine Learning, pp.\ 8016--8024. PMLR, 2021

    2021

  18. [26]

    Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi

    Hadi Nekoei, Xutong Zhao, Janarthanan Rajendran, Miao Liu, and Sarath Chandar. Towards Few-shot Coordination: Revisiting Ad-hoc Teamplay Challenge In the Game of Hanabi . In Conference on Lifelong Learning Agents, pp.\ 861--877. PMLR, 2023

    2023

  19. [27]

    Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wrist

    Hai Nguyen, Tadashi Kozuno, Cristian C Beltran-Hernandez, and Masashi Hamaya. Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wrist . arXiv preprint arXiv:2402.18002, 2024

    2024 arXiv

  20. [28]

    Dec-POMDPs with delayed communication

    Frans A Oliehoek, Matthijs TJ Spaan, Nikos Vlassis, et al. Dec-POMDPs with delayed communication . In The 2nd Workshop on Multi-agent Sequential Decision-Making in Uncertain Domains. Citeseer, 2007

    2007

  21. [29]

    Vehicle Communication Strategies for Simulated Highway Driving

    Cinjon Resnick, Ilya Kulikov, Kyunghyun Cho, and Jason Weston. Vehicle Communication Strategies for Simulated Highway Driving . arXiv preprint arXiv:1804.07178, 2018

    2018 arXiv

  22. [30]

    Continuous MDP Homomorphisms and Homomorphic Policy Gradient

    Sahand Rezaei-Shoshtari, Rosie Zhao, Prakash Panangaden, David Meger, and Doina Precup. Continuous MDP Homomorphisms and Homomorphic Policy Gradient . Advances in Neural Information Processing Systems, 35: 0 20189--20204, 2022

    2022

  23. [31]

    JaxMARL: Multi-Agent RL Environments and Algorithms in JAX

    Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ingvarsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, Saptarashmi Bandyopadhyay, Mikayel Samvelyan, Minqi Jiang, Robert Tjarko Lange, Shimon Whiteson, Bruno La...

    2023 arXiv

  24. [32]

    Some Studies in Machine Learning Using the Game of Checkers

    Arthur L Samuel. Some Studies in Machine Learning Using the Game of Checkers . IBM Journal of research and development, 3 0 (3): 0 210--229, 1959

    1959

  25. [33]

    Policy Gradient Methods for Reinforcement Learning with Function Approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation . Advances in Neural Information Processing Systems, 12, 1999

    1999

  26. [34]

    Multi-Agent Reinforcement Learning: Independent vs

    Ming Tan. Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents . In International Conference on Machine Mearning, pp.\ 330--337, 1993

    1993

  27. [35]

    Temporal difference learning and TD-Gammon

    Gerald Tesauro et al. Temporal difference learning and TD-Gammon . Communications of the ACM, 38 0 (3): 0 58--68, 1995

    1995

  28. [36]

    A New Formalism, Method and Open Issues for Zero-Shot Coordination

    Johannes Treutlein, Michael Dennis, Caspar Oesterheld, and Jakob Foerster. A New Formalism, Method and Open Issues for Zero-Shot Coordination . In International Conference on Machine Learning, pp.\ 10413--10423. PMLR, 2021

    2021

  29. [37]

    MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning

    Elise Van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning . Advances in Neural Information Processing Systems, 33: 0 4199--4210, 2020

    2020

  30. [38]

    Multi-Agent MDP Homomorphic Networks

    Elise van der Pol, Herke van Hoof, Frans A Oliehoek, and Max Welling. Multi-Agent MDP Homomorphic Networks . In International Conference on Learning Representations, 2021

    2021

  31. [39]

    The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games . Advances in Neural Information Processing Systems, 35: 0 24611--24624, 2022

    2022

  32. [40]

    Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning

    Xin Yu, Rongye Shi, Pu Feng, Yongkai Tian, Simin Li, Shuhao Liao, and Wenjun Wu. Leveraging Partial Symmetry for Multi-Agent Reinforcement Learning . In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 17583--17590, 2024

    2024

  33. [41]

    Sample Efficient Grasp Learning Using Equivariant Models

    Xupeng Zhu, Dian Wang, Ondrej Biza, Guanang Su, Robin Walters, and Robert Platt. Sample Efficient Grasp Learning Using Equivariant Models . arXiv preprint arXiv:2202.09468, 2022

    2022 arXiv

  34. [42]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  35. [43]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  36. [44]

    J^w >V ؅m./9gOݳ3 <3

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.