{"id":"7f04f1c4-72d6-4dbc-995c-55d7551417d8","arxiv_id":"2502.01711","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Expected return symmetries, transformations that preserve the expected return of optimal policies, contain environment symmetries as a subgroup and improve zero-shot coordination in Hanabi and Overcooked V2.","lead":"This paper introduces expected return symmetries, a new class of transformations in cooperative multi-agent games that preserve the returns of optimal policies, and shows they can be learned from interaction without knowing the environment's true symmetries. Agents trained to be compatible under these symmetries coordinate much better with unseen partners in Hanabi and Overcooked V2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact ΦER is defined only on self-play-optimal policies, while OP is optimized over all policies; the guarantees in Sec. 3.1 do not apply to OPΦER-optimal policies.","rationale":"The reader's weakest_assumption focuses on the approximation of ΦER by a finite pool Π' and a soft-invertible neural parameterization. That is a legitimate concern, but there is a more fundamental gap upstream: even if the learned transformations were exact members of the true ΦER, the OP objective is optimized over policies outside Πα*, where ΦER provides no preservation guarantee. The paper acknowledges that ΦER may not fully satisfy Item 2 and cites Section 4.4 as empirical evidence, but that evidence is gathered after training with the learned symmetries and does not test the equality [π] = ΠΦER*. This does not undermine the reported empirical XP improvements—Table 1 shows strong gains—so the verdict remains CONDITIONAL: the empirical direction is plausible, but the theoretical framing needs either a proof that OPΦER-optimal policies fall inside Πα* (or that their orbits are well-behaved), or an explicit statement that the ZSC benefit is an empirical finding rather than a consequence of the group definition. I therefore agree partially with the reader: the approximation gap is real, but the domain mismatch between ΦER and OP is the more load-bearing issue.","tokens_in":23108,"tokens_out":8766,"duration_ms":85375,"concrete_test":"On a small Dec-POMDP where exact ΦER can be enumerated (e.g., cat/dog), compute the exact OPΦER-optimal policy π* and its equivalence class [π*] under ΦER. Check whether [π*] equals the full set of OPΦER-optimal policies ΠΦER*, and compare maxπ OPΦER(π) with XP(OPΦER*) (Eq. 5). If the two values differ, the equality condition in Eq. (7) fails and the OP objective does not target the true ZSC objective for ER symmetries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Definition 4 defines ΦER as transformations preserving the set Πα* of self-play-optimal Boltzmann-exploratory policies. But the OP objective (Eq. 4) is maximized over the full policy space Π, and the resulting OPΦER-optimal policies are generally not in Πα*: in the cat/dog example the OPΦER-optimal policy is the grounded policy, which is deliberately suboptimal in self-play. For such π ∉ Πα*, nothing in the definition of ΦER constrains φ(π); the Appendix B invariance proofs and the 'Item 2' optimality argument in Sec. 3.1 apply only to policies in Πα*. The paper attempts to close this gap empirically in Sec. 4.4, but the check OPΦER(φ(π)) ≈ OPΦER(π) only shows that the orbit has similar OP-values; it does not establish the stronger condition [π] = ΠΦER* needed for equality in Eq. (7). Without that, maximizing OPΦER need not correspond to maximizing expected XP between independently trained agents, and the theoretical link between the ER-symmetry group and ZSC is incomplete even with exact ΦER.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces expected return symmetries (ΦER), defined as the transformations in Ψ that preserve the set of self-play-optimal Boltzmann-exploratory policies, and argues that this group contains Dec-POMDP symmetries as a subgroup. The authors propose three practical objectives and associated algorithms for learning approximate ER symmetries from agent-environment interaction, and evaluate the resulting other-play variants in four environments. They report that agents trained with OP over learned ER symmetries achieve substantially better zero-shot coordination than self-play or OP over Dec-POMDP symmetries, e.g., Hanabi cross-play of 21.64 versus 8.61 for OP with ΦMDP. The central claims are that ΦER is a broader and learnable symmetry class and that it improves ZSC without privileged environment information.","tokens_in":23326,"tokens_out":3532,"duration_ms":35613,"significance":"If the central claims hold, the paper makes a useful contribution: it identifies a broader symmetry class than Dec-POMDP symmetries, gives a model-free discovery procedure, and provides evidence of practical ZSC gains in several environments. The group-theoretic lemmas in Appendix B are clean, and the main experiments evaluate learned transformations on independently retrained other-play policies, so the reported cross-play improvements are not merely an artifact of the symmetry-fitting procedure itself. The strengths of the paper are its clear formal setup, the substantial empirical gains, and the reproducible code links for several environments. The main weaknesses are that the formal guarantees do not apply to the policies actually produced by the OPΦER learning rule, and that the learned transformations are not certified to be group elements.","major_comments":[{"comment":"Definition 4 constrains transformations only on the set Πα* of self-play-optimal Boltzmann-exploratory policies, but the OP objective in Eq. (4) is maximized over the full policy space Π, and OPΦER-optimal policies are generally not in Πα*. The cat/dog example makes this explicit: the OPΦER-optimal policy is the grounded policy, which is deliberately suboptimal in self-play. For such π ∉ Πα*, the definition of ΦER imposes no constraint on φ(π), so the invariance proofs in Appendix B do not apply to the policies selected by OPΦER. Consequently, the derivation leading to Eq. (7), and the claim that satisfying Items 1 and 2 makes OPΦ equal to expected XP among OPΦ-optimal policies, is incomplete: it requires [π] = ΠΦ* for π ∈ ΠΦ*, which is not established. The empirical check in §4.4, OPΦER(φ(π)) ≈ OPΦER(π), only shows that the orbit of π under the learned transformations has similar OP values; it does not show that the equivalence class of an OPΦER-optimal policy coincides with the set of all OPΦER-optimal policies. This is load-bearing because the theoretical link between the ER-symmetry group and zero-shot coordination depends on it.","section":"§3.2 (Definition 4) and §3.1 (Eq. 7)"},{"comment":"The learned observation transformations are not shown to be bijections on O, or even maps from O to O. In Eq. (10), the invertibility term E_o d(o, φ_θ^2(o))^2 is computed over a continuous extension of O, and the text notes that this extension is needed to avoid a locally constant gradient. The parameterization is a feed-forward neural network with ReLU output in the large-scale experiments, so after training there is no guarantee that φ_O,θ is invertible or that it maps observations to observations. Since membership in ΦER requires an element of Ψ, which consists of bijective maps on S, A, and O, the learned transformations are not certified to lie in ΦER. The group-theoretic guarantees of Appendix B therefore do not apply to the symmetries used in the OP objective. Appendix D gives approximate checks of compositionality and invertibility, but these are measured on a finite set of unseen policies and do not provide a certificate for the actual training procedure.","section":"§3.3 (Eq. 10) and Appendix D"},{"comment":"The learned symmetries are selected by maximizing expected return on a finite pool Π′ of approximately optimal policies, e.g., k = 6 self-play seeds in Hanabi. This creates a selection bias: a transformation that happens to preserve returns on Π′ may not preserve returns on other self-play-optimal policies or on the policies encountered by OP. The paper acknowledges that the broader Π′ is, the more representative it is, but it does not provide any bound or statistical test connecting the finite-pool objective to preservation on Πα*. The practical claim that the method 'learns' ΦER is therefore only an approximation, and the experimental gains could partly come from generic policy diversity rather than from membership in ΦER. This concern is not fatal to the empirical results, but it should be stated more carefully as an approximation with no formal guarantee.","section":"§4.4, Eq. (9), and Algorithms 1–3"}],"minor_comments":[{"comment":"The observation invariance condition contains a typographical error: 'U(ϕ(o_{t+1}) |, ϕ(s_{t+1}), ϕ(a_t))' has an extra comma after the conditioning bar.","section":"§2.3, Eq. (3)"},{"comment":"The text calls Eq. (9) an 'equivalent objective' to Eq. (8), but this equivalence holds only approximately because the policies in Π′ are approximately self-play-optimal; the paper should state this as an approximation rather than an equivalence.","section":"§3.3, Eqs. (8)–(9)"},{"comment":"The pseudocode in Algorithms 1 and 2 loops over all tuples of local action transpositions, but the Hanabi experiments randomly sample 64 transpositions; the algorithms should indicate that random subsampling is part of the described procedure.","section":"§3.3, Algorithms 1 and 2"},{"comment":"The symmetrizer S is defined in the experiment section but not in the background; since it plays a role in the reported cross-play results, a brief formal definition in Section 2 would improve readability.","section":"§4.4, Symmetrizer definition"}],"recommendation":"major_revision","confidential_remarks":"The paper is already accepted at ICLR 2025, but as a journal referee I would not regard the current manuscript as ready without revision. The main issue is not the empirical demonstrations, which are interesting and fairly convincing, but the gap between the formal definition of ΦER and the policies produced by the OPΦER learning rule. I would not require new experiments to address this gap if the authors explicitly weaken the theoretical claims and present the method as an approximate, empirically motivated approach. The second major concern, about certifying that learned transformations are bijections, also needs an explicit acknowledgment of the limitation rather than a claim of group membership."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the ICLR paper on expected return symmetries. The new idea is real: instead of hand-specified environment symmetries, they define ΦER as transformations of the policy space that preserve self-play optimal Boltzmann policies, and they show it contains ΦMDP as a subgroup and can be learned model-free. The Hanabi result is impressive—mean cross-play 21.64 with OPΦER vs 8.61 for OPΦMDP—and the cat/dog example is a clean proof of concept. The group-theoretic appendix is solid as far as it goes, and the code links and hyperparameters make it reproducible in principle.\n\nThe soft spots are proportionate. The main one is the stress-test note: Definition 4 only constrains φ on Πα*, the self-play-optimal Boltzmann policies, but the OP objective is maximized over all policies, and OPΦER-optimal policies are generally not in Πα*. The Appendix B invariance proofs and the 'Item 2' argument in Sec 3.1 only apply to policies in Πα*. The paper tries to close this empirically in Sec 4.4, but checking OPΦER(φ(π)) ≈ OPΦER(π) on the optimal policies only shows those orbits have similar OP-values; it doesn't establish [π] = ΠΦER* needed for equality in Eq. (7). So the theoretical link between ΦER and ZSC is incomplete even for exact ΦER. That's not fatal—the empirical gains are real—but the narrative overstates the theoretical closure.\n\nTwo minor issues: the claims that off-belief learning and cognitive hierarchies fail are asserted without reported experiments (they support this only in small examples), and the three-lever and cat/dog sections lack quantitative cross-play tables. Also, the learned transformations are only approximate members of ΦER; the soft invertibility penalty doesn't guarantee bijections, so the group-theoretic guarantees may not apply to what's actually used. These are worth noting but don't undermine the main empirical direction.\n\nOverall: this is a useful paper for anyone working on zero-shot coordination or symmetry-based RL. It deserves a serious referee. It's already published at ICLR, but if this were going through review today, I'd send it to referees with a request to focus on the gap between ΦER and OPΦER-optimal policies, and to ask for either a theoretical bridge or a more guarded statement of the guarantees.","headline":"A genuinely new symmetry class for ZSC with strong empirical results, but the theoretical guarantee doesn't cover the policies that OPΦER actually produces.","tokens_in":23867,"tokens_out":2329,"would_cite":true,"duration_ms":21194,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper defines expected return symmetries—transformations preserving optimal-policy returns—and shows they can be learned and used to improve zero-shot coordination beyond environment symmetries.","keywords":["expected return symmetries","zero-shot coordination","other-play","Dec-POMDP","symmetry discovery","multi-agent reinforcement learning","Boltzmann exploration","Hanabi"],"falsifier":"Train a held-out set of self-play-optimal Boltzmann-exploratory policies and apply the learned transformations to them: if the expected return of $\\phi(\\pi)$ is systematically below $J(\\pi)$, or if replacing the learned transformations with random policy permutations matched for diversity gives the same cross-play improvement, then the transformations are not expected return symmetries and the coordination gain is not due to the group.","tokens_in":22867,"feed_emoji":"🤝","tokens_out":9667,"duration_ms":78607,"temperature":0.7,"pith_summary":"This paper defines expected return symmetries: transformations of agents' actions and observations that map every self-play-optimal policy with Boltzmann (softmax) exploration to another policy with the same expected return. These transformations form a group, and the usual Dec-POMDP (environment) symmetries are a subgroup, so the new group captures coordination-equivalent policies that no environment relabeling can express—for example, two opposite but equally good communication conventions in a game whose rewards break every naive symmetry. The paper shows how to discover approximate members of this group from ordinary agent–environment interaction, with no access to transition or reward functions or ground-truth symmetries, and then use them in the other-play objective so independently trained agents become compatible. Across the iterated lever game, a cat/dog communication game, Overcooked V2, and Hanabi, agents trained under expected return symmetries achieve better zero-shot coordination than agents trained under environment symmetries, with Hanabi within-population cross-play rising from 8.61 to 21.64 at roughly unchanged self-play scores.","feed_headline":"Learned return symmetries lift Hanabi cross-play to 21.64","feed_subtitle":"Model-free discovery of hidden policy equivalences beats the 8.61 scored by environment symmetries.","key_machinery":"The central object is the expected return symmetry group $\\Phi_{\\mathrm{ER}}$, the set of transformations of actions and observations that carry every self-play-optimal Boltzmann-exploratory policy to another such policy. Boltzmann-exploratory policies here are softmax policies over local action values at temperature $\\alpha$, a choice that ensures suboptimal actions are distinguished by their quality rather than lumped together. The argument runs through the other-play objective $\\mathrm{OP}_\\Phi(\\pi) = \\mathbb{E}_{\\tilde\\pi \\in [\\pi]}[\\mathrm{XP}(\\pi, \\tilde\\pi)]$, where $[\\pi]$ is the equivalence class under $\\Phi$; using $\\Phi_{\\mathrm{ER}}$ makes this objective average over diverse optimal conventions. The discovery machinery is gradient-based: with a fixed pool $\\Pi'$ of approximately optimal policies, the paper learns $\\phi_\\theta$ by maximizing $J(\\phi_\\theta(\\pi))$ (Equation 9) or cross-play between policy pairs (Equation 11), searching over action transpositions and a learned observation network, and Equation 10 adds penalties that push the learned maps toward compositionality and invertibility.","core_discovery":"The central claim is that the right symmetry group for zero-shot coordination is not the group of relabelings that leave the environment dynamics and rewards invariant, but the larger group $\\Phi_{\\mathrm{ER}}$ of transformations that preserve expected return on the set $\\Pi^\\alpha_*$ of self-play-optimal Boltzmann-exploratory policies. Formally, $\\Phi_{\\mathrm{ER}} = \\{\\phi \\in \\Psi : \\phi(\\pi) \\in \\Pi^\\alpha_* \\text{ for all } \\pi \\in \\Pi^\\alpha_*\\}$, which the paper proves is a group under composition and contains the Dec-POMDP symmetries $\\Phi_{\\mathrm{MDP}}$ as a subgroup. Because $\\Phi_{\\mathrm{ER}}$ puts more, and more diverse, optimal policies into the same equivalence class, other-play over $\\Phi_{\\mathrm{ER}}$ trains each agent to be compatible with partners that share its optimality but break conventions differently, which is exactly the failure mode of mutually incompatible symmetry breaking. The paper further claims that approximate elements of $\\Phi_{\\mathrm{ER}}$ can be learned from interaction alone by maximizing the expected return of transformed fixed optimal policies, with regularization toward compositionality and invertibility, and that this learned group is what delivers the improved zero-shot coordination in the experiments.","pith_inferences":["If outcome-based symmetry is the right abstraction, the same recipe could be carried to single-agent reinforcement learning: transformations that preserve optimal value would justify data augmentation without knowing the dynamics, and one could test whether policy-space symmetrization improves generalization to unseen tasks.","The paper fixes action transformations to transpositions and learns observation transformations; a natural extension is to learn joint transformations over continuous or structured action and observation spaces, where bijections are harder, and to check whether the cross-play gains persist.","A direct test of the mechanism would compare other-play under learned expected return symmetries against other-play under an equal number of random transformations chosen for the same policy diversity; if the random baseline matches the cross-play gains, the gains come from diversity rather than from the group structure.","The regularized learning objective enforces invertibility only softly, so a stricter parameterization with exact bijectivity could turn the approximate group-theoretic guarantees into quantitative bounds on cross-play degradation."],"forward_implications":["Other-play with a symmetry group that preserves optimal policies, rather than environment structure, generalizes beyond label relabelings, so independently trained agents can coordinate on any of several equivalent but differently broken optimal conventions.","Because $\\Phi_{\\mathrm{MDP}} \\subseteq \\Phi_{\\mathrm{ER}}$, all previous gains from environment-symmetry training are retained, and additional gains appear in games with no non-trivial Dec-POMDP symmetries, such as the cat/dog game.","Symmetry-based coordination no longer requires privileged knowledge of the environment's transition, reward, or symmetry structure; a model-free agent can discover approximate expected return symmetries from its own interactions.","A small set of learned transformations can be enough: in Hanabi, 11 learned expected return symmetries produced a within-population cross-play score of 21.64, versus 8.61 for agents using all 120 Dec-POMDP symmetries, while self-play scores stayed near 23.7–24.0.","The approach works in settings where off-belief learning and cognitive-hierarchy methods fail, including the iterated lever game, the cat/dog game, and Overcooked V2."],"supporting_citations":[{"why":"Introduces the other-play objective and Dec-POMDP symmetry breaking that this paper generalizes to $\\Phi_{\\mathrm{ER}}$.","marker":"Hu et al. (2020)"},{"why":"Provides the Dec-POMDP formalism used to define states, observations, transitions, and local policies.","marker":"Oliehoek et al. (2007)"},{"why":"Serves as the source of the iterated three-lever game and the symmetrizer operator used in experiments.","marker":"Treutlein et al. (2021)"},{"why":"Defines off-belief learning and the cat/dog communication game, a setting where the paper shows its method succeeds while OBL fails.","marker":"Hu et al. (2021)"},{"why":"Supplies the policy-gradient update used in Algorithm 1 to learn observation transformations.","marker":"Sutton et al. (1999)"},{"why":"Provides the simulation code and baseline implementations used in the Hanabi and Overcooked experiments.","marker":"Rutherford et al. (2023)"},{"why":"Establishes Hanabi as a benchmark for cooperative partial-observability research.","marker":"Bard et al. (2020)"},{"why":"Introduces Overcooked V2, the asymmetric-information coordination benchmark used for evaluation.","marker":"Gessler et al. (2025)"},{"why":"Supplies the PPO-based training algorithm used to train the self-play and other-play populations.","marker":"Yu et al. (2022)"},{"why":"Supplies the independent Q-learning baseline used in the cat/dog experiment.","marker":"Tan (1993)"}],"fun_headline_variants":["Return symmetries lift Hanabi cross-play to 21.64","Expected return symmetries beat environment symmetries in Hanabi","Broader symmetry group boosts zero-shot coordination to 21.64","Hidden symmetry discovery improves Hanabi zero-shot scores","New symmetry class lifts cross-play in Hanabi to 21.64"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the transformations trained on a finite pool of good policies really are symmetries of the whole set of optimal policies, and that the approximate invertibility penalty makes them true bijections; if either fails, the group-theoretic coordination guarantees lapse.","fun_headline_variants_meta":{"raw":{"variants":["Return symmetries lift Hanabi cross-play to 21.64","Expected return symmetries beat environment symmetries in Hanabi","Broader symmetry group boosts zero-shot coordination to 21.64","Hidden symmetry discovery improves Hanabi zero-shot scores","New symmetry class lifts cross-play in Hanabi to 21.64"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000641,"raw_usage":{"total_tokens":2981,"prompt_tokens":1009,"completion_tokens":1972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":1886}},"tokens_in":625,"tokens_out":1972,"duration_ms":14352,"temperature":1.0,"reasoning_tokens":1886,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T15:17:48.682156+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a held-out set of self-play-optimal Boltzmann-exploratory policies and apply the learned transformations to them: if the expected return of $\\phi(\\pi)$ is systematically below $J(\\pi)$, or if replacing the learned transformations with random policy permutations matched for diversity gives the same cross-play improvement, then the transformations are not expected return symmetries and the coordination gain is not due to the group.","supporting_citations":[{"cited_title":"Dec-POMDPs with delayed communication","cited_arxiv_id":null,"evidence_quote":"Provides the Dec-POMDP formalism used to define states, observations, transitions, and local policies."},{"cited_title":"A New Formalism, Method and Open Issues for Zero-Shot Coordination","cited_arxiv_id":null,"evidence_quote":"Serves as the source of the iterated three-lever game and the symmetrizer operator used in experiments."},{"cited_title":"Policy Gradient Methods for Reinforcement Learning with Function Approximation","cited_arxiv_id":null,"evidence_quote":"Supplies the policy-gradient update used in Algorithm 1 to learn observation transformations."},{"cited_title":"The Hanabi Challenge: A New Frontier for AI Research","cited_arxiv_id":null,"evidence_quote":"Establishes Hanabi as a benchmark for cooperative partial-observability research."},{"cited_title":"OvercookedV2: Rethinking Overcooked for Zero-Shot Coordination","cited_arxiv_id":null,"evidence_quote":"Introduces Overcooked V2, the asymmetric-information coordination benchmark used for evaluation."},{"cited_title":"Multi-Agent Reinforcement Learning: Independent vs","cited_arxiv_id":null,"evidence_quote":"Supplies the independent Q-learning baseline used in the cat/dog experiment."}],"review_version":1}