{"id":"c3d9b17b-841d-4273-9828-2fa7cef09995","arxiv_id":"2504.12714","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Training a self-play agent across many procedurally generated cooperative tasks yields better zero-shot coordination with novel partners and novel layouts than training on one task with many partners.","lead":"Cross-environment Cooperation trains a single self-play agent on billions of procedurally generated Overcooked layouts, and the trained agent coordinates with new partners better than single-task population-based baselines. The paper argues environment diversity, not partner diversity, is the key ingredient for zero-shot human-AI coordination.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'novel environment' claim is not fully tested because the Overcooked generator reuses the same five wall topologies in training and test; a check with truly unseen wall structures is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the procedural generator reuses the five original wall structures, so the held-out Overcooked layouts are not topologically novel. This is the single most important threat to the paper's headline claim because the phrase 'novel environments' is what separates CEC from prior single-task ZSC work. The paper has real independent support: released code, Jax-based fast training, toy ablations, partial-observability and multi-task variants, EGT analysis, and human experiments with subjective ratings. Those results support a more limited conclusion about generalizing across object placements and rotations within familiar wall topologies, but they do not currently establish coordination in genuinely novel spatial layouts. The abstract also overstates the human-play reward comparison with E3T: in Figure 9, E3T obtains higher reward than CEC, while CEC wins on subjective survey metrics. This is a secondary precision issue rather than a separate mechanism-level flaw. Overall, the concern does not justify rejection because the paradigm is plausible and the reported comparisons are informative; it does justify keeping the paper CONDITIONAL pending a test with held-out wall topologies or equivalent evidence of true environment novelty.","tokens_in":21673,"tokens_out":4149,"duration_ms":49425,"concrete_test":"Generate a new held-out test set with wall topologies that are not among the five base layouts (e.g., using random wall placement with a reachability solver, or holding out one entire base topology from CEC training) and evaluate CEC and the baselines on those layouts. If CEC's XP reward collapses to near the level of single-task baselines, the reported advantage depends on training on the five wall structures rather than on general cross-environment coordination norms.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that environment diversity alone, through CEC, enables zero-shot coordination with genuinely novel partners and novel environments. The least secure premise is that the Overcooked evaluation environments are actually novel. In Section A.1 and Algorithm 1, each generated layout first samples one of the five original wall structures uniformly (Lbase∼U({Coordination Ring, Counter Circuit, ...})), then re-randomizes object placements, rotations, and padding; the only safeguard against exact leakage is checking that goal/pot/plate/onion positions differ from the held-out levels. The '100 procedurally generated held-out grids' are sampled by this same generator, so they share wall-topology support with CEC's training distribution. CEC trains for 3 billion steps on all five wall structures; its XP success on the five original layouts and on the 100 generated grids may therefore reflect familiarity with those wall layouts plus object-placement augmentation, rather than a general coordination norm that transfers to never-seen spatial layouts. The toy Dual Destination result has the same structure: 'procedural' variation randomizes starting positions and goal cells in an otherwise wall-free grid, so no novel topology is tested there either. If the generator were allowed to sample wall structures outside the five base layouts, the claim of zero-shot coordination in novel environments would be directly tested; as reported, the Overcooked evidence demonstrates generalization within five wall structures across object arrangements and rotations, which is a weaker statement than the abstract's 'new problems' and 'novel environments.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Cross-Environment Cooperation (CEC), a self-play training paradigm that replaces partner diversity with environment diversity, and evaluates it on a toy Dual Destination game and a JAX-based procedurally generated Overcooked benchmark. The authors train a single recurrent IPPO policy on a distribution of procedurally generated tasks and compare cross-play performance with single-task self-play, FCP, E3T, and a CEC-finetuned variant, in both AI-AI and human-AI settings. They report that CEC outperforms PBT baselines in cross-play on the original five layouts and on 100 generated grids, and that human participants rate CEC as more cooperative despite somewhat lower task score than E3T.","tokens_in":21891,"tokens_out":4333,"duration_ms":48909,"significance":"If the main result holds, the paper provides a useful counterpoint to population-based training: it suggests that increasing environment diversity during self-play can improve zero-shot coordination with unseen partners, and it contributes a fast JAX procedural generator (up to 10 million steps per minute on a single GPU, with roughly 10^17 layouts claimed) plus open source code. The simulated and human experiments are extensive, with multiple seeds, several baselines, an empirical game-theoretic analysis, and a human study with subjective cooperation metrics. The main caveat is that environment novelty is not fully tested, because the generator samples wall structures from the same five layouts that are used in training and evaluation; the central novelty claim therefore needs additional experimental support.","major_comments":[{"comment":"The claim of zero-shot coordination in novel environments is not fully supported as stated. Algorithm 1 first samples L_base uniformly from the same five wall layouts used as evaluation layouts, and the only leakage check in the post-processing step is that object positions differ from the held-out levels G_h. Consequently, the 100 'procedurally generated held-out grids' used in Figure 6 and Figure 22 share their wall-topology support with CEC's training distribution, while the five original layouts are not held out at the topology level either. To support the abstract's claim of generalization to 'many new problems', the evaluation should include wall structures that were never used as base predicates during training, for example new wall topologies or a deliberately held-out set of topologies.","section":"Section 4 / Section A.1 / Algorithm 1"},{"comment":"The headline comparison between environment diversity and partner diversity is confounded. CEC trains on the same five wall structures on which the single-task baselines are evaluated, whereas FCP, E3T, and IPPO are trained on only one layout. The cross-play gap observed in Figure 6 may therefore partly reflect training-set coverage of the test layouts rather than a general advantage of environment diversity over partner diversity. A cleaner test would train a population-based method on the same environment distribution, or hold out entire wall structures from CEC, and compare methods with matched environment coverage.","section":"Section 6, Q1 / Figure 6"},{"comment":"The statement that 'we hold out those five layouts from the CEC generator' is inaccurate as written. Section A.1 says the generator samples each of the five original layouts as the base predicate uniformly at random, and the resampling condition only checks that goal, pot, plate, and onion positions differ from held-out levels. This internal inconsistency should be corrected, and the experiments should be revised so that the '5 Heldout Grids' panel is not read as evidence of generalization to never-seen wall topologies.","section":"Section 5, Evaluation Protocol / Section 6"}],"minor_comments":[{"comment":"The abstract says CEC 'outperforms competitive baselines quantitatively and qualitatively when collaborating with real people', but Figure 9 shows CEC's task score is lower than E3T's and only the subjective ratings favor CEC; the wording should be adjusted to match the reported results.","section":"Abstract"},{"comment":"The heading 'Is increasing environment diversity more effective than increasing partner diversity of ZSC?' should read 'for ZSC'.","section":"Section 6, Q1 heading"},{"comment":"The phrase 'single level ad-hoc single level ad-hoc collaboration performance' contains a duplicated phrase and should be rewritten.","section":"Section 4"},{"comment":"The sentence about methods being 'swerve trained on' appears to contain a typo and should read 'methods were trained on'.","section":"Section A.3"},{"comment":"The paragraph begins with a sentence fragment: 'From.' should be removed or completed.","section":"Section A.6"},{"comment":"The limitations paragraph lists two open questions in one sentence; numbering them would improve readability, and the fact that CEC has not converged should be acknowledged in the main text when interpreting the quantitative comparisons.","section":"Section A.9"},{"comment":"The label '5 Heldout Grids' is misleading given the generator's base-topology leakage; the caption should specify exactly what was held out (object placements) and distinguish it from a true held-out-topology evaluation.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's infrastructure and empirical scope are strong, and the core idea is worth publishing if the environment-novelty claim is made rigorous. The base-topology leakage described in Section A.1 is a load-bearing issue because the paper repeatedly claims generalization to new environments; an additional experiment with truly unseen wall topologies is needed before acceptance. I would not reject because the fix is well-defined and within the scope of the existing codebase and experimental setup."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is real, but the packaging needs a fix. Training a single self-play policy across procedurally varied environments (CEC) clearly beats the partner-diversity baselines on cross-play with new partners in these experiments, and the human evaluation is more careful than most in this area. The stress-test note is right, though: the 'novel environment' claim is weaker than the abstract says. In A.1 and Algorithm 1, every generated Overcooked layout first samples one of the same five wall structures used for the held-out levels, then re-randomizes objects, rotations, and padding. So the 100 'held-out' grids share wall-topology support with CEC's training distribution. CEC demonstrates generalization across object arrangements and rotations within five known wall skeletons, not zero-shot transfer to unseen spatial topologies. The Dual Destination toy has no walls at all, so it does not add topological novelty either. That matters for Q2 and for the abstract's 'new problems' and 'novel environments' language.\n\nWhat is genuinely new: the Jax generator (1.16e17 solvable layouts, 10M steps/min), the scale of the comparison (3B steps, six seeds), and the human study showing CEC preferred on subjective measures while staying competitive on reward. The EGT analysis is a nice addition. The paper also cites McKee et al. honestly and goes beyond it with humans and scale. The ablations (partial observability, multi-task variant, CEC+E3T, no-recurrence control) are useful, and the limitations section is candid.\n\nSoft spots beyond the main one: the abstract says CEC 'outperforms competitive baselines quantitatively and qualitatively' with humans, but the human-play reward numbers in Figure 9 show E3T ahead; CEC wins on subjective ratings and collisions. That overstatement should be fixed. Minor: the solvability guarantee is only reachability-based, and rotating/padding generated grids may hurt single-task baselines beyond the intended distribution shift. CEC is also not converged at 3B steps, so the ceiling is unknown; the authors say this themselves.\n\nBottom line: for a ZSC/multi-agent RL audience, this is a useful empirical contribution and a solid foundation for follow-up work. It deserves a serious referee, not a desk reject. I'd send it to review with a required revision: hold out entire wall topologies, or add truly new wall structures, and align the abstract with the actual human-results comparison.","headline":"CEC is a real empirical step for zero-shot coordination, but the novel-environment claim is undercut by reuse of five wall topologies and the abstract overstates human reward versus E3T.","tokens_in":22461,"tokens_out":3896,"would_cite":true,"duration_ms":41827,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a single self-play policy on a distribution of procedurally generated cooperative tasks yields zero-shot coordination with novel partners and novel environments, and this environment diversity outperforms population-based…","keywords":["zero-shot coordination","multi-agent reinforcement learning","cross-environment cooperation","environment diversity","procedural generation","self-play","human-AI coordination","Overcooked"],"falsifier":"Train a CEC agent with a generator that samples wall topologies from a disjoint held-out family never seen during training (not just new object positions on the five familiar wall structures), and measure cross-play reward on those held-out layouts; if performance falls to single-task-baseline levels, the reported environment generalization is driven by exposure to the training layout support rather than by general cooperative norms.","tokens_in":21472,"feed_emoji":"🍳","tokens_out":7202,"duration_ms":70672,"temperature":0.7,"pith_summary":"The paper tries to establish that environment diversity can substitute for partner diversity in zero-shot coordination: a single self-play policy trained across many procedurally generated cooperative tasks learns general cooperative norms that transfer to novel partners and novel tasks. This matters because current state-of-the-art methods train against populations of partners on one task and fail when the task changes even slightly; practical agents would otherwise need retraining for every new kitchen, road, or room. The authors demonstrate the effect in a simple gridworld and in a scaled-up cooking game, and show that their agents are preferred by human participants on subjective cooperation metrics despite never seeing human data. If correct, the result offers a route toward generalist cooperative AI that does not require collecting human demonstrations or maintaining partner populations.","feed_headline":"Diverse tasks teach cooperative agents better than diverse partners","feed_subtitle":"A single self-play policy trained across many generated kitchens generalizes to new partners and layouts, and humans prefer it.","key_machinery":"The load-bearing mechanism is the procedural task distribution itself: a generator samples one of five base wall structures, removes objects, randomizes goal, plate, pot, and onion placements on reachable walls, randomizes agent start positions, and randomly rotates the grid, yielding billions of solvable layouts. A single recurrent (LSTM) IPPO policy trains in self-play against itself on this distribution, with the cross-environment objective forcing the policy to solve the same kinds of coordination problems in different spatial arrangements. The recurrence lets the policy adapt within an episode, which the authors argue is what lets a single frozen network act differently with different partners and in different environments. This replaces the partner population of PBT with a task distribution while using the same total compute.","core_discovery":"The central claim is that optimizing the cross-environment objective $J(\\pi_C)=\\mathbb{E}_{m\\sim\\mathcal{M}}[S(\\pi_C,\\pi_C,m)]$ — self-play on a distribution of tasks rather than against a population of partners — yields a policy whose cross-play performance with novel partners is higher than population-based training on a single task, and whose behavior transfers to novel tasks and to human partners. In the toy Dual Destination game, CEC scores near an oracle cross-play bound, while self-play and fictitious co-play fail on novel layouts. In Overcooked, single-task methods receive zero reward on 100 held-out procedural layouts, whereas CEC achieves substantial cross-play reward; fine-tuning CEC on a target layout improves single-task cross-play but reduces novel-layout generalization. Human studies on two layouts show CEC outperforms FCP on cooperation score, approaches the reward trajectory of specialized agents, and is rated higher on adaptation, consistency, and enjoyability.","pith_inferences":["A stricter procedural split that withholds wall topologies, not just object placements, would likely shrink the reported generalization gap; part of CEC's advantage may come from having seen the same five wall structures in training rather than from truly general norms.","Combining partner diversity with environment diversity underperforms vanilla CEC in their experiments, but the reported learning curves suggest this is a capacity and convergence issue; longer training or larger networks may unlock both axes together.","The same environment-diversity principle should transfer to partially observable and communication-heavy games; the paper's partial-observation Dual Destination result supports this, but hidden-state conventions like those in Hanabi remain untested.","CEC's collision-avoidance behavior could be measured directly as a learned social norm (e.g., yielding frequency) and tested for transfer to a different task family, rather than relying only on human preference ratings."],"forward_implications":["A single CEC policy can be reused as pretraining: fine-tuning on a target layout improves cross-play reward there beyond FCP and IPPO, at the cost of reduced generalization to novel layouts.","Population-based single-task methods score zero on 100 held-out procedural Overcooked layouts, so deploying them where the task varies even slightly would require retraining per layout.","Without any human data, CEC approaches the human-AI reward of a specialized agent that trained on the test layout, and beats all baselines on human ratings of adaptation, consistency, and enjoyability.","Because CEC trains one policy instead of a population, the same compute budget can be spent on many environment samples rather than many partner seeds, making the approach cheaper to scale."],"supporting_citations":[{"why":"Supplies the Overcooked benchmark and human-AI coordination setup that the baselines and human evaluations build on.","marker":"Carroll et al., 2020"},{"why":"Defines Fictitious Co-Play (FCP), the population-based baseline CEC is compared against for cross-partner generalization.","marker":"Strouse et al., 2022"},{"why":"Provides E3T, the state-of-the-art single-task ZSC baseline and the main reward competitor in human studies.","marker":"Yan et al., 2023"},{"why":"Provides IPPO, the independent self-play algorithm used to train CEC and the self-play baselines.","marker":"de Witt et al., 2020"},{"why":"Prior evidence that environment diversity improves collaboration on novel levels in multi-agent RL, which CEC extends toward human ad-hoc partners.","marker":"McKee et al., 2022"},{"why":"Supplies the Jax-based Overcooked implementation that the paper extends with its procedural generator.","marker":"Rutherford et al., 2023"},{"why":"Maximum-entropy population-based training baseline for zero-shot human-AI coordination used for comparison.","marker":"Zhao et al., 2022"}],"fun_headline_variants":["Diverse tasks outperform diverse partners for zero-shot coordination","Task variety beats partner variety for general coordination","Train across many tasks to coordinate with any new partner","General cooperative AI emerges from task diversity, not partner diversity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the held-out test layouts being genuinely novel environments, but the procedural generator builds every layout from the same five wall structures the agents train on, changing only object placements and rotations; if wall structure is what limits generalization, the advantage may shrink once test layouts contain unseen wall topologies.","fun_headline_variants_meta":{"raw":{"variants":["Diverse tasks outperform diverse partners for zero-shot coordination","Task variety beats partner variety for general coordination","Train across many tasks to coordinate with any new partner","General cooperative AI emerges from task diversity, not partner diversity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2310,"prompt_tokens":902,"completion_tokens":1408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1346}},"tokens_in":518,"tokens_out":1408,"duration_ms":12772,"temperature":1.0,"reasoning_tokens":1346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:24:05.171257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a CEC agent with a generator that samples wall topologies from a disjoint held-out family never seen during training (not just new object positions on the five familiar wall structures), and measure cross-play reward on those held-out layouts; if performance falls to single-task-baseline levels, the reported environment generalization is driven by exposure to the training layout support rather than by general cooperative norms.","supporting_citations":[{"cited_title":"R., Leibo, J","cited_arxiv_id":null,"evidence_quote":"Prior evidence that environment diversity improves collaboration on novel levels in multi-agent RL, which CEC extends toward human ad-hoc partners."}],"review_version":1}