{"id":"63a77816-9557-4153-ad5a-428c93adf312","arxiv_id":"2508.06336","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"UPD trains cooperative agents by adaptively selecting randomly generated partners with high return variance, outperforming population-based and population-free baselines in foraging, cooking, and with humans.","lead":"This paper introduces Unsupervised Partner Design (UPD), a way to train AI agents to cooperate with unfamiliar partners by generating training partners on the fly and picking which ones to practice with based on how much they help the agent learn. Across cooking and foraging games and a study with human players, agents trained this way scored higher and felt more adaptive than agents trained with existing methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The key ablation for the learnability mechanism is confounded: UPD w/o ℓ removes both scoring and buffer/refresh, so the specific contribution of learnability-based selection is not established.","rationale":"The reader's weakest assumption is the theoretical mapping from return variance to expected policy improvement. That is a real weakness, but I think the more load-bearing issue is empirical: the paper's own ablation does not cleanly test the learnability mechanism. UPD w/o ℓ is not a single-component ablation; it removes the scoring function together with the entire buffer/refresh pipeline. Therefore the paper provides no direct evidence that learnability-based selection, as opposed to the surrounding generation/selection infrastructure, causes the improvement. This is more damaging than the theoretical mismatch alone because even a perfect theory would not establish the causal role of the criterion without a controlled comparison. The central abstract claim is about adaptive selection via learnability, not just about randomized generation; if that mechanism is inert, the claim is overclaimed. I do not think this warrants rejection: the method is well evaluated, the transparency note is honest, and the empirical results are useful. The appropriate response is to keep the conditional verdict, requiring either a clean random-from-buffer ablation or a tempered claim about the role of learnability. This is why I recommend UNCHANGED relative to the reader's CONDITIONAL verdict. I credit the paper for the large-scale empirical evaluation, the human study, and the OGC extension; those stand regardless of the mechanism attribution.","tokens_in":28578,"tokens_out":10385,"duration_ms":124441,"concrete_test":"Run a controlled ablation that isolates selection: keep Algorithm 1 identical (same K=4000 generated candidates, same buffer size |B|=512, N=10, R=4, ρ=0.5, Dirichlet bias masks), but replace line 11's top-|B| selection with a uniform random draw of |B| candidates from B_temp. Train on the same five Overcooked-AI layouts with 6 seeds and compare to Table 1. If random-from-buffer is statistically indistinguishable from UPD (94.4), the learnability criterion is not the operative ingredient; if it drops toward UPD w/o ℓ or E3T, the learnability criterion matters.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty claimed in the abstract is adaptive partner selection \"based on a learnability criterion.\" But the only ablation aimed at this component, UPD w/o ℓ (§5.2.2, Table 1), removes learnability scoring and also \"samples random partners per rollout,\" meaning it does not use the scored buffer, the refresh schedule, or the top-|B| admission step at all. Full UPD uses all of those. Thus the observed gap between UPD (94.4±2.3) and UPD w/o ℓ (90.8±6.0) cannot be attributed specifically to the learnability criterion; it could come from the buffer/refresh infrastructure alone. The difference is also small relative to the gap over E3T (78.8±12.5), and UPD w/o bias (87.9±3.6) is close as well. So the data are fully consistent with the alternative hypothesis that randomized large-scale partner generation plus bias masking—an upgraded E3T—is the active ingredient, and learnability selection contributes little. The theoretical bridge in §4.1 does not resolve this: it transfers a result about the variance of a scalar advantage signal (Foster et al., 2026) to episodic return variance in a non-stationary induced game, and no controlled experiment isolates return variance as the causal selection signal. This matters because the paper's advertised conceptual contribution is the learnability-based selection mechanism; if that mechanism is not responsible for the reported gains, the central claim is overstated even though the trained agents are empirically strong.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Unsupervised Partner Design (UPD), a population-free method for training ad-hoc teamwork agents. UPD generates candidate partner policies on the fly by mixing the ego policy with a biased random policy using a stochastic mixing coefficient, scores candidates by the variance of episodic returns (a learnability signal), maintains a buffer of high-scoring partners, and trains the ego policy with PPO. The same mechanism is extended to joint partner-environment curricula (JUPD) using a coefficient-of-variation-squared score. The method is evaluated on Level-Based Foraging, Overcooked-AI, the Overcooked Generalisation Challenge, and in a human-AI study, with 282 trained policies overall. The central claims are that UPD achieves robust ad-hoc teamwork without pre-trained partner populations or manual parameter tuning, and that the adaptive learnability-based selection is the mechanism responsible for the gains.","tokens_in":28992,"tokens_out":8174,"duration_ms":92345,"significance":"If the learnability-based partner-selection mechanism is indeed responsible for the reported gains, this is a valuable contribution: it extends unsupervised environment design to the partner-policy space, removes the need for pre-trained partner populations, and shows a simple adaptive curriculum can compete with or outperform population-based methods across multiple benchmarks. The empirical effort is substantial and is a clear strength: multiple seeds, diverse evaluation partners, an additional ROTATE comparison, per-partner results, curriculum-dynamics analyses, alternative learnability functions, a human study with corrected multiple-comparison statistics, and a transparency note about earlier baseline issues. The paper also honestly discusses the possibility that large-scale randomized partner generation alone may be the active ingredient. However, the evidence isolating the specific contribution of the learnability criterion is incomplete, and the tuning-related claims are broader than what the experiments support. The significance is therefore conditional on resolving these attribution issues.","major_comments":[{"comment":"The only ablation targeting learnability is confounded. The text states that UPD w/o ℓ \"removes learnability scoring and instead samples random partners per rollout,\" which eliminates not only scoring but also the buffer-admission (top-|B|) and refresh (R) steps of Algorithm 1. The observed gap (UPD 94.4±2.3 vs. UPD w/o ℓ 90.8±6.0) can therefore be attributed to the buffer/refresh infrastructure or to the interaction of ϵ~U(0,1) and bias masking, not specifically to the variance signal. This matters because the abstract's central claim is adaptive selection \"based on a learnability criterion.\" The paper needs an ablation that keeps the buffer/refresh pipeline but selects partners randomly, and/or an ablation that keeps learnability scoring but removes the buffer, to isolate the mechanism. Without this, the data are fully consistent with the alternative—raised by the authors themselves in","section":"§5.2.2, Table 1"},{"comment":"The formal connection between return variance and expected policy improvement is not established. The paragraph invokes Foster et al. (2026), whose result concerns the variance of the scalar learning signal used in advantage estimation, and then equates this with Varτ[R(τ)] in the induced single-agent game Gπp,θ. Episodic return variance is not the same as the variance of the per-step advantage estimator, and the induced game is non-stationary because πego changes during training even between buffer refreshes. As written, Eq. (5) is a heuristic. Since the learnability criterion is the advertised conceptual novelty, the paper should either state the precise transfer assumptions or provide a direct derivation for this setting. This issue is compounded by the confounded ablation in Table 1, so the empirical evidence for the criterion is currently weak.","section":"§4.1, Eq. (5)"},{"comment":"The claim that UPD removes \"manual parameter tuning\" is overstated. UPD introduces several curriculum hyperparameters (|B|, N, R, ρ, α, pbias, δ, K) and the paper uses different values across tasks: LBF uses |B|=64, N=5; Overcooked-AI uses |B|=512, N=10; OGC uses buffer sizes 4,096/16,384, ρ=1.0, and R=2. Appendix E, Table 7 describes a hyperparameter search whose choices were made using evaluation with a BRDiv population as a proxy, and Appendix D shows that per-layout fine-tuning of curriculum parameters improves UPD from 94.4 to 102.3 average return. The claim would be defensible if narrowed to \"removes per-layout tuning of the E3T mixture coefficient ϵ in Overcooked-AI\" or if accompanied by sensitivity evidence showing that default hyperparameters are robust across tasks. As written, the abstract overstates the elimination of manual tuning.","section":"Abstract, §5.2.1, Appendix E"}],"minor_comments":[{"comment":"Rename \"UPD w/o ℓ\" to something like \"UPD w/o ℓ (random partners, no buffer)\" to make clear that this ablation removes both learnability scoring and the buffer/refresh infrastructure.","section":"§5.2.2, Table 1"},{"comment":"Table 2 uses \"DR-DR\" but Figure 15 uses \"DR\"; align the naming.","section":"§6, Figure 15"},{"comment":"The stability constant δ is used in the main text but defined only in Appendix E. Add a one-line definition in the main text.","section":"§4.2, Eq. (6)"},{"comment":"The significance stars are informative, but reporting exact p-values and effect sizes (e.g., median/mean differences with confidence intervals) for the human study would improve transparency, especially given the small sample size (n=12).","section":"§5.3, Figure 7"},{"comment":"Line 5 says \"for each desired partner\" without specifying the number of candidates; the number K (the SFL batch size) is only given in Appendix E. State K directly in the algorithm or its caption.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is strong and the paper is honest about limitations, including a transparency note about earlier baseline issues. My main concern is attribution: the advertised learnability mechanism is not isolated by the current ablations, and the theoretical bridge is not yet convincing. This is fixable with an additional isolating ablation and a more careful statement of the theoretical claim, so I recommend major revision rather than rejection. The tuning claim in the abstract should also be qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper on a simple, useful idea—generate partners online via random mixtures of ego and biased random policies, score them by return variance, and train against the high-variance ones. The method is population-free, works across LBF, Overcooked, and the OGC, and the human study is a genuine plus. The paper is worth a serious referee. But the advertised central mechanism—the learnability filter—is not cleanly isolated, and the theory cited for it doesn't quite apply.\n\nWhat's new: applying SFL-style learnability to partner policies, the bias-mask generator, and the joint partner-environment extension (JUPD). Those are real increments over E3T and prior UED work. The empirical work is extensive: 282 trained policies, multiple seeds, ablations, a per-partner breakdown, and a careful double-blind human study with proper multiple-comparison corrections. The transparency note about fixing earlier baseline bugs is a point in their favor.\n\nSoft spots. First, the ablations. UPD w/o ℓ removes not just scoring but the whole buffer/refresh machinery and samples random partners per rollout. So the 3.6-point average gap between UPD and UPD w/o ℓ cannot be attributed specifically to learnability selection; it could come from the buffer infrastructure. The paper itself acknowledges that large-scale generation alone is a strong baseline, and includes helpful analysis in the appendix showing that the convention-breaking dynamics do require the learnability filter. That partially rescues the mechanism, but the headline claim \"selects them adaptively based on a learnability criterion\" overstates what the ablation establishes.\n\nSecond, the theory. Section 4.1 invokes Foster et al. (2026) that expected policy improvement is proportional to variance of the scalar learning signal. That's about advantage variance, not episodic return variance in a non-stationary induced game. The paper says \"applying the result ... implies\" but that transfer is not proven. Minor in practice—the empirical results stand—but the bridge is rhetorical, not formal.\n\nThird, the tuning-free claim. They search over UPD hyperparameters (buffer size, N, refresh frequency) in Appendix E.1 and report a fine-tuned version that does better. So \"no manual parameter tuning\" should be softened to \"one shared configuration across layouts, found with a modest search.\"\n\nWho this is for: anyone working on zero-shot coordination or ad-hoc teamwork, especially with human-AI evaluation. The practical recipe is cheap and reproducible.\n\nRecommendation: send it to review. The empirical core is sound and the method is useful; the learnability mechanism needs a cleaner ablation or a substantially toned-down claim. A referee can sort that out, but the paper should not be desk-rejected.","headline":"The empirical package is strong and the method is useful, but the headline mechanism—learnability-based selection—is not cleanly isolated, and the theory cited for it doesn't quite apply.","tokens_in":29458,"tokens_out":2601,"would_cite":true,"duration_ms":27650,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unsupervised Partner Design (UPD) claims that robust ad-hoc teamwork can be trained by generating partners on the fly and selecting them by return variance, removing pre-trained partner populations and per-task tuning, and that the same rul","keywords":["ad-hoc teamwork","multi-agent reinforcement learning","unsupervised environment design","learnability","curriculum learning","zero-shot coordination","partner generation","Overcooked-AI"],"falsifier":"Train two UPD variants whose scoring rollouts are matched for mean return but differ in return variance, and measure the ego policy's one-step PPO improvement; the paper's learnability justification predicts the high-variance partners produce larger improvement. At the method level, compare UPD with UPD without the learnability filter at matched compute on the layout where the filter matters most (Asymmetric Advantages, reported 181.4 vs 164.0): if filtering does not beat unfiltered generation, the central learnability claim is falsified.","tokens_in":28505,"feed_emoji":"🧑🍳","tokens_out":12538,"duration_ms":119787,"temperature":0.7,"pith_summary":"Unsupervised Partner Design (UPD) targets ad-hoc teamwork—cooperating with partners never encountered during training. The paper's central claim is that robustness to unseen partners does not require the standard expensive ingredients: a pre-trained population of diverse partner policies or hand-tuned mixing coefficients. Instead, UPD generates candidate partners on the fly, scores each by the variance of returns it produces with the current agent, and trains only against the most learnable ones, turning partner selection into an adaptive curriculum. The same scoring rule, renormalized across reward scales, also selects procedurally generated levels jointly with partners. If the claim holds, training for ad-hoc teamwork becomes nearly as cheap as self-play while matching or beating population-based methods in both artificial-partner and human evaluations.","feed_headline":"No partner library: agents design their training partners on the fly","feed_subtitle":"Variance-based learnability builds an adaptive partner curriculum that beats population methods and humans prefer it.","key_machinery":"The load-bearing mechanism is the pair (generator, scorer). The partner generator samples a mixing coefficient $\\epsilon \\sim U(0,1)$ and, with probability 0.5, a persistent action-bias mask $m\\sim\\mathrm{Dirichlet}(\\alpha \\mathbf{1}_A)$, producing a partner policy $\\pi_p = \\epsilon \\pi_{r,m} + (1-\\epsilon)\\pi_{\\mathrm{ego}}$. The scorer then estimates $\\ell_{\\mathrm{var}} = \\mathrm{Var}_{\\tau\\sim G_{\\pi_p,\\theta}}[R(\\tau)]$ from rollout returns (or the mean-normalized coefficient-of-variation-squared variant in joint settings) and admits the top-$|B|$ candidates to a buffer refreshed every $R$ training loops. This turns the induced single-agent game $G_{\\pi_p,\\theta}$ into a training instan","core_discovery":"UPD's central claim is that a partner can be treated as a training instance to be generated and filtered, exactly as unsupervised environment design treats levels. The paper defines a stochastic partner generator that mixes the ego policy with a biased random policy, samples many such partners, scores each by the variance of episodic return when paired with the current ego, and keeps only the highest-variance candidates in a replay buffer for PPO updates. This learnability filter is the whole method: it replaces the pre-trained partner population of fictitious co-play and maximum-entropy population training and the fixed mixture coefficient of the E3T baseline with an adaptive curriculum tha","pith_inferences":["Because UPD's partner space is defined by a stochastic generator, richer partner distributions (learned latent partner models or stored populations) could plug into the same learnability filter; the paper names this as future work, which suggests UPD is best read as a general curriculum layer rather than a single partner recipe.","The emergent bias switches imply a testable bridge to convention-breaking methods: UPD-trained agents should cooperate better with partners that share none of the ego's learned action conventions than self-play agents do, because learnability actively selects such partners.","The JUPD results leave open whether joint selection is additive: a natural ablation the paper does not run is levels chosen by SFL with partners chosen uniformly, or vice versa, which would isolate the gain that comes from selecting both dimensions at once.","The appendix's transparency note reports that an earlier version contained baseline configuration issues; after correction baselines improved but the paper states conclusions were unchanged, so the current comparative numbers are the ones to rely on."],"forward_implications":["Ad-hoc teamwork training becomes single-stage and population-free: UPD needs only the ego policy plus a stochastic partner generator, removing the $O(N C)$ partner-pretraining cost; the paper's cost analysis gives a break-even population size of about 2.5 in the simulation-dominated limit, so any practical population of size 3 or more is more expensive.","One hyperparameter configuration transfers across all five Overcooked layouts, whereas E3T requires a per-layout sweep of its mixture coefficient; this removes task-specific tuning for population-free ad-hoc teamwork.","The same mechanism solves joint partner-and-level generalization: JUPD jointly selects partners and procedurally generated levels and outperforms random level-partner sampling, self-play cross-environment cooperation, and SFL-level selection combined with E3T partners on the Overcooked Generalisation Challenge.","Learnability-based selection produces emergent convention-breaking dynamics—partner action biases switch direction over training—so UPD implicitly avoids overfitting to a single coordination equilibrium without an explicit mechanism.","In a human-AI study, UPD-trained agents achieve higher returns and are rated significantly more adaptive, more human-like, and less frustrating than self-play, MEP, and E3T agents."],"supporting_citations":[{"why":"Defines E3T, the population-free baseline UPD extends by randomizing the mixing coefficient and adding bias masks to partner generation.","marker":"Yan et al. (2023)"},{"why":"Supplies the sampling-for-learnability curriculum idea and the buffer/ratio hyperparameters UPD inherits for partner scoring.","marker":"Rutherford et al. (2024)"},{"why":"Extends learnability scoring to continuous rewards via return variance, motivating UPD's variance-based selection and the CV2 variant.","marker":"Monette et al. (2025)"},{"why":"Provides the formal link between variance of the learning signal and expected policy improvement used to justify return-variance partner selection.","marker":"Foster et al. (2026)"},{"why":"Introduces unsupervised environment design, the paradigm UPD transfers from level parameters to partner policies.","marker":"Dennis et al. (2020)"},{"why":"Supplies the ROTATE baseline and the BRDiv/planning evaluation-partner populations used in Level-Based Foraging and Overcooked-AI.","marker":"Wang et al. (2025)"},{"why":"Defines MEP, a population-based maximum-entropy baseline that UPD is compared against.","marker":"Zhao et al. (2023)"},{"why":"Defines fictitious co-play, the standard pre-trained-partner-population approach UPD avoids.","marker":"Strouse et al. (2021)"},{"why":"Provides the Overcooked-AI benchmark, its five layouts, and observations about human action biases used in partner generation.","marker":"Carroll et al. (2019)"},{"why":"Defines the Overcooked Generalisation Challenge benchmark used to evaluate joint partner-and-level generalization.","marker":"Ruhdorfer et al. (2025b)"}],"fun_headline_variants":["Partners generated on the fly, filtered by learnability","No partner library: agents pick their own training partners","Adaptive partner curriculum beats population methods","Variance-based partner selection for robust teamwork","Unsupervised design of training partners for ad-hoc teams"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the variance of returns across rollouts with a fixed partner tracks how much the ego policy is expected to improve by training with that partner; if that proxy fails, the learnability filter loses its justification.","fun_headline_variants_meta":{"raw":{"variants":["Partners generated on the fly, filtered by learnability","No partner library: agents pick their own training partners","Adaptive partner curriculum beats population methods","Variance-based partner selection for robust teamwork","Unsupervised design of training partners for ad-hoc teams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00094,"raw_usage":{"total_tokens":3803,"prompt_tokens":641,"completion_tokens":3162,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":385,"completion_tokens_details":{"reasoning_tokens":3089}},"tokens_in":385,"tokens_out":3162,"duration_ms":25521,"temperature":1.0,"reasoning_tokens":3089,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:46:38.179625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two UPD variants whose scoring rollouts are matched for mean return but differ in return variance, and measure the ego policy's one-step PPO improvement; the paper's learnability justification predicts the high-variance partners produce larger improvement. At the method level, compare UPD with UPD without the learnability filter at matched compute on the layout where the filter matters most (Asymmetric Advantages, reported 181.4 vs 164.0): if filtering does not beat unfiltered generation, the central learnability claim is falsified.","supporting_citations":[],"review_version":1}