{"id":"b8462f4a-09ea-4e56-a04c-fd934af72d43","arxiv_id":"1908.02269","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding mutual-action-predictability (TeamReg) or synchronized sub-policy switching (CoachReg) as a training-time objective improves cooperative multi-agent RL performance over MADDPG baselines on several sparse-reward tasks.","lead":"Two policy regularizers for cooperative multi-agent reinforcement learning are proposed: one makes each agent's actions predictable to teammates, and one forces all agents to switch together among shared behavior modes. The synchronizing method improves scores on most tested sparse-reward tasks, including a simplified Google Research Football game.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The CoachReg/TeamReg headline margins may owe more to the extra tuning dimensions in the per-environment best-of-50 random search than to the coordination regularizers themselves; the paper provides no nested validation or significance test to rule this out.","rationale":"The paper's central empirical claim lives in Table 1 and the football Table 2, so the comparison protocol is the load-bearing element. The reader's weakest assumption identifies the same primary concern: the proposed methods have three extra tuning coefficients, yet the best-of-50 selection protocol is used without any correction or statistical test. I do not see an internal inconsistency in the derivations; the equations and ablations are coherent. I also do not treat the synchronization-attribution concern as the most load-bearing one, because the policy-mask ablation is a meaningful control for the mask architecture. The missing code and the absence of significance tests make the tuning-budget concern impossible to dismiss from the text alone. The recommended verdict remains conditional, matching the reader's verdict: the methods are plausible and the results are useful, but acceptance should be tied to code release and a check that the reported margins are not an artifact of the extra tuning dimensions.","tokens_in":18481,"tokens_out":7485,"duration_ms":96670,"concrete_test":"Perform a nested validation over the three search seeds: for each algorithm and environment, leave out one seed, select the hyperparameter configuration with the highest mean return on the other two seeds, and record the held-out seed's return; repeat for each seed. Recompute the Table 1 and football margins from these held-out means. If CoachReg's margin over MADDPG and over the policy-mask ablation collapses to within noise, or is not consistently positive across all held-out folds, the reported superiority is a selection artifact rather than a robust effect. This directly penalizes the three extra λ knobs because overfitting the tuning seeds will not transfer to held-out seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 7 and Appendix E compare methods by running 50 random hyperparameter configurations per algorithm-environment pair, selecting the best configuration, and retraining it with 10 seeds (Appendix E.1, E.2). All algorithms share four search dimensions (αθ, ωφ, τ, ηnoise), but TeamReg and CoachReg also tune λ1, λ2, and λ3, with CoachReg's λ3 given its own range (Appendix E.1 Table 2). Because the selected configuration is chosen on the same three search seeds used for the initial comparison, the extra dimensions give the proposed methods additional chances to hit a configuration that is good for the specific task and seed set. The best-of-50 score is then reported as the headline number (Table 1; football Table 2). The policy-mask ablation controls for the mask architecture, but it does not control for the amount of tuning freedom, since no ablated variant tunes λs. Thus the claim that CoachReg 'consistently yields significant improvements' (Section 7) is not yet supported by a comparison neutral to tuning budget, and no significance tests are supplied to separate selection noise from algorithmic advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper argues that coordinating agents' policies can guide exploration in multi-agent reinforcement learning and proposes two policy regularization methods, TeamReg and CoachReg, that augment CTDE algorithms (exemplified by MADDPG). TeamReg adds an inter-agent action predictability objective (Eqs. 3-5); CoachReg adds a centralized coach that selects a shared policy mask that agents are regularized to predict and follow (Eqs. 6-10). The methods are evaluated on four sparse-reward continuous-control tasks and on Google Research Football 3v2, with hyperparameter search (50 configurations, 3 seeds; 10 seeds for final comparison), ablations, and behavioral analysis of masks and predictability. The paper reports that CoachReg consistently improves performance and TeamReg improves performance on two tasks but is harmful on the adversarial COMPROMISE task.","tokens_in":18756,"tokens_out":5391,"duration_ms":56583,"significance":"The contribution is potentially significant: if the reported gains are real, the paper provides simple, generally applicable regularizers that can be plugged into existing MARL algorithms, with careful behavioral validation that the proxies (predictability, mask synchronization) are indeed enforced. The experimental effort is substantial for the era (3 CPU-years, reproducibility-oriented appendices, scaling experiments to 6 agents). The toy Markov game in Section 3 provides an intuitive motivating result. However, the headline empirical claims require stronger statistical support and a tuning-budget-neutral comparison to rule out that the gains are an artifact of the extra hyperparameters searched for the proposed methods.","major_comments":[{"comment":"The comparison between methods is not neutral to tuning budget: TeamReg and CoachReg have three additional hyperparameters (lambda_1, lambda_2, lambda_3) that are included in the per-environment best-of-50 search, while the ablations and baselines either fix or lack these coefficients. The headline numbers in Table 1 and Table 2 are therefore selected from a larger effective configuration space, and the reported margins may reflect tuning flexibility rather than a robust algorithmic advantage. I request a nested validation protocol (e.g., selecting hyperparameters on a separate subset of seeds, or an equal-query budget per algorithm) or an explicit demonstration that the conclusion is insensitive to the number of tuning dimensions.","section":"Appendix E.1-E.2, Table 1"},{"comment":"The paper makes repeated claims of 'significant' improvements without providing significance tests or effect-size statistics. For example, on SPREAD the CoachReg return (210±12) is statistically indistinguishable from MADDPG+policy mask (221±11) and TeamReg (216±12); on CHASE the CoachReg and MADDPG+sharing intervals overlap; and on BOUNCE the gap between CoachReg (7.4±1.2) and MADDPG (4.0±1.6) is underpowered with only 10 seeds. The football result (Table 2) similarly reports one nonzero mean with no test across seeds. Please add per-task significance tests (e.g., paired bootstrap over the 10 seeds) and adjust the wording of the claims accordingly.","section":"Section 7.1, Table 1; Section 7.4, Table 2"},{"comment":"The football evaluation does not include the two ablation variants (MADDPG+agent modelling and MADDPG+policy mask), so it is not possible to determine whether the CoachReg gain comes from synchronous mask selection rather than the coach network or the mask architecture alone. The paper's own Section 7.3 shows the policy-mask ablation is an important control; without it on football, the discrete-action generalization claim is incomplete.","section":"Section 7.4, Table 2"}],"minor_comments":[{"comment":"The symbols lambda_1, lambda_2, and lambda_3 are reused for different objectives in TeamReg and CoachReg; please use distinct symbols or clearly state in the text that the coefficients play analogous but not identical roles.","section":"Equations (5), (9)"},{"comment":"The paper states that the categorical mask distribution is reparameterized with the Gumbel-softmax, but the precise estimator (e.g., straight-through versus soft relaxation) is not specified; this matters for the gradient of the mask sampling step.","section":"Section 4.2, Equation (6)"},{"comment":"The hyperparameter tables contain formatting glitches (e.g., '0 .71', '0 .083') and should be cleaned; additionally, the large relative critic learning rates (omega_phi values such as 53 and 79) deserve a brief explanation in the training details.","section":"Appendix E.3, Tables 3-6"},{"comment":"In Figure 1, the notation 'a=(1,1)' and the dashed versus solid line legend are not explained in the caption; the reader must infer the coordinated versus unconstrained action spaces.","section":"Figure 1"},{"comment":"The Hamming proximity for 'MADDPG + policy mask' is maximized over all mask equivalences, which can inflate the reported proximity; this procedure should be stated in the main text where the ablation comparison is mentioned.","section":"Appendix G.3"}],"recommendation":"major_revision","confidential_remarks":"The tuning-budget confound is the core reason for the major revision; the authors should be given the opportunity to address it with nested validation or a controlled tuning budget. The paper is likely to be influential in the MARL community, but the current evidence is not sufficient for the strong claims made, especially given the lack of significance tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know about this paper. First, the CoachReg idea is actually novel: a central coach selects policy masks at every timestep, with agents regularized to predict the mask from local observations, while the coach is removed at test time. That differs from MAVEN's first-timestep mode selection and from feudal approaches, and the masking mechanism itself is clean. Second, the empirical claims are less solid than the prose suggests. The comparison gives CoachReg and TeamReg three extra hyperparameters (lambda_1..3) in the same 50-configuration random search, so selecting the best config per environment gives them more chances to land on a good setting. The policy-mask ablation does not tune lambda, so it doesn't control for tuning budget. Without nested validation or significance tests, the statement that CoachReg 'consistently yields significant improvements' (Section 7) is not supported.\n\nWhat the paper does well: it defines two concrete coordination proxies and enforces them as auxiliary objectives, not as fitted predictions, so there's no circularity. The experiments are thorough for the era: 50 configs x 3 seeds for search, 10 seeds for final, ablations, behavior analysis showing synchronized mask entropy and Hamming proximity, plus a scalability study. It also honestly reports that TeamReg is harmful on COMPROMISE and explains the failure mode. That's good practice.\n\nThe soft spots, in rough order of importance: (1) the tuning-budget asymmetry, which could explain part of the margin on SPREAD and BOUNCE, where standard errors overlap; (2) no significance tests anywhere; (3) football results have no ablations or analysis, though the gap (0.088 vs 0.004–0.006) is large; (4) no code or data linked, despite a promise to release upon publication. None of these is fatal on its own. The central mechanism is plausible and the ablation shows synchronized masks emerge even without the coach, which supports the attribution to synchronization rather than to the mask architecture alone.\n\nWho should read it: MARL researchers working on coordination, exploration, or CTDE. It would be a useful reading group paper.\n\nRecommendation: it deserved a serious, technical peer review. I'd accept it with major revision: release code, add significance tests, and run a tuning-budget-matched comparison—for example, by giving baselines the same number of random draws through dummy hyperparameters or splitting the search seeds from the evaluation seeds. As it stands, the idea is worth citing, but the headline numbers should be read with caution.","headline":"CoachReg is a genuinely new coordination mechanism with useful results, but the headline margins are undercut by an asymmetric tuning budget and the absence of significance tests.","tokens_in":19268,"tokens_out":2937,"would_cite":true,"duration_ms":30276,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Coordinating exploration through policy regularizers improves sparse-reward multi-agent reinforcement learning, with synchronized sub-policy selection the most consistent.","keywords":["multi-agent reinforcement learning","coordination","policy regularization","MADDPG","centralized training decentralized execution","sparse rewards","policy masks","agent modelling"],"falsifier":"A decisive check would rerun the comparison with every baseline given the same number of free tuning parameters, and would replace CoachReg's coach with a random but shared mask schedule; if the margin disappears or the random schedule matches CoachReg, the claimed mechanism is not what drives the result.","tokens_in":18279,"feed_emoji":"🤝","tokens_out":9724,"duration_ms":92648,"temperature":0.7,"pith_summary":"The paper tries to establish that coordinated exploration, not just centralized critics, is what makes cooperative multi-agent reinforcement learning work under sparse rewards, and that coordination can be injected as a policy regularizer. It proposes TeamReg, which trains each agent to predict teammates' actions and to act predictably, and CoachReg, which trains a central coach to pick a shared sub-policy mask that agents learn to match from their local observations. On four continuous-control tasks and a discrete-action football task, CoachReg consistently improves final returns over MADDPG and ablations, while TeamReg helps on purely cooperative tasks but can destabilize mixed tasks. If right, this means hard cooperative tasks can be made learnable without task-specific reward shaping.","feed_headline":"Synchronized sub-policy switching unlocks sparse-reward cooperation","feed_subtitle":"A training-only coach aligns agents' policy switches and beats standard RL on football and sparse tasks.","key_machinery":"The carrier of CoachReg is the policy mask: a one-hot vector of size K (fixed at 4) that modulates the first hidden layer of an agent's policy network through structured dropout, with only units whose index is congruent to the selected mask kept. A coach network, used only in training, outputs a mask distribution from joint observations; each agent has a linear layer producing its own mask distribution from local observations. The coach and agents are tied by two objectives, a KL divergence that makes agents' masks match the coach's, and a policy-gradient objective evaluated under the coach's mask, and the sampled masks are made differentiable via Gumbel-softmax. TeamReg's carrier is the team-spirit objective pair: each agent predicts teammates' actions from its own observation, and is regularized to be predictable in return. These mechanisms are what inject the coordination inductive bias into MADDPG.","core_discovery":"Adding coordination-promoting regularizers to MADDPG biases exploration toward joint behaviors and raises final returns on several sparse-reward cooperative tasks. CoachReg, the strongest method, enforces synchronous sub-policy selection: a training-only coach chooses a policy mask from the joint observation, agents are regularized to predict the same mask from their own observations, and both coach and agents are optimized so that following the coach's mask yields high returns. At test time the coach is removed and agents still switch masks synchronously. The paper reports, for example, 0.088±0.017 average return on the 3v2 football task versus 0.004–0.006 for baselines, and shows that unsynchronized policy masks alone do not match CoachReg. TeamReg's gains are presented as task-dependent, with a failure mode in the only task with a competitive component.","pith_inferences":["The tuning-flexibility caveat means the reported margins could shrink if baselines were given the same number of free hyperparameters; a matched-tuning re-run would settle this.","The discrete mask mechanism suggests CoachReg could plausibly compose with hierarchical or options-based RL to learn temporally extended joint strategies rather than per-step switches.","A random-shared-mask control would test whether synchronization itself, rather than situation recognition by the coach, is the active ingredient in CoachReg's gains."],"forward_implications":["Sparse-reward cooperative tasks that currently require hand-designed reward shaping or curricula can be approached by adding a synchronization regularizer to an off-the-shelf CTDE algorithm.","Decentralized execution is preserved: the coach and extra prediction heads are used only during training and can be removed at deployment.","Unsynchronized policy masks are not a substitute; the paper's ablation shows mask structure alone rarely beats plain MADDPG, so the synchronous selection is the operative ingredient.","TeamReg should be restricted to fully cooperative settings; on tasks with an adversarial component it can produce a dominant agent and lower average return."],"supporting_citations":[{"why":"Supplies MADDPG, the base centralized-training decentralized-execution algorithm that both regularizers extend.","marker":"[24]"},{"why":"Provides the multi-agent particle environment used for the continuous-control evaluations and the new sparse-reward tasks.","marker":"[26]"},{"why":"Supplies the discrete-action football environment whose 3v2 result is the strongest CoachReg advantage.","marker":"[20]"},{"why":"Gives the Gumbel-softmax reparameterization that makes sampled policy masks differentiable for CoachReg.","marker":"[15]"},{"why":"Defines dropout, the mechanism through which a policy mask modulates an agent's hidden layer.","marker":"[30]"},{"why":"Underlies the single-agent DDPG baseline against which MADDPG and the regularizers are compared.","marker":"[22]"},{"why":"Provides the tabular Q-learning used in the motivation experiment that coordinated policies learn faster.","marker":"[33]"}],"fun_headline_variants":["Hidden coach syncs policies, lifting sparse-reward returns","CoachReg: policy synchrony from a training-only coach","Synchronized sub-policy selection beats MARL baselines","Coach-guided policy sync improves cooperative MARL","Train-time coach syncs policies, aiding sparse-reward tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that choosing each algorithm's best of 50 random hyperparameter settings isolates algorithmic merit, even though CoachReg and TeamReg have three extra tuning weights, and that CoachReg's gains come from mask synchronization rather than from the extra coach network itself.","fun_headline_variants_meta":{"raw":{"variants":["Hidden coach syncs policies, lifting sparse-reward returns","CoachReg: policy synchrony from a training-only coach","Synchronized sub-policy selection beats MARL baselines","Coach-guided policy sync improves cooperative MARL","Train-time coach syncs policies, aiding sparse-reward tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00196,"raw_usage":{"total_tokens":7622,"prompt_tokens":868,"completion_tokens":6754,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":6674}},"tokens_in":484,"tokens_out":6754,"duration_ms":52702,"temperature":1.0,"reasoning_tokens":6674,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:48:23.540390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check would rerun the comparison with every baseline given the same number of free tuning parameters, and would replace CoachReg's coach with a random but shared mask schedule; if the margin disappears or the random schedule matches CoachReg, the claimed mechanism is not what drives the result.","supporting_citations":[{"cited_title":"Multi- agent actor-critic for mixed cooperative-competitive environments","cited_arxiv_id":null,"evidence_quote":"Supplies MADDPG, the base centralized-training decentralized-execution algorithm that both regularizers extend."},{"cited_title":"Emergence of grounded compositional language in multi- agent populations","cited_arxiv_id":null,"evidence_quote":"Provides the multi-agent particle environment used for the continuous-control evaluations and the new sparse-reward tasks."},{"cited_title":"MADDPG + policy mask","cited_arxiv_id":null,"evidence_quote":"Provides the tabular Q-learning used in the motivation experiment that coordinated policies learn faster."}],"review_version":1}