{"id":"4f161748-18d1-473c-a05c-298533340f7d","arxiv_id":"2412.15619","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EMAI trains masking agents to randomize low-importance agents while preserving reward, and reads agent importance from the learned masking probabilities.","lead":"This paper presents EMAI, which identifies the most important agents in a trained multi-agent system by learning which agents can have their actions randomized with the least impact on the team reward. The method is tested on StarCraft, Google Research Football, and particle environments, aiming to help users understand, attack, or patch multi-agent policies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on an unvalidated mapping from masking probability to single-agent counterfactual importance; the fidelity metric is aligned with the training objective, and Eq. 9's loss appears misspecified.","rationale":"The paper addresses a real gap (per-agent, per-time-step importance in black-box MAS) and its MARL-based formulation with CTDE is a plausible architecture. The practical applications (attacks, patching) and the user study provide some independent evidence that the ranking has utility. However, the central claim—that EMAI achieves superior fidelity because its masking probability accurately reflects single-agent counterfactual importance—rests on a chain that is not validated. The training objective optimizes a joint masking policy with a sparsity bonus, so the probability is not a direct estimate of individual counterfactual impact. The reader flagged the β coupling; I agree and add that Eq. 9 appears to be an incorrect estimator of J(πθ), which makes the training objective itself questionable. Without code, hyperparameters, or a ground-truth counterfactual correlation study, the 11–118% fidelity improvements are best interpreted as showing that EMAI's policy learned its own training signal, not necessarily that it produces a valid explanation. The reader's CONDITIONAL verdict is therefore appropriate: require code release, corrected/validated loss, hyperparameter reporting, and an independent ground-truth evaluation before the claim of superior fidelity is accepted.","tokens_in":944,"tokens_out":862,"duration_ms":64563,"concrete_test":"In SMAC-8m, sample 100 held-out states; for each agent i, randomize only that agent's action (all others follow the original policy π), run 500 rollouts, and compute the mean absolute reward change. Compare this ground-truth counterfactual ranking to EMAI's 1 − P(mask) via Spearman correlation. Repeat for β ∈ {0.01, 0.1, 1} (and λ if reported). If correlation is low (<0.5) or highly β-sensitive, the readout is not a faithful importance measure. Independently re-derive Eq. 9 from Eqs. 3–5; if the sum over t is not a typo, recompute the reported RRD with a corrected loss and check whether the headline improvements persist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"EMAI's explanation is the masking probability πθ(a^m_i=1|o_i), but this probability is learned under a joint masking policy with a sparsity bonus β·Σa^m_i (Eq. 5). The readout therefore conflates (i) the marginal counterfactual effect of randomizing agent i alone, (ii) interaction effects with other simultaneously masked agents, and (iii) the pressure to mask many agents from β. The paper reports no ground-truth check of this probability against Monte Carlo single-agent action-randomization impact, and no sensitivity analysis for β. Moreover, Eq. 9 defines Ld as E[(J(π) − Σ_t γ^t(Qtot(o_t, a^m_t) − Rm(a^m_t)))^2]. Since Qtot already estimates discounted future return from time t, summing γ^t Qtot over t double-counts and is dimensionally inconsistent; if this loss is actually used, the training may not minimize |J(π) − J(πθ)|, severing the link between the readout and the paper's stated importance definition. The fidelity metric (RRD) randomizes the single most critical agent, which directly rewards the policy for what it was trained to avoid, so the high RRD may be self-confirming rather than evidence of a valid explanation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EMAI, a post-hoc explanation method for multi-agent systems that estimates the importance of each individual agent at each time step. The method defines importance via counterfactual reasoning: randomizing an important agent's action should cause a large reward drop. To avoid enumerating all agents, EMAI learns a masking policy that decides, at each step, which target agents should have their actions replaced by random actions. The masking probabilities are then interpreted as inverse importance scores. The learning problem is formulated as a decentralized POMDP and solved with a CTDE framework using a QMIX-style value decomposition. The authors evaluate EMAI on seven tasks from SMAC, GRF, and MPE, comparing against StateMask, VB, and GBA. They report that EMAI achieves higher fidelity (measured by relative reward difference, RRD) and better guidance for understanding policies, launching attacks, and patching policies.","tokens_in":14736,"tokens_out":3522,"duration_ms":33012,"significance":"Agent-level importance explanation for black-box multi-agent systems is a relevant and underexplored problem with practical implications for diagnosing policy failures and targeting interventions. Modeling the explanation search as a MARL problem is a novel and potentially scalable way to avoid brute-force counterfactual enumeration. However, the current experimental evidence is undermined by a near-tautological relationship between the training objective and the fidelity metric, an unreported sparsity hyperparameter, and a loss function that appears misspecified. If these issues are corrected and the method is validated against independent ground-truth counterfactual importance, the approach could be a solid contribution to explainable multi-agent reinforcement learning.","major_comments":[{"comment":"The fidelity metric RRD measures the reward change when the identified most critical agent is randomized. This is essentially the same quantity that EMAI's training objective (Eq. 2) minimizes: the expected reward difference before and after randomizing selected agents' actions. Because the masking probabilities directly determine which agent is selected in the fidelity test, the reported high RRD is at least partly constructed by the training objective rather than independently verified. To break this circularity, the authors should compare EMAI's importance scores against a ground-truth single-agent counterfactual estimate obtained by Monte Carlo randomization of each agent individually, and report the correlation or rank agreement.","section":"Fidelity Evaluation, Eq. (2) and Eq. (5)"},{"comment":"The sparsity weight β in Eq. (5) is never reported, and no sensitivity or ablation analysis is provided. β controls the trade-off between preserving the original reward and masking as many agents as possible, and it directly shapes the learned masking probabilities. Without a reported value or a sweep over β, the fidelity and practicability results are conditional on an unspecified hyperparameter, and the mapping from masking probability to a meaningful single-agent importance score is not validated.","section":"Approach, Eq. (5), Experiments"},{"comment":"The loss function Ld in Eq. (9) appears misspecified. It computes the squared error between the constant J(π) and the sum Σ_t γ^t (Qtot(o, a^m_t) − Rm(a^m_t)). Since Qtot already estimates the discounted future return from time t onward, summing the discounted Qtot values over all time steps double-counts rewards and is dimensionally inconsistent. If this loss is actually used in training, it is unclear that minimizing Ld corresponds to minimizing |J(π) − J(πθ)|. The authors should derive Eq. (9) carefully or replace it with a correct surrogate that provably bounds or approximates the counterfactual reward difference.","section":"Approach, Eq. (9)"}],"minor_comments":[{"comment":"There are typos: 'provideed' should be 'provided' and 'demonstratee' should be 'demonstrate'.","section":"Abstract"},{"comment":"The figure reports only means and standard deviations; for several tasks the error bars overlap (e.g., SMAC-1c3s5z). Please provide statistical significance tests, such as paired bootstrap confidence intervals, to support the claim of consistent improvements.","section":"Experiments, Figure 2"},{"comment":"The hyperparameters λ (in Eq. 10) and β (in Eq. 5) are not listed. Please include their values and the training details (e.g., learning rate, network architecture) for reproducibility.","section":"Experiments, Experimental Setup"},{"comment":"The user study reports that 75% of participants preferred EMAI and 58% found it helpful, but no confidence intervals, inter-rater agreement, or details of the questionnaire are provided. Please elaborate on the study design.","section":"Understanding Policies"},{"comment":"The notation ω(Q_i(o_i, a_i^m; θ)) is vague; clarify whether ω is a sum, a monotonic neural network, or another aggregation function.","section":"Approach, Eq. (7)"},{"comment":"The similarity threshold d_th is not specified in the text or tables. Please include its value or describe how it is selected.","section":"Patching Policies"},{"comment":"Since Shapley value is discussed in related work, consider adding a Shapley-based agent attribution baseline or explicitly justify why it is not applicable in this black-box, per-time-step setting.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The core idea of learning a masking policy for agent-level explanation is interesting, but the evaluation is too closely aligned with the training objective to support the paper's central claim. The issues with Eq. (9) and the unreported β are concrete and fixable, but they require substantive work rather than minor edits. If the authors can provide an independent validation of the importance scores and correct the loss derivation, the paper could become a solid contribution to the explainable RL literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. EMAI takes on a real gap—per-agent, per-time-step importance for black-box multi-agent policies—and the idea of training masking agents as a MARL problem is a legitimate, non-obvious way to approach it. That part is worth reading. But the headline fidelity numbers are weaker than they look: the RRD metric measures almost exactly what the training objective minimizes, and the paper never reports the sparsity weight β that shapes the importance readout, never validates the readout against a true single-agent counterfactual, and ships no code.\n\nWhat's new: previous post-hoc RL explanation work largely targets features or time steps, not agents. EMAI frames the agent-level target, uses CTDE to train a joint masking policy over which agents to randomize, and defines importance as masking probability. The black-box assumption is honest. The experiments span three benchmarks and seven tasks, and the applications—policy understanding, attacks, patching—are a useful extension beyond a single fidelity number. The user study is small but a good instinct. The citation pattern looks fine; the relevant post-hoc explanation and credit-assignment literature is covered.\n\nNow the soft spots, in order of size. First, the fidelity metric RRD randomizes the single most important agent selected by the explainer. That is basically the quantity Eq. 2 tells the masking agent to be good at. So the 11–118% improvement over baselines is partly a measurement of how well EMAI optimizes its own objective, not of whether the explanation is correct in any independent sense. Second, the importance readout is the masking probability learned under Eq. 5 with a sparsity bonus β. The value of β is never reported or swept. If β is large, the policy masks many agents and the probabilities stop being a clean ranking. No ground-truth check by Monte Carlo single-agent randomization appears anywhere. Third, Eq. 9 as written sums γ^t Qtot over all t; since Qtot already estimates discounted return, this double-counts and is dimensionally inconsistent. If that loss is used literally, the link between the training objective and the stated importance definition breaks. This needs clarification or a fix. Smaller issues: λ and d_th are unreported, no code release, and the user study lacks statistical testing.\n\nOverall: the core idea is plausible and the paper is a serious first attempt at this explanation target. But the evidence as presented does not yet establish that masking probability equals counterfactual importance. Send it to serious review. Ask for code, hyperparameters, sensitivity on β and λ, a correction or clarification of Eq. 9, and an independent fidelity check.","headline":"A real gap and a plausible MARL-based method, but the evaluation is partly self-confirming and the key hyperparameters aren't reported.","tokens_in":15307,"tokens_out":3129,"would_cite":true,"duration_ms":30623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that agent importance in a black-box multi-agent system can be learned as a masking probability, with less masking meaning more importance.","keywords":["agent-level explanation","counterfactual reasoning","multi-agent reinforcement learning","masking policy","explainable reinforcement learning","agent importance","fidelity evaluation","centralized training with decentralized execution"],"falsifier":"Take a small benchmark, fix a set of states, and for each state randomize each agent's action alone many times to measure the true average reward drop, then compare that ranking with EMAI's masking probabilities. If the ranking changes when the sparsity weight $\\beta$ is varied or when agents are masked in groups rather than singly, the masking probability is not a stable measure of individual counterfactual importance.","tokens_in":14268,"feed_emoji":"🤖","tokens_out":7757,"duration_ms":63194,"temperature":0.7,"pith_summary":"The paper tries to establish that in a trained multi-agent team, each agent's importance at a given state can be read out as a learned masking probability: the less likely a learned masking policy is to randomize an agent's action, the more that agent matters to the team's reward. It frames this as a multi-agent reinforcement-learning problem, training masking agents whose only job is to preserve the original expected reward while randomizing as many target agents as possible. Across seven tasks from well-known benchmarks, the resulting explanations beat three baselines on fidelity, with relative improvements from 11% to 118%, and outperform them when used to guide policy understanding, adversarial attacks, and policy patching. If the claim holds, black-box multi-agent policies become inspectable at the level of individual agents and time-steps, which would make targeted interventions such as finding lazy agents, attacking critical units, or patching weak links more efficient.","feed_headline":"Counterfactual masks rank which agents matter most","feed_subtitle":"A reward-preserving masking policy outscores baseline explanations on seven multi-agent tasks and guides attacks and patches.","key_machinery":"The central object is the masking policy $\\pi_\\theta$: a set of $n$ agents whose action space is $\\{\\mathrm{keep}, \\mathrm{randomize}\\}$, one for each target agent. The load-bearing inversion is the readout of importance from masking probability, because the policy is trained to make randomization impact reward as little as possible, it randomizes only agents whose random actions are cheap, so the agents left with low masking probability are the important ones. The policy is trained as a DEC-POMDP with the reward-preservation objective $|J(\\pi)-J(\\pi_\\theta)|$, a sparsity bonus $\\beta$ times the number of masked agents, and a one-step temporal-difference loss in which a central critic $Q^{tot}(o,a^m)$, built with non-negative weights to satisfy the Individual-Global-Max principle, estimates the joint value of masking actions.","core_discovery":"The paper claims that the counterfactual quantity \"expected reward lost when an agent's action is randomized\" can be characterized by a learned policy instead of by enumerating random perturbations. EMAI trains $n$ masking agents, one per target agent, that output a binary action at each time-step: keep the target agent's action or replace it with a random action. The training objective is to minimize the absolute difference between the original expected reward $J(\\pi)$ and the expected reward under masking $J(\\pi_\\theta)$, plus a sparsity reward $\\beta \\sum_i a^m_{t,i}$ that encourages masking more agents; the masking probability assigned to each agent is then reported as the inverse importance. Because joint masking decisions interact across agents and across time, the masking policies are learned with centralized training and decentralized execution, using a central critic $Q^{tot}(o,a^m)$ whose value is decomposed into per-agent values. In experiments on seven tasks spanning StarCraft battles, football, and particle navigation, the method records the highest relative reward difference (RRD) among all compared explanations, with improvements of 11% to 118%; in downstream tests, attacks guided by its importance rankings reduce team rewards more, and patches guided by them improve team rewards more, than baseline-guided interventions.","pith_inferences":["If the masking-probability readout tracks true single-agent counterfactual impact beyond the tested tasks, an untested corollary is that importance rankings could be assembled into a surrogate for Shapley-style attribution without enumerating exponentially many coalitions.","The same learned counterfactual signal could be used during training, not only after it, as a curriculum signal that randomizes or overprotects high-importance agents to stress-test coordination.","The reported attack improvements cut both ways: the rankings that help defenders patch policies could also be mined by adversaries, so deployment of such explanations should be paired with robustness analysis of the underlying policy.","A direct testable extension would be to compare EMAI's rankings with brute-force single-agent randomization on a small environment, and if the match is close the method could serve as a scalable estimator of causal importance in larger systems."],"forward_implications":["Users can now ask which agent is critical at any state of a black-box multi-agent system, not just which observation region or time-step matters.","Attackers can concentrate perturbations on a few high-importance agents, achieving larger reward reductions than attacks guided by existing explanations.","Defenders can patch policies by replacing the actions of critical agents with recorded high-reward actions, yielding consistent reward gains across all seven tested tasks.","The same importance ranking exposes low-contributing agents, pointing to targeted retraining or credit-assignment changes.","The approach is black-box compatible: it needs only observations and actions of the target agents, not their parameters or value functions."],"supporting_citations":[{"why":"Supplies the state-mask baseline, the Monte Carlo estimate for J(pi), and the fidelity-comparison convention that EMAI extends from states to agents.","marker":"(Cheng et al. 2023)"},{"why":"Provides the counterfactual principle that perturbing the most influential elements changes the outcome most, which motivates the importance definition.","marker":"(Goyal et al. 2019)"},{"why":"Contributes the monotonic non-negative value decomposition that lets EMAI train decentralized masking policies with a centralized critic.","marker":"(Rashid et al. 2018)"},{"why":"Relates counterfactual reasoning to credit assignment in multi-agent reinforcement learning, the conceptual bridge the paper adapts to post-hoc explanation.","marker":"(Foerster et al. 2018)"},{"why":"Defines the RRD fidelity metric used to compare explanations across environments.","marker":"(Yu et al. 2023)"},{"why":"Provides the StarCraft Multi-Agent Challenge tasks used in the fidelity, attack, and patch experiments.","marker":"(Samvelyan et al. 2019)"},{"why":"Provides the Google Research Football tasks used to test explanations under sparse, adversarial rewards.","marker":"(Kurach et al. 2020)"},{"why":"Provides the Multi-Agent Particle Environments used to test explanations in dense-reward cooperative tasks.","marker":"(Lowe et al. 2017)"},{"why":"Motivates the practical attack-and-patch evaluation by showing how explanations can guide security interventions in reinforcement learning.","marker":"(Guo et al. 2021)"}],"fun_headline_variants":["Counterfactual masks learn agent importance for team strategy","Masking agents' actions reveals their true value","Reward-preserving masks rank agent importance better","Counterfactual agent importance guides attacks and patches","Learned masking explains agent importance in MAS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the probability a trained masking policy assigns to randomizing an agent's action faithfully ranks how much that agent's random action actually lowers the team's expected reward, even though the ranking is trained with a hand-set sparsity weight $\\beta$ that encourages masking many agents.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual masks learn agent importance for team strategy","Masking agents' actions reveals their true value","Reward-preserving masks rank agent importance better","Counterfactual agent importance guides attacks and patches","Learned masking explains agent importance in MAS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1585,"prompt_tokens":982,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":598,"tokens_out":603,"duration_ms":5832,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:15:27.013470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small benchmark, fix a set of states, and for each state randomize each agent's action alone many times to measure the true average reward drop, then compare that ranking with EMAI's masking probabilities. If the ranking changes when the sparsity weight $\\beta$ is varied or when agents are masked in groups rather than singly, the masking probability is not a stable measure of individual counterfactual importance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the state-mask baseline, the Monte Carlo estimate for J(pi), and the fidelity-comparison convention that EMAI extends from states to agents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the counterfactual principle that perturbing the most influential elements changes the outcome most, which motivates the importance definition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the monotonic non-negative value decomposition that lets EMAI train decentralized masking policies with a centralized critic."},{"cited_title":"N.; Farquhar, G.; Afouras, T.; Nardelli, N.; and Whiteson, S","cited_arxiv_id":null,"evidence_quote":"Relates counterfactual reasoning to credit assignment in multi-agent reinforcement learning, the conceptual bridge the paper adapts to post-hoc explanation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the StarCraft Multi-Agent Challenge tasks used in the fidelity, attack, and patch experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Google Research Football tasks used to test explanations under sparse, adversarial rewards."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates the practical attack-and-patch evaluation by showing how explanations can guide security interventions in reinforcement learning."}],"review_version":1}