{"id":"fc828c9f-3c5f-4878-9e5b-34e1759f6ef3","arxiv_id":"2608.12921","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A post-hoc explainer masks edges of LLM multi-agent communication graphs, measures which removals preserve task outcomes, and learns a fast predictor that prunes redundant links at deployment.","lead":"This paper presents E2-Explainer, a method that identifies compact subgraphs of LLM multi-agent communication topologies by testing which edges, when removed, hurt task performance. These subgraphs can be executed directly, cutting token usage by roughly 20 to 25 percent while keeping accuracy about the same.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Budget E25+N20 is selected after a full sweep scored on the test set (Table 7), so the headline accuracy-maintenance result may reflect selection on test noise.","rationale":"The reader's stated weakest assumption, single-edge removals versus multi-edge coalitions, is real but secondary: the paper evaluates the predicted subgraph end-to-end, so even a suboptimal causal attribution can still yield a working subgraph. The more load-bearing problem is that the main quantitative headline is selected after the evaluation set is observed. Table 7 uses the same evaluation queries as Table 1, and the appendix explicitly says the sweep is used to select the operating point; no validation split is described for this selection. The high sensitivity across neighboring budgets indicates that the reported accuracy gain is not robust. This reinforces, rather than overturns, the CONDITIONAL verdict: the method is plausible and the transfer results are suggestive, but the main accuracy-maintenance claim requires a frozen-budget confirmation. The reader did mention budget selection on test data in the rationale, although the formal weakest_assumption field names a different mechanism, hence 'partial' agreement.","tokens_in":22056,"tokens_out":7463,"duration_ms":86179,"concrete_test":"Freeze the operating budget before any test-set evaluation: either pre-register E25+N20 or select it by running the same budget sweep on the 40 calibration queries per dataset only, then keep that choice fixed. Rerun Table 1 for all four topology optimizers with the budget frozen and test data untouched until final evaluation, and report both the selected budget and the resulting average accuracy and weighted token delta. If the average accuracy gain over the four generators is no longer positive, or if the token reduction drops materially below 20%, the central pruning claim is partly an artifact of test-set budget selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The appendix 'Additional Experimental Results' reports Table 7, a sweep over six budgets on the same evaluation queries used in Table 1 (MMLU 153, GSM8K 1,319, MultiArith 600, SVAMP 1,000, AQuA 254, HumanEval 124). The text says the sweep is used to 'select one common operating point,' and E25+N20 is then fixed for all four topology optimizers. No separate validation split is used for this choice; the 40 calibration queries per dataset are used only to train the explainer. Thus the main 'preserves or improves accuracy' claim is selected after the test-set outcomes are observed. Adjacent budgets behave very differently: E50+N20 changes average accuracy by -0.62 and E25+N40 by -0.91, while the reported E25+N20 gives +1.09. The token reductions are consistent across budgets, so the cost-reduction part of the claim is less affected; what is at risk is the accuracy-maintenance part. This is a more immediate threat to the central claim than the multi-edge-coalition limitation, which the appendix openly acknowledges and which the end-to-end evaluation partially addresses. The paper provides no evidence that E25+N20 would have been chosen without peeking at test accuracy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes E2-Explainer, a post-hoc framework for identifying compact communication subgraphs in LLM-based multi-agent systems. The method estimates per-edge preservation utilities by single-edge masking interventions, combining task-score changes with semantic entropy of the final response (Eqs. (3)-(7)), selects budgeted subgraphs via TopK and an executability-constrained projection (Eqs. (9)-(10)), and distills the resulting subgraphs into an amortized explainer for test-time one-shot prediction. Experiments on six benchmarks with four topology optimizers report weighted token reductions of 20.1%-25.6% with average accuracy changes of +0.44 to +1.09 points, as well as cross-generator, cross-scale, and hand-crafted-topology transfer experiments.","tokens_in":22326,"tokens_out":6973,"duration_ms":67770,"significance":"If the central claims hold, the paper makes a practical contribution: a model-agnostic, amortized post-hoc method that can reduce communication cost in LLM-MAS without requiring generator internals or repeated interventions at deployment. The paper is unusually transparent about its protocol: it provides pseudocode (Algorithms 1-2), explicit token accounting, detailed qualitative trace studies that openly note the independent-execution caveat, and an appendix with budget sweeps and transfer tests. These strengths make the work falsifiable and reproducible in principle. However, the headline accuracy-preservation claim currently rests on two load-bearing supports that need reinforcement: the default budget is selected after observing evaluation-set outcomes, and the causal attribution is built from first-order single-edge effects with no multi-edge validation. The significance of the contribution is therefore real but not yet fully established.","major_comments":[{"comment":"The default operating point E25+N20 is selected by a budget sweep whose reported accuracies are computed on the same benchmark evaluation queries used in Table 1 (MMLU 153, GSM8K 1,319, MultiArith 600, SVAMP 1,000, AQuA 254, HumanEval 124), and no separate validation split is mentioned for this choice. Table 7 shows that neighboring budgets behave very differently: E50+N20 changes average accuracy by -0.62 points and E25+N40 by -0.91 points, whereas the selected E25+N20 reports +1.09 points. Because the headline 'preserves or improves accuracy' claim is chosen after observing these test-set outcomes, it may reflect selection on test noise; the token-reduction part of the claim is less affected because token reductions are consistent across budgets. The authors should either select the budget on a held-out split or present all sweep points as primary outcomes with appropriate multiple-comparison awareness.","section":"Experiment; Table 7; 'Additional Experimental Results'"},{"comment":"The subgraph supervision is constructed from single-edge removal interventions, and Eq. (9) selects edges by their individual utilities; the appendix acknowledges that multi-edge coalitions are not enumerated. This first-order assumption is load-bearing for the causal-pruning claim: if edges are jointly redundant or jointly important, the TopK selection can keep a mutually redundant set and drop a combination that matters. The end-to-end results partially address this, but no experiment compares the first-order TopK subgraphs against a multi-edge ablation, a greedy backward-elimination baseline, or random subgraphs of the same budget. Adding such a comparison, even at small scale, would substantiate the claim that the identified subgraphs are 'critical' rather than merely first-order plausible.","section":"Eqs. (7)-(9); Appendix 'Scope of the causal attribution'"},{"comment":"No error bars or confidence intervals are reported for any accuracy or token delta. Several headline differences are small relative to the evaluation sizes: for example, a 1.09-point improvement on MMLU corresponds to roughly two questions out of 153, and the ARG-Designer result on GSM8K drops by 2.24 points. With only three runs and no variance information, it is not possible to tell whether the accuracy-maintenance claim is statistically distinguishable from noise. The authors should report per-dataset variance (standard deviations or bootstrap intervals) for both accuracy and token usage.","section":"Table 1; Appendix 'Implementation Settings' (reported evaluation runs=3)"}],"minor_comments":[{"comment":"The notation 'e·' for normalization in Eq. (7) is not defined in the main text; the appendix reveals that the task term is left unnormalized while the semantic term is divided by log M, which should be stated at first use for reproducibility.","section":"Eq. (7) and Appendix 'Score normalization'"},{"comment":"The sentence 'We train a single explainer using only G-Designer-generated candidate graphs' is in tension with the appendix statement 'We train one dataset-specific explainer for each benchmark'; please clarify whether 'single' means one per dataset.","section":"Experiment, 'Candidate generators and calibration'"},{"comment":"The computation of 'weighted' token reduction is not specified; state the weighting scheme (for example, weighting by total tokens per dataset) explicitly in the main text.","section":"Tables 1 and 7"},{"comment":"The notation (Q_e_G, Y_G,e) for the original-graph evaluation is confusing and appears to attach an edge index e to a graph-level score; rename to something like (Q(x,G), Y_G).","section":"Algorithm 1, line 5"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper before citing it for 'LLM-MAS topologies can be pruned with little accuracy loss': the token-reduction numbers are probably real, but the accuracy-maintenance headline rests on a budget chosen after seeing the test results.\n\nThe genuinely new piece is the integration: a Granger-style edge-attribution that masks one communication edge at a time, measures task-score and semantic-entropy changes, ranks edges, then distills the resulting subgraphs into an amortized explainer that predicts subgraphs for unseen topologies in one forward pass. That is a reasonable and useful design. The cross-generator and cross-scale transfer results (a G-Designer-trained explainer applied to ARG-Designer, OFA-MAS, AgentPrune, and 6-agent graphs) are a real plus, and the qualitative execution traces are unusually candid, including cases where pruning turns a correct answer into a wrong one. Those traces alone make the paper worth reading.\n\nThe soft spots are real, though. The most important one is the budget sweep. Table 7 sweeps six budgets on the same evaluation queries used in Table 1, and E25+N20 is then fixed for all experiments. That is test-set selection. Neighboring budgets give much worse average accuracy (-0.62 for E50+N20, -0.91 for E25+N40), so the +1.09 point gain at E25+N20 could be a selection artifact. There are also no error bars anywhere in the main tables; the token reductions are consistent across budgets, so those are more credible, but the accuracy deltas are small enough that noise is a real concern.\n\nThe causal attribution is also first-order: single-edge masks only, with the appendix explicitly noting that multi-edge coalitions are not enumerated. That means the derived subgraphs are not guaranteed to be near-optimal, and the end-to-end evaluation does not test this directly against a simple random-pruning baseline at the same budget. A comparison to random or degree-based pruning at matched token budgets would have strengthened the paper a lot. The authors also do not release code or data, which limits reproducibility.\n\nNet: the paper is a serious, honest attempt at a useful problem, and the transfer results plus qualitative cases are valuable. But the central 'maintains accuracy' claim needs a validation-set budget selection, error bars, and a stronger baseline before I would trust it. The cost-reduction finding is more robust.\n\nI would send this to a serious referee, not desk reject. It deserves peer review because the idea is new enough and the empirical scaffold is substantial, even though the main claim needs tightening. I would not cite the accuracy-maintenance result in its current form, but I would cite the method and the transfer findings if code becomes available.\n\nRecommendation: engage with it, but ask for a resubmission with fixed budget selection on a held-out split, error bars, a matched-budget baseline, and code release.","headline":"Solid engineering contribution to LLM-MAS pruning, but the accuracy-maintenance claim is undercut by test-set budget selection and missing error bars.","tokens_in":22838,"tokens_out":3622,"would_cite":false,"duration_ms":33659,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating edge importance as its causal contribution to the task outcome reveals compact subgraphs that preserve multi-agent LLM performance while cutting token use by roughly a quarter.","keywords":["LLM-based multi-agent systems","communication topology","causal attribution","Granger causality","graph pruning","post-hoc explainability","amortized explainer","semantic entropy"],"falsifier":"Run E2-Explainer on a small five- or six-agent graph, then enumerate every edge subset of the same budget and compare task accuracy with the predicted subgraph; if some budget-sized subset preserves accuracy while all its edges have zero or negative single-edge utility, the first-order Granger ranking has missed a joint effect. Because the paper's appendix explicitly does not enumerate multi-edge coalitions, this check would settle whether the pruning claim is complete.","tokens_in":21866,"feed_emoji":"✂️","tokens_out":8818,"duration_ms":82131,"temperature":0.7,"pith_summary":"This paper tries to establish that communication graphs produced by LLM-based multi-agent systems contain a compact 'causal core' of edges that actually carries the collaboration, and that this core can be recovered after the fact even when the topology generator is a black box. The authors model edge importance with a Granger-style criterion: an edge is important if blocking it changes the task score or destabilizes the final response. They then train a small explainer to predict, for a new graph and a pruning budget, a subgraph built from those per-edge contributions. In experiments, executing the chosen subgraphs cuts weighted token usage by 20.1 to 25.6 percent across four topology optimizers while average accuracy rises by 0.44 to 1.09 points. If right, the method gives a general way to lower communication costs of existing multi-agent systems without retraining agents or topology generators.","feed_headline":"Causal pruning cuts multi-agent LLM token costs by a quarter","feed_subtitle":"Removing causally unimportant communication edges preserves or improves accuracy across four topology optimizers.","key_machinery":"The load-bearing object is the edge-level Granger-style preservation utility, $u_e = \\mathrm{clip}_{[0,1]}(\\alpha\\,\\tilde{u}^{\\mathrm{task}}_e + \\beta\\,\\tilde{u}^{\\mathrm{sem}}_e)$. Here $\\tilde{u}^{\\mathrm{task}}_e = \\max(0, Q(x,G)-Q(x,G_{-e}))$ is the nonnegative change in task score from blocking edge $e$'s message, and $\\tilde{u}^{\\mathrm{sem}}_e$ is the nonnegative increase in semantic entropy — the uncertainty of the set of final responses after grouping semantically equivalent answers — caused by the same intervention. These utilities order all edges; the budget converts to retention counts, TopK selects the highest-utility edges, and the validity projection $\\mathcal{R}_{\\mathrm{valid}}$ turns the selection into an executable subgraph that never adds new links and preserves acyclicity. An amortized explainer $F_\\theta(x,G,b)$ is trained with binary cross-entropy on the edge and node membership of these subgraphs, so at deployment it outputs a budget-specific subgraph in one forward pass with no edge-masking evaluations.","core_discovery":"The paper's central claim is that optimized communication topologies can be explained by attributing each edge's contribution to task preservation, and that the resulting compact subgraphs are not just explanations but executable efficiency gains. For each edge $e$, the authors block the message along $e$ while holding everything else fixed, and measure $\\Delta^{\\mathrm{task}}_e = Q(x,G)-Q(x,G_{-e})$, keeping nonnegative values; the auxiliary signal is the increase in semantic entropy $\\bar{H}$ of the final response across five stochastic executions. The combined utilities rank all edges, a budgeted TopK selection plus validity projection yields the explanation subgraph, and an amortized explainer $F_\\theta(x,G,b)$ learns to predict that subgraph for unseen topologies without repeated interventions. Across six benchmarks and four topology optimizers, the paper reports that this procedure reduces weighted online tokens by 20.1–25.6% and improves average accuracy by 0.44–1.09 points, with transfer to unseen generators, larger agent counts, and hand-crafted topologies.","pith_inferences":["Beyond the paper: because the method ranks edges by single-edge interventions, the pruning claim would be on firmer ground if checked against multi-edge ablations; the paper's own appendix notes that multi-edge coalitions are not enumerated.","Beyond the paper: the consistent accuracy gains, especially on AQuA and HumanEval, suggest that some optimized topologies contain actively distracting edges, so causal pruning acts partly as denoising; a testable extension is to see whether this denoising effect grows with backbone strength or agent count.","Beyond the paper: if the single-edge ranking is reliable, the same amortized explainer could be used as a per-query pre-execution gate that selects a subgraph before any agent message is sent, turning the reported token savings into a deployment-time latency optimization; the paper evaluates the final subgraph but does not study this online selection regime."],"forward_implications":["Executing the predicted subgraphs on G-Designer, AgentPrune, OFA-MAS, and ARG-Designer graphs cuts weighted token use by 20.1%–25.6% while average accuracy rises by 0.44–1.09 points.","An explainer trained only on G-Designer graphs transfers without retraining to graphs from other generators and to hand-crafted topologies, so the learned edge-relevance cues are not generator-specific adjacency memorization.","The causal task signal is the primary component: removing it drops MMLU accuracy from 79.74 to 75.82 and HumanEval from 90.69 to 86.88, while the semantic-entropy signal alone is weaker but complementary.","The default budget E25+N20 improves average accuracy on five of six benchmarks; pushing node pruning to 40% cuts 42.63% of tokens but drops overall accuracy, mainly from a 6.52-point fall on HumanEval."],"supporting_citations":[{"why":"Supplies the causal criterion the paper adapts: a variable is causally informative if access to it improves prediction of another.","marker":"Granger 1969"},{"why":"Cited as the origin of the Wiener–Granger principle, the conceptual basis for edge-level causal contribution.","marker":"Wiener 1956"},{"why":"GNNExplainer motivates identifying compact subgraphs that preserve graph-model behavior; E2 transfers this idea from differentiable GNNs to frozen LLM-MAS communication graphs.","marker":"Ying et al. 2019"},{"why":"PGExplainer supplies the amortized graph-explainer paradigm that E2-Explainer's distillation step builds on.","marker":"Luo et al. 2020"},{"why":"Semantic entropy is the auxiliary signal used in Eq. (5) for response stability under sparse or coarse task rewards.","marker":"Kuhn, Gal, and Farquhar 2023"},{"why":"Validates semantic entropy as an LLM uncertainty measure, supporting its use as a preservation signal when task scores are identical across edge masks.","marker":"Farquhar et al. 2024"},{"why":"G-Designer is the primary topology generator used for calibration; its graphs are the source of the Figure 1 random-masking observation that motivates the existence of compact task-preserving subgraphs.","marker":"Zhang et al. 2025b"}],"fun_headline_variants":["Explainer prunes LLM agent links, saving 25% tokens","Causal edge pruning trims LLM team chatter, lifts accuracy","E2-Explainer: explainable pruning cuts agent tokens 20-25%","Critical subgraph discovery slashes multi-agent LLM costs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim assumes that each edge's importance is fully captured by removing it one at a time with all other edges present, so edges that matter only in combination can be misranked by the method.","fun_headline_variants_meta":{"raw":{"variants":["Explainer prunes LLM agent links, saving 25% tokens","Causal edge pruning trims LLM team chatter, lifts accuracy","E2-Explainer: explainable pruning cuts agent tokens 20-25%","Critical subgraph discovery slashes multi-agent LLM costs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1411,"prompt_tokens":984,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":349}},"tokens_in":600,"tokens_out":427,"duration_ms":4546,"temperature":1.0,"reasoning_tokens":349,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:32:11.560089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run E2-Explainer on a small five- or six-agent graph, then enumerate every edge subset of the same budget and compare task accuracy with the predicted subgraph; if some budget-sized subset preserves accuracy while all its edges have zero or negative single-edge utility, the first-order Granger ranking has missed a joint effect. Because the paper's appendix explicitly does not enumerate multi-edge coalitions, this check would settle whether the pruning claim is complete.","supporting_citations":[],"review_version":1}