{"id":"746a0896-151c-4c1b-b42e-a5c64adc7405","arxiv_id":"2602.04234","paper_version":6,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across six benchmarks and five topologies, early-round token entropy predicts which LLM agent team will answer correctly, and an entropy-based selector improves pass@k accuracy.","lead":"Tracing the token-by-token uncertainty of open-weights LLM agents, the paper finds that a single agent beats multi-agent teams in about 43% of the configurations tested. It argues that the first round of interaction largely decides success, and uses that early uncertainty signal to select better answers from multiple candidates.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal arrow untested: SHAP correlations are read as 'harms/benefits,' but the paper's own Appendix H (83.4% possible anchoring) and Appendix J (causal identification deferred) leave the antecedent-entropy premise unsupported.","rationale":"The reader's weakest assumption—entropy as causally antecedent rather than epiphenomenal—is exactly the load-bearing point. The paper's own Appendix H and J concede that causal identification is incomplete, and the 83.4% possible-anchoring rate directly undermines the stronger reading of the entropy→accuracy link. My stress test does not identify a different or additional concern that would change the verdict: the descriptive and predictive contributions remain credible, but the central causal claim requires either softened language or a within-problem intervention/design. Since the reader already returned CONDITIONAL, I see no reason to move the verdict; the proposed matched-decode test would either validate or retire the causal framing.","tokens_in":54513,"tokens_out":4554,"duration_ms":56278,"concrete_test":"Re-analyze the existing traces with a within-problem matched design. For each problem that has multiple trajectories in a (model, architecture, dataset) cell, split trajectories into quartiles by round-1 peak entropy and compute the accuracy gap between top and bottom quartiles, stratifying by problem and controlling for answer token count, total tokens, and base-model correctness. Better: generate 20 repeated decodes per problem under the paper's sampling settings and estimate a logistic mixed-effects model with random intercepts for problem; test whether the round-1 peak-entropy coefficient is significantly negative. If the within-problem slope is not robustly negative, the headline 'peak entropy universally harmful' is an artifact of problem difficulty or sampling quality rather than a causal or robust conditional effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'MAS effectiveness is largely determined by early-round uncertainty dynamics' and 'peak entropy universally harmful across architectures'—requires that entropy be causally antecedent to correctness. The paper's evidence is correlational: an XGBoost/LightGBM ensemble is trained on entropy features to predict per-sample correctness, and SHAP signs are interpreted as 'harms' and 'benefits' (§4.3–4.4). This inference is not secured. Appendix H.4 reports that 83.4% of round-1→2 transitions show entropy decrease without accuracy gain ('possible anchoring'), with only 6.2% 'genuine improvement'; Appendix J states that 'strict causal mechanisms... remain inherently challenging' and defers causal identification to future work. Because token entropy is measured from the same trajectories that produce the answer, high round-1 peak entropy could simply mark hard problems or low-quality sampling rather than cause failure. The controlled SAS-vs-MAS decomposition in Appendix H isolates role and interaction effects on entropy, but it does not test the effect of entropy on correctness; it explicitly leaves the entropy→accuracy direction open. Without a within-problem comparison—for example, high- versus low-entropy decodes of the same question—the load-bearing premise that entropy comes first is untested. The descriptive findings (SAS competitiveness, first-round dominance of predictive features, Entropy Judger cross-validation accuracy) can survive this concern, but the abstract's causal language and the 'universally harmful' framing cannot as currently stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies entropy dynamics in LLM-based multi-agent systems (MAS) built on five open-weight models across six benchmarks and four topologies. It extracts 245 hierarchical entropy features, trains an XGBoost/LightGBM ensemble (the Entropy Judger) to predict per-sample correctness, and uses SHAP analysis to derive three observations: certainty preference, base-model entropy as a limiting factor, and task/architecture-dependent entropy roles. It reports that a single agent achieves the best accuracy in 43.3% of cases, that first-round entropy dynamics dominate prediction, and that entropy-based pass@k selection improves accuracy. The appendices include extensive robustness checks (temperature, model scale, tool-calling tasks, RL-trained base models) and a causal-analysis appendix that explicitly acknowledges unresolved causal identification and reports an 83.4% 'possible anchoring' rate in round 1→2 transitions.","tokens_in":54864,"tokens_out":3550,"duration_ms":39386,"significance":"If the descriptive claims hold, the paper provides a useful, reproducible diagnostic for when MAS collaboration pays off and when it does not, based on openly available models and a large dataset (44,780 samples). Strengths include the favorable sample-to-feature ratio, stratified 5-fold CV, ablation studies, cross-method importance validation, and explicit checks at 14B scale, different temperatures, and a tool-calling agentic benchmark. The authors are also unusually honest in Appendices H and J about the limits of causal inference. However, the headline contributions are framed causally ('peak entropy directly harms', 'base entropy causally drives MAS performance') while the evidence is correlational, and a claimed algorithmic benefit (pass@k selection) is not quantified in the manuscript. The central descriptive findings are defensible, but the causal and universal claims need substantial reframing or additional evidence.","major_comments":[{"comment":"The abstract and Section 4.4 state that peak entropy 'directly harms' correctness and that lower base-model entropy 'causally drive[s]' MAS performance. The evidence is SHAP-based correlation from a trained classifier, not causal identification. Appendix J itself says 'strict causal mechanisms... remain inherently challenging' and defers causal identification to future work. Appendix H.4 reports 83.4% of round-1→2 entropy-decrease transitions without accuracy gain ('possible anchoring'). The causal language must be removed or substantiated with within-problem interventions (e.g., same question decoded under high- vs low-entropy traces) or explicit counterfactual manipulations.","section":"Abstract, §4.3–4.4, Appendix J"},{"comment":"The Entropy Judger's central practical contribution is pass@k selection, and the main text claims it 'consistently improves accuracy across all MAS configurations and tasks.' However, no quantitative result for this selection is reported anywhere in the main text or the appendix. Appendix F.5 only describes the protocol (choose argmax predicted correctness) and mentions baselines (Random, Oracle) without giving numbers. The claimed accuracy improvement is therefore unsupported as written. Please add a table/figure reporting pass@k selection accuracy against random and oracle baselines across configurations.","section":"§5.5, Appendix F.5"},{"comment":"The claim that 'peak entropy is universally harmful across architectures' is overstated. Figure 4(d) shows an across-architecture reversal: the single-agent architecture has the highest values of round-1 peak/entropy-dispersion features yet the highest accuracy, while sequential (lowest accuracy) has the lowest feature averages. The text acknowledges that the entropy–performance relationship depends on 'architectural capacity to control uncertainty,' which is a conditional rather than universal claim. The headline 'universally harmful' should be replaced with a qualified statement such as 'harmful within each architecture after controlling for architectural capacity', or the figure should be reconciled explicitly with the universal claim.","section":"§5.2, Figure 4(d)"},{"comment":"The 'Base Uncertainty' observation claims that lower base-model entropy 'directly benefits' MAS performance. The Gbase-H analysis measures base entropy from the same base model trajectory that also produces the base correct/incorrect label; base entropy is likely a proxy for base capability. The authors do include base correctness in Gbase-full, but the causal claim in §4.4 is based on Gbase-H alone. A partial-correlation analysis controlling for base correctness, or an intervention (e.g., temperature/contrastive decoding on the base model), is needed before 'directly benefits' can be defended. Otherwise the claim should be phrased as 'base-model entropy is a strong predictor correlated with MAS accuracy'.","section":"§4.4, Figure 2"}],"minor_comments":[{"comment":"The abstract uses 'causally drive' while the contribution list in Section 1 says 'strongly correlate with correctness.' This inconsistency should be resolved in favor of the more accurate correlational framing throughout.","section":"Abstract vs. §4.4"},{"comment":"'The basedenotes...' is a typo; also the caption does not mention that debate is excluded on HumanEval (Appendix B.1), which affects the interpretation of '30 scenarios' in §4.4.","section":"Figure 1 caption"},{"comment":"The text reports '254 features' and then '245 trainable features' after excluding 9 experimental identifiers. Clarify explicitly which 9 are excluded and why they are not counted among the 245 used in Table 1.","section":"§4.2"},{"comment":"Global average ECE is 0.391, with LLaMA models above 0.5. This substantial miscalibration means low entropy does not reliably indicate correctness for these models; this should be mentioned in the main text wherever 'low entropy is beneficial' is claimed, not only in the appendix.","section":"Appendix G"},{"comment":"The phrase 'single agents channel higher entropy into genuine exploration' is a mechanistic interpretation not directly supported by the data; suggest softening to 'single agents can tolerate higher entropy without accuracy loss' or provide direct evidence.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically solid for its core correlational analysis and is transparent about its limitations, which is commendable. The main obstacles are (1) the causal framing in the abstract and key observations that goes beyond the evidence, and (2) the missing quantitative support for the pass@k selection claim. Both are fixable with rewriting and additional experiments/reporting, so I do not recommend rejection. The paper would also benefit from a tighter definition of 'universally harmful' given the cross-architecture reversal in Figure 4(d)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The descriptive contributions are real: across five open models and six benchmarks, single-agent systems match or beat the best MAS in 43.3% of cases, and first-round entropy features dominate a trained classifier's predictions of correctness. The entropy feature engineering is thorough (245 features, large sample-to-feature ratio, 5-fold CV, ablations, calibration, plus a controlled SAS-vs-MAS decomposition in Appendix H). The Entropy Judger idea is practical: selecting among pass@k candidates without ground truth. I also appreciate the honest appendices: H reports 83.4% of round-1-to-2 entropy drops come without accuracy gains (possible anchoring), and J explicitly defers causal identification. That is more transparent than most papers. The soft spot is exactly the one the stress-test flags. The abstract says peak entropy 'causally drive[s]' and Section 4.4 says 'directly benefits' — but the evidence is correlational. SHAP signs on a fitted classifier do not establish that entropy precedes correctness. Given the paper's own anchoring numbers and the high base-model-correctness feature that dominates everything else, the causal language is not supported. This is fixable: reword to 'is associated with' or 'predicts,' and add within-problem comparisons (e.g., different decodes of the same question) if they want to make causal claims. The 43.3% headline also lacks error bars on the small AIME sets, and the 'consistent accuracy improvements' for Entropy Judger are asserted but not quantified with actual pass@k selection numbers in the visible text. Those are minor relative to the causal overreach but should still be fixed. Bottom line: the work is serious, well-run, and worth engaging, but the headline claims overstate the evidence. A referee should demand the causal language be pulled back and the pass@k numbers be reported. I'd send it to peer review rather than desk reject — it's a solid empirical study with a fixable mismatch between claims and evidence.","headline":"Solid empirical study of MAS entropy dynamics whose descriptive findings survive scrutiny, but the abstract's causal language overstates what the evidence supports.","tokens_in":666,"tokens_out":1657,"would_cite":true,"duration_ms":34925,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the success of LLM-based multi-agent systems is largely set in the first round of interaction: peak token-level entropy harms correctness across every architecture, and a single agent is the best choice in 43.3% of cas","keywords":["multi-agent systems","entropy","uncertainty quantification","large language models","reasoning","single-agent vs multi-agent","entropy judger","LLM collaboration"],"falsifier":"A reader could split the paper's released data by whether the base model already answered correctly and re-run the correlation between round-1 peak entropy and MAS correctness within each split; if the negative correlation vanishes when base-model correctness is held fixed, the 'peak entropy harms MAS' claim reduces to a proxy for problem difficulty rather than a causal mechanism.","tokens_in":54354,"feed_emoji":"🤖","tokens_out":5047,"duration_ms":49620,"temperature":0.7,"pith_summary":"The paper tries to establish that the success or failure of LLM-based multi-agent systems can be explained by the entropy—the token-level uncertainty—of their early reasoning, rather than by the number of agents or the collaboration topology. It reports that a single agent matches or beats every multi-agent architecture in 26 of 30 model-benchmark combinations, and that the decisive signal is the first round of interaction: peak entropy is harmful, stable low entropy helps, and higher base-model entropy hurts the team. If correct, this gives practitioners a cheap, label-free diagnostic for when to deploy a team at all, and the paper's Entropy Judger uses exactly this signal to pick better answers from pass@k candidates.","feed_headline":"Peak uncertainty in round one sinks AI agent teams","feed_subtitle":"Single agents win 43% of matchups; low, stable entropy predicts success.","key_machinery":"Hierarchical entropy features: 245 features computed from token-level Shannon entropy of every agent's output, aggregated at agent, round, sample, and system levels. These feed an ensemble of gradient-boosted tree classifiers trained to predict per-sample correctness from the entropy trace; feature-attribution analysis then identifies which entropy patterns push predictions toward success or failure. The Entropy Judger is this classifier used for label-free pass@k selection.","core_discovery":"The paper's central claim is that MAS effectiveness is largely determined by early-round uncertainty dynamics, with peak entropy universally harmful across architectures. Across five open-source LLMs and six benchmarks, a single agent achieves the highest accuracy in 43.3% of cases, and in 86.7% of scenarios it matches or exceeds at least one MAS architecture. Three principles emerge: Certainty Preference (low, stable cross-agent entropy correlates with correctness), Base Uncertainty (lower base-model entropy directly benefits MAS performance), and Task Awareness (optimal entropy profiles shift with task difficulty and architecture). The paper further claims that the first round is decisive","pith_inferences":["If the entropy signal is this informative, an adaptive router that runs a single agent when first-round entropy is low and only convenes a team when divergence is moderate could get most of the team's benefit at a fraction of the cost; the paper does not test this but its data make it a natural next step.","The paper's own appendix shows 83.4% of round-1 to round-2 transitions reduce entropy without improving accuracy, which suggests much of the observed convergence may be agents anchoring on each other—so Certainty Preference may be partly about suppressing disagreement rather than about finding the truth.","The RL-trained base model results imply that entropy's meaning is training-dependent: the same entropy value can signal productive exploration or noise depending on the model, so a single entropy threshold will not transfer across base models and selection rules should be recalibrated per model family."],"forward_implications":["Multi-agent collaboration should not be assumed superior; single-agent baselines are the reference point, and teams only pay off under specific early-uncertainty conditions.","First-round entropy peaks are a universal warning sign; adding more interaction rounds rarely recovers from initial misalignment and often degrades accuracy.","The Entropy Judger provides a label-free way to select high-quality outputs from multiple candidates, with consistent accuracy gains across architectures and tasks.","Base-model entropy is a strong predictor of team performance, so choosing a base model with lower reasoning uncertainty is a direct lever for MAS success."],"fun_headline_variants":["Early entropy decides AI agent team success","Round-one uncertainty predicts agent team fate","Single agent beats AI teams 43% of the time","Low, stable entropy: key to AI agent success","Peak entropy in first round cripples agent teams"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the entropy measured during reasoning causes performance differences—not merely that it happens to correlate with them—so that increasing peak entropy or stabilizing entropy would change outcomes as the paper's SHAP directions suggest.","fun_headline_variants_meta":{"raw":{"variants":["Early entropy decides AI agent team success","Round-one uncertainty predicts agent team fate","Single agent beats AI teams 43% of the time","Low, stable entropy: key to AI agent success","Peak entropy in first round cripples agent teams"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000413,"raw_usage":{"total_tokens":1993,"prompt_tokens":782,"completion_tokens":1211,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":526,"completion_tokens_details":{"reasoning_tokens":1140}},"tokens_in":526,"tokens_out":1211,"duration_ms":43205,"temperature":1.0,"reasoning_tokens":1140,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:40:46.102293+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could split the paper's released data by whether the base model already answered correctly and re-run the correlation between round-1 peak entropy and MAS correctness within each split; if the negative correlation vanishes when base-model correctness is held fixed, the 'peak entropy harms MAS' claim reduces to a proxy for problem difficulty rather than a causal mechanism.","supporting_citations":[],"review_version":2}