{"id":"ac9c94d3-ff22-4743-8273-17befe6f5be3","arxiv_id":"2508.11995","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AgentCDM uses two-stage RL training, first with ACH reasoning scaffolding then with the scaffold gradually removed, to make a Qwen-7B decision agent outperform voting, dictatorial, and prompted-reasoning baselines on MMLU, MMLU-Pro, and ARC-Challenge.","lead":"This paper trains a single 'decision agent' to combine answers from several AI agents using a structured reasoning protocol borrowed from intelligence analysis. The trained agent beats simpler voting and single-decider setups on three multiple-choice benchmarks, by up to 29 points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The same full AgentCDM framework is assigned three different ARC-Challenge scores (88.4, 89.5, 96.0) across Tables 1, 2, and 4; until reconciled, the empirical core of the paper is unstable.","rationale":"The reader's weakest assumption is the unstated train/test split. That is a real concern, but the paper contains a sharper and more specific problem: the same method receives materially different ARC-Challenge scores in different tables. Because the central claims are empirical—state-of-the-art accuracy, cross-dataset generalization, and the necessity of both training stages—each of those tables is load-bearing. An unexplained 6.5-to-7.6-point spread across tables means a reader cannot currently verify which number is the true performance. This is not an objection to the method's conceptual design; the ACH-inspired framework is plausible and the two-stage RL idea is clear. It is an objection to the evidentiary basis. The paper's own limitations section appropriately notes sensitivity to execution-agent output quality and open questions about adversarial settings, but those do not undermine the in-domain claim. The absence of code, seeds, and variance estimates compounds the problem, but the specific numeric contradiction is the most concrete and checkable issue. A conditional verdict is appropriate: the authors can resolve the concern by disclosing and standardizing the evaluation protocol for all tables. If the discrepancy turns out to reflect a typo, the fix is trivial; if it reflects unreported configuration differences, the comparisons must be recomputed. Either way, the current manuscript does not yet support unconditional acceptance.","tokens_in":14862,"tokens_out":9178,"duration_ms":96298,"concrete_test":"Obtain the AgentCDM checkpoint and run it on the ARC-Challenge test split under the exact configuration implied by Table 1's 'AgentCDM' row (execution agents, number of agents, sampling, and split). Then run the same checkpoint under the configuration implied by Table 4's 'Full' row (same training dataset, evaluated on the ARC-Challenge test split). Report both scores, the random seed, and the full experimental configuration. If the two scores differ by more than a rounding error, determine which configuration produced the claimed state-of-the-art result and recompute Tables 1, 2, and 4 under a single consistent protocol; if they are the same, correct the inconsistent table entries.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.1 claims that AgentCDM 'consistently and significantly outperforms all baselines,' and Section 5.2 claims strong cross-dataset generalization. The empirical anchor for these claims is inconsistent for ARC-Challenge. In Table 1, the average row for AgentCDM reports 88.4 for ARC-Challenge. In Table 2, the diagonal entry for training on ARC-Challenge and evaluating on ARC-Challenge reports 89.5. In Table 4, the 'Full' ablation, explicitly defined as 'our complete two-stage AgentCDM framework,' reports 96.0 on ARC-Challenge. The 96.0 figure is 6.5 to 7.6 points higher than the other two. No experimental configuration difference is disclosed—execution-agent pool, number of agents, train/test split, or prompt variant—that would reconcile these as different settings. Table 1 grounds the state-of-the-art claim, Table 2 grounds the generalization claim, and Table 4 grounds the two-stage synergy claim, so at least one of these three empirical pillars is not represented by a reproducible number. This is more immediately load-bearing than the train/test split ambiguity: even after the held-out protocol is clarified, the reader cannot tell which ARC number is the true performance of the method. The paper also never states the split used for in-domain RL training; if the accuracy reward is computed on the same split later reported as test, the reported gains would be fitting artifacts. Both issues are fixable by disclosure, but the current manuscript does not provide enough information to verify the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AgentCDM, a framework for collaborative decision-making in LLM-based multi-agent systems. A decision agent is trained in two reinforcement-learning stages: Stage 1 uses an explicit ACH-inspired reasoning protocol as a scaffold, rewarded by format, accuracy, and pattern-matched ACH adherence; Stage 2 replaces the explicit protocol with a curriculum over full and simplified ACH prompts and adds a soft ACH reward based on BGE-M3 cosine similarity. The trained Qwen-7B decision agent is evaluated on MMLU, MMLU-Pro, and ARC-Challenge against dictatorial, voting, and ACH-prompted baselines, plus cross-dataset transfer, scalability, and ablation experiments. The paper reports large gains, e.g., +29.3 points over a Mistral-7B single-agent baseline on MMLU-Pro, and claims state-of-the-art performance and strong generalization.","tokens_in":15151,"tokens_out":8401,"duration_ms":84129,"significance":"The core idea is timely and the comparisons are mostly well chosen; in particular, comparing AgentCDM against Qwen-7B-R1 under the same ACH protocol is the right control and shows a consistent advantage on the reported averages. If the numbers are reproducible, the two-stage scaffold-then-soften RL curriculum is a useful contribution to multi-agent decision-making, and the cross-dataset transfer results are notable. However, several load-bearing empirical details are missing or internally inconsistent, so the current manuscript does not yet establish the claimed state-of-the-art performance or generalization.","major_comments":[{"comment":"The ARC-Challenge accuracy reported for the same AgentCDM system is internally inconsistent: Table 1's ARC average row gives 88.4, Table 2's diagonal entry (training on ARC-Challenge, evaluating on ARC-Challenge) gives 89.5, and Table 4's 'Full' row, defined as the complete two-stage AgentCDM framework, gives 96.0. No experimental configuration difference (execution-agent pool, number of agents, prompt variant, or train/eval split) is disclosed that would explain a gap of 6.5–7.6 points. Because Table 1 grounds the state-of-the-art claim, Table 2 grounds the cross-dataset generalization claim, and Table 4 grounds the two-stage synergy claim, the reader cannot tell which number is the true performance; please reconcile the three tables or explicitly label the different configurations.","section":"Tables 1, 2, and 4"},{"comment":"The paper never states the train/test split used for the RL training in the main experiments. The implementation paragraph says only that batches sample P=256 prompts, and the phrase 'previously unseen test sets' appears only in the cross-dataset section. If the accuracy reward in Eqs. (3) and (6) is computed on the same MMLU/MMLU-Pro/ARC-Challenge items later reported as evaluation, the reported gains are fitting artifacts. Please state explicitly which split was used for training, validation (if any), and testing, and confirm that no test-set item contributed to reward computation.","section":"Models and Implementation Details / Main Results"},{"comment":"The blanket claim that AgentCDM 'consistently and significantly outperforms all baselines' is contradicted by the table's own GPT-4 rows: on MMLU, Informed Dictatorial scores 85.7 versus AgentCDM's 85.6, and on MMLU-Pro, Informed Dictatorial scores 71.2 versus AgentCDM's 71.0. The claim should be restricted to aggregate/average performance or to non-GPT-4 execution pools, or the rows should be recomputed and the text corrected.","section":"Main Results (Table 1)"},{"comment":"Stage 2 is described as 'Scaffolding Removal and Autonomous Exploration,' but the Stage-2 reward still contains R_softACH, which explicitly rewards similarity to the authors' own ACH protocol. Consequently, the ablation label 'Exploration-Only' is not unguided exploration, and the conclusion that unguided exploration is inefficient is not supported. Please clarify what is actually removed in Stage 2, and consider an additional ablation without any ACH-shaped reward to substantiate the autonomy claim.","section":"Stage Two (Eq. 6) and Ablation Study (Table 4)"},{"comment":"No variance estimates, number of seeds, or significance tests are reported; all tables give single-run accuracy values. The text repeatedly says AgentCDM 'significantly outperforms' baselines, but several comparisons are within one point (e.g., Table 1, GPT-4 MMLU row), and the claim of statistical significance is unsubstantiated. Please report standard deviations across runs/seeds and, where appropriate, significance tests, or soften the wording to describe the observed run-level improvements.","section":"Main Results and Ablation Study (Tables 1 and 4)"}],"minor_comments":[{"comment":"The scalability plot has no error bars or per-condition sample sizes, so the claimed 'dual phenomenon' may be partly noise; please add variability information or state the number of queries per point.","section":"Robustness and Scalability Analysis (Figure 4)"},{"comment":"The text refers to Appendices A, B, and C for implementation and benchmark details, but no appendix content appears in the manuscript; please include these sections or remove the references.","section":"Appendix references"},{"comment":"Table 3 reports only two conditions and does not describe how the three random outputs were sampled from the heterogeneous pool or how many queries were used; please add these details.","section":"Robustness and Scalability Analysis (Table 3)"},{"comment":"The BGE-M3 citation is given as 'Multi-Granularity 2024'; please use the proper bibliographic entry for the M3-Embedding paper.","section":"References"},{"comment":"The phrase 'state-of-the-art' is stronger than the comparison set supports, since the baselines are limited to the GEDI configurations and no comparison to other recently proposed MAS decision or aggregation methods is included; consider qualifying the claim.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core, but the empirical record is not stable: the ARC-Challenge numbers are inconsistent across the three central tables, and the train/test split is never disclosed. These issues are fixable by disclosure and re-analysis, so I recommend major revision rather than rejection. I would also ask the authors to provide code or checkpoints to facilitate verification of the reported gains."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one thing to know: AgentCDM is a real idea, but the paper currently cannot support its own headline numbers. The same claimed configuration gets 88.4 on ARC-Challenge in Table 1, 89.5 in Table 2, 93.0 in Table 3, and 96.0 in Table 4, with no disclosed difference in setup. Since Table 1 anchors the SOTA claim, Table 2 the generalization claim, and Table 4 the two-stage synergy claim, at least one of the three central empirical pillars is not represented by a reproducible number. That is the first thing I'd fix.\n\nWhat's actually new: applying ACH to the decision agent in an LLM MAS, and especially the two-stage RL curriculum—explicit ACH scaffolding with pattern-matching rewards, then a soft ACH reward based on embedding similarity plus curriculum annealing that removes the scaffold. That is a sensible adaptation of DeepSeek-R1-style rule-based RL to the multi-agent setting, and I don't know of prior work doing exactly this. The control comparison that matters—AgentCDM vs Qwen-7B-R1 under the same ACH prompt—shows consistent gains, which is evidence the training, not just the prompt, is doing something. The cross-dataset transfer experiment is also a good design, and the conclusion that harder training data transfers better is interesting.\n\nSoft spots in proportion: the inconsistent ARC numbers are load-bearing. The train/test split for the RL training is never stated; only the cross-dataset section mentions 'previously unseen test sets.' If accuracy rewards were computed on prompts later reported as test, the gains are fitting artifacts. Single runs, no variance or significance tests, no code or data release. These are fixable by disclosure and a few repeated seeds. The ACH-reward circularity worry is less serious than it looks, because the accuracy reward is external; the soft ACH reward mainly shapes the text to resemble the protocol, which would matter more if there were no accuracy signal.\n\nWho this is for: anyone working on aggregation in LLM-based MAS, and people interested in RL curricula for reasoning. The method section is worth reading. But the experimental claims as written are not verifiable, so I wouldn't cite the numbers yet.\n\nRecommendation: send it to peer review, but with a clear request to reconcile the ARC scores, state the exact train/test split, and add variance estimates. The idea is worth referee time; the current reporting isn't.","headline":"The training recipe is genuinely new, but the reported ARC-Challenge numbers disagree across tables—the empirical core needs fixing before the claims can be trusted.","tokens_in":15708,"tokens_out":4457,"would_cite":false,"duration_ms":43249,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgentCDM claims that a small decision agent trained to reason through competing hypotheses, evidence matrices, and falsification can beat voting and untrained dictatorial arbiters in LLM multi-agent systems.","keywords":["multi-agent systems","collaborative decision-making","Analysis of Competing Hypotheses","reinforcement learning","GRPO","LLM reasoning","MMLU-Pro","structured reasoning"],"falsifier":"Run the identical two-stage training with the evaluation split explicitly quarantined and documented; if MMLU-Pro accuracy drops from 65.2 toward the ACH-protocol-only level (58.0) or the +29.3 gain over single-agent shrinks to single digits, then the headline gains are training-set leakage rather than learned reasoning.","tokens_in":14658,"feed_emoji":"🧠","tokens_out":8450,"duration_ms":84144,"temperature":0.7,"pith_summary":"The paper tries to establish that the final step of an LLM multi-agent system—the step that turns several candidate answers into one output—is itself trainable, and that a 7B model trained for that role can outperform voting rules, untrained arbiters, and even stronger single models. The training recipe borrows the Analysis of Competing Hypotheses (ACH) protocol from cognitive science: propose hypotheses, log evidence, build a hypothesis-evidence matrix, then deliberately try to falsify the leading answer before reporting. AgentCDM first rewards strict adherence to that protocol, then rewards semantic similarity to it while fading the scaffold, so the model internalizes the reasoning instead of parroting it. On MMLU, MMLU-Pro, and ARC-Challenge the reported accuracy gains over single agents reach +29.3 points on MMLU-Pro, and a model trained on MMLU-Pro transfers to other datasets. If the result holds, multi-agent system design should focus as much on how decisions are made as on how agents talk to each other.","feed_headline":"Trained 7B arbiter lifts multi-agent accuracy by 29 points","feed_subtitle":"ACH-style hypothesis testing plus two-stage RL makes the small model beat voting as group decision-maker.","key_machinery":"The load-bearing mechanism is the ACH decision protocol plus the two-stage reinforcement-learning schedule that internalizes it. Stage 1 trains with a seven-step scaffold—enumerate hypotheses from each agent's answer, extract tagged evidence, fill a hypothesis-evidence matrix with consistent/inconsistent/irrelevant marks, refine the matrix, pick the hypothesis with the fewest inconsistencies, challenge it with the strongest counter-argument, and produce a final analytic report—while a pattern-matching reward verifies the presence of those steps inside <think> tags. Stage 2 replaces strict pattern matching with a semantic-similarity reward computed from embedding cosine similarity using the BGE-M3 model, and uses a cosine annealing schedule to sample from 'full ACH' versus 'simplified ACH' prompts, gradually withdrawing the scaffold so the model learns to reason this way unprompted. The optimizer is GRPO with a composite reward of format, answer accuracy, and ACH adherence in both stages.","core_discovery":"The central claim, on the paper's own terms, is that replacing an unstructured 'decider' prompt with a learned ACH-structured decision policy causes a large, consistent jump in collaborative decision accuracy. The trained decision agent receives the query plus the execution agents' answers, runs through hypothesis generation and evidence evaluation internally, and returns a final answer; the composite rule-based reward (format plus answer correctness plus ACH adherence) supplies all learning signal, so no human-annotated reasoning traces are used in training. The most striking reported results are on MMLU-Pro, where AgentCDM improves average accuracy from 47.9 single-agent to 65.2, and cross-dataset transfer where training on MMLU-Pro yields 80.8 on MMLU and 94.0 on ARC-Challenge, higher than training on those datasets directly. The ablation attributes this to the two stages working together: scaffolding-only collapses, exploration-only plateaus, full model wins.","pith_inferences":["A natural extension the paper leaves open is to make the decision agent also choose how many execution agents to consult; the scalability results suggest this could improve weak-executor settings. This is an editorial inference, not a paper claim.","The transfer result suggests that the difficulty of training data, not just its domain coverage, drives the learned decision skill; a direct test would be to train on a deliberately disagreement-rich subset of MMLU-Pro rather than the full set. This is an extrapolation, not reported.","The Stage-1 pattern-matching ACH reward may be doing double duty as a format regularizer that reduces output variance; comparing against a format-only reward would isolate how much of the gain is structural reasoning versus output stability. This is an inference the paper does not test."],"forward_implications":["Collaborative decision-making in MAS can be improved by training the decision agent itself, not just by choosing a better aggregation rule or prompting strategy.","A single decision agent trained on one hard benchmark can transfer to other benchmarks, which would let system builders train decision policies once on a deliberately difficult, disagreement-rich dataset and deploy them elsewhere.","The two-stage schedule is load-bearing: skipping either stage measurably hurts, suggesting that scaffold-then-anneal is a general recipe for internalizing structured reasoning protocols.","The scaling result implies that adding more execution agents is not automatically helpful; system designers may need to tune agent count to agent capability, or let the decision agent handle heterogeneous noisy inputs."],"supporting_citations":[{"why":"Supplies the ACH protocol that the paper's structured reasoning scaffold is built from.","marker":"(Heuer 1999)"},{"why":"Provides the rule-based RL and GRPO training recipe that AgentCDM adapts.","marker":"(Guo et al. 2025)"},{"why":"Defines the dictatorial and voting baseline configurations AgentCDM is compared against.","marker":"(Zhao, Wang, and Peng 2024)"},{"why":"Contributes the MMLU benchmark used for training and evaluation.","marker":"(Hendrycks et al. 2020)"},{"why":"Contributes MMLU-Pro, the hardest benchmark and the source of the largest reported gains.","marker":"(Wang et al. 2024)"},{"why":"Contributes ARC-Challenge as the science-reasoning benchmark.","marker":"(Clark et al. 2018)"},{"why":"Provides the Qwen base model weights from which AgentCDM is trained.","marker":"(Bai et al. 2023)"},{"why":"Provides the BGE-M3 embeddings used for the Stage-2 soft semantic ACH reward.","marker":"(Multi-Granularity 2024)"}],"fun_headline_variants":["ACH reasoning lifts 7B arbiter by 29 accuracy points","Hypothesis testing turns 7B model into top multi-agent decider","ACH-inspired training cuts voting bias in agent teams","Two-stage RL makes 7B arbiter outvote the group","From votes to hypotheses: structured reasoning wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training prompts for the main experiments are assumed to come from held-out training splits of MMLU, MMLU-Pro, and ARC-Challenge, but the paper never states the split and only says \"previously unseen test sets\" in the cross-dataset section.","fun_headline_variants_meta":{"raw":{"variants":["ACH reasoning lifts 7B arbiter by 29 accuracy points","Hypothesis testing turns 7B model into top multi-agent decider","ACH-inspired training cuts voting bias in agent teams","Two-stage RL makes 7B arbiter outvote the group","From votes to hypotheses: structured reasoning wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000768,"raw_usage":{"total_tokens":3411,"prompt_tokens":957,"completion_tokens":2454,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2371}},"tokens_in":573,"tokens_out":2454,"duration_ms":20672,"temperature":1.0,"reasoning_tokens":2371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:26:21.233567+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical two-stage training with the evaluation split explicitly quarantined and documented; if MMLU-Pro accuracy drops from 65.2 toward the ACH-protocol-only level (58.0) or the +29.3 gain over single-agent shrinks to single digits, then the headline gains are training-set leakage rather than learned reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ACH protocol that the paper's structured reasoning scaffold is built from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BGE-M3 embeddings used for the Stage-2 soft semantic ACH reward."}],"review_version":2}