{"id":"170ad116-226e-4091-a103-91c8baa491d8","arxiv_id":"2608.11420","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A seven-round multi-agent LLM pipeline improves differential diagnosis recall on the Open-XDDx benchmark, with gains concentrated in the hardest cases.","lead":"This paper introduces SCoT, a seven-round multi-agent LLM pipeline for medical differential diagnosis, where five specialist personas debate and vote on diagnoses. On 570 medical cases, SCoT improves recall over single-model baselines on most models, and the authors argue the gain comes from agent diversity, not extra compute.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best-of-n control is not compute-matched: SCoT uses far more LLM calls and tokens, and majority-vote aggregation suppresses monolithic recall, so the claimed social-scaling advantage is not established.","rationale":"The reader's verdict identifies the best-of-n control as the weakest assumption; I agree. The paper's strongest claim is the recall advantage of SCoT over monolithic inference at matched compute. The only direct test of that claim is the best-of-35 majority-vote control on Qwen-2.5-32B. That control does not match SCoT's compute: SCoT issues far more LLM calls (counting the three sub-rounds of round 5 and the voting and can't-miss rounds) and each call carries a long, growing context, while best-of-35 uses 35 identical short prompts. Token counts are not reported, but the ratio is likely an order of magnitude. Furthermore, the aggregation rule is biased: majority vote over 35 samples yields a short consensus list, whereas SCoT's Borda-weighted final differential includes many candidates, so recall is measured on a much larger candidate set. A fair monolithic control would need to match token budget and use a union or reranked aggregation. Without that, the 7-percentage-point recall gap between SCoT and best-of-35 could be explained by compute and aggregation alone. The paper's supporting arguments — single-agent SCoT degrading recall, difficulty-conditioned gains — do not rescue the central claim because they do not control for compute either. The Haiku 4.5 result comparing 570 baseline cases to 100 SCoT cases is another validity issue, but it is exploratory and not the load-bearing comparison. An honest assessment: the empirical pattern is interesting and worth pursuing, but the current evidence does not establish that social interaction, rather than compute and candidate-set size, drives the reported gains. The verdict of REJECT remains appropriate, with the path to acceptance being a token-matched, aggregation-fair control.","tokens_in":16071,"tokens_out":6798,"duration_ms":54481,"concrete_test":"Run a token-matched monolithic control on Qwen-2.5-32B across all 570 Open-XDDx cases. First measure the median total input+output tokens consumed by SCoT N=5 per case (sum over all agents and rounds). Then set k so that k times the median tokens per single best-of-n response equals the SCoT median token budget; k will likely be 100–500, not 35. Aggregate the k samples three ways: (a) majority vote, (b) union of all unique diagnoses, and (c) judge-based reranking of the union using the same judge prompt as the paper. Compare macro recall against SCoT's 0.607. If the union or judge baseline meets or exceeds SCoT recall, the central claim fails; if SCoT still leads under all three aggregations at matched tokens, the social-scaling claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that SCoT's recall advantage is 'not reproduced by monolithic inference alone' and that gains come from agent heterogeneity rather than compute — rests on the best-of-n control in Sections 5.4 and 6.2. That control is not compute-matched. SCoT N=5 runs five agents through seven rounds, with round 5 containing three sub-rounds and round 6 requiring each agent to rank the master list; this yields at least 40–45 LLM calls per case, each with rapidly growing context (the shared master list and full preceding deliberations). Best-of-35 uses exactly 35 calls, each with a short, fixed prompt (case plus instruction). Token totals differ by an order of magnitude or more. In addition, SCoT aggregates via credibility-weighted Borda count over the full candidate pool, while best-of-n is aggregated by majority vote, which selects a small consensus set and inherently caps recall relative to a union of sampled diagnoses. The paper reports best-of-35+judge recall of 0.499 versus SCoT's 0.607, but the judge is still applied to only 35 samples with no token match. Because compute and aggregation both favor SCoT in this comparison, the observed gap cannot be attributed to 'social' deliberation. The paper's own single-agent SCoT result (recall −5.4 pp, Figure 2) shows the pipeline structure alone is recall-negative; the multi-agent gain may simply be ensemble diversity plus a more recall-favorable aggregation rule. The Haiku 4.5 comparison (570-case baseline vs 100-case SCoT, Table 2) is a separate validity problem, but the Qwen best-of-n control is the load-bearing failure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Social Chain of Thought (SCoT), a multi-round, multi-agent architecture for medical differential diagnosis. Five persona-conditioned specialist agents generated by a single backend LLM deliberate over seven rounds (specialist ranking, symptom management, independent differentials, master list, three refinement sub-rounds, credibility-weighted Borda voting, and a final 'can't miss' safety round) to produce a ranked differential diagnosis. The authors evaluate SCoT on the Open-XDDx benchmark (570 physician-curated vignettes) across eight backend models, comparing against single-agent baselines, a single-agent SCoT ablation, and best-of-n repeated sampling with and without a judge. They report consistent recall gains of 4–12 percentage points across most backends, with larger gains on harder cases, and interpret the results as evidence that SCoT's recall advantage comes from structured multi-agent heterogeneity rather than from additional compute or repeated monolithic sampling.","tokens_in":16333,"tokens_out":5653,"duration_ms":50204,"significance":"If the central claim were established, the paper would make a meaningful contribution: it shows a concrete task domain (medical differential diagnosis) where a structured multi-agent interaction protocol with a single backend model outperforms repeated sampling at matched or greater inference budgets, and it provides a reproducible pipeline with open-weight models. The paper also contains useful ablations (agent count, temperature, single-agent pipeline) and a difficulty-conditioned analysis that identifies where the architecture helps most. However, the paper's headline claim—that SCoT's recall advantage is not reproduced by monolithic inference and scaling—rests on a best-of-n control that is not compute-matched and on an aggregation rule that is compared asymmetrically. The cross-model results also contain a case-count mismatch for the Claude Haiku 4.5 condition. These issues are load-bearing; until they are fixed, the social-scaling interpretation is not supported.","major_comments":[{"comment":"","section":"§5.4 and §6.2, Figure 3"},{"comment":"","section":"Table 2 and Figure 1"},{"comment":"","section":"Appendix F"},{"comment":"","section":"§6.3 and Figure 5"}],"minor_comments":[{"comment":"","section":"Abstract"},{"comment":"","section":"§5.2 vs Appendix F.4"},{"comment":"","section":"Table 2 caption"},{"comment":"","section":"Figure 5"},{"comment":"","section":"Appendix D"},{"comment":"","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper's core claim—that SCoT's recall advantage cannot be reproduced by monolithic inference and scaling—is interesting and well-motivated, but the current evidence does not support it. The best-of-n control is not compute-matched in terms of tokens or calls, and the aggregation rules differ, so the observed gap could be an artifact of compute or of the majority-vote recall cap. The Haiku 4.5 case-count mismatch also undercuts the cross-model comparison. These issues are fixable with additional experiments, but they are load-bearing; if the authors cannot provide a properly compute-matched control (including a union or equivalent aggregation baseline), the central claim should be withdrawn or substantially softened. I also suggest asking the authors to provide a clear definition of the 'convergence pathway' attribution rule and to report sensitivity analyses for the hand-tuned voting weights. The manuscript claims an open-source framework but does not include a repository link; the editor may wish to request code and prompts for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SCoT is a seven-round multi-agent pipeline for medical differential diagnosis, and the paper's most honest contribution is the difficulty-conditioned finding: recall gains concentrate in the cases where a single-agent baseline fails completely, and late-round refinement matters most there. That pattern appears consistently across models and is worth taking seriously.\n\nWhat is genuinely new: the pipeline design itself — Delphi-inspired rounds, persona-conditioned specialist panels, credibility-weighted Borda voting — is detailed and reproducible in principle. The single-agent SCoT ablation is a useful negative control, and the error-correlation analysis is a real attempt to isolate the mechanism behind the gains. The paper is also candid in its Limitations section about the exploratory frontier runs, the small dataset, and compute accessibility. The citation practice looks ordinary; the related work is positioned honestly.\n\nThe soft spot is the load-bearing control. The abstract and contributions claim the recall advantage 'is not reproduced by monolithic inference alone' and that SCoT beats repeated sampling at matched compute. The best-of-n comparison does not establish that. SCoT N=5 runs five agents through seven rounds, with round 5 containing three sub-rounds and round 6 requiring every agent to rank the master list; that is at least 40-45 LLM calls per case with rapidly growing context. Best-of-35 is 35 calls with a short fixed prompt; token totals differ by an order of magnitude. On top of that, SCoT aggregates via credibility-weighted Borda count over the full candidate pool, while best-of-n is aggregated by majority vote, which selects a small consensus set and inherently caps recall. Both differences favor SCoT. The paper asserts 'comparable compute' but provides no token accounting.\n\nSmaller problems: the Haiku 4.5 comparison mixes a 570-case baseline with 100-case SCoT runs; there are no significance tests or confidence intervals; and the voting weights in Appendix F are hand-tuned with no sensitivity analysis. None alone sink the paper, but together they mean the 'social scaling' claim outruns the evidence.\n\nWho this is for: anyone working on multi-agent LLM systems or clinical decision support. The hard-case analysis is the genuine signal. I would not cite it for the headline claim until the control is fixed, but it would make a good reading-group case study in how to mis-match compute in an ablation. My recommendation: send to peer review with a request for major revision — token-matched best-of-n, a union or Borda aggregation baseline for the monolithic samples, full-case frontier runs, and significance testing. If the hard-case gap survives a compute-matched comparison, this becomes a real result. As it stands, it is an interesting hypothesis with a flawed demonstration.","headline":"The hard-case diagnostic gains are plausible and worth chasing, but the central claim that SCoT beats monolithic best-of-n at matched compute is not supported by the evidence.","tokens_in":16925,"tokens_out":4239,"would_cite":false,"duration_ms":35524,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A seven-round \"doctor panel\" built from one LLM—five persona-conditioned specialist agents who propose, challenge, and vote on diagnoses—recovers more correct diagnoses from the same model than single-agent inference or repeated sampling…","keywords":["multi-agent LLM systems","medical differential diagnosis","Social Chain of Thought","persona conditioning","test-time scaling","best-of-n sampling","diagnostic recall","deliberative consensus"],"falsifier":"Run SCoT and a single-model repeated-sampling control on the same 570 cases with matched total generated tokens and union-based aggregation (not majority vote). If the single model reaches or exceeds SCoT's 0.607 recall under matched compute, the social-scaling claim is falsified; if it still falls short at several times the token budget, the claim is strengthened.","tokens_in":15785,"feed_emoji":"🩺","tokens_out":8740,"duration_ms":124305,"temperature":0.7,"pith_summary":"This paper tries to establish that a panel of five LLM agents, each assigned a medical-specialist persona and made to deliberate over seven rounds, recovers more correct diagnoses from the same underlying model than a single agent can, and that this gain is not merely extra compute. The claim matters because differential diagnosis is a deliberative task where the space of plausible candidates must be broadened before it is narrowed, and millions of users already ask LLMs health questions. Across eight backend models, SCoT improves recall by four to twelve percentage points over single-agent baselines on 570 physician-curated cases, with the largest gains on cases where monolithic inference fails completely. The paper also claims a capability floor: a 1.5B-parameter model degrades under SCoT, while 3B and larger models benefit. If right, SCoT offers a transparent, open-weight architecture for getting more diagnostic value out of a fixed model without training or larger parameters.","feed_headline":"Five doctor personas lift diagnosis recall beyond repeated sampling","feed_subtitle":"Five specialist agents add 4–12 recall points on 570 medical cases and rescue complete failures.","key_machinery":"The central object is the SCoT pipeline itself—a seven-round deliberative scaffold (with the refinement round split into three sub-rounds) in which the backend LLM first generates five specialist personas tailored to the vignette, then has them act as a panel. Rounds 1–3 elicit relevance ranking, triage, and independent differentials; round 4 compiles a deduplicated master list; round 5 runs three debate sub-rounds (support/challenge/neutral, cross-agent challenge, defense/update); round 6 aggregates rankings by credibility-weighted Borda voting; round 7 adds a \"can't-miss\" safety check for life-threatening diagnoses. The mechanism doing the work is persona-conditioned heterogeneity plus structured consensus: persona conditioning decorrelates errors, the shared candidate list lets agents revise in response to one another, and the voting step converts a broadened candidate space into a ranked differential, so that recall comes from breadth and precision from selection.","core_discovery":"On the 570-case benchmark used in this paper, SCoT with five specialist personas improves macro recall by 4.4–12.3 percentage points over the same model's single-agent baseline across capable backends, while the smallest tested model loses 3.7 points. The decisive control uses Qwen-2.5-32B: best-of-5 reaches 0.524 recall, best-of-35 reaches 0.537, best-of-35 with a judge reaches 0.499, and all three lower precision and F1 compared with SCoT's 0.607 recall, 0.509 precision, and 0.531 F1. A single agent run through the SCoT pipeline loses recall (−5.44 points) but gains precision (+10.74 points), indicating that the pipeline's voting mechanism filters rather than broadens without heterogeneity. SCoT's lift is difficulty-conditioned: in the hardest quartile, F1 rises 15.1 points (0.212 to 0.363), while the easiest quartile shows net decreases; in the 89 cases where baselines recovered zero ground-truth diagnoses, SCoT recovered at least one in 53.9% of them, with 36% of its true positives arriving through late refinement.","pith_inferences":["A natural next test is token-matched rather than pass-matched compute: count the total prompts and generated tokens SCoT consumes, then give a single model that same budget via repeated sampling with union-of-samples recall. If the gap narrows, the \"social\" part of the gain shrinks; if it persists, SCoT's claim is robust.","SCoT's difficulty-conditioned results suggest a routing rule the paper does not itself propose: reserve multi-agent deliberation for cases where a monolithic baseline is uncertain, and keep the cheap single-pass path for easy cases.","The reported error-correlation results imply persona conditioning accounts for only a modest decorrelation; a counterfactual that isolates conversation itself—same personas, same rounds, but no cross-agent messages—would separate the social mechanism from static panel diversity.","SCoT-like panels may transfer to other high-stakes reasoning tasks where candidate sets must be broadened before consensus is applied, such as failure diagnosis in engineering systems; the paper mentions future work beyond diagnostics but does not test it."],"forward_implications":["If SCoT's claim holds, a fixed open-weight LLM can gain 4–12 percentage points of diagnostic recall without fine-tuning or a larger model, by spending inference budget on structured multi-agent deliberation.","Repeated sampling with the same model does not automatically recover the same recall at the compare budgets tested, so test-time scaling alone is not a substitute for heterogeneous multi-agent structure.","Deployment can be difficulty-aware: cases that a monolithic baseline handles confidently can stay on the cheaper single-pass path, while low-confidence or hard cases get the SCoT panel, because SCoT's gains concentrate there and it can hurt precision on easy cases.","There is a capability floor for social scaling; small models below the threshold (around 1.5B parameters in the Qwen family) should not be placed in multi-agent deliberation, since SCoT makes them worse.","The architecture adds transparency: each round records which specialist proposed, challenged, and revised which diagnoses, producing an audit trail of how the final differential was formed."],"supporting_citations":[{"why":"Supplies the 570-case physician-curated benchmark and the dual-inference baseline (53% recall) that SCoT is evaluated against and extends.","marker":"Zhou et al. (2025)"},{"why":"Justifies the best-of-n repeated-sampling procedure used as the monolithic compute-equivalence control.","marker":"Brown et al. (2024)"},{"why":"Motivates the core premise that single-model self-evaluation propagates correlation bias, hence the need for intentional heterogeneity.","marker":"Huang et al. (2024)"},{"why":"Provides the deliberative-headroom prediction that smaller models gain most from multi-agent systems and informs the refinement-debate design.","marker":"Yang et al. (2025)"},{"why":"Inspires the Symptom Management triage round in the SCoT pipeline.","marker":"Tu et al. (2024)"},{"why":"Supplies the Delphi-style iterative consensus methodology adapted for SCoT's refinement and voting rounds.","marker":"Graham et al. (2003)"},{"why":"Establishes the distinction between heterogeneity and resource scaling that SCoT claims to operationalize.","marker":"Cemri et al. (2025)"}],"fun_headline_variants":["Multi-agent doctor chat beats repeated sampling on hard diagnoses","Doctor personas rescue half of otherwise zero-recall cases","Five specialist personas beat best-of-35 sampling on tough cases","Multi-agent chat adds 15 F1 points on hardest diagnoses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the best-of-n experiment gives the monolithic model a fair, compute-matched chance against SCoT; if SCoT actually consumes far more prompts and tokens per case, the headline \"no advantage for monolithic inference\" may be an artifact of unequal effort, and the majority-vote aggregation may further disadvantage repeated sampling.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent doctor chat beats repeated sampling on hard diagnoses","Doctor personas rescue half of otherwise zero-recall cases","Five specialist personas beat best-of-35 sampling on tough cases","Multi-agent chat adds 15 F1 points on hardest diagnoses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001013,"raw_usage":{"total_tokens":4301,"prompt_tokens":993,"completion_tokens":3308,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":3241}},"tokens_in":609,"tokens_out":3308,"duration_ms":19968,"temperature":1.0,"reasoning_tokens":3241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:12:15.125625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SCoT and a single-model repeated-sampling control on the same 570 cases with matched total generated tokens and union-based aggregation (not majority vote). If the single model reaches or exceeds SCoT's 0.607 recall under matched compute, the social-scaling claim is falsified; if it still falls short at several times the token budget, the claim is strengthened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Delphi-style iterative consensus methodology adapted for SCoT's refinement and voting rounds."}],"review_version":1}