{"id":"15a5f165-f76b-4262-8d7b-46aec5aae1ad","arxiv_id":"2608.12585","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Reasoning Jury aggregates independent defect judgments from a jury of LLMs through consolidation or moderated deliberation, beating frontier single-model judges on Hard2Verify and DeltaBench at a fraction of the cost.","lead":"A panel of several open-weight language models, arguing under a moderator, finds flawed reasoning steps in long AI math solutions more accurately and much more cheaply than using one frontier model as judge. The system, called Reasoning Jury, also flags steps so a model can retry and fix mistakes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Hard2Verify margin may be inflated by design choices made on the same 200 records; the DeltaBench replication is much smaller and lacks a reported CI, so the headline jury-vs-frontier gap is not yet established as pre-specified.","rationale":"The concern is real but not disqualifying. The paper's own evidence reduces its force: the prompt ablation with gpt-5.4 (83.9 vs 85.3) shows the chosen prompt was not selected to maximize Balanced F1 on Hard2Verify; the homogeneous-jury result is explicitly framed as a diversity control rather than a tuned headline; the DeltaBench transfer, though smaller, is qualitatively consistent; and the retry experiment validates the downstream usefulness of the diagnosis field against an external AIME answer key. Bootstrap CIs on Hard2Verify also show the main 8.6-point gap is well outside sampling noise. What remains unsettled is not whether a cheap open-weight jury can beat frontier single judges, but whether the 82.3 vs 73.7 magnitude is robust to the configuration choices made on the same 200 records. The reader's conditional acceptance is exactly right: the central claim should not be rejected, but it should not be treated as fully settled until a held-out evaluation or pre-registered split is provided. Secondary reproducibility concerns (no public code or commit hash despite detailed artifact paths) reinforce the conditional stance but are not the main logical weakness.","tokens_in":34045,"tokens_out":10385,"duration_ms":115670,"concrete_test":"Randomly split Hard2Verify's 200 records into a 100-record dev set and a 100-record test set before any further experiments. Fix the complete configuration exactly as reported (Section B Phase 1 prompt, minor+ threshold, propagation-aware scoring, 3x gpt-oss-120b jurors with same-model moderator, deliberation) and pre-register it. Run the jury and the opus-4.6 single-judge baseline exactly once on the held-out 100-record test split, reporting the Balanced F1 gap with a 10,000-sample record-level bootstrap confidence interval. If the test-split gap is positive beyond the CI, the selection concern is resolved; if it shrinks substantially toward zero, the headline margin was partly produced by on-benchmark configuration choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the 82.3 vs 73.7 Balanced F1 gap between a 3x gpt-oss-120b jury and a single opus-4.6 judge on Hard2Verify (Tables 4 and 7). The load-bearing assumption is that this is an honest, pre-specified measurement. Section 3.1 decides between two prompts by running both on Hard2Verify, and all headline numbers are then reported on the same 200 records. The minor+ threshold and the propagation-aware scoring convention (Section G; prompt addendum in Section B) also shape every score and are not documented as fixed before labels were seen. No held-out development split or pre-registered configuration is described. The independent DeltaBench result (Section 4.6) supports the direction: the same jury beats opus 61.5 vs 58.4 and roughly matches gpt-5.4 (61.8), but the margin is about a third of the Hard2Verify margin, no bootstrap interval is given, and the paired sample drops 10 records where opus failed. If the Hard2Verify scores influenced the prompt, threshold, or the highlighted open-weight panel, the central 'significant outperformance' claim is optimistic and may not transfer at the advertised magnitude.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Reasoning Jury, a two-phase pipeline in which several LLM jurors independently audit a step-segmented reasoning trace and a Phase 2 mechanism (consolidation or moderated deliberation) produces a consensus verdict with per-step defect localization, severity, and evidence. The evaluation uses Hard2Verify (200 records) and DeltaBench (1,226 paired records), comparing solo frontier judges with juries of open-weight models. The headline results are that a homogeneous 3× gpt-oss-120b jury reaches 82.3 Balanced F1 on Hard2Verify versus 73.7 for opus-4.6 solo, that the same jury reaches 61.5 on DeltaBench versus 58.4 for opus, and that the jury costs 8–16% of the frontier judge at list prices. The paper also reports exploratory defect profiling on AIME2026 and a defect-guided retry experiment in which rich feedback improves retry accuracy from 71.2% to 76.2%.","tokens_in":34239,"tokens_out":7019,"duration_ms":65124,"significance":"If the reported results hold, the paper makes a practically important contribution: it identifies a configuration in which three samples of a modest open-weight model, moderated by the same model, outperform much more expensive frontier judges on step-level reasoning-defect detection while producing structured, auditable verdicts. The strengths are the explicit comparison against union and majority-vote baselines, record-level bootstrap intervals on the main benchmark, a second benchmark replication, a paired McNemar-based retry analysis, full prompts in the appendices, and unusually candid limitations (the explanation fields and severity labels are not validated against ground truth). The main benchmark result is not yet established as a pre-specified comparison, because key configuration choices are made on the same 200 records and the headline jury score lacks a confidence interval; the DeltaBench replication also lacks uncertainty quantification. With those additions the claim would be convincing.","major_comments":[{"comment":"The main Hard2Verify result is reported on the same 200 records that were used to choose the Phase 1 prompt: §3.1 compares the Hard2Verify prompt (85.3 Balanced F1 with gpt-5.4) against the paper's prompt (83.9) and then uses the latter for all subsequent experiments, with no held-out development split described. The minor+ severity threshold, the specificity self-check cutoff in §B, and the propagation-aware scoring convention in §G are additional configuration choices that affect every score and are not documented as fixed before the test labels were examined. Because the central claim is the 82.3 versus 73.7 gap, the authors should either show that these choices were pre-specified and not selected on Hard2Verify labels, or provide a confirmatory run on a held-out split or an independent benchmark. The direction of any bias is not obvious, since the prompt selection actually chose a lower-scoring prompt for gpt-5.4, but the possibility of label-based selection remains.","section":"§3.1, §4.5"},{"comment":"The headline homogeneous 3× gpt-oss-120b deliberation score of 82.3 is reported in Table 7 without a bootstrap confidence interval, whereas Table 4 reports ± intervals for the other juries. The opus-4.6 solo score is 73.7±3.7, so the 8.6-point headline gap needs an interval around 82.3 and ideally a paired comparison over records to support the word 'significantly' in the abstract. Please add record-level bootstrap intervals for the homogeneous panels and a paired significance test for jury-versus-opus on Hard2Verify.","section":"§4.5, Table 7"},{"comment":"The DeltaBench generalization result is presented as a replication but has no uncertainty quantification: the 61.5 versus 58.4 Balanced F1 gap has no confidence interval or paired significance test, and §M reports that the paired sample drops 10 records where opus could not complete, with the excluded records concentrated in the longest math traces. Given the smaller gap relative to Hard2Verify and the non-random exclusion, the authors should report bootstrap or paired intervals and analyze the sensitivity of the conclusion to inclusion or exclusion of those 10 records.","section":"§4.6, Table 12"}],"minor_comments":[{"comment":"The abstract states the cost is '8 to 15%' of the frontier judge, while Table 2 reports 16% for deliberation and the introduction says '8 to 16%'; please make the cost range consistent across the abstract, introduction, and results.","section":"Abstract and §4.8"},{"comment":"The main text says the DeltaBench evaluation is on '1,226 records' and §M clarifies that this is the paired subset after removing 10 records where opus failed; please state this exclusion and its rationale directly in Section 4.6.","section":"§4.6 and §M"},{"comment":"Section 3.2 says open-weight models are served on a local vLLM cluster, but Table 2 prices gpt-oss-120b using AWS Bedrock list rates; please state explicitly whether the cost analysis reflects list prices or actual serving costs, and whether local serving would change the reported ratios.","section":"§4.8, Table 2"},{"comment":"The sentence '49steps outputs and27 full outputs contain no parseable integer' appears to mean 49 step-anchor outputs and 27 rich-feedback outputs; please make the condition labels unambiguous.","section":"§N.1"},{"comment":"The caption says the open-weight jury outperforms frontier models 'by up to 12 points', but Table 7 shows a 13.0-point lift for the homogeneous gpt-oss jury over its best solo sample and Table 1 shows lifts up to 15.4 points; please reconcile the stated maximum.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.AI and the appendices are unusually complete. The main risk is that the headline Hard2Verify result may reflect selection on the test set; I would ask the authors for a confirmatory analysis on a held-out split or on an additional benchmark before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the read on 2608.12585. The paper gets one big thing right: a panel of cheap open-weight judges—three samples of gpt-oss-120b—with a moderated deliberation beats frontier single-model judges at step-level reasoning defect detection on Hard2Verify by a wide margin (82.3 vs 73.7 balanced F1 against opus-4.6), and at roughly 8 to 15 percent of the dollar cost. That is a genuinely useful result for anyone building data filters, RL rewards, or runtime feedback for long reasoning traces.\n\nThe paper also does its homework. It reports bootstrap intervals on the main benchmark, tests on a second benchmark (DeltaBench), compares against union and majority-vote baselines, and shows that much of the gain comes from pooling independent judgments. The diversity-control experiment is clean: homogeneous panels still get large deliberation lifts, but the ceiling tracks base-model strength. The authors are explicit about what they don't validate—the factual accuracy of the what_went_wrong text, severity calibration, downstream usefulness—and that honesty is real.\n\nThe soft spot is the one your stress-test note flagged. The main benchmark is roughly 200 records, and the prompt, the minor+ threshold, and the propagation-aware scoring convention all appear to have been chosen after looking at Hard2Verify results. Section 3.1 compares two prompts on that benchmark, and the headline numbers are reported on the same records. Without a described held-out development split, the 82.3 vs 73.7 margin may be optimistic. DeltaBench reassures me the direction is real: the same jury beats opus by 3 points (61.5 vs 58.4) and lands next to gpt-5.4. But that result lacks a confidence interval and drops 10 records where opus failed, so it's a lighter pillar. I'd want the authors to either release a pre-specified configuration or at least demonstrate that the prompt and threshold choices transfer to a held-out slice.\n\nMinor quibbles: the retry experiment has no blind-retry condition (acknowledged), and no public code or commit hash despite detailed artifact paths. Neither breaks the central claim.\n\nWho this is for: anyone doing LLM-as-a-judge, process reward modeling, or reasoning-data curation. It deserves a serious referee. I'd send it to peer review and ask for the development protocol to be clarified and the DeltaBench interval added.","headline":"A practical open-weight jury for step-level reasoning defect detection with honest baselines and clear limitations; the headline margin may be optimistic because key choices were made on the same 200-record benchmark, but the direction is supported by a smaller second benchmark.","tokens_in":34803,"tokens_out":4860,"would_cite":true,"duration_ms":37213,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A moderated jury of cheap open-weight language models, deliberating under a trace-blind moderator, catches step-level reasoning defects more accurately than single frontier judges and at a fraction of the cost.","keywords":["LLM-as-a-judge","reasoning trace evaluation","multi-model consensus","deliberation","step-level defect detection","open-weight models","Balanced F1","defect-guided retry"],"falsifier":"Take a fresh, never-seen set of long reasoning traces with human step-level defect labels (or a held-out split of Hard2Verify), freeze the Phase 1 prompt, severity threshold, and propagation rule before reading any labels, then run the 3×gpt-oss-120b jury and an opus-4.6 solo judge on the same records. If the jury's Balanced F1 does not exceed opus-4.6 by at least the margin reported here (roughly 8.6 points), the central claim that an open-weight jury outperforms frontier single judges at reasoning-defect detection fails.","tokens_in":33819,"feed_emoji":"⚖️","tokens_out":9735,"duration_ms":72804,"temperature":0.7,"pith_summary":"Reasoning Jury claims that step-level defect detection in long reasoning traces—the signal needed to filter training data, shape reinforcement learning rewards, and evaluate reasoning models—is better done by a panel of independent language-model judges whose verdicts are combined through moderated consensus than by any single model, even a frontier one. On Hard2Verify, a homogeneous jury of three gpt-oss-120b samples reaches 82.3 Balanced F1 in deliberation, against 73.7 for opus-4.6 solo and 69.5–71.2 for the other frontier judges, while costing between 8% and 16% of opus-4.6's dollar price. The same pattern holds on DeltaBench, where the jury at 61.5 Balanced F1 matches gpt-5.4 and beats opus-4.6. A further experiment shows the structured verdicts are usable: when a generating model retries flagged traces, adding the jury's diagnoses and severity raises accuracy from 71.2% with step locations alone to 76.2%.","feed_headline":"Open-weight jury beats frontier models at spotting reasoning flaws","feed_subtitle":"Three gpt-oss-120b jurors reach 82.3 Balanced F1 versus 73.7 for a lone opus-4.6, at 8–16% of the dollar cost.","key_machinery":"The load-bearing object is the [STEP-x] segmented trace: every reasoning trace is split at sentence-boundary double-newlines into numbered steps, and every juror verdict cites specific steps. Phase 1 gives each juror the problem, the segmented trace, and the final answer and asks for JSON defects with step references, a self-contained what_went_wrong, a severity (neutral/minor/major/fatal), and quoted evidence, enforced by a genericness test and specificity self-check. Phase 2 has two modes: Consolidation, where a judge sees the trace plus all Phase 1 verdicts and verifies, merges, and fills gaps; and Deliberation, where a moderator—blind to the problem, trace, and solution—selects speakers, issues process instructions, tracks a claim-level consensus state, detects convergence, and extracts final consensus defects with confidence and dissents. This two-phase structure is what turns a bag of independent opinions into a single grounded, severity-tagged verdict.","core_discovery":"The paper's central claim is that a panel of open-weight LLM judges with a moderated consensus phase can identify defects in long step-segmented reasoning traces more reliably than a single frontier model judge, and at a small fraction of the cost. In Phase 1 each juror independently emits structured defect findings (step references, explanation, severity, evidence); in Phase 2 either a consolidator merges them or a moderator runs a deliberation in which jurors critique each other and may revise, with the moderator blind to the problem and trace. On Hard2Verify the deliberated 3×gpt-oss-120b jury scores 82.3 Balanced F1 versus 73.7 for opus-4.6 solo, 71.2 for sonnet-4.6, 69.5 for gemini-3.1-pro, and 83.9 for gpt-5.4; the jury costs $12.35 in deliberation mode and $6.50 in consolidation mode for 200 records, against $79.13 for the opus pass. The authors attribute the gain primarily to pooling independent judgments (recall recovery), with deliberation also adjudicating and deduplicating, and they show that homogeneous juries (three samples of one model) retain most of the lift, so model diversity is not the driver.","pith_inferences":["If the jury's advantage is driven by pooling and debate rather than by the specific models, the same two-phase structure could be applied to other fine-grained evaluation tasks—such as grading code patches, legal or medical reasoning, or detecting hallucinated citations—where single-model judges are known to miss errors.","The paper deliberately does not isolate the moderator's trace-blindness; an extension would vary moderator access (full trace vs transcript only) to measure how much of the deliberation gain comes from preventing content bias versus from simple speaker scheduling.","The retry experiment suggests a scalable offline loop: use the jury to flag defective traces, have the generator retry only those, and keep original answers otherwise; the reported 8.5-point corpus gain is a lower bound because there is no blind-retry control, so a careful head-to-head against 'retry everything' would sharpen the value of jury-gated retries.","Because the jury is cheap and open-weight, the same pipeline could serve as an RL reward and defect-feedback generator for training reasoning models, provided the latency of deliberation is acceptable for off-policy training; the paper notes this is not yet usable for on-policy RL."],"forward_implications":["On Hard2Verify, an open-weight jury with deliberation reaches 82.3 Balanced F1, surpassing opus-4.6 (73.7), sonnet-4.6 (71.2), and gemini-3.1-pro (69.5) and landing within two points of gpt-5.4 (83.9), at roughly 8–16% of the frontier judge's dollar cost.","A homogeneous jury—three independent samples of gpt-oss-120b with the same model as moderator—captures most of the gain (+13.0 Balanced F1 over its best solo), so model diversity is not required; the ceiling is set by the underlying model's capability.","Deliberation mostly improves recall, consolidation trades recall for precision, and a simple union of Phase 1 findings captures much of the improvement; majority voting collapses because jurors rarely flag the exact same steps.","The pattern generalizes to DeltaBench: the 3×gpt-oss-120b jury's deliberated consensus (61.5 Balanced F1) beats opus-4.6 solo (58.4) and matches gpt-5.4 (61.8).","Feeding jury diagnoses and severity back to a model on retry lifts accuracy over step locations alone (76.2% vs 71.2%), and a selective-retry policy reaches 88.5% corpus accuracy versus 80.0% before retry."],"supporting_citations":[{"why":"Hard2Verify provides the 200 human-annotated records, gold error-step labels, and Balanced F1 protocol on which the headline comparisons are scored.","marker":"[25]"},{"why":"DeltaBench supplies the second set of long chain-of-thought traces with human defect annotations used to test generalization beyond Hard2Verify.","marker":"[15]"},{"why":"The gpt-oss-120b model card identifies the open-weight juror and its list pricing, which makes the cost-fraction argument concrete.","marker":"[22]"},{"why":"The opus-4.6 system card defines the frontier single-judge baseline and its per-million-token pricing for the accuracy and cost comparisons.","marker":"[2]"},{"why":"The panel-of-LLM-judges idea that Reasoning Jury adapts to step-grounded reasoning-defect detection.","marker":"[4]"},{"why":"The diverse-model panel approach the paper specializes and compares against single-judge baselines.","marker":"[28]"}],"fun_headline_variants":["Jury of open models outperforms top single judges","Cheaper jury of LLMs finds more reasoning flaws","Open-weight panel tops frontier lone judges","Multi-model consensus spots reasoning errors better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline results assume that the Phase 1 prompt, the 'minor and above' severity threshold, and the scoring rule that counts a defect as propagating to later steps were not tuned against Hard2Verify's gold labels; if those choices were selected on the test set, the Balanced F1 gains would be optimistic and might not transfer to new traces or benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Jury of open models outperforms top single judges","Cheaper jury of LLMs finds more reasoning flaws","Open-weight panel tops frontier lone judges","Multi-model consensus spots reasoning errors better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000464,"raw_usage":{"total_tokens":2424,"prompt_tokens":1156,"completion_tokens":1268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":772,"completion_tokens_details":{"reasoning_tokens":1211}},"tokens_in":772,"tokens_out":1268,"duration_ms":8277,"temperature":1.0,"reasoning_tokens":1211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:04:56.938002+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh, never-seen set of long reasoning traces with human step-level defect labels (or a held-out split of Hard2Verify), freeze the Phase 1 prompt, severity threshold, and propagation rule before reading any labels, then run the 3×gpt-oss-120b jury and an opus-4.6 solo judge on the same records. If the jury's Balanced F1 does not exceed opus-4.6 by at least the margin reported here (roughly 8.6 points), the central claim that an open-weight jury outperforms frontier single judges at reasoning-defect detection fails.","supporting_citations":[{"cited_title":"Hard2Verify: A step-level verification benchmark for open-ended frontier math","cited_arxiv_id":null,"evidence_quote":"Hard2Verify provides the 200 human-annotated records, gold error-step labels, and Balanced F1 protocol on which the headline comparisons are scored."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeltaBench supplies the second set of long chain-of-thought traces with human defect annotations used to test generalization beyond Hard2Verify."},{"cited_title":"Claude opus 4.6 system card","cited_arxiv_id":null,"evidence_quote":"The opus-4.6 system card defines the frontier single-judge baseline and its per-million-token pricing for the accuracy and cost comparisons."}],"review_version":1}