{"id":"42fa6450-eb70-4b88-b775-121e783c0829","arxiv_id":"2608.07813","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On frozen candidate pools, constraining an LLM judge with an evidence-certificate gate is worth more than the judge is; unconstrained, the same judge can be worse than majority vote.","lead":"This paper studies what happens when an LLM is used not just to grade answers but to choose which answer a system returns. It finds the same judge can hurt accuracy under full authority and help when its power is limited by an evidence check.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The repair candidate r in Eq. (3) is never specified; its generation appears to consume extra inference budget, so the 'no change to budget' claim and the paired-selector comparison are unverified for the derive-repair branch that produces the main GSM8K gain.","rationale":"The paper's central claim has two parts: the negative result (an unconstrained scalar judge buys little or hurts) and the positive result (the same judge under EL-DGR helps). The negative result is well supported by matched pools and honest reporting. The positive result, however, depends critically on the repair branch, because the only strongly significant contrast (GSM8K, p=0.0026) is driven by derive-repair wins, and the confirmation audit attributes 4 of 8 overrides to repairs. If r is produced by an additional inference call, then EL-DGR is not the same judge, candidates, and budget as the baselines; it is the same judge plus an extra generator and extra compute. That would not disprove the broader message that admissibility constraints can help, but it would invalidate the specific paired comparison and the abstract's 'no change to budget' phrasing. The reader's weakest_assumption already names the unspecified repair generation; this stress test elevates it to the primary concern because it bears on internal validity rather than statistical power. The certificate-precision and overlap concerns are also real but secondary: the certificate is explicitly conservative and its admitted limitations are acknowledged, whereas the repair generator is not described at all. The recommended verdict remains CONDITIONAL because the issue is resolvable by inspecting the released code; no rejection is warranted on the current text alone, but acceptance requires the missing specification or a matched-compute rerun.","tokens_in":11251,"tokens_out":4738,"duration_ms":43467,"concrete_test":"Inspect the released artifact (Appendix C) and locate the code path that builds r. If r is produced by prompting deepseek-r1:7b or another model, or by an external solver, rerun the GSM8K full-budget comparison with a matched baseline: give majority vote and the scalar judge the same extra derive call (or restrict EL-DGR's r to the frozen pool, e.g., the best certified candidate among the existing G completions). If the +2.8pp GSM8K gain and the 4/30 repair overrides vanish when compute is matched, the 'decision rule alone' conclusion is unsupported. If the code shows r is already a frozen-pool candidate, rerun the analysis with an explicit assertion that no additional forward passes are used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison is between selectors that consume 'the identical frozen pool' (Section 5), and the abstract claims 'no change to the judge, candidates, or the budget.' But Eq. (3) selects r, an 'independently derived and checked repair,' and Section 6.3 reports a separate derive-gate-repair call costing 2.91s per pilot question and 1.36s / 4.99s per full-budget question. The paper never specifies what produces r. If r is generated by an additional LLM call or solver, then EL-DGR is not budget-matched to majority or scalar judge: it has access to a new candidate and new inference that the baselines are denied. The main significant result (GSM8K +2.8pp, p=0.0026) comes from the arithmetic derive-repair branch (17 paired wins, 3 losses), and on the confirmation split 4 of 8 overrides are repairs. Without knowing whether r is a frozen candidate, a verifier's output, or an extra generative call, the improvement cannot be attributed to the decision rule; it may be an artifact of additional compute. Section 4.3's phrase 'the verifier must expose a derived numeric answer' presupposes the generator rather than specifying it. This is not a statistical quibble: it threatens the internal validity of the headline 'no change to budget' and the claim that judge usefulness, not judge accuracy, is what changed. If r is instead drawn from the frozen pool, this concern dissolves; the paper currently provides no way to tell.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies LLM judges as selectors over a frozen candidate pool. It defines a scalar judge baseline, a majority-vote baseline, and EL-DGR, a decision rule that uses an extractive evidence certificate (retrieval) or an executable numeric consistency check (arithmetic) to constrain when the judge's preferred candidate may override the consensus candidate. On 500 GSM8K and 300 HotpotQA questions generated from four frozen policies, the scalar judge adds little over majority (Table 2), while EL-DGR improves over first candidate and majority (GSM8K +2.8 pp, exact McNemar p=0.0026; HotpotQA +2.0 EM, borderline). A 30-question frozen-rule confirmation split shows the scalar judge 10 EM points below majority, while EL-DGR is the best selector tested (Table 3). The paper also reports that the same seven-channel decomposition used as a gated training reward is null (Section 7). The headline conclusion is that judge usefulness depends less on judge accuracy than on the decision rule, and that bounding the judge's blast radius can be cheaper than improving the judge.","tokens_in":11551,"tokens_out":6895,"duration_ms":57954,"significance":"If the repair candidate is properly specified and the budget accounting is clarified, this is a valuable and clean empirical result: it isolates the decision rule as the manipulated variable, uses matched paired tests, and ships per-question predictions and code (Appendix C). The negative training result and the corrected channel-drop ablations are reported with appropriate caution, and Section 8 explicitly acknowledges certificate scope and statistical power. The practical message, that constraining a judge's authority can matter more than improving its accuracy, is falsifiable and directly relevant to LLM-as-judge deployment. The main reservation is that the derivation of the repair candidate r is unspecified, and the 'no change to the budget' claim therefore cannot yet be verified as stated.","major_comments":[{"comment":"The manuscript never specifies how the repair candidate r is generated. Eq. (3) returns r when c(r,E)=1 and both c(m,E)=c(s,E)=0, but the text only says r is 'an independently derived and checked repair' (Section 3) and 'the verifier must expose a derived numeric answer' (Section 4.3). This matters because the headline GSM8K gain is driven by the arithmetic derive-repair branch (17 paired wins vs. 3 losses, Section 6.1), and the confirmation audit attributes 4 of 8 overrides to repairs (Section 6.3). If r is produced by an additional LLM call or solver, then EL-DGR consumes extra inference budget that the scalar-judge and majority baselines do not, so the abstract's 'no change to the judge, the candidates, or the budget' is not established. If r is a member of the frozen pool, the authors should state this explicitly and make Eq. (3) select from the pool; the current text provides no way to tell which case holds.","section":"Section 4.3, Eq. (3); Sections 6.1 and 6.3"},{"comment":"The strict extractive certificate c(a,E)=1[a nonempty and a occurs verbatim in E] is the load-bearing admissibility filter for HotpotQA, and the paper acknowledges that exact-substring matching rejects supported paraphrases, but it never measures the certificate's precision or recall on the frozen pools. A concrete test would be to report, per pool and per question, how often a gold correct answer occurs verbatim in the BM25 K=5 evidence, and how often a non-gold candidate does. Without that measurement, the claim that the judge can only cause a regression by preferring an answer that is both wrong and verbatim-present in the evidence while the correct consensus is absent remains a structural bound rather than a verified property of this benchmark. The decision audit showing 0 correct-to-incorrect flips on n=30 is reassuring but too small to establish the mechanism.","section":"Section 8, 'Certificate scope'; Eq. (3)"},{"comment":"The cost accounting in Section 6.3 is in tension with the abstract's budget-neutrality claim. The paper reports that the 'scalar and derive-gate-repair calls' average 1.36s per GSM8K and 4.99s per HotpotQA question and that this is 'an inference-time accuracy-latency trade-off, not a free lunch,' which implies an extra call that the majority and first-candidate baselines do not make. If 'budget' in the abstract means the candidate-generation budget only, that should be stated explicitly; if it means total inference budget, the comparison is not budget-matched. The paper should specify exactly which calls each selector performs and how the derive-gate-repair call is charged, including whether r generation is included in the reported latencies.","section":"Section 6.3, Cost; Abstract and Section 1 budget claims"}],"minor_comments":[{"comment":"The 'Matched here' column uses dashes without a legend; add a note that dashes mean the system was not rerun on the frozen candidate pool.","section":"Table 5"},{"comment":"The legend order 'GRPO ORM PRM RAMSD' should be aligned with the line styles, and 'RA-MSD' should be spelled consistently instead of 'RAMSD'.","section":"Figure 2"},{"comment":"The sentence reporting '1.77s and 2.91s per pilot question (4.68s total)' should clarify whether these are mean latencies and whether the derive-gate-repair call includes generation of r.","section":"Section 6.3"},{"comment":"The model name 'deepseek-r1:7b' should be written as 'DeepSeek-R1-7B' in prose, and the reference to 'V ACS' in the bibliography contains a stray space in the title.","section":"References and running text"}],"recommendation":"major_revision","confidential_remarks":"The paired-selector design is the right one, and the honest reporting of null results is a genuine strength. The main gate to publication is the unspecified repair candidate and the ambiguous use of 'budget.' If the authors can show that r is drawn from the frozen pool, or can respecify the claim as 'no change to candidate generation budget' and re-run the matched comparison with an explicit r generator, the result would be publishable. I would ask the editor to require this clarification rather than desk-reject; the issue is fixable in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper you want to know about: Zhang et al. show that the decision rule around an LLM judge matters more than the judge's accuracy. On a frozen candidate pool, an unconstrained scalar judge is nearly worthless over majority vote (+1 pp GSM8K, +0.34 EM HotpotQA) and is the worst selector on a 30-question confirmation split. The same judge, placed under a non-compensatory evidence-admissibility rule (EL-DGR), becomes the best selector: +2.8 pp on GSM8K (p=0.0026) and +2.0 EM on HotpotQA (borderline), with an audit showing 8 overrides and zero correct-to-incorrect flips. The paired design is right: all selectors consume the same frozen candidates, and the authors honestly report the null gated-reward training result and a collapsed channel-drop ablation.\n\nWhat's genuinely new is the authority-versus-accuracy framing, treating judge usefulness as a property of the decision rule rather than of judge accuracy. That distinction is worth carrying forward.\n\nThe soft spots are load-bearing. Eq. (3) selects a repair candidate r that is never specified. It's an 'independently derived and checked repair,' and the paper reports extra latency for a 'derive-gate-repair call' (2.91s per pilot, 4.99s per full-budget HotpotQA). This contradicts the abstract's 'no change to the budget.' If r comes from an additional LLM call or solver, EL-DGR has access to a new candidate and extra compute that the baselines don't, and the main GSM8K gain from the arithmetic repair branch cannot be attributed solely to the decision rule. The paper needs to state exactly how r is produced; if it's drawn from the frozen pool, the concern dissolves.\n\nSecondary issues: the HotpotQA full-budget contrasts are borderline (p=0.07), and the paper never says whether the 300-question evaluation set overlaps the 120-question development set. The n=30 confirmation split is underpowered, as the authors acknowledge.\n\nThis is a solid paper with an important gap. It deserves a serious referee, but not acceptance as-is. The authors must specify the repair generation, report certificate precision/recall, and clarify dataset disjointness. I'd bring it to reading group and would cite the authority-versus-accuracy framing.","headline":"The authority-versus-accuracy framing is genuinely new and the paired design is right, but the unspecified repair candidate and budget mismatch undercut the headline claim; deserves a serious referee.","tokens_in":12121,"tokens_out":4183,"would_cite":true,"duration_ms":31762,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM judge embedded in a reasoning pipeline can be worse than no judge, but the same judge becomes the best selector tested when a cheap evidence certificate restricts what it may overrule; judge usefulness is a property of the decision…","keywords":["LLM-as-a-judge","decision rules","non-compensatory selection","evidence certificates","retrieval-augmented reasoning","reasoning pipelines","GRPO","HotpotQA"],"falsifier":"Take the same frozen pools and replace the exact-substring certificate with a paraphrase-tolerant one, such as NLI entailment against the retrieved evidence, while keeping the judge, candidates, and budget identical. If the gain over the scalar judge disappears or reverses, the paper's improvement is an artifact of exact-match coverage on these particular questions; if the gain survives, the non-compensatory structure is the operative cause.","tokens_in":1970,"feed_emoji":"⚖️","tokens_out":2443,"duration_ms":82567,"temperature":0.7,"pith_summary":"The paper argues that when an LLM judge is wired into a reasoning pipeline, its damage and value are set by the decision rule around it, not by its raw score. On matched frozen candidate pools, giving a scalar judge full authority buys almost nothing over majority vote and can even hurt. The authors then subordinate the same judge to a non-compensatory rule, Evidence-Locked Derive-Gate-Repair (EL-DGR), which permits the judge to override consensus only when the alternative is backed by an extractive evidence certificate, and permits a repair only when nothing else is certified. Under that rule the same judge becomes the best selector tested, improving first-candidate accuracy by 2.8 points on GSM8K and 2.0 EM on HotpotQA, with zero correct-to-incorrect flips. The larger claim is a design principle: bound the judge's blast radius rather than spend budget making it accurate.","feed_headline":"A cheap evidence gate turns a harmful LLM judge into the best selector","feed_subtitle":"Same judge, same candidates, no retraining: an evidence certificate lifts it from worst to best.","key_machinery":"The central mechanism is the evidence certificate and the lexicographic EL-DGR selection rule of Equation (3): return the repair $r$ when only $r$ is certified; return the judge's preference $s$ when $s$ is certified and the consensus $m$ is not; otherwise return $m$. For retrieval QA the certificate is $c(a,E)=\\mathbf{1}[a \\neq \\emptyset \\wedge a \\text{ occurs verbatim in } E]$; for arithmetic it is an executable check that a parsed numeric repair passes a consistency check. The rule is non-compensatory because the scalar judge's score is never allowed to trade off against the certificate; scores only rank candidates within a feasibility stratum. The complementary negative mechanism is the paper's definition of $(\\eta,\\delta)$-conflation, which formalizes how a step-level quality score can be high while every retrieved document contradicts the step.","core_discovery":"A fixed scalar LLM judge, taken as given with all its biases, becomes a reliable selection instrument only when its authority is structurally constrained. EL-DGR partitions candidate answers into feasibility strata via a cheap certificate: for retrieval QA, the answer must occur verbatim in the retrieved evidence; for arithmetic, an independently derived numeric repair must parse and pass a consistency check. The judge's scalar preference ranks candidates only within a stratum; it can never promote an uncertified high-scoring answer over an evidence-supported consensus. On a frozen-rule 30-question confirmation split, the same judge that was 10 points below majority vote under full authority is the best selector under EL-DGR (26.7 vs. 20.0 EM), and the decision audit shows only 8 consensus overrides with no correct consensus converted to error. The authors also report that decomposing the scalar into seven scored channels and gating a training reward with them is null; the decomposition pays off only when used as a feasibility partition that cannot be summed away.","pith_inferences":["The same design discipline should transfer to any pipeline where an LLM judge gates a cheaply verifiable output, such as code-execution checks or tool selection: the certificate's false-accept rate, not judge accuracy, becomes the primary design metric.","A direct follow-up experiment is to measure the certificate's precision and recall on the frozen pools; the paper admits exact-substring matching rejects paraphrases, so a paraphrase-tolerant certificate is the natural comparison.","Because the paper does not train on the lexicographic group rank induced by Equation (3), testing that rank as an explicit training reward would separate whether the benefit is purely a selection artifact or a usable learning signal.","The null training result suggests that any recombination of decomposed scores into one scalar may reinherit the conflation it was meant to remove, which predicts that process-supervision gains reported elsewhere come from protocol details rather than from the decomposition itself."],"forward_implications":["Full judge authority over a frozen pool is worth little or negative: the scalar judge gains +1.0 pp over majority on GSM8K, +0.34 EM on HotpotQA, and is 10 EM points below majority on the frozen-rule confirmation split.","The same judge under EL-DGR improves over every matched selector on both datasets, including +2.8 pp over first-candidate on GSM8K ($p=0.0026$) and +2.0 EM on HotpotQA ($p=0.070$), with no extra candidates and at most about five seconds per question of verifier latency.","A bounded judge should be deployed with a stated default, a required certificate, and a measured override rate broken down by outcome; the paper's audit reports 22/30 defaults preserved and 8 overrides with zero correct-to-incorrect flips.","Decomposing a judge into seven scored channels only helps when the decomposition is used as a feasibility partition; using the same channels as a gated training reward is empirically null and no single channel is individually necessary ($p=1.0$).","Judge usefulness and judge accuracy are loosely coupled, so before spending budget on a more accurate judge, a pipeline should constrain what the current judge may overrule."],"supporting_citations":[{"why":"Supplies the LLM-as-a-judge paradigm and pairwise-agreement benchmark that the paper contrasts with its judge-plus-rule perspective.","marker":"[Zheng et al., 2023]"},{"why":"Provides the self-consistency majority-vote baseline that EL-DGR and the scalar judge must beat.","marker":"[Wang et al., 2023]"},{"why":"Defines the GSM8K benchmark whose held-out split is used to measure arithmetic selection.","marker":"[Cobbe et al., 2021]"},{"why":"Defines HotpotQA and its official EM and F1 metrics used in the retrieval evaluation.","marker":"[Yang et al., 2018]"},{"why":"Exemplifies the scalar step-level process reward that the paper argues cannot separate logical validity from factual support.","marker":"[Lightman et al., 2023]"},{"why":"Supplies the claim-verification grounding idea behind the factual channel, moved from offline evaluation into an online decision rule.","marker":"[Min et al., 2023]"},{"why":"Motivates holding judge biases fixed rather than debiased, since self-preference is built into the judge.","marker":"[Panickssery et al., 2024]"},{"why":"Documents judge-reliability protocols that the paper contrasts with its judge-plus-rule framing.","marker":"[Gu et al., 2024]"}],"fun_headline_variants":["Evidence-locked judge outperforms majority vote on reasoning tasks","Constraint, not accuracy, makes LLM judge best selector","EL-DGR: evidence gate flips harmful judge to top picker","When a judge shouldn't decide: evidence rules beat all","Bias stays, authority bounded: EL-DGR wins with same judge"],"cache_read_input_tokens":14080,"weakest_assumption_plain":"The result stands on the certificate separating right from wrong answers in the retrieved evidence: the correct answer must appear verbatim in the top-5 retrieved documents whenever the consensus is wrong, wrong answers must not appear when the consensus is right, and for arithmetic the independently derived repair must parse and pass its consistency check.","fun_headline_variants_meta":{"raw":{"variants":["Evidence-locked judge outperforms majority vote on reasoning tasks","Constraint, not accuracy, makes LLM judge best selector","EL-DGR: evidence gate flips harmful judge to top picker","When a judge shouldn't decide: evidence rules beat all","Bias stays, authority bounded: EL-DGR wins with same judge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2534,"prompt_tokens":1176,"completion_tokens":1358,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":792,"completion_tokens_details":{"reasoning_tokens":1271}},"tokens_in":792,"tokens_out":1358,"duration_ms":12089,"temperature":1.0,"reasoning_tokens":1271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:13:02.258348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same frozen pools and replace the exact-substring certificate with a paraphrase-tolerant one, such as NLI entailment against the retrieved evidence, while keeping the judge, candidates, and budget identical. If the gain over the scalar judge disappears or reverses, the paper's improvement is an artifact of exact-match coverage on these particular questions; if the gain survives, the non-compensatory structure is the operative cause.","supporting_citations":[{"cited_title":"Hotpotqa: A dataset for diverse, explainable multi-hop question answering","cited_arxiv_id":null,"evidence_quote":"Defines HotpotQA and its official EM and F1 metrics used in the retrieval evaluation."},{"cited_title":"Factscore: Fine-grained atomic evaluation of factual precision in long form text generation","cited_arxiv_id":null,"evidence_quote":"Supplies the claim-verification grounding idea behind the factual channel, moved from offline evaluation into an online decision rule."}],"review_version":1}