{"id":"16db3861-bf59-4d82-9728-fbe5d0dec0d8","arxiv_id":"2608.05353","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"Making a persisted evidence record the exclusive input to a later LLM verdict reduces preference agreement and order robustness relative to one-call structured judging.","lead":"This paper tests whether freezing an LLM judge's written evidence and feeding only that record to a later decision call hurts evaluation quality. Across 24,000 judgments it finds that locked evidence lowers agreement with human preferences and increases answer-order flips, while simply asking for structured evidence in one call does not.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Untested control separates answer removal from context removal; 'should not replace source answers' may be overgeneralized.","rationale":"The paper's empirical pattern is strong and the negative result is credible. The reader correctly flags the bundled intervention and non-deterministic decoding as a fragile premise. My concern is more specific: the decision call in evidence lock loses the task context, not just the candidate answers. The abstract and discussion frame the conclusion as 'should not replace the source answers,' but the operationalization replaces the entire source input, including the question. This mismatch means the central practical claim is not directly tested. A control that keeps the context (or provides source access) while presenting the frozen evidence would isolate whether the degradation is due to removing the answers or merely due to the impoverished decision input. Adding such a control is a reasonable condition for fully endorsing the 'replace source answers' conclusion. The paper's own limitations acknowledge the bundling, so my read does not reject the work; it asks for a specific control to sharpen the claim.","tokens_in":6230,"tokens_out":13039,"duration_ms":118145,"concrete_test":"Run a new protocol on the same 1,500 comparison IDs and two judges: Evidence Lock + Source Access (ELSA). First call extracts evidence exactly as in evidence lock. Second call receives the original task context and both answers in the same format as standard judging, plus the locked evidence, and returns A/B/tie. Compute ELSA vs. structured and ELSA vs. evidence lock contrasts for agreement and order inconsistency. If ELSA is significantly better than evidence lock and close to structured, the degradation is attributable to source removal. If ELSA resembles evidence lock, the information-bottleneck/answer-replacement account is not supported, and the order-inconsistency difference may be driven by output length or stochasticity. As a secondary check, run each protocol three times on a 100-comparison subset with the same order to measure same-order repeatability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's key contrast (evidence lock vs. structured judging) bundles at least three changes: an extra call, a longer total output, and removal of both the original candidate answers and the task context from the decision call. The abstract's practical takeaway—'persisted evidence should not replace the source answers'—is specifically about the answers, but the experiment removes the context as well (Appendix A: decision call 'Using only the locked evidence and criteria below'). A judge that receives only a model-extracted evidence record, without the original question, may be unable to evaluate the criteria; the observed 4–6 pp agreement drop could therefore reflect context deprivation rather than evidence replacing the answers. The paper acknowledges the bundled intervention but does not include a control that retains source access (or even only the task context) at decision time. The reader's non-determinism concern is a related facet: longer, multi-call outputs could also inflate order inconsistency via stochasticity alone. Without a source-access control, the central recommendation that the record should 'not replace the original pair' is underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a repeated-measures experiment on LLM-as-a-judge protocols. Across 1,500 comparisons sampled from HelpSteer3, FeedbackQA, and CoVal, each evaluated under four protocols (standard pairwise, structured one-call, two-call evidence lock, three-call pointwise lock), in both answer orders, with Claude Sonnet 4.5 and GPT-5, the authors measure agreement with released human preferences and answer-order inconsistency. They find that structured one-call judging is close to standard judging, while evidence lock reduces preference agreement by 4.3 and 5.6 percentage points relative to structured judging and increases answer-order inconsistency by 9.5 and 8.0 points. Pointwise locking is also harmful. The paper concludes that persisted evidence can support auditability but should not replace the source answers at decision time.","tokens_in":6376,"tokens_out":7213,"duration_ms":69306,"significance":"If the main finding is robust, this is a valuable negative result for the design of auditable LLM-as-a-judge pipelines. The study has clear strengths: a paired, repeated-measures design over 24,000 judgments; bootstrap confidence intervals computed over comparison IDs; replication across six judge/dataset cells; and an artifact that includes prompts, seeded comparison IDs, both answer orders, stage records, and comparison-level bootstrap code. The analysis is direct and non-circular: agreement and inconsistency are computed against released external human labels, and there is no fitted model or derived quantity that could manufacture the contrasts. The paper is also appropriately cautious about not inferring internal reasoning order from visible output order. The main weakness is that the headline intervention is bundled, and the abstract's practical recommendation is stated more narrowly than the experimental manipulation supports.","major_comments":[{"comment":"The central contrast between evidence lock and structured judging removes both the candidate answers and the task context from the decision call: Appendix A's evidence-lock decision prompt says \"Using only the locked evidence and criteria below, select A, B, or tie,\" while the structured call retains the original context and both answers. The observed 4.3/5.6 percentage-point agreement drop and 9.5/8.0-point inconsistency increase therefore cannot establish the abstract's specific recommendation that persisted evidence \"should not replace the source answers\"; the degradation could equally come from losing the task context. Section 6's suggestion that a final judge can inspect both the artifact and the source answers is a plausible design but is not directly tested. I recommend adding at least one decision-time control that retains source access (e.g., locked evidence plus the original context without the answers, or locked evidence plus the answers without the context) so that answer removal and context removal are separated.","section":"§5.1 and Appendix A"},{"comment":"Because neither vendor accepts temperature 0 with reasoning enabled, decoding is nondeterministic, and locked protocols make more calls with substantially longer outputs (Section 5.4 reports mean completion tokens of 4,183 for evidence lock and 6,825 for pointwise lock, versus 1,279 for standard and 1,864 for structured judging). The AB/BA inconsistency metric therefore compares protocols that differ not only in access to source answers and context but also in the amount of sampling noise per judgment. Without a same-order test-retest control (e.g., running a subset of comparisons twice under the same display order in each protocol), the increased inconsistency cannot be attributed uniquely to the frozen interface; it may partly reflect decoding stochasticity amplified by longer, multi-call outputs. I recommend adding a repeat-reliability control or explicitly modeling protocol length and number of calls as covariates.","section":"§4.2 and §5.2"}],"minor_comments":[{"comment":"The per-dataset results are presented as point estimates only; since the bootstrap infrastructure already exists, reporting the six cell-level contrasts with 95% intervals would help readers judge whether the negative effect is homogeneous across sources or driven by a particular dataset.","section":"§5.3"},{"comment":"The paper does not report tie rates by protocol. Because agreement is computed against strict A/B labels, a protocol that produces more ties will show lower agreement even if its underlying preference accuracy is unchanged. Reporting tie rates would clarify whether evidence lock reduces correct preferences or merely increases abstentions.","section":"§5.1"},{"comment":"Table 3 reports unsupported-claims rates without confidence intervals; the text notes that the evidence-lock change is inconclusive, but intervals for all three contrasts would make the exploratory analysis easier to interpret.","section":"Appendix C"},{"comment":"The reference to Shi et al. contains \"V osoughi\" with an extra space; it should read \"Vosoughi.\"","section":"References"},{"comment":"The paired contrast labels run into the numeric columns (e.g., \"Structured−standard+0.7\"); adding explicit column separators or spacing would improve readability.","section":"Table 2"},{"comment":"The prompt templates do not specify which content goes into the system message versus the user message. Since proprietary API behavior can depend on role boundaries, specifying this split would improve reproducibility.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed, reproducible negative result with a clean paired design and appropriate bootstrap inference. My main reservation is that the abstract's practical takeaway ('should not replace the source answers') is underdetermined by the experimental manipulation, which removes both answers and context, and the nondeterministic decoding concern further weakens the causal reading of the inconsistency result. I would be comfortable with major revision if the authors add at least one control separating answer removal from context removal, or substantially narrow the stated claim to the full evidence-lock workflow. The nondeterminism control is strongly advisable but perhaps could be handled as a supplementary reliability analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is a solid empirical negative result with a headline that slightly overreaches its own design. The core finding—making a persisted evidence record the exclusive input to a later verdict call reduces agreement with human preferences and increases order inconsistency—holds up. The paired repeated-measures setup across 24,000 judgments, two judges, three datasets, and both answer orders is careful. Bootstrap CIs over comparison IDs exclude zero for the main contrasts, and the direction replicates in all six judge/dataset cells. Structured one-call judging staying close to standard judging is a useful control: it shows explicit evidence elicitation isn't the problem; freezing and exclusivity are. The paper also ships prompts, seeded comparison IDs, and bootstrap code, which is real reproducibility evidence.\n\nThe soft spot is the gap between the mechanism tested and the practical takeaway. Evidence lock bundles three changes: an extra call, a longer multi-call output, and removal of both the original candidate answers and the task context from the decision call. The abstract says persisted evidence 'should not replace the source answers,' but the experiment also withholds the original question. A judge given only a model-extracted record, without the context, may simply lack the information to apply the criteria. The paper acknowledges the bundle in its limitations and calls the information-bottleneck account an interpretation, not a uniquely identified mechanism. That's honest, but the title and abstract push past it. A control that retains source access—or even just the task context—at decision time would have made the 'don't replace the answers' claim precise. Its absence is a moderate flaw, not a fatal one.\n\nThe non-deterministic decoding concern is real but minor. Since all protocols run under the same nondeterminism, and length differences are tied to the protocol itself, the effect could partly be stochasticity interacting with output length. The paper flags this too. I don't think it undermines the measured workflow-level effect.\n\nWho benefits: anyone building LLM-as-judge pipelines or studying evaluation protocols. It's a practical caution that two-call evidence persistence, as commonly implemented for auditability, is not free. The paper deserves a serious referee. I'd accept it with a request to soften the abstract to say 'persisted records should not replace source access' rather than specifically 'the source answers,' and ideally add a control.\n\nRecommendation: send to peer review, expect revision, and treat the main negative result as established.","headline":"A well-designed negative result showing frozen evidence records hurt LLM-judge agreement, but the abstract's 'don't replace source answers' takeaway overreaches because the locked condition also removes task context.","tokens_in":6872,"tokens_out":1818,"would_cite":true,"duration_ms":18208,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Persisting evidence and passing only that record to the final decision reduces LLM judges' agreement with released human preferences by 4 to 6 percentage points and raises answer-order inconsistency by 8 to 10 points.","keywords":["LLM-as-a-judge","pairwise preference","evidence lock","answer-order inconsistency","preference agreement","structured evaluation","information bottleneck","auditability"],"falsifier":"Run evidence-lock, structured judging, and an added control in which the decision call receives the locked evidence plus the original context and both answers; if the control matches standard judging on agreement and order inconsistency, source removal is the active cause, and if it still degrades, the frozen artifact itself carries the harm.","tokens_in":6017,"feed_emoji":"⚖️","tokens_out":6624,"duration_ms":53207,"temperature":0.7,"pith_summary":"This paper asks whether an LLM judge can be safely split into two stages: first write down evidence about two candidate answers, then make the verdict using only that written record. Across 24,000 judgments on three preference datasets and two reasoning-capable models, the answer is no: making the persisted record the exclusive input lowers agreement with released human preferences and makes the verdict more sensitive to answer order. The paper isolates the reason by comparing evidence lock with structured one-call judging, which also asks for evidence but lets the final call see the original answers. The finding matters for anyone building auditable evaluation pipelines: persisted evidence can support logging and review, but it should not replace the source answers at decision time.","feed_headline":"Locked evidence drops LLM judge agreement by 4–6 points","feed_subtitle":"24,000 judgments suggest persisted records aid audits, not verdicts: keeping source answers preserves quality.","key_machinery":"The central object is the evidence-lock protocol, a two-call intervention that makes information flow observable: the first call reads the context and both answers and writes structured evidence without selecting a winner; the second call receives only that record, with the original context and answers absent. Its contrast term is structured one-call judging, which also elicits criteria and evidence but keeps source access at verdict time. The difference between these two protocols is the load-bearing comparison: both use explicit evidence, only the lock freezes the artifact and removes original answers, so degradation in the lock isolates the information bottleneck without relying on claims about hidden reasoning order. Pointwise lock extends the same logic by separating the two answers during extraction.","core_discovery":"The paper's central claim is that a frozen natural-language evidence record is not a sufficient interface for a later pairwise verdict. The evidence-lock protocol, in which a first call extracts criteria and evidence without choosing a winner and a second call sees only the serialized record, reduces preference agreement by 4.3 and 5.6 percentage points relative to structured one-call judging for Claude Sonnet 4.5 and GPT-5, and raises answer-order inconsistency by 9.5 and 8.0 points. Pointwise lock, which evaluates each answer separately and then compares frozen assessments, is also harmful. Because structured judging, which requests the same kind of evidence but keeps source access, remains close to standard judging, the degradation is attributed to removing source access from the final call, not to requiring evidence in the first place. The authors state the practical conclusion directly: persisted evidence can support auditability, but it should not replace the source answers at decision time.","pith_inferences":["A direct test of the bottleneck account would be a control where the second call receives both the locked record and the original answers; the paper does not run this control, but its logic predicts agreement and order robustness should recover.","Because the vendor APIs do not allow greedy decoding, the longer, multi-call locked protocols may carry extra generation instability; repeated sampling per comparison would separate that instability from loss of source access.","The frozen-interface result plausibly extends beyond judging to any chained-agent pipeline that passes a summary to a later decision step, suggesting such pipelines should preserve access to raw context when later decisions matter."],"forward_implications":["If the central claim holds, audit-friendly evaluation systems should show the final judge both the persisted evidence record and the original answers.","Structured one-call evidence elicitation remains a safe way to request criteria and evidence, since it stays close to standard judging on agreement and order robustness.","Locked protocols buy nothing for evaluation quality: they use more calls and tokens while lowering agreement and raising order inconsistency.","Evaluating the two answers separately does not remove order sensitivity, because pointwise lock stays substantially less robust than standard judging.","The degradation generalizes across all six judge-dataset combinations tested, so it is not a quirk of one dataset or one judge."],"supporting_citations":[{"why":"Establishes LLM-based pairwise evaluation and documents position bias, providing the standard-judging baseline the paper compares against.","marker":"(Zheng et al., 2023)"},{"why":"Supplies the systematic position-bias study and counterfactual answer-order swap used to measure order inconsistency.","marker":"(Shi et al., 2025)"},{"why":"Introduces G-Eval-style structured evaluation with explicit reasoning, the template for the structured one-call protocol.","marker":"(Liu et al., 2023)"},{"why":"Motivates fine-grained criteria-based evaluation that the structured and locked protocols draw on.","marker":"(Kim et al., 2024)"},{"why":"Places an explicit plan before the final evaluation, the closest prior workflow to evidence lock and the motivation for testing frozen artifacts.","marker":"(Saha et al., 2025)"},{"why":"Shows visible explanations need not faithfully reflect decision order, motivating an intervention defined by call boundaries rather than hidden reasoning.","marker":"(Turpin et al., 2023)"}],"fun_headline_variants":["Evidence locking cuts LLM judge accuracy by up to 6 points","Frozen evidence records worsen LLM judge verdicts","Keep source answers: evidence records fail as judge input","Evidence locking impairs LLM judges; structured elicitation works","LLM judges: persisted evidence is for audits, not verdicts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on the assumption that the proprietary APIs' nondeterministic, non-greedy decoding does not interact with the longer, multi-call structure of locked protocols to produce the observed drops in agreement and rises in order inconsistency.","fun_headline_variants_meta":{"raw":{"variants":["Evidence locking cuts LLM judge accuracy by up to 6 points","Frozen evidence records worsen LLM judge verdicts","Keep source answers: evidence records fail as judge input","Evidence locking impairs LLM judges; structured elicitation works","LLM judges: persisted evidence is for audits, not verdicts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2380,"prompt_tokens":924,"completion_tokens":1456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1386}},"tokens_in":540,"tokens_out":1456,"duration_ms":10332,"temperature":1.0,"reasoning_tokens":1386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:43:34.055708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run evidence-lock, structured judging, and an added control in which the decision call receives the locked evidence plus the original context and both answers; if the control matches standard judging on agreement and order inconsistency, source removal is the active cause, and if it still degrades, the frozen artifact itself carries the harm.","supporting_citations":[{"cited_title":"and Zhang, Hao and Gonzalez, Joseph E","cited_arxiv_id":null,"evidence_quote":"Establishes LLM-based pairwise evaluation and documents position bias, providing the standard-judging baseline the paper compares against."},{"cited_title":"Learning to Plan & Reason for Evaluation with Thinking-","cited_arxiv_id":null,"evidence_quote":"Places an explicit plan before the final evaluation, the closest prior workflow to evidence lock and the motivation for testing frozen artifacts."}],"review_version":3}