{"id":"75c662e3-ccf7-4d9e-9471-a2dade3f3496","arxiv_id":"2607.01240","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Supplying the true error count in a prompt raises LLM Count-F1 by ~0.22 but edit-level F0.5 by only ~0.04, so count agreement mostly measures prompt compliance.","lead":"This paper shows that when an evaluation prompt tells an LLM how many errors a text contains, the model tends to echo that number, inflating count-based F1 scores even though its actual error-finding quality barely improves. The result matters because LLM proofreading and document-review evaluations often use error counts as a quality proxy, and this study shows that proxy can be distorted by the prompt itself.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The replication's edit-F0.5 change (+0.042 vs +0.218 Count-F1) rests on a custom exact-tuple, per-sentence-best-reference scorer; the paper's own description-derived pipeline shows Claude S.4.6 with no inflation gap — official ERRANT multi-reference re-scoring should confirm the claim.","rationale":"The paper's central claim is that the prompt supplies the target used by Count-F1, so high count agreement is not evidence of better localization. The structural half of this claim is a design fact: in the Anchored condition the reference count appears verbatim in the prompt, so near-ceiling Count-F1 is expected. The empirical half — localization does not improve commensurately — is carried by the replication's differential (+0.218 Count-F1 vs +0.042 edit F0.5), so the edit measurement is the load-bearing piece. I examined the candidates: the unverifiable proxy and single-inference design primarily threaten the GPT/Claude-vs-Gemini family attribution and reproducibility, but the compliance phenomenon itself would survive even re-labeled endpoints; the Gemini 2.5 missing-block asymmetry affects one of six models and is disclosed. The strongest threat to the central claim is the edit-scoring pipeline itself. The paper's own description-derived overlap analysis shows Claude S.4.6 with Blind→Anchored changes of +0.235 (count) and +0.232 (edit) — the inflation gap disappears for that model — whereas the ERRANT exact-tuple replication shows +0.156 vs +0.024. Because the passage set and scoring pipeline both differ, this does not refute the replication, but it shows the headline differential is not stable across the paper's own measurement choices, and the replication deliberately uses exact tuple matching with per-sentence best-of-two-reference selection rather than the official ERRANT/M2 max-match multi-reference procedure. A re-scoring with the official scorer is the decisive, low-cost check: it isolates whether the +0.042 edit change reflects genuine near-invariance of localization or an artifact of strict matching and the empty-prediction policy. The GPT/Claude compliance results and the four clean GPT/Claude replication contrasts keep the qualitative recommendation (blind prompts, span-aware metrics) intact even under a pessimistic outcome, so this is a verification concern rather than a refutation. The reader's CONDITIONAL verdict already accommodates additional verification; I therefore keep it unchanged while flagging the specific re-scoring as the priority.","tokens_in":16346,"tokens_out":20921,"duration_ms":199393,"concrete_test":"Re-score the released Path-B corrected texts (83 passages, all models, Blind and Anchored) with ERRANT 3.0.0's official multi-reference scorer using both CoNLL-2014 annotators with max-match (errant_m2 -multi / the M2 scorer's -max_match), and recompute the Blind→Anchored edit F0.5 change per model and the six-model average. Compare with the paper's exact-tuple/per-sentence-best values (average +0.042; Table 13). If the official scorer raises the average change materially (e.g., to roughly +0.1 or more), the 'no commensurate localization gain' claim needs quantitative softening; if it stays near +0.04–0.05, the claim is robust. Also verify whether imputing Gemini 2.5's 21 extra missing Anchored blocks changes its negative edit change materially.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — prompt-supplied counts inflate Count-F1 without commensurate localization gains — rests on the 83-passage replication contrast: Blind→Anchored raises Count-F1 by +0.218 but two-reference ERRANT edit F0.5 by only +0.042. That edit-change estimate is pipeline-sensitive, and the paper's own two pipelines disagree. On the 123-passage description-derived overlap metric (Table 10), Claude S.4.6 shows Blind→Anchored changes of +0.235 Count-F1 vs +0.232 edit F0.5 — effectively no inflation gap — while the 83-passage ERRANT exact-tuple pipeline (Table 13) shows +0.156 vs +0.024 for the same model. Gemini 3.1 shows a +0.159 edit gain in the replication with a bootstrap contrast crossing zero (Table 12). The replication's edit metric is a custom exact-tuple matcher with per-sentence best-of-two-reference selection, explicitly 'not the official M2 max-match procedure' (Limitations), and missing corrected-text blocks are scored as empty predictions, asymmetrically penalizing Gemini 2.5's Anchored condition (44/83 vs 23/83 missing blocks, §A.5). Because the edit-change magnitude is the load-bearing empirical quantity for 'no commensurate localization gain,' its sensitivity to scoring choices is the central threat: if the official ERRANT multi-reference (max-match, alignment-based) scorer credits Anchored-condition improvements that exact tuples miss, the differential narrows. The paper's transparency about these choices mitigates but does not resolve the concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ErrorBench tests whether a supplied error count distorts count-based evaluation of LLM error detection. The paper runs six LLMs over five prompt conditions on 123 retained CoNLL-2014 passages (plus an 83-passage corrected-text replication), holding passages fixed while varying the stated count from Blind to Anchored to Mislead-Over/Under. It reports that Mislead-Over versus Mislead-Under moves reported counts by 82–94% of the four-error prompt difference for GPT/Claude models and by only 0.14–0.21 for Gemini models; that Anchored prompts push Count-F1 near 1.0; and that in the replication Blind→Anchored raises Count-F1 by 0.218 on average while a two-reference ERRANT-based edit F0.5 rises by only 0.042, with a paired-bootstrap contrast above zero for five of six models. The paper concludes that count agreement supplied by the prompt is not evidence of better localization and recommends blind prompts plus span-aware metrics.","tokens_in":16690,"tokens_out":6035,"duration_ms":58163,"significance":"If the empirical estimates hold, this is a useful and practical evaluation-methodology result: it demonstrates a concrete failure mode in count-based LLM proofreading evaluation and gives actionable safeguards. The study has genuine strengths: paired within-passage prompt contrasts, BH correction, a seeded paired bootstrap, a second evaluation path using ERRANT, a documented two-reference sensitivity rule, and an unusually candid Limitations section. The paper is transparent about the custom nature of its span scoring and about missing-output policies. However, the paper's central quantitative claim—that prompt-supplied counts inflate Count-F1 without commensurate localization gains—depends on an edit-scoring pipeline whose model-level conclusions are not fully corroborated by the paper's own alternative pipeline. The manuscript therefore needs a targeted rescoring or a substantial tempering of the claim, but the underlying evaluation-design warning is defensible.","major_comments":[{"comment":"The load-bearing estimate—Blind→Anchored raises Count-F1 by 0.218 but edit F0.5 by only 0.042—uses a custom exact-tuple matcher with per-sentence higher-F0.5 reference selection, explicitly 'not the official M2 max-match procedure.' Exact-tuple matching is stricter than official ERRANT/M2 alignment and could miss Anchored-condition edits that match a reference under max-match scoring. The paper's own description-derived pipeline (Table 10) already disagrees at the model level: for Claude Sonnet 4.6, Blind→Anchored changes Count-F1 by +0.235 and overlap F0.5 by +0.232, i.e., essentially no inflation gap. Please rescore the 83-passage outputs with official ERRANT multi-reference scoring and report model-level contrasts, or restrict the abstract's 'without commensurate localization' claim to the custom scoring pipeline.","section":"§4.6, Table 12, Limitations"},{"comment":"Missing corrected-text blocks are scored as empty predictions, and this policy is strongly asymmetric for Gemini 2.5: 23/83 Blind cells are missing but 44/83 Anchored cells are missing. This mechanically lowers Anchored edit F0.5 for that model, inflating both its count-vs-edit contrast and the six-model average reported in the abstract. The bootstrap contrast for Gemini 2.5 (+0.156, CI [0.080, 0.240]) may be partly an artifact of this asymmetry. Please report the average and per-model contrasts with Gemini 2.5 excluded, and/or with missing blocks treated as missing rather than empty, to show the headline +0.218/+0.042 gap is not driven by the asymmetric omission pattern.","section":"§A.5, Table 13"}],"minor_comments":[{"comment":"'whicherrors' is missing a space ('which errors').","section":"Introduction"},{"comment":"The caption reads 'M2F0.5' but the metric is 'description-derived M2-style'; adding 'description-derived' avoids implying the official CoNLL-2014 M2 scorer.","section":"§4.5, Table 4 caption"},{"comment":"The sentence 'The high count is repeated exactly in 93–100% of GPT/Claude outputs and 0% of Gemini outputs' is computed from one temperature-0 inference per cell through a proxy. This should be phrased as 'in this protocol's single-sample outputs' to avoid overstating run-to-run stability.","section":"§4.2"},{"comment":"'Path B' is used without definition; define it at first mention (the corrected-text replication) rather than only in the appendix.","section":"§3.1 / §4.6"}],"recommendation":"major_revision","confidential_remarks":"This is an honest and unusually transparent paper, and the direction of the finding is likely correct. The main risk is that the headline differential is generated by a custom exact-tuple scorer combined with an asymmetric missing-block policy; the paper's own alternative pipeline fails to show the differential for one model. If the authors add official ERRANT multi-reference rescoring of the 83-passage outputs and a sensitivity analysis around missing blocks, the paper would be much stronger. I would not reject on the current evidence, but the load-bearing empirical estimate needs reinforcement before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful thing: this paper demonstrates cleanly that when you tell an LLM the expected error count, Count-F1 against that count goes up a lot while span-level edit metrics improve only a little. The paired Mislead-Over vs Mislead-Under contrast (82–94% of the four-error prompt shift for GPT/Claude, ~0.15 for Gemini) is a strong, simple demonstration that these models are echoing the supplied number. The 83-passage replication with ERRANT extraction fits the same story: Blind→Anchored raises Count-F1 by 0.22 on average but edit F0.5 by only 0.04, with bootstrap contrast above zero for five of six models. That is a real, citable result for evaluation design.\n\nThe paper also earns credit for being transparent: it flags the custom exact-tuple scoring, the single temperature-0 inference per cell, the proxy routing, and the post-collection boundary exclusions. The case studies are useful.\n\nNow the soft spots. The exact magnitude of the count–edit gap is not as stable as the abstract implies. The paper's own description-derived pipeline (123 passages) shows a much smaller average gap (roughly +0.16 vs +0.06), and one model, Claude Sonnet 4.6, shows essentially no gap there (Count +0.235, edit +0.232). In the Path B replication, the gap for that model is large (+0.156 vs +0.024), but the two pipelines disagree. That doesn't kill the thesis — the aggregate direction still holds — but it means the \"+0.22 vs +0.04\" headline is pipeline-dependent. A re-scoring with the official ERRANT multi-reference max-match procedure would settle it.\n\nAlso, the metadata abstract's \"up to 0.79/0.96 F1 Inflation\" is misleading: those are raw cross-metric differences the body itself cautions against interpreting that way. The full-text abstract avoids them, so the posted abstract should be reconciled. Finally, the Gemini 2.5 missing corrected-text blocks (44/83 in Anchored vs 23/83 in Blind) asymmetrically penalize the Anchored edit score, so that model's contrast is partly an artifact of missing outputs, as the paper acknowledges.\n\nNone of this undermines the practical recommendation: don't put expected counts in evaluation prompts, and pair count metrics with span-aware ones. The evidence is solid for the four GPT/Claude endpoints and plausible more generally. This is a paper a serious editor should send to peer review — it's methodologically interesting, honestly reported, and the community needs the reminder. I'd cite it (with a caveat about the scoring pipeline) and bring it to reading group.","headline":"A well-designed stress test showing prompt-supplied error counts inflate count-based F1 far more than span-level metrics, though the exact magnitude is pipeline-sensitive and the posted abstract overstates the raw gap.","tokens_in":17267,"tokens_out":4611,"would_cite":true,"duration_ms":40192,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stating the expected error count in a prompt raises count-based F1 by 0.22 while true edit-level detection rises by only 0.04.","keywords":["count-based F1","numeric anchoring","prompt sensitivity","LLM evaluation","error detection","span localization","ERRANT","F1 inflation"],"falsifier":"Re-run the 83-passage Blind→Anchored experiment with multiple temperature-0 draws per cell through a direct, verifiable API route. If the mean Count-F1 gain no longer exceeds the mean ERRANT F0.5 gain by a margin comparable to the reported 0.218 vs 0.042, or if the bootstrap contrast crosses zero, the central claim fails. A pre-registered replication using the official M2 scorer rather than custom matching would test whether the gap survives standard evaluation tooling.","tokens_in":16131,"feed_emoji":"🧮","tokens_out":6338,"duration_ms":55479,"temperature":0.7,"pith_summary":"This paper argues that count-based F1, a common metric for LLM error detection, can be inflated by simply telling the model how many errors to expect. In a controlled stress test across six large language models and five prompt conditions, supplying the true count raises Count-F1 by about 0.22 on average, while a span-aware edit score (ERRANT F0.5) rises by only 0.04 under the same prompt change. Because the prompt itself supplies the target the metric measures, high count agreement is not evidence that the model localizes errors better. The authors introduce ErrorBench, a protocol for detecting this evaluation failure, and recommend blind prompts plus span-aware metrics for proofreading evaluations.","feed_headline":"Stating error counts inflates LLM detection F1 by 0.22","feed_subtitle":"Under the same prompt change, span-level edit F0.5 rises only 0.04 — so high count agreement is not better localization.","key_machinery":"The central mechanism is the prompt-supplied numeric anchor: a single sentence ('This text contains exactly N errors') that the model treats as a target. The count-based F1 metric then scores the model against the very number the prompt provided, so agreement can be placed in the prompt rather than earned by the model. The paper isolates this by comparing Blind, Informed, Anchored, and two Misleading count conditions on identical passages, and corroborates it with an ERRANT-based edit score that measures span-level localization.","core_discovery":"The paper's central discovery is that count-overlap F1 is a compliance metric as much as a detection metric: when a prompt states 'this text has exactly N errors,' the model's reported count tracks N, and Count-F1 approaches 0.99–1.00, while the corresponding improvement in edit-level localization (measured by ERRANT F0.5 against two references) is roughly five times smaller. The quantitative anchor is the Blind→Anchored contrast: on 83 corrected-text passages, Count-F1 rises by 0.218 on average, edit F0.5 by 0.042, with a paired bootstrap contrast above zero for five of six models. The paper frames this as 'F1 Inflation' — the apparent score gain is an artifact of the prompt, not improved d","pith_inferences":["A natural next experiment is a dose–response curve: varying the supplied count by more than ±2 would show whether the reported count follows the anchor linearly or saturates; the paper's Mislead conditions only test ±2.","The small count response of the Gemini family may reflect a stronger undercount prior rather than resistance to anchoring; the paper itself flags this confound. A follow-up using balanced priors could separate compliance from prior.","If the inflation pattern transfers beyond grammar error detection, count-based accuracy in other audit tasks (code-review defect counts, factual error counts) may also be inflated by prompts that state expected numbers."],"forward_implications":["LLM proofreading and document-review evaluations should stop putting expected error counts in prompts; doing so turns count metrics into measures of instruction-following.","Count-F1 and exact-count accuracy should be reported as auxiliary calibration metrics, not as evidence of detection quality.","Evaluations should pair count metrics with span-aware scores (M2, ERRANT, or equivalent) to avoid inflated conclusions.","Deployed review systems should monitor how often outputs reproduce counts supplied by users or templates, since this behavior can mask real detection failures."],"fun_headline_variants":["Error-count prompts inflate LLM F1 by 0.22, not localization","Stating 'N errors' boosts detection F1 0.22 but edit F0.5 only 0.04","LLM F1 inflation: numeric anchors raise count score 0.22, span score 0.04","Pre-stated error counts overstate LLM detection F1 by 0.22","Prompting with error counts inflates F1, not correctness: 0.22 vs 0.04"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central contrast assumes that one temperature-0 inference per cell, delivered through a third-party API proxy, is a faithful sample of each model's behavior, and that treating missing corrected-text blocks as empty predictions does not unfairly tilt the edit-score comparison.","fun_headline_variants_meta":{"raw":{"variants":["Error-count prompts inflate LLM F1 by 0.22, not localization","Stating 'N errors' boosts detection F1 0.22 but edit F0.5 only 0.04","LLM F1 inflation: numeric anchors raise count score 0.22, span score 0.04","Pre-stated error counts overstate LLM detection F1 by 0.22","Prompting with error counts inflates F1, not correctness: 0.22 vs 0.04"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00036,"raw_usage":{"total_tokens":1805,"prompt_tokens":791,"completion_tokens":1014,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":885}},"tokens_in":535,"tokens_out":1014,"duration_ms":9197,"temperature":1.0,"reasoning_tokens":885,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T14:58:43.654222+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 83-passage Blind→Anchored experiment with multiple temperature-0 draws per cell through a direct, verifiable API route. If the mean Count-F1 gain no longer exceeds the mean ERRANT F0.5 gain by a margin comparable to the reported 0.218 vs 0.042, or if the bootstrap contrast crosses zero, the central claim fails. A pre-registered replication using the official M2 scorer rather than custom matching would test whether the gap survives standard evaluation tooling.","supporting_citations":[],"review_version":2}