{"id":"ebe1ccf0-d1ff-4e36-b501-e919e3dab09b","arxiv_id":"2608.12571","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Frontier LLMs are near-perfect at detecting wrong-case legal citations but miss most same-case wrong-pinpoint errors, because they match legal topic instead of verifying page-level support.","lead":"This paper tests whether large language models can tell when a legal citation points to a real case but the cited page does not support the claim it appears to back. Across 14 model configurations, models almost always catch a swapped-in wrong case, but they miss a large share of errors that change only the pinpoint page, accepting the citation because the topic matches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hard-corruption labels rest on 23 citations per dataset and the paper concedes some wrong-page corruptions may remain on point; if even 10–15% do, BriefMe Hard recall near 82% is close to the label ceiling and the claimed capability gap shrinks.","rationale":"The paper is a well-designed controlled study. I looked for a more fundamental flaw and did not find one: the corruption taxonomy is clear, the results are consistent across fourteen configurations and two datasets, and the page-grounded intervention supports the proposed mechanism. The load-bearing weak point is the ground-truth label for Hard corruptions. The human validation is small (23 per dataset), non-stratified, and one annotator agrees with heuristics only 69.6% of the time on BriefMe. Since Section 8 admits some corrupted citations may remain on point, the measured recall is a lower bound. On BriefMe, the top Hard recall values are close to the 84% label-agreement ceiling, so the strongest version of the claim—'even with reasoning, GPT-5.4 still misses 18% on briefs'—could be explained substantially by label noise. The CLERC numbers are far enough below the ceiling that the qualitative conclusion survives. This is exactly the condition the reader flagged, so I agree. The concrete check above would settle it. No change to the verdict: it should remain conditional on releasing the exact test files and providing a fuller, stratified human validation.","tokens_in":26898,"tokens_out":8082,"duration_ms":79350,"concrete_test":"Have three legal annotators label at least 100 Hard corrupted examples per dataset plus 50 original citations, blind to heuristic labels, with the Section D instructions. For each Hard example, record whether the quoted proposition or a close paraphrase appears on the corrupted page. Recompute Tables 2/3 Hard recall excluding examples any annotator judged on point, and again for unanimous 'not on point.' If adjusted BriefMe recall for GPT-5.4 (reasoning) and Gemini models stays near 82%, the central claim stands; if it drops materially, the Easy-to-Hard gap and the 'misses 18%' headline must be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is the Easy-vs-Hard recall gap (93–100% vs 37–61% on CLERC, 52–83% on BriefMe). This gap is measured against heuristic labels that assume every rule-based corruption is genuinely not on point. Section 3.3 validates this with only 23 citations per dataset, and Section 8 concedes that 'some corrupted citations may nevertheless remain on point by chance.' Table 10 shows average annotator agreement with the heuristic labels of only 88% (CLERC) and 84% (BriefMe), with one annotator at 77.3% and 69.6%, respectively. If a fraction e of Hard examples are actually on point, measured recall is capped at 1−e. On BriefMe, the best Hard recall is 82.7% (Gemini 2.5 Pro, Gemini 3.1 Pro) and GPT-5.4 with reasoning reaches 82.0%; with e=0.16 the ceiling is 84%, so these models are essentially at ceiling and the 'still misses 18%' conclusion may be mostly label noise. On CLERC, the 37–61% range remains well below an 84–88% ceiling, so the qualitative conclusion survives, but the size of the gap and the brief/opinion comparison are partly artifacts of unmeasured label error. The paper reports no per-example confirmation that the quoted proposition or its equivalent is absent from the corrupted page for the full Hard set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies proposition-level legal citation support verification: whether a citation to a real case actually supports the proposition for which it is offered. Using controlled rule-based perturbations of citations from CLERC (court opinions) and BriefMe (legal briefs), it creates Easy (wrong case), Medium (same-document different case), and Hard (wrong pinpoint page within the same case) corruptions, then evaluates fourteen LLM configurations. The central finding is a large difficulty gap: models catch 93–100% of Easy corruptions but only 37–61% (CLERC) and 52–83% (BriefMe) of Hard pinpoint-page corruptions. A page-grounded prompt improves Hard recall by 7–36 percentage points but consistently raises false positive rates. The paper also contributes a three-category citation taxonomy, a human validation study, and an error analysis showing that models often fabricate rationales claiming the cited page contains text that is not there.","tokens_in":27051,"tokens_out":6860,"duration_ms":71168,"significance":"If the quantitative result holds, this is a useful and important negative result for legal NLP and AI safety: it separates database-level citation checking from proposition-level support verification and gives a concrete, controlled measurement of where frontier LLMs fail. The paper's strengths are its clean corruption taxonomy, the use of two public legal corpora, the breadth of fourteen model configurations across three families, and the concrete error analysis with verbatim GPT-5 rationales. The Easy/Hard gap is large and consistent enough that the qualitative conclusion is robust even under the label-quality caveats discussed below. The main quantitative claims—especially the BriefMe recall ceiling and the briefs-vs-opinions comparison—need additional validation before they can be taken at face value.","major_comments":[{"comment":"The Hard-corruption gold labels rest on only 23 human-validated citations per dataset, with average agreement with the heuristic labels of 88% on CLERC and 84% on BriefMe, and Section 8 concedes that some corrupted citations may remain on point by chance. Since measured Hard recall is the fraction of corrupted examples flagged as erroneous, any fraction e of corruptions that are actually on point caps achievable recall at 1−e; on BriefMe, where the best baseline Hard recall is 82.7% and GPT-5.4 with reasoning reaches 82.0%, this places the strongest models essentially at the label ceiling, so the 'still misses 18%' headline may be largely annotation noise. The CLERC range (37–61%) remains well below an 84–88% ceiling, so the qualitative conclusion survives, but the reported gap size and the briefs-vs-opinions comparison are partly artifacts of unmeasured label error. Please provide full-set per-example confirmation for the Hard set (at minimum for the quoted-language cases), or report recall bounds adjusted for a directly measured e, and rephrase the BriefMe-specific claims accordingly.","section":"§3.3, §8, Tables 3 and 10"},{"comment":"The claim that briefs produce a stronger signal than opinions compares CLERC and BriefMe under different protocols: CLERC supplies preceding-paragraph context (src_document) while BriefMe does not, pinpoint coverage differs (36% vs. 23% of retained citations in Table 6), and the Hard sets are constructed differently because BriefMe has no Medium corruption. The 12–26 percentage-point advantage on Hard recall could therefore reflect contextual or sampling differences rather than document type, and Section 7 itself describes proposition specificity as an untested hypothesis. Please either rephrase this as a descriptive cross-dataset observation or control for these confounds (for example, with matched subsets) before presenting it as the answer to RQ3.","section":"§5.1–§7 (RQ3)"},{"comment":"The page-grounded prompt was designed after inspecting GPT-5 false negatives on exactly the same test set on which it is then evaluated, and no held-out prompt-development split is reported. The improvement figures in Tables 4 and 5 therefore conflate the intervention's effect with selection on the observed failure modes; the qualitative recall/FPR trade-off is the robust part, but the point estimates should be presented as in-sample prompt engineering or validated on a held-out subset before being stated as a general result.","section":"§6.2 and Appendix B.2"}],"minor_comments":[{"comment":"Section D says that for BriefMe 7 of the 10 triple-annotated cases were 'Medium corruptions,' but Section 3.2 states that Medium corruption is not available for BriefMe because the dataset lacks preceding paragraphs; please clarify whether these are same-document substitutions under a different name or correct the terminology.","section":"§D vs. §3.2"},{"comment":"Please report the number of examples in each difficulty cell; percentages without counts make it impossible to gauge uncertainty, especially near the label ceiling where a few examples can move a recall estimate by several points.","section":"Tables 2–5"},{"comment":"The abstract and introduction give corpus sizes of roughly 2,000 (CLERC) and 750 (BriefMe) citations, while Table 6 reports 2,965 and 641 retained case-law citations; these numbers should be aligned.","section":"Abstract and Table 6"},{"comment":"Given that only 10 citations per dataset were triple-annotated, reporting a confidence interval around Gwet's AC1 would help readers calibrate how much weight to place on the agreement metrics.","section":"§3.3 and Table 9"}],"recommendation":"major_revision","confidential_remarks":"The label-quality issue is the main obstacle to accepting the paper as is. I do not see a circularity problem, and the CLERC result is strong enough that the central qualitative claim is very likely correct; however, the BriefMe headline and the document-type comparison are currently stated more strongly than the validation data support. If the authors add full Hard-set validation or report conservative corrected recall bounds, the paper should be acceptable; otherwise the abstract and conclusion need to be materially softened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper does something genuinely new. It turns proposition-level citation support into a controlled task by corrupting real Bluebook citations at three difficulty levels, and the headline result—models catch 93–100% of wrong-case swaps but only 37–61% of wrong-pinpoint corruptions on court opinions—is consistent across fourteen configurations and two datasets. The strongest part is the error analysis. When models miss a corrupt pinpoint, they often claim the cited page “expressly states” language that is not there. That is not a vague topical-overlap excuse; it is active fabrication inside the verification rationale. The page-grounded prompt intervention also earns its place: it clearly trades recall against false positives, which is a useful diagnostic even if it is not a fix.\n\nThe soft spots are real but concentrated. The Hard-corruption labels rest on 23 citations per dataset, only 10 triple-annotated, and agreement with the heuristic labels averages 84–88%, with one annotator at 69.6% on BriefMe. The paper concedes some wrong-page corruptions may still be on point. That matters more than the paper admits. Measured recall on Hard examples is capped at 1 minus the fraction of labels that are actually on point, and on BriefMe the best recall is 82.7%. If even 15% of the Hard labels are wrong, the ceiling is about 85%, so the “still misses 18%” framing is close to label noise. The CLERC result is much safer: 37–61% is far below any plausible ceiling, so the qualitative conclusion survives there. But the brief/opinion comparison and the precise BriefMe numbers are partly artifacts of unmeasured label error. The paper also reports no variance across stochastic model runs and does not ship the evaluation files, which makes independent replication harder.\n\nWho is this for: anyone building or auditing legal AI tools, and researchers working on citation verification or hallucination detection. The contribution is a new evaluation method with a robust finding on opinions, plus a cautionary note about how easily models confound topic with support. The paper deserves a serious referee. I would send it out, but with a clear revision request: release the exact eval files and code, report variance across runs, and expand the human validation with stratified sampling by corruption type. If the authors do that, I would cite this in my own work.\n\nNet: send it to review, conditional on the data and code actually shipping.","headline":"The wrong-pinpoint finding is real and worth taking seriously, but the briefs result is partly a label-noise artifact; deserve a serious referee if the eval files and code ship.","tokens_in":27709,"tokens_out":1548,"would_cite":true,"duration_ms":18755,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current LLMs catch 93–100% of wrong-case substitutions but only 37–61% of same-case wrong-page citations, relying on topical overlap.","keywords":["Legal NLP","citation verification","LLM evaluation","hallucination detection","legal AI safety","pinpoint citations","proposition-level support","legal hallucination"],"falsifier":"Select the Hard corrupted examples the models accept and have fresh legal experts, blind to the corruption, read the full cited page and judge whether it supports the asserted proposition. If a substantial share, say more than 20%, of the pages judged not on point by the construction turn out to support the proposition, the reported recall numbers are inflated by label error; if expert agreement with the corruption labels holds on a larger sample, the models' topical-overlap failure is real.","tokens_in":26556,"feed_emoji":"⚖️","tokens_out":7097,"duration_ms":58630,"temperature":0.7,"pith_summary":"The paper tries to establish a distinction between two legal citation tasks: catching citations to a wrong case, and catching citations that point to a real case but the wrong pinpoint page. It creates controlled corruptions of real citations from court opinions and legal briefs and runs fourteen frontier LLM configurations on them. Models catch 93–100% of wrong-case substitutions but only 37–61% of wrong-pinpoint corruptions on opinions and 52–83% on briefs. When they fail, they accept the citation based on topical overlap and even claim the cited page contains text it does not. The paper's conclusion is that recognizing the right legal topic and verifying proposition-level support are distinct capabilities, and current models conflate them.","feed_headline":"Frontier LLMs miss up to 63% of wrong-page legal cites","feed_subtitle":"Even the best model misses 40% of pinpoint-page errors in opinions, so AI citechecking cannot verify support.","key_machinery":"The load-bearing object is the difficulty-graded corruption of Bluebook pinpoint citations. A valid citation is altered in one of three ways: replace the case with an unrelated one (Easy), replace it with a different case cited elsewhere in the same document (Medium), or keep the case and change only the pinpoint page number (Hard). Because only the page changes in Hard examples, the cited content stays topically and doctrinally related to the proposition while losing page-level support, isolating the exact capability being measured. The evaluation pipeline also uses a three-category citation taxonomy (substantive, procedural, secondary) to restrict the test set to citations that actually claim support, plus a page-grounded prompt that instructs the model to check verbatim quotes and distinguish topic from support.","core_discovery":"The central discovery is that proposition-level citation support verification is a separate, harder capability than fabrication detection, and current LLMs do not have it. On easy wrong-case corruptions, models are near-saturated, with recall of 93–100%. On hard corruptions that keep the same case and change only the pinpoint page, recall drops to 36.5–60.6% on court opinions and 51.5–82.7% on briefs. Scale and extended reasoning narrow the gap but do not close it: with high reasoning effort, the strongest evaluated setting still misses about 40% of pinpoint mismatches on opinions and 18% on briefs. The paper shows the failure is not a refusal to check but a substituted check: in roughly two-thirds of false negatives the model states that the cited page \"expressly states\" or \"explicitly says\" language that is not there, and among false negatives with verbatim quotes the quoted text is absent from the cited page 92% of the time. Prompting the model to verify page-level support raises Hard recall by 7–36 percentage points but increases false positive rates across every model, so the intervention makes models more skeptical, not selectively skeptical.","pith_inferences":["Beyond the paper, a deterministic check for verbatim quoted text on the cited page would catch most false negatives and could be implemented without an LLM, since 92% of missed Hard examples contain quotes absent from the page.","Beyond the paper, treating support verification as a two-stage pipeline, first retrieve the pinpoint page and then judge entailment with the proposition, is a testable alternative to monolithic prompting; the paper's verbatim-quote finding predicts the retrieval stage alone would close most of the gap.","Beyond the paper, the error pattern suggests confidence scores from models that accept a wrong-pinpoint citation may be systematically miscalibrated, so a calibration study under the same corruption scheme would tell whether any thresholding can separate accepted-wrong from accepted-right citations.","Beyond the paper, the same difficulty gradient should appear in other domains where a citation pinpoints a specific part of a large document, such as statutes, regulations, or scientific articles, so the findings are testable outside legal corpora."],"forward_implications":["Database lookups that only confirm a case exists will keep missing the harder error class, since the failing citations name real cases with correct reporters and pinpoint formats.","Even the strongest evaluated configuration cannot be relied on as a standalone proposition-level citechecker: it misses about 40% of wrong-pinpoint citations in court opinions and 18% in briefs.","Prompting for page-level verification is diagnostic, not corrective: it buys 7–36 points of Hard recall but raises false positive rates by 1–25 points, so a prompted model flags roughly one in four valid citations on some settings.","Recognizing the correct legal topic is a prerequisite for support verification but not sufficient; systems must add a second step that locates relevant content on the cited page and then judges support.","Document type matters: the same models perform 12–26 points better on briefs than on court opinions, suggesting proposition specificity, not model capability alone, drives difficulty."],"supporting_citations":[{"why":"Defines the two-way taxonomy of fabrication versus misrepresentation that motivates the task and grounds the Hard corruption setting.","marker":"(Dahl et al., 2024)"},{"why":"Supplies the CLERC court-opinion corpus whose pinpoint citations are corrupted into Easy, Medium, and Hard examples.","marker":"(Hou et al., 2024)"},{"why":"Supplies the BriefMe legal-brief corpus used for the second evaluation set.","marker":"(Woo et al., 2025)"},{"why":"Bluebook pinpoint structure is what lets Hard corruptions change only the page while keeping the case and reporter intact.","marker":"(The Harvard Law Review Association and others, 2020)"},{"why":"Provides the AC1 inter-annotator agreement measure used to validate that the corruptions are judged not on point.","marker":"(Gwet, 2008)"},{"why":"Mata v. Avianca is the motivating case showing fabricated citations visible to database lookup, against which the harder misrepresentation task is defined.","marker":"(United States District Court for the Southern District of New York, 2023)"}],"fun_headline_variants":["LLMs catch 93-100% wrong cases but miss 63% wrong pages","Scale doesn't fix LLMs' pinpoint citation errors","Legal AI can't tell topic match from page support","LLMs ace fake cites, miss real ones on wrong pages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that changing only the pinpoint page number within an otherwise valid citation produces a citation that is genuinely not on point; that assumption rests on human labels for only 23 citations per dataset with 84–88% agreement with the heuristic labels, and the paper concedes that some corrupted citations may still be on point by chance, so label noise could make the reported Easy-to-Hard gap look larger than the models' true behavior.","fun_headline_variants_meta":{"raw":{"variants":["LLMs catch 93-100% wrong cases but miss 63% wrong pages","Scale doesn't fix LLMs' pinpoint citation errors","Legal AI can't tell topic match from page support","LLMs ace fake cites, miss real ones on wrong pages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000667,"raw_usage":{"total_tokens":3091,"prompt_tokens":1044,"completion_tokens":2047,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":1975}},"tokens_in":660,"tokens_out":2047,"duration_ms":14682,"temperature":1.0,"reasoning_tokens":1975,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:04:42.972155+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select the Hard corrupted examples the models accept and have fresh legal experts, blind to the corruption, read the full cited page and judge whether it supports the asserted proposition. If a substantial share, say more than 20%, of the pages judged not on point by the construction turn out to support the proposition, the reported recall numbers are inflated by label error; if expert agreement with the corruption labels holds on a larger sample, the models' topical-overlap failure is real.","supporting_citations":[],"review_version":1}