REVIEW 3 major objections 4 minor 12 references
Is this Citation on Point?
T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Current LLMs catch 93–100% of wrong-case substitutions but only 37–61% of same-case wrong-page citations, relying on topical overlap.
desk verdict The wrong-pinpoint finding is real and worth taking seriously, but the briefs result is partly a label-noise artifact; deserve a serious referee if the eval files and code ship. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the difficulty-graded corruption of Bluebook pinpoint citations. A valid citation is altered in one of three ways: replace the case with an unrelated one (Easy), replace it with a different case cited elsewhere in the same document (Medium), or keep the case and change only the pinpoint page number (Hard). Because only the page changes in Hard examples, the cited content stays topically and doctrinally related to the proposition while losing page-level support, isolating the exact capability being measured. The evaluation pipeline also uses a three-category citation taxonomy (substantive, procedural, secondary) to restrict the test set to citations that actually claim support, plus a page-grounded prompt that instructs the model to check verbatim quotes and distinguish topic from support.
What would settle it
Select the Hard corrupted examples the models accept and have fresh legal experts, blind to the corruption, read the full cited page and judge whether it supports the asserted proposition. If a substantial share, say more than 20%, of the pages judged not on point by the construction turn out to support the proposition, the reported recall numbers are inflated by label error; if expert agreement with the corruption labels holds on a larger sample, the models' topical-overlap failure is real.
Extended reading notes
Core claim
The central discovery is that proposition-level citation support verification is a separate, harder capability than fabrication detection, and current LLMs do not have it. On easy wrong-case corruptions, models are near-saturated, with recall of 93–100%. On hard corruptions that keep the same case and change only the pinpoint page, recall drops to 36.5–60.6% on court opinions and 51.5–82.7% on briefs. Scale and extended reasoning narrow the gap but do not close it: with high reasoning effort, the strongest evaluated setting still misses about 40% of pinpoint mismatches on opinions and 18% on briefs. The paper shows the failure is not a refusal to check but a substituted check: in roughly two-thirds of false negatives the model states that the cited page "expressly states" or "explicitly says" language that is not there, and among false negatives with verbatim quotes the quoted text is absent from the cited page 92% of the time. Prompting the model to verify page-level support raises Hard recall by 7–36 percentage points but increases false positive rates across every model, so the intervention makes models more skeptical, not selectively skeptical.
Load-bearing premise
The evaluation assumes that changing only the pinpoint page number within an otherwise valid citation produces a citation that is genuinely not on point; that assumption rests on human labels for only 23 citations per dataset with 84–88% agreement with the heuristic labels, and the paper concedes that some corrupted citations may still be on point by chance, so label noise could make the reported Easy-to-Hard gap look larger than the models' true behavior.
Editorial extensions
If this is right
- Database lookups that only confirm a case exists will keep missing the harder error class, since the failing citations name real cases with correct reporters and pinpoint formats.
- Even the strongest evaluated configuration cannot be relied on as a standalone proposition-level citechecker: it misses about 40% of wrong-pinpoint citations in court opinions and 18% in briefs.
- Prompting for page-level verification is diagnostic, not corrective: it buys 7–36 points of Hard recall but raises false positive rates by 1–25 points, so a prompted model flags roughly one in four valid citations on some settings.
- Recognizing the correct legal topic is a prerequisite for support verification but not sufficient; systems must add a second step that locates relevant content on the cited page and then judges support.
- Document type matters: the same models perform 12–26 points better on briefs than on court opinions, suggesting proposition specificity, not model capability alone, drives difficulty.
Reading between the lines
- Beyond the paper, a deterministic check for verbatim quoted text on the cited page would catch most false negatives and could be implemented without an LLM, since 92% of missed Hard examples contain quotes absent from the page.
- Beyond the paper, treating support verification as a two-stage pipeline, first retrieve the pinpoint page and then judge entailment with the proposition, is a testable alternative to monolithic prompting; the paper's verbatim-quote finding predicts the retrieval stage alone would close most of the gap.
- Beyond the paper, the error pattern suggests confidence scores from models that accept a wrong-pinpoint citation may be systematically miscalibrated, so a calibration study under the same corruption scheme would tell whether any thresholding can separate accepted-wrong from accepted-right citations.
- Beyond the paper, the same difficulty gradient should appear in other domains where a citation pinpoints a specific part of a large document, such as statutes, regulations, or scientific articles, so the findings are testable outside legal corpora.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies proposition-level legal citation support verification: whether a citation to a real case actually supports the proposition for which it is offered. Using controlled rule-based perturbations of citations from CLERC (court opinions) and BriefMe (legal briefs), it creates Easy (wrong case), Medium (same-document different case), and Hard (wrong pinpoint page within the same case) corruptions, then evaluates fourteen LLM configurations. The central finding is a large difficulty gap: models catch 93–100% of Easy corruptions but only 37–61% (CLERC) and 52–83% (BriefMe) of Hard pinpoint-page corruptions. A page-grounded prompt improves Hard recall by 7–36 percentage points but consistently raises false positive rates. The paper also contributes a three-category citation taxonomy, a human validation study, and an error analysis showing that models often fabricate rationales claiming the cited page contains text that is not there.
Significance. If the quantitative result holds, this is a useful and important negative result for legal NLP and AI safety: it separates database-level citation checking from proposition-level support verification and gives a concrete, controlled measurement of where frontier LLMs fail. The paper's strengths are its clean corruption taxonomy, the use of two public legal corpora, the breadth of fourteen model configurations across three families, and the concrete error analysis with verbatim GPT-5 rationales. The Easy/Hard gap is large and consistent enough that the qualitative conclusion is robust even under the label-quality caveats discussed below. The main quantitative claims—especially the BriefMe recall ceiling and the briefs-vs-opinions comparison—need additional validation before they can be taken at face value.
major comments (3)
- [§3.3, §8, Tables 3 and 10] The Hard-corruption gold labels rest on only 23 human-validated citations per dataset, with average agreement with the heuristic labels of 88% on CLERC and 84% on BriefMe, and Section 8 concedes that some corrupted citations may remain on point by chance. Since measured Hard recall is the fraction of corrupted examples flagged as erroneous, any fraction e of corruptions that are actually on point caps achievable recall at 1−e; on BriefMe, where the best baseline Hard recall is 82.7% and GPT-5.4 with reasoning reaches 82.0%, this places the strongest models essentially at the label ceiling, so the 'still misses 18%' headline may be largely annotation noise. The CLERC range (37–61%) remains well below an 84–88% ceiling, so the qualitative conclusion survives, but the reported gap size and the briefs-vs-opinions comparison are partly artifacts of unmeasured label error. Please provide full-set per-example confirmation for the Hard set (at minimum for the quoted-language cases), or report recall bounds adjusted for a directly measured e, and rephrase the BriefMe-specific claims accordingly.
- [§5.1–§7 (RQ3)] The claim that briefs produce a stronger signal than opinions compares CLERC and BriefMe under different protocols: CLERC supplies preceding-paragraph context (src_document) while BriefMe does not, pinpoint coverage differs (36% vs. 23% of retained citations in Table 6), and the Hard sets are constructed differently because BriefMe has no Medium corruption. The 12–26 percentage-point advantage on Hard recall could therefore reflect contextual or sampling differences rather than document type, and Section 7 itself describes proposition specificity as an untested hypothesis. Please either rephrase this as a descriptive cross-dataset observation or control for these confounds (for example, with matched subsets) before presenting it as the answer to RQ3.
- [§6.2 and Appendix B.2] The page-grounded prompt was designed after inspecting GPT-5 false negatives on exactly the same test set on which it is then evaluated, and no held-out prompt-development split is reported. The improvement figures in Tables 4 and 5 therefore conflate the intervention's effect with selection on the observed failure modes; the qualitative recall/FPR trade-off is the robust part, but the point estimates should be presented as in-sample prompt engineering or validated on a held-out subset before being stated as a general result.
minor comments (4)
- [§D vs. §3.2] Section D says that for BriefMe 7 of the 10 triple-annotated cases were 'Medium corruptions,' but Section 3.2 states that Medium corruption is not available for BriefMe because the dataset lacks preceding paragraphs; please clarify whether these are same-document substitutions under a different name or correct the terminology.
- [Tables 2–5] Please report the number of examples in each difficulty cell; percentages without counts make it impossible to gauge uncertainty, especially near the label ceiling where a few examples can move a recall estimate by several points.
- [Abstract and Table 6] The abstract and introduction give corpus sizes of roughly 2,000 (CLERC) and 750 (BriefMe) citations, while Table 6 reports 2,965 and 641 retained case-law citations; these numbers should be aligned.
- [§3.3 and Table 9] Given that only 10 citations per dataset were triple-annotated, reporting a confidence interval around Gwet's AC1 would help readers calibrate how much weight to place on the agreement metrics.
Circularity Check
No significant circularity; the evaluation is self-contained and the single self-citation is not load-bearing.
full rationale
The paper's central claim, that LLMs detect wrong-case substitutions but largely miss same-case wrong-pinpoint corruptions, is an experimental measurement against labels produced by rule-based corruption and human annotation, independent of the fourteen evaluated model configurations. No fitted parameter is renamed as a prediction: the page-grounded prompt is an intervention measured on the same test set, not a parameter fit, and the paper explicitly reports the resulting false-positive-rate tradeoff rather than presenting a validated predictive model. The only self-citation (Pradhan et al., 2025) supports the choice of Gwet's AC1 over Fleiss's kappa; the same sentence independently cites Brennan & Prediger (1981), and the metric choice does not enter the main recall/FPR claims. Section 8 honestly concedes that 'some corrupted citations may nevertheless remain on point by chance,' which is a label-noise limitation, not circularity: imperfect ground truth is a validity concern, not a reduction of the results to their inputs. The error-analysis claims about 'topical matching' are interpretations of model rationales (Section 6.1, Table 11), not derivations from the construction. The task definition, corruption strategies, human validation, and evaluation are all separate links in the chain, so there is no self-definitional or fitted-input circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Rule-based corruptions (replacing case or changing pinpoint page) yield citations that are not on point for the surrounding proposition.
- domain assumption A citation is on point only when the cited page (or full document for non-pinpoint citations) supports the specific proposition, not merely the legal topic.
- domain assumption Substantive citations are the relevant subset and can be identified automatically with an LLM classifier.
- domain assumption The extracted tgt_content for a pinpoint citation accurately reflects the content of the cited page.
Cite this review
Pith. "Pith review of Is this Citation on Point?." pith.science (2026). https://pith.science/paper/SNQHSAOZ
@misc{pith2026260812571,
author = {Pith},
title = {Pith review of: Is this Citation on Point?},
year = {2026},
howpublished = {\url{https://pith.science/paper/SNQHSAOZ}},
note = {Machine review of arXiv:2608.12571}
}
read the original abstract
In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora, either replacing the cited case or changing only the pinpoint page within the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.
Figures
Reference graph
Works this paper leans on
-
[1]
src_paragraph: The source paragraph with the citation enclosed in <CITE> ,→</CITE> tags
-
[2]
doi: 10.18653/v1/2023.emnlp-main.127
Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.127. URL https:// aclanthology.org/2023.emnlp-main.127/ . (Cited on 3) Google DeepMind. Gemini 2.5: Our most capable model,
-
[3]
Distinguish topic from support: A citation is on point only if the ,→tgt_content supports the SPECIFIC claim being made, not merely if it ,→discusses the same area of law. Positive Example: [Same as baseline prompt above.] Negative Example (topically related but wrong page): src_paragraph: "The court held that ‘it is the duty of courts to harmonize ,→jury...
work page 1943
-
[7]
tgt_content: Content from the cited document or specific pages indicated ,→by the pinpoint citation. −Use the IRAC framework: −Issue: Clearly state if the cited document addresses the specific legal ,→issue raised by the src_paragraph. −Rule: Identify the legal rule or principle in the cited content relevant ,→to the issue. −Analysis: Analyze whether the ...
work page 2007
-
[8]
,→If the quoted text is not present, the citation is NOT on point ,→regardless of topical relevance
Verbatim quote check: If the src_paragraph quotes specific language, then ,→verify that the quoted language actually appears in the tgt_content. ,→If the quoted text is not present, the citation is NOT on point ,→regardless of topical relevance
-
[9]
The tgt_content represents the ,→specific page(s) cited
Page−level verification: Do not accept a citation merely because the cited ,→case is about the right legal topic. The tgt_content represents the ,→specific page(s) cited. Verify that THIS specific content−−not the ,→case generally−−supports the specific proposition
-
[11]
Fabricated verification.In roughly two-thirds of false negatives, models do not merely accept the citation passively; they claim the cited page “expressly states,” “explicitly says,” or “directly supports” the proposition. The model constructs a plausible-sounding legal rationale asserting it verified content that does not appear on the cited page. A prac...
-
[12]
merely reflects the un- remarkable proposition that statements must be analyzed in context
Topical matching in place of propositional verification.Because Hard corruptions swap pages within the same case, the target content is always topically related: same dispute, often the same statute or doctrine. The model treats this overlap as sufficient rather than checking whether the page states the proposition being cited. This explains the Easy-to-H...
work page 2024
Show all 12 references
-
[2020]
doi: 10.18653/v1/2020.emnlp-main.609
Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.609. URL https:// aclanthology.org/2020.emnlp-main.609/ . (Cited on 13) Wojtasik, K., Dolega, T., and Piasecki, M. CiteVerifier: How Good Are Citation Verifiers and How to Use Them? InInternational Con...
2020 arXiv
-
[2023]
doi: 10.18653/v1/2023.emnlp-main.398
Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.398. URL https:// aclanthology.org/2023.emnlp-main.398/ . (Cited on 13) Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., and Szpektor, I. TrueTeacher: Learning factual consistency evaluation with larg...
2023 doi
-
[2025]
(Cited on 4) Greenawalt, K
URL https://deepmind.google/tech nologies/gemini/. (Cited on 4) Greenawalt, K. Precedent: Importance and context. In Statutory and Common Law Interpretation. Oxford Uni- versity Press, 11 2012a. ISBN 9780199756148. doi: 10.1093/acprof:oso/9780199756148.003.0008. URL 9 Is this ...
-
[2026]
fabricated
(Cited on 2, 13) Yuan, Z., Shi, K., Zhang, Z., Sun, L., Chawla, N. V ., and Ye, Y . CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era.arXiv preprint arXiv:2602.23452, 2026. URL https://arxiv.org/abs/2602.23452. (Cited ...
2024 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.