Pith. sign in

REVIEW 3 major objections 6 minor 26 references

Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorization

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Modern LLMs identify legal holdings as well as specialized legal models without any fine-tuning.

desk verdict Useful zero-shot results on CaseHOLD with a clever but limited memorization control; worth a look, but don't lean on the anti-memorization claim. read the letter →

arxiv 2505.02172 v3 pith:2C3RJRXA submitted 2025-05-04 cs.CL

Chuck Arvin This is my paper · ORCID
classification cs.CL
keywords LargeLanguageModelsCaseHOLDlegalholdingszero-shotevaluationmodelscalingmemorizationcitationanonymizationNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether general-purpose large language models can do a core legal-reading task—identifying the holding a citation stands for—without any legal training, and whether their scores reflect real understanding or memorized case law. It evaluates eight off-the-shelf models, from 3B to 90B+ parameters, on the full 5,314-question CaseHOLD test set and finds that the best, GPT-4o, reaches a macro F1 of 0.744, outperforming several published models built specifically for legal text. Performance rises with model size within every family tested. To address the memorization worry, the paper builds a citation anonymization test that swaps every case name and citation for a plausible fake; scores stay high (0.728), with 88% of answers unchanged, which the paper reads as evidence against rote memorization. The practical stakes are that legal analytics may no longer require expensive domain-specific training, while benchmark designers gain a reusable check for contamination.

What carries the argument

The carrying mechanism is the citation anonymization test, a two-step procedure: first a language model rewrites each citing passage so that all case names, jurisdictions, years, and page numbers become invented but legally formatted equivalents, while the <HOLDING> tag, punctuation, and legal reasoning stay exactly in place; then the evaluated model receives the rewritten passage under the same zero-shot chain-of-thought prompt and must pick the completion. The test assumes that a model leaning on memorized identifiers should stumble once those identifiers are faked, whereas a model using the legal content should not. The rewrite is deliberately large—a median of 91 Levenshtein edits, roughly 10% of the prompt—so that the change is far beyond the small perturbations known to shift LLM answers.

What would settle it

A direct way to test the claim: fully paraphrase the entire prompt text of all 5,314 questions, not just the citations, and see whether macro F1 stays near 0.728; a large drop would show the earlier robustness was tied to the citation-swap procedure rather than to true non-memorization.

Watch

Extended reading notes

Core claim

The central claim is that a modern general-purpose LLM, prompted zero-shot with chain-of-thought reasoning, identifies legal holdings on CaseHOLD at a level competitive with custom legal models: GPT-4o scores 0.744 macro F1, ahead of the best result reported in the original CaseHOLD paper and of several published specialized legal models, while Nova Pro reaches 0.720. The same capability scales with model size across the Llama, Nova, and GPT-4o families. The paper's second claim is that this performance is not explained by memorization of case names or citations: after a cheaper model rewrites every prompt with fictitious but consistent case names, courts, years, and page numbers—median Levenshtein distance 91 edits, about 10% of the prompt—the macro F1 remains 0.728 and 88% of individual answers are unchanged. The paper presents these results as evidence against rote memorization, while explicitly noting the test is not conclusive.

Load-bearing premise

The conclusion stands only if rote memorization of the test set would have to pass through the case names and citations that the anonymization replaces; if a model instead memorized whole test passages, the fake citation swap would not erase the trigger, and the paper concedes that its test is not conclusive.

Editorial extensions

If this is right

  • Zero-shot general-purpose LLMs can now reach the published performance bar of custom legal models on CaseHOLD, so a large legal corpus and fine-tuning pipeline is no longer required to reach that benchmark.
  • Within each model family, larger models score higher, so continued general-purpose scaling should push legal-holding identification higher without legal-specific training.
  • The citation anonymization test gives legal NLP a reusable way to check whether benchmark scores reflect memorized case details rather than reasoning from the prompt text.
  • When models disagree on a question, accuracy is lower for every model, so ensemble or mixture-of-expert aggregation is a natural next step for these benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if the anonymization robustness generalizes to other legal benchmarks, contamination from publicly available judicial opinions may be less crippling than feared, because the models seem to use the legal reasoning in the prompt rather than the case identifier.
  • A testable extension would be to apply the same rewriting procedure to the answer choices rather than only the prompt text, since memorized choice wording is another possible shortcut.
  • Another extension: measure memorization directly by asking models to complete masked passages of well-known cases and checking whether outputs match CaseHOLD test text.
  • The scaling effect implies that performance on legal holdings may keep improving as frontier models grow, which would change the cost-benefit of building domain-specific legal models.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper evaluates eight zero-shot large language models (roughly 3B to 90B+ parameters) on the CaseHOLD legal holding identification benchmark. It reports macro F1 scores, with GPT-4o at 0.744 and Amazon Nova Pro at 0.720, and argues these are competitive with published fine-tuned legal models. The paper also observes within-family scaling of performance with model size and introduces a citation anonymization procedure that replaces case names and citations with fictitious equivalents; under this perturbation the macro F1 remains 0.728, which the authors interpret as evidence against rote memorization. Additional analyses cover answer agreement across models and question difficulty measured by the number of unique model choices.

Significance. If the core measurements are correct, the paper provides a useful reference point for legal NLP: a clean zero-shot evaluation over the full CaseHOLD test set with temperature 0, a transparent regex parsing scheme, a documented random tie-break for the under-1% unparseable responses, and public code and data. The comparison against published fine-tuned models is a valuable contribution, and the scaling observation across three model families is suggestive. The main weakness is the memorization claim: the anonymization test is too narrow to support the abstract's statement that performance is 'not due to rote memorization.' The paper's strongest contribution is the systematic benchmark measurement, not the no-memorization conclusion, which needs to be either substantially strengthened or appropriately qualified.

major comments (3)
  1. [3 (Prompt 6, Figure 7)] The citation anonymization test in Section 3 cannot support the abstract's claim that performance is 'not due to rote memorization.' The intervention replaces only case names, years, courts, and page numbers, affecting roughly 10% of the prompt by the paper's own median Levenshtein estimate of 91 edits, while leaving the other 90% of the context and, critically, all five answer choices untouched. A model that memorized the test item, the correct holding sentence, or a lexical mapping from legal content words to an answer option would be completely unaffected. The drop from 0.744 to 0.728 is therefore not diagnostic of memorization. The paper itself concedes the test is 'not conclusive' (Section 3), so the abstract and Key Findings should be softened, or a stronger control (e.g., paraphrasing the full prompt and answer choices, or evaluating on a non-public set of holdings) should be added.
  2. [3 (Prompt 6)] Prompt 6 uses GPT-4o-mini to generate the anonymized prompts, and the paper asserts that these rewrites preserve the semantic content and difficulty of the originals without validating this assumption. Since the same class of models is being evaluated, this introduces a self-referential component: any systematic simplification or obfuscation by the anonymizer changes the difficulty in a way that the aggregate F1 comparison cannot detect. The paper should either validate difficulty preservation (e.g., human annotation, a second independent anonymizer, or per-item difficulty checks) or explicitly restrict the claim to 'robust to citation replacement' rather than 'not rote memorization.'
  3. [2.3, Figure 5] The scaling claim rests on a single deterministic run per model with no confidence intervals, bootstrapping, or significance testing. With three to four points per family, the observed monotonicity is suggestive but not quantified, and the comparison of GPT-4o (0.744) with published models (e.g., 0.770 in [13]) would be more informative with uncertainty bounds. I recommend adding bootstrap confidence intervals over the 5,314 test items and, if feasible, multiple API repetitions or random seeds, and reporting effect sizes.
minor comments (6)
  1. [Abstract, Section 1, Section 2.3] The reported GPT-4o macro F1 is 0.744 in the Abstract and Section 2.3 but 0.742 in the first bullet of Section 1; please reconcile these numbers.
  2. [Throughout] Model naming is inconsistent (e.g., 'GPT4o' vs. 'GPT-4o' and 'AmazonNovaPro' vs. 'NovaPro'); standardize the names, especially in Figure 5 axis labels and Table 1 captions.
  3. [2.2] The parsing rule is described as extracting 'ANSWER: X' with regex \bANSWER:\s*([A-E]); please clarify whether the letter is case-sensitive and how lower-case answers (e.g., 'a') are handled.
  4. [Figure 7] The 'changes in red' are difficult to see in black-and-white print; consider using a different annotation (e.g., underline or bold) or a table of changed tokens.
  5. [Table 1] The accuracy column appears to aggregate across all models; a per-model version, or an explicit note that accuracy is averaged over models, would make the table self-explanatory.
  6. [4] The conclusion says 'suggesting the models are doing more than mere memorization of case names,' which is a more defensible claim than the abstract's 'not due to rote memorization' and should be used consistently throughout the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark scores, scaling trend, and anonymization comparison are measured against external data rather than derived from fitted parameters or self-citations.

full rationale

The paper's central empirical claims are (1) zero-shot macro F1 scores on CaseHOLD, (2) scaling with model size, and (3) robustness to citation anonymization. None of these are derived from fitted parameters: the CaseHOLD test set is an external benchmark, and the reported scores are direct measurements at temperature 0, not predictions produced by fitting a model to a subset of the same data. The scaling observation is an empirical trend across the Llama, Amazon Nova, and GPT-4o families, and it is not used to define or construct the F1 scores. The citation-anonymization experiment in Section 3 replaces case names and citations using Prompt 6 with GPT-4o-mini, then re-measures F1; the conclusion is a measured comparison (0.744 vs 0.728), not an analytic identity. The paper's own statement 'Though this is not conclusive' (Section 3) is a limitation about the strength of the no-memorization inference, not a sign that the inference is built into its assumptions by definition. The paper cites prior benchmarks ([2], [13], [25]) as external comparison points and does not rely on any self-citation chain or on a uniqueness theorem from the author's prior work. The only arguable weakness is that the anonymization test leaves answer choices and most content words unchanged, so it cannot rule out passage-level memorization; however, that is an external-validity limitation, not circularity, and it does not affect the measured F1 values. Accordingly, no circular step is exhibited and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. Its only new construct is the citation anonymization procedure, which is a method rather than an entity. The main domain assumptions are the correctness of benchmark labels and the semantic equivalence of anonymized prompts, the latter being the weakest.

assumptions (3)
  • domain assumption The CaseHOLD test set ground-truth labels are correct.
    The paper relies on the benchmark labels as ground truth for computing macro F1, without auditing them. This is standard practice.
  • ad hoc to paper GPT-4o-mini's anonymization preserves the semantic content and difficulty of the original prompts.
    The memorization test depends on this assumption, which is supported only by Levenshtein distance and the examples shown, not by human evaluation.
  • standard math Macro F1 computed by averaging per-class F1 is a valid measure of performance for this multi-class task.
    The paper uses macro F1 as in prior CaseHOLD work; the choice is standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorization." pith.science (2026). https://pith.science/paper/2C3RJRXA

@misc{pith2026250502172,
  author       = {Pith},
  title        = {Pith review of: Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2C3RJRXA}},
  note         = {Machine review of arXiv:2505.02172}
}
read the original abstract

As large language models (LLMs) continue to advance in capabilities, it is essential to assess how they perform on established benchmarks. In this study, we present a suite of experiments to assess the performance of modern LLMs (ranging from 3B to 90B+ parameters) on CaseHOLD, a legal benchmark dataset for identifying case holdings. Our experiments demonstrate scaling effects - performance on this task improves with model size, with more capable models like GPT4o and AmazonNovaPro achieving macro F1 scores of 0.744 and 0.720 respectively. These scores are competitive with the best published results on this dataset, and do not require any technically sophisticated model training, fine-tuning or few-shot prompting. To ensure that these strong results are not due to memorization of judicial opinions contained in the training data, we develop and utilize a novel citation anonymization test that preserves semantic meaning while ensuring case names and citations are fictitious. Models maintain strong performance under these conditions (macro F1 of 0.728), suggesting the performance is not due to rote memorization. These findings demonstrate both the promise and current limitations of LLMs for legal tasks with important implications for the development and measurement of automated legal analytics and legal benchmarks.

Figures

Figures reproduced from arXiv: 2505.02172 by the authors.

Figure 1
Figure 1. All models agree on the correct answer. As GPT4o [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 4
Figure 4. Match rate for answers between different models. LLMs [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 2
Figure 2. The models reach three distinct answers. As [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Macro F1 Scores on the CaseHOLD test set. Model perfor [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 7
Figure 7. Figure 7: Original and “anonymized” citation prompts with [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 9 canonical work pages

  1. [13]

    Joel Niklaus, Veton Matoshi, Matthias Sturmer, Ilias Chalkidis, and Daniel E. Ho. 2024. MultiLegalPile: A 689GB Multilingual Legal Corpus. arXiv:2306.02069 [cs.CL] https://arxiv.org/abs/2306.02069

    arXiv 2024

  2. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  3. [2]

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopou- los, Daniel Martin Katz, and Nikolaos Aletras. 2022. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. arXiv:2110.00976 [cs.CL] https://arxiv.org/abs/2110.00976

    arXiv 2022

  4. [3]

    Zhaopeng Feng, Yan Zhang, Hao Li, Bei Wu, Jiayu Liao, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, and Zuozhu Liu. 2024. TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinement. arXiv:2402.16379 [cs.CL] https://arxiv.org/abs/2402.16379

    arXiv 2024

  5. [4]

    Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zam- brano, et al. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.Advances in Neural Information Pro- cessing Systems36 (2024)

    work page 2024

  6. [5]

    Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. SoK: Memorization in General-Purpose Large Language Models. arXiv:2310.18362 [cs.CL] https://arxiv.org/abs/2310.18362

    arXiv 2023

  7. [6]

    Krass, Lucia Zheng, Neel Guha, Christopher D

    Peter Henderson, Mark S. Krass, Lucia Zheng, Neel Guha, Christopher D. Man- ning, Dan Jurafsky, and Daniel E. Ho. 2022. Pile of Law: Learning Respon- sible Data Filtering from the Law and a 256GB Open-Source Legal Dataset. arXiv:2207.00220 [cs.CL] https://arxiv.org/abs/2207.00220

    arXiv 2022

  8. [7]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs.LG] https: //arxiv.org/abs/2001.08361

    arXiv 2020

Show all 26 references
  1. [8]

    Bommarito II

    Daniel Martin Katz, Dirk Hartung, Lauritz Gerlach, Abhik Jana, and Michael J. Bommarito II. 2023. Natural Language Processing in the Legal Domain. arXiv:2302.12039 [cs.CL] https://arxiv.org/abs/2302.12039

    2023 arXiv

  2. [9]

    Heiko Koziolek, Sten Grüner, Rhaban Hark, Virendra Ashiwal, Sofia Linsbauer, and Nafise Eskandani. 2024. LLM-based and retrieval-augmented control code generation. InProceedings of the 1st International Workshop on Large Language Models for Code. 22–29

    2024

  3. [10]

    Weixin Liang, Yaohui Zhang, Zhengxuan Wu, Haley Lepp, Wenlong Ji, Xuandong Zhao, Hancheng Cao, Sheng Liu, Siyu He, Zhi Huang, Diyi Yang, Christopher Potts, Christopher D Manning, and James Y. Zou. 2024. Mapping the Increasing Use of LLMs in Scientific Papers. arXiv:2404.01268 ...

    2024 arXiv

  4. [11]

    Mellinkoff

    D. Mellinkoff. 2004.The Language of the Law. Resource Publications. https: //books.google.com/books?id=YBRLAwAAQBAJ

    2004

  5. [12]

    Meta. 2024. Llama 3.2: Revolutionizing edge AI and vision with open, customiz- able models. (2024). https://ai.meta.com/blog/llama-3-2-connect-2024-vision- edge-mobile-devices/

    2024

  6. [14]

    The State Bar of California. 2024. PRACTICAL GUIDANCE FOR THE USE OF GENERATIVE ARTIFICIAL INTELLIGENCE IN THE PRACTICE OF LAW. https://www.calbar.ca.gov/Portals/0/documents/ethics/Generative-AI- Practical-Guidance.pdf

    2024

  7. [15]

    OpenAI. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL] https://arxiv.org/ abs/2410.21276

    2024 arXiv

  8. [16]

    Abel Salinas and Fred Morstatter. 2024. The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance. arXiv:2401.03729 [cs.CL] https://arxiv.org/abs/2401.03729

    2024 arXiv

  9. [17]

    Amazon Web Services. 2024. Introducing Amazon Nova foundation models: Frontier intelligence and industry leading price performance. (2024). https://aws.amazon.com/blogs/aws/introducing-amazon-nova-frontier- intelligence-and-industry-leading-price-performance/

    2024

  10. [18]

    Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022. Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities. arXiv:2206.10883 [cs.CL] https://arxiv.org/abs/2206.10883

    2022 arXiv

  11. [19]

    Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024. Exploring LLM Prompting Strategies for Joint Essay Scoring and Feedback Gen- eration.arXiv preprint arXiv:2404.15845(2024)

    2024 arXiv

  12. [20]

    The New York Times. 2024. The New York Times vs Microsoft et al. https: //nytco-assets.nytimes.com/2023/12/NYT_Complaint_Dec2023.pdf

    2024

  13. [21]

    William Webber

    John Tredennick and Dr. William Webber. 2024. An Introduc- tion to Large Language Models for Ediscovery Professionals. MIT Computational Law Report(2024). https://law.mit.edu/pub/ anintroductiontolargelanguagemodelsforediscoveryprofessionals/release/1

    2024

  14. [22]

    John Villasenor. 2024. Generative Artificial Intelligence and the Practice of Law: Impact, Opportunities, and Risks.Minnesota Journal of Law, Science and Technology(2024). https://scholarship.law.umn.edu/cgi/viewcontent.cgi?article= 1563&context=mjlst

    2024

  15. [23]

    Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. 2024. Planning In Natural Language Improves LLM Search For Code Generation. arXiv:2409.03733 [cs.LG] https://arxiv.org/abs/2409.03733

    2024 arXiv

  16. [24]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903 [cs.CL] https: //arxiv.org/abs/2201.11903

    2023 arXiv

  17. [25]

    Anderson, Peter Henderson, and Daniel E

    Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset. arXiv:2104.08671 [cs.CL] https://arxiv.org/abs/ 2104.08671

    2021 arXiv

  18. [26]

    Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis. arXiv:2304.04675 [cs.CL] https://arxiv.org/abs/2304.04675

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.