Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study

Ehsan Shareghi; Jiuzhou Han; Paul Burgess

REVIEW 4 major objections 4 minor 36 references

Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Legal citation prediction is far from solved: the best approaches, instruction-tuned LLMs and re-rankers over reason-of-citation retrieval, reach only about 52% accuracy.

desk verdict Useful benchmark and credible instruction-tuning results, but the retrieval numbers are inflated by test-query RoCs sitting in the RoC Aggregations index. read the letter →

arxiv 2412.06272 v2 pith:4HGQ33UR submitted 2024-12-09 cs.CL cs.AIcs.IR

Jiuzhou Han , Paul Burgess , Ehsan Shareghi This is my paper · ORCID

classification cs.CLcs.AIcs.IR

keywords legalcitationpredictionAustralianlawbenchmarkdatasetinstructiontuningretrieval-augmentedgenerationreason-of-citationdomain-specificpre-trainingre-ranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish how to predict the case citation a legal passage should reference, a task at the heart of precedent-based legal reasoning. It introduces AusLaw Citation Benchmark, a 55,005-instance dataset built from NSW Caselaw with 18,677 unique citations, and benchmarks four families of solutions: prompting general and law-specialised LLMs, retrieval-only pipelines, instruction fine-tuning, and hybrid retrieval-generation methods. The paper's central finding is that neither general nor law-specialised pre-training is remotely sufficient for this fine-grained task: zero-shot prompting gives near-zero accuracy even for a 54B law model, while instruction tuning on task-specific data lifts accuracy to 51.7% (Cite-SaulLM-7B) and 46.2% (Cite-LLaMA-3.1-8B). The best result, 52.1% ACC@1, comes from a trained re-ranker operating over top-5 candidates retrieved from databases indexed by aggregated 'Reasons-of-Citation'. The paper concludes that a gap of nearly 50% remains, making the benchmark a rigorous test-bed for future legal-domain research.

What carries the argument

The load-bearing object is the Reason-of-Citation (RoC): an LLM-generated prose description, produced for every citation instance from the citing sentence, the preceding sentence, and the full text of the cited case, stating why the case is being cited (e.g., 'The cited case is referenced to establish the standard required to demonstrate a likelihood of injury under the Trees Act'). RoCs do three jobs in the paper: (1) concatenated per citation as 'RoC Aggregations', they form the best database granularity for dense retrieval, outperforming full case text and catchwords; (2) they serve as the generation target for instruction-tuned models, which are trained to emit both the RoC and the citation; and (3) they provide the candidate description input for training the re-rankers that achieve the best reported ACC@1 of 52.1%.

What would settle it

Take a random sample of, say, 500 test instances, replace the automatically generated RoCs with versions written or verified by practising lawyers, and re-run the retrieval, re-ranker, and instruction-tuned pipelines; if ACC@1 and the ranking of methods shift materially, the reported benchmark difficulty and method comparison are artifacts of RoC quality rather than of the underlying task.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that legal citation prediction is not solved by domain pre-training, general LLM scale, or off-the-shelf retrieval, but is substantially advanced by two ingredients: task-specific instruction tuning and the right index granularity. The authors show that SaulLM-54B, despite being pre-trained on 94B tokens of legal text including the very NSW corpus used here, scores 2% or less when prompted zero-shot, while Claude Sonnet 3.5 with the leaked citation reason reaches only 15.5%. Instruction tuning the same or smaller backbones on the benchmark's training set (using LoRA) produces a jump to 51.7% for Cite-SaulLM-7B and 46.2% for Cite-LLaMA-3.1-8B. On the retrieval side, indexing each citation by concatenating all its LLM-generated Reasons-of-Citation ('RoC Aggregations') outperforms full-case text and catchwords by a wide margin, and a re-ranker trained on RoC-based candidates raises retrieval-only accuracy from 35.1% to 52.1% ACC@1. A companion experiment shows that pre-training on just 0.5B tokens of Australian law (Cite-AusLawLLM-7B, 52.0%) slightly beats pre-training on 94B tokens of mixed-jurisdiction law (Cite-SaulLM-7B, 51.7%), suggesting jurisdiction-specific data is more valuable than sheer legal-corpus size for this task.

Load-bearing premise

The entire pipeline leans on the automatically generated Reason-of-Citation descriptions being accurate and complete enough to serve both as the retrieval index and as the supervision signal; only a subset of these descriptions was manually checked.

Editorial extensions

If this is right

Task-specific instruction tuning, not domain pre-training or model scale, is the main performance lever for fine-grained legal tasks like citation prediction; even a generic 8B model beats a 70B general instruct model and a 54B law model.
Legal retrieval systems should index citations by aggregated reasons-for-citation rather than by full case text or catchwords; the paper reports up to 70% variation in accuracy across index granularities.
Trained re-rankers over top-5 retrieval candidates convert a substantial share of the top-5 accuracy (up to about 60%) into top-1 accuracy, making re-ranking the strongest component of the hybrid pipelines.
Jurisdiction-specific pre-training on a small corpus (0.5B tokens) can match or exceed much larger mixed-jurisdiction legal pre-training, pointing toward a cheaper path for building sovereign legal models.
The 52% ceiling on this benchmark implies that legal citation prediction needs mechanisms beyond semantic similarity and parametric memory, such as reasoning over precedent structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A sensitivity analysis that re-runs all pipelines with human-verified RoCs would quantify how much of the benchmark's difficulty depends on the automatically generated RoC descriptions.
Extending the RoC approach to statutes and to passages citing multiple authorities would test whether the same components, RoC aggregation and re-ranking, dominate across citation types.
The jurisdiction pre-training result suggests a testable conjecture: a model pre-trained only on a given jurisdiction's caselaw should beat a mixed-jurisdiction model by a larger margin as the target task becomes more jurisdiction-specific.
The near-50% gap suggests citation prediction may require explicit modeling of precedent networks (overruling, distinguishing, following) rather than only local passage-to-case similarity; enriching the benchmark with such relational metadata would probe this.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Thanks for the note. I read the paper and agree with the stress-test: the RoC Aggregations index almost certainly contains the test instances' own RoCs, so the retrieval, RAG, and re-ranker numbers are inflated. This is the main thing you should know.

The benchmark itself is a genuine contribution. AusLaw is the first large-scale citation prediction dataset for Australian law, and the release includes the data, code, and trained models. The paper also does a thorough job of comparing zero-shot prompting, retrieval, fine-tuning, and hybrids. The finding that task-specific instruction tuning jumps a near-zero baseline to ~50% is credible and useful, and the result that domain-specific pre-training alone doesn't help is worth reporting. Those points stand even after the leakage is accounted for because the instruction-tuned models never see the RoC Aggregations index.

The soft spot is load-bearing though. Section 3 describes generating an RoC for every citation instance, then Section 4.2 aggregates all RoCs per cited case into the retrieval index. The 1k test set is extracted from the same 55k instances after RoCs are generated. So for a test query, its own RoC is sitting in the index under the gold citation. That makes the similarity search trivially easier for that granularity. Full Cases and Catchwords don't contain query-specific text, so the granularity comparison in Table 1 is confounded, and every hybrid that retrieves from RoC Aggregations—query expansion, voting ensemble, RAG, re-ranker—gets an unfair boost. The paper never states that test RoCs were removed from the index. That's an omission, not an intentional cheat, but it undermines the central claim that RoC Aggregations is the best index and that the re-ranker's 52.1% is the top result.

The fix is straightforward: exclude test-instance RoCs from the database and recompute. I'd also like to see variance across runs, since the numbers jump around a lot between settings, and a clearer statement about how many RoCs were manually checked. Those are minor compared with the leakage.

Who should read it? Anyone building legal retrieval or evaluation benchmarks. The dataset is useful; the methodology needs a major revision before the results can be trusted. I'd send it to review—the benchmark deserves referee time—but the authors need to redo the retrieval experiments and be explicit about the index construction.

Referee Report

4 major / 4 minor

Summary. The paper introduces AusLaw Citation Benchmark, a dataset of 55,005 Australian legal citation instances with 18,677 unique citations, and systematically evaluates LLM-based approaches to legal citation prediction. The methods compared include zero-shot prompting of general and law-specialised LLMs, retrieval-only pipelines with different embeddings and index granularities, instruction fine-tuning, and hybrid retrieval-augmented strategies (query expansion, voting ensembles, RAG, and trained re-rankers). The main empirical claims are that zero-shot LLMs perform near 0% accuracy, instruction-tuned domain models (Cite-SaulLM-7B) reach 51.7% ACC@1, and a re-ranker over an aggregated Reason-of-Citation (RoC) index reaches 52.1% ACC@1, leaving a roughly 50% gap. The paper argues that index granularity and domain-specialised embeddings are critical, and that jurisdictional pre-training on a small Australian corpus can match or beat much larger mixed-law pre-training.

Significance. If the empirical findings are reliable, the benchmark would be a valuable resource: it is, to the authors' knowledge, the first large-scale citation prediction benchmark for Australian law, and the systematic comparison across prompting, fine-tuning, and retrieval is useful for the community. The release of data, code, and trained models is a concrete strength, and the paper makes a clear distinction between open-world and closed-world settings. However, the central quantitative claims depend on the correctness of the evaluation protocol, and the current protocol has a load-bearing leakage problem in the RoC Aggregations index that likely inflates the reported retrieval, RAG, and re-ranker numbers. The fine-tuning results are less directly affected but still lack variance reporting and test/train overlap analysis. The benchmark's value is therefore conditional on a corrected and re-run evaluation.

major comments (4)

[Sec. 3 and Sec. 4.2] The RoC Aggregations retrieval index leaks test-query information. Section 3 states that for each of the 55,005 citation instances an LLM generates a RoC from the exact sentence containing the citation, the preceding sentence, and the full text of the cited case. The test set of 1k instances is then extracted from these same 55,005 instances. Section 4.2 defines RoC Aggregations as the concatenation of all RoCs for a cited case. Consequently, for each test instance, the RoC generated from that very test instance's text is included in the database entry of the gold citation. Because retrieval queries are the same (Si-1, Mask(Si)) text from which the test RoC was generated, the gold entry contains a near-duplicate of the query. This inflates all Table 1 results that index RoC Aggregations, including retrieval-only (ACC@1 27.1/29.5), query expansion (up to 35.1), voting ensemble (up to 48.2), RAG (42.2/42.9), and the re-ranker (up to 52.1), relative to Full Cases and Catchwords granularities that contain no test-query text. The paper never states that test-instance RoCs were excluded from the index. This compromises the central claims that RoC Aggregations is the best index granularity and that the re-ranker is the best method; the experiments must be re-run with a sanitised index.
[Sec. 5 and Table 1] No variance or significance testing is reported for any of the main results. All ACC@1 values in Tables 1 and 2 appear to come from single runs, yet the paper draws fine-grained conclusions such as 'Cite-AusLawLLM-7B slightly outperforms Cite-SaulLM-7B' (Sec. 5.2) when the difference is 52.0 vs. 51.7, and the best re-ranker is 52.1 vs. 51.7 for the fine-tuned model. With a test set of 1,000 instances, a 0.4-point difference is well within typical run-to-run variance for LoRA fine-tuning and for stochastic retrieval. The authors should report multiple seeds with means and standard deviations, or otherwise perform a significance test, before making claims about the ranking of the best methods.
[Sec. 3, Appendix A.2] The relationship between training and test instances at the cited-case level is not addressed. The paper says 'we extracted 1k citations as test set' without specifying whether these are 1,000 unique citation cases or 1,000 instances, and without stating whether the remaining training instances share cited cases with the test set. If the same case appears in both training and test sets (with different citing contexts), the fine-tuned models may memorise case-specific patterns, inflating their accuracy. If, instead, the test set consists of unique cited cases, the frequency distribution of the test set would be very different from the overall benchmark (since 54% of citations appear only once), which affects the interpretation of Figure 1 and the difficulty of the task. The paper should clarify the sampling procedure and report overlap statistics.
[Sec. 3, Sec. 4.3] The quality of the LLM-generated RoCs is not adequately validated. The paper states only that 'we manually checked a subset of the LLM-generated descriptions for quality assurance and optimizing the prompt wording' and gives no size of the subset, no annotation protocol, no inter-annotator agreement, and no error rate. Since RoCs serve three roles — retrieval index entries (Sec. 4.2), supervision targets for instruction-tuned models (Sec. 4.1), and inputs to re-ranker training (Sec. 4.3) — noisy or biased RoCs can change method rankings and the benchmark's difficulty. The authors should report quantitative QA statistics and, ideally, release the QA annotations along with the benchmark.

minor comments (4)

[Sec. 5.1] There are typographical errors: 'Lawps-specialised' should be 'law-specialised', 'emeddings' should be 'embeddings', and the phrase 'pre-training pre-training' in Sec. 5.2 is duplicated.
[Table 1] Some cells in Table 1 omit ACC@5 values without explanation (e.g., the LLM-only rows and the re-ranker rows). Either fill in all values or add a note that ACC@5 is not applicable for those settings.
[Sec. 4.2] The comparison between text-embedding-3-large (3072 dimensions) and AusLaw-embedding (384 dimensions) is acknowledged as unfair, but the paper does not report retrieval latency or storage cost. A brief note on computational overhead would make the comparison more actionable.
[Appendix A.3] Hyperparameters such as learning rate, LoRA rank, and number of epochs are fixed without any ablation or sensitivity analysis. A short discussion of how these were chosen (e.g., a small validation set) would strengthen reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Test-set RoCs leak into the RoC Aggregations retrieval index, inflating retrieval and re-ranker results.

self definitional [Section 3 (dataset construction, RoC generation and test split) + Section 4.2 (RoC Aggregations index); Table 1]
"We further utilised an LLM to generate an auxiliary description of ck, based on (FullText_ck, Si_ck, Si−1_ck)... Each reference to a citation ck in the data results in a unique new RoCck... This resulted in the final dataset of 55,005 instances... From this final set, we extracted 1k citations as test set, and used the rest for training... (3) Aggregation of all its corresponding M RoCs (RoC Aggregations = concat(RoC1_ck, RoC2_ck, . . . ,RoCM_ck))."

By construction, every test query's own sentence pair (Si−1_ck, Si_ck) was used to generate a RoC for the gold citation before the 1k test split was carved out. The Section 4.2 database entry for that gold citation is the concatenation of all M RoCs for ck, so it includes the RoC generated from the test query itself. When the retrieval query Text = (Si−1_ck, Mask(Si_ck)) is embedded and compared against the RoC Aggregations index, the gold entry already contains a paraphrase of the query, making the top-1 match a form of query-to-itself similarity. This inflates the RoC Aggregations retrieval-only numbers (27.1/29.5 ACC@1 in Table 1) and the re-ranker numbers (up to 52.1) that rely on retrieving that same gold RoC among the top-5.

full rationale

Aside from the RoC Aggregations index leakage, the derivation chain is largely self-contained: zero-shot LLM results and instruction-tuned results are evaluated on the plain masked query with no fitted constants; the Text+RoC LLM-only rows are explicitly labelled as an upper-bound leakage study; the pre-tuning comparison is an empirical measurement. There are no load-bearing self-citations or imported uniqueness theorems. However, the central retrieval conclusion — that database granularity matters and that the trained re-ranker achieves the best 52.1% — is compromised because the gold citation's RoC Aggregations entry contains the RoC generated from the very test query being retrieved. This makes the retrieval/re-ranker numbers partially a self-match rather than a prediction. Score 6: partial circularity confined to the retrieval/re-ranker family; the instruction-tuning headline (Cite-SaulLM-7B 51.7%) stands independently.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the quality of the LLM-generated RoCs and the train/test splitting strategy. Several training hyperparameters and retrieval choices are fixed without sensitivity analysis, and the RoC is an internally produced entity whose accuracy is not independently verified.

free parameters (5)

Learning rate for LoRA fine-tuning = 2e-4
Chosen manually for instruction tuning and re-ranker training; affects reported accuracies.
Number of epochs = 10
Fixed across fine-tuning experiments; not swept.
LoRA rank (r) = 16
Typical choice; affects capacity of adapter.
Top-k retrieval count = 5
Used for all retrieval and re-ranking experiments; not varied.
Test set size = 1k citations
Chosen for evaluation cost; split strategy between train and test is not fully specified.

assumptions (4)

domain assumption The sentence containing the citation and the preceding sentence provide enough context to predict the cited case.
This frames the task; the paper uses this context as input for all methods (Section 3).
domain assumption The LLM-generated Reason-of-Citation descriptions are accurate enough after manual subset QA.
RoC aggregations are the best retrieval granularity and re-rankers are trained on RoCs; no quantitative QA results are reported (Section 5.1).
domain assumption Citation prediction can be posed as single-citation classification with a masked case name.
The paper acknowledges this is a simplification in its limitations; real cases often cite multiple authorities.
domain assumption Open Australian Legal Corpus is a suitable source for a representative Australian case law benchmark.
The dataset is built entirely from the NSW Caselaw subset of this corpus.

invented entities (1)

Reason-of-Citation (RoC)
purpose: An LLM-generated description of why a case is cited, used as the primary index representation for retrieval and as auxiliary supervision for fine-tuned models and re-rankers.
The RoC is generated by an LLM with manual spot-checking; no external or quantitative validation of its quality is provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study." pith.science (2026). https://pith.science/paper/4HGQ33UR

@misc{pith2026241206272,
  author       = {Pith},
  title        = {Pith review of: Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4HGQ33UR}},
  note         = {Machine review of arXiv:2412.06272}
}

read the original abstract

Large Language Models (LLMs) have demonstrated strong potential across legal tasks, yet the problem of legal citation prediction remains under-explored. At its core, this task demands fine-grained contextual understanding and precise identification of relevant legislation or precedent. We introduce the AusLaw Citation Benchmark, a real-world dataset comprising 55k Australian legal instances and 18,677 unique citations which to the best of our knowledge is the first of its scale and scope. We then conduct a systematic benchmarking across a range of solutions: (i) standard prompting of both general and law-specialised LLMs, (ii) retrieval-only pipelines with both generic and domain-specific embeddings, (iii) supervised fine-tuning, and (iv) several hybrid strategies that combine LLMs with retrieval augmentation through query expansion, voting ensembles, or re-ranking. Results show that neither general nor law-specific LLMs suffice as stand-alone solutions, with performance near zero. Instruction tuning (of even a generic open-source LLM) on task-specific dataset is among the best performing solutions. We highlight that database granularity along with the type of embeddings play a critical role in retrieval-based approaches, with hybrid methods which utilise a trained re-ranker delivering the best results. Despite this, a performance gap of nearly 50% remains, underscoring the value of this challenging benchmark as a rigorous test-bed for future research in legal-domain.

Figures

Figures reproduced from arXiv: 2412.06272 by the authors.

**Figure 1.** Average Accuracy and Number of Unique Cases for various citation frequency buckets. Accuracies per bin are based on the following settings from [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

**Figure 2.** Examples of Catchwords from different cases [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

**Figure 3.** (Top) Frequency distribution of unique cases in the data. The red vertical dashed line marks the last case [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 11 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

2024
[3]

Paul Axel-Lute. 1982. Legal citation form: Theory and practice. Law Library Journal, 75:1--27

work page 1982
[4]

Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. 2023. Blt: Can large language models handle basic legal text? arXiv preprint arXiv:2311.09693

work page Pith review arXiv 2023
[5]

Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.261 LEGAL - BERT : The muppets straight out of law school . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898--2904, Online. Association for Computational Linguistics

doi:10.18653/v1/2020.findings-emnlp.261 2020
[6]

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. https://doi.org/10.18653/v1/2022.acl-long.297 L ex GLUE : A benchmark dataset for legal language understanding in E nglish . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

doi:10.18653/v1/2022.acl-long.297 2022
[7]

Cohere. 2024. https://docs.cohere.com/v2/docs/command-r-plus Command r+

work page 2024
[8]

Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Filipe Coimbra Pereira de Melo, Gabriel Hautreux, Etienne Malaboeuf, Johanne Charpentier, Dominic Culver, and Michael Desa. 2024 a . https://openreview.net/forum?id=NLUYZ4ZqNq Saul LM -54b & saul LM -141b: Scaling up domain adaptation for the legal domain . In The Thirty-eighth Annual Conference on Neural In...

work page 2024

Show all 36 references

[9]

Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andr \' e F. T. Martins, Fabrizio Esposito, Vera L \' u cia Raposo, Sofia Morgado, and Michael Desa. 2024 b . https://doi.org/10.48550/ARXIV.2403.03883 Saullm-7b: A pioneering large langua...

2024 doi arXiv
[10]

Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E Ho. 2024. Large legal fictions: Profiling legal hallucinations in large language models. Journal of Legal Analysis, 16(1):64--93

2024
[11]

Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2024. Applicability of large language models and generative models for legal case judgement summarization. Artificial Intelligence and Law, pages 1--44

2024
[12]

Ricardo Dominguez-Olmedo, Vedant Nanda, Rediet Abebe, Stefan Bechtold, Christoph Engel, Jens Frankenreiter, Krishna Gummadi, Moritz Hardt, and Michael Livermore. 2024. Lawma: The power of specialization for legal tasks. arXiv preprint arXiv:2407.16615

2024 arXiv
[13]

Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. 2023. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289

2023 arXiv
[14]

Aaron Grattafiori and 1 others. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

2024 arXiv
[15]

Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2024. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. A...

2024
[16]

Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268

2021 arXiv
[17]

Nils Holzenberger, Andrew Blair - Stanek, and Benjamin Van Durme. 2020. https://arxiv.org/abs/2005.05257 A dataset for statutory reasoning in tax law entailment and question answering . CoRR, abs/2005.05257

2020 arXiv
[18]

Abe Hou, William Jurayj, Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2024 a . Gaps or hallucinations? scrutinizing machine-generated legal analysis for fine-grained text evaluations. In Proceedings of the Natural Legal Language Processing Workshop 2024, pag...

2024
[19]

Abe Bohan Hou, Orion Weller, Guanghui Qin, Eugene Yang, Dawn Lawrie, Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2024 b . https://arxiv.org/pdf/2406.17186 Clerc: A dataset for legal case retrieval and retrieval-augmented analysis generation . ArXiv

2024 arXiv
[20]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

2022
[21]

Yinghao Hu, Leilei Gan, Wenyi Xiao, Kun Kuang, and Fei Wu. 2025. Fine-tuning large language models for improving factuality in legal question answering. arXiv preprint arXiv:2501.06521

2025 arXiv
[22]

Cong Jiang and Xiaolei Yang. 2023. https://doi.org/10.1145/3594536.3595170 Legal syllogism prompting: Teaching large language models for legal judgment prediction . In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL '23, page 41...

2023
[23]

Abhinav Joshi, Shounak Paul, Akshat Sharma, Pawan Goyal, Saptarshi Ghosh, and Ashutosh Modi. 2024. Il-tur: Benchmark for indian legal text understanding and reasoning. arXiv preprint arXiv:2407.05399

2024 arXiv
[24]

Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, and 1 others. 2024 a . Legalagentbench: Evaluating llm agents in legal domain. arXiv preprint arXiv:2412.17259

2024 arXiv
[25]

Haitao Li, You Chen, Qingyao Ai, Yueyue Wu, Ruizhe Zhang, and Yiqun Liu. 2024 b . Lexeval: A comprehensive chinese legal benchmark for evaluating large language models. arXiv preprint arXiv:2409.20288

2024 arXiv
[26]

Fangyu Liu, Ivan Vuli \'c , Anna Korhonen, and Nigel Collier. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.109 Fast, effective, and self-supervised: Transforming masked language models into universal lexical and sentence encoders . In Proceedings of the 2021 Conference on...

2021 doi
[27]

Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D Manning, and Daniel E Ho. 2024. Hallucination-free? assessing the reliability of leading ai legal research tools. arXiv preprint arXiv:2405.20362

2024 arXiv
[28]

OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Gpt-4o

2024
[29]

Xiao Peng and Liang Chen. 2024. Athena: Retrieval-augmented legal judgment prediction with large language models. arXiv preprint arXiv:2410.11195

2024 arXiv
[30]

Nicholas Pipitone and Ghita Houir Alami. 2024. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain. arXiv preprint arXiv:2408.10343

2024 arXiv
[31]

Thiago Dal Pont, Federico Galli, Andrea Loreggia, Giuseppe Pisano, Riccardo Rovatti, and Giovanni Sartor. 2023. Legal summarisation through llms: The prodigit project. arXiv preprint arXiv:2308.04416

2023 arXiv
[32]

Frederick Schauer. 1987. Precedent. Stanford Law Review, 39(3):571--605

1987
[33]

Dong Shu, Haoran Zhao, Xukun Liu, David Demeter, Mengnan Du, and Yongfeng Zhang. 2024. https://doi.org/10.1145/3627673.3680020 Lawllm: Law large language model for the US legal system . In Proceedings of the 33rd ACM International Conference on Information and Knowledge Manage...

2024
[34]

Steven H Wang, Antoine Scardigli, Leonard Tang, Wei Chen, Dimitry Levkin, Anya Chen, Spencer Ball, Thomas Woodside, Oliver Zhang, and Dan Hendrycks. 2023. Maud: An expert-annotated legal nlp dataset for merger agreement understanding. arXiv preprint arXiv:2301.00876

2023 arXiv
[35]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[36]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

2024

[3] [3]

Paul Axel-Lute. 1982. Legal citation form: Theory and practice. Law Library Journal, 75:1--27

work page 1982

[4] [4]

Andrew Blair-Stanek, Nils Holzenberger, and Benjamin Van Durme. 2023. Blt: Can large language models handle basic legal text? arXiv preprint arXiv:2311.09693

work page Pith review arXiv 2023

[5] [5]

Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.261 LEGAL - BERT : The muppets straight out of law school . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898--2904, Online. Association for Computational Linguistics

doi:10.18653/v1/2020.findings-emnlp.261 2020

[6] [6]

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. https://doi.org/10.18653/v1/2022.acl-long.297 L ex GLUE : A benchmark dataset for legal language understanding in E nglish . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

doi:10.18653/v1/2022.acl-long.297 2022

[7] [7]

Cohere. 2024. https://docs.cohere.com/v2/docs/command-r-plus Command r+

work page 2024

[8] [8]

Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Filipe Coimbra Pereira de Melo, Gabriel Hautreux, Etienne Malaboeuf, Johanne Charpentier, Dominic Culver, and Michael Desa. 2024 a . https://openreview.net/forum?id=NLUYZ4ZqNq Saul LM -54b & saul LM -141b: Scaling up domain adaptation for the legal domain . In The Thirty-eighth Annual Conference on Neural In...

work page 2024