Pith. sign in

REVIEW 5 major objections 5 minor 127 references

A Survey of Classification Tasks and Approaches for Legal Contracts

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Legal contract classification has seven recurring tasks, fourteen English-language datasets, and three method families, and this paper is the first to survey them together.

desk verdict A useful first survey with a sound taxonomy, but the summary tables contain numeric errors that need correction before the paper can serve as a reliable reference. read the letter →

arxiv 2507.21108 v1 pith:ZLY2LEEI submitted 2025-07-09 cs.CL cs.AI

Amrita Singh , Aditya Joshi , Jiaojiao Jiang , Hye-young Paik This is my paper · ORCID
classification cs.CLcs.AI
keywords legalcontractclassificationNLPcontractualtransformermodelsdeeplearningsurveytaxonomydatasetsnaturallanguageinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Legal contract classification is the task of labeling clauses, provisions, or entire contracts with categories such as topic, risk, obligation type, or ambiguity; automating it matters because manual review is time-consuming, costly, and error-prone. This survey claims to be the first comprehensive review focused exclusively on that task, organizing the field into seven classification tasks, fourteen English-language datasets (eleven public, one non-public, two proprietary), and a methodology taxonomy covering traditional machine learning, classical deep learning, and transformer-based approaches. It synthesizes 35 studies, lays out the evaluation metrics used, and reports best-achieved performance per task in tabular form. A sympathetic reader would use this as a map: which tasks exist, which labeled data supports each, and which method families currently lead on what.

What carries the argument

The organizing machinery is a pair of taxonomies plus summary tables. The task taxonomy (Table 3) defines the seven classification tasks with example clauses and labels; the dataset inventory (Table 4) groups fourteen datasets under those tasks with source, size, annotation scheme, and class counts; the methodology taxonomy (Figure 4 and Tables 5-12) classifies approaches into traditional machine learning, classical deep learning, and transformer-based methods, with transformer methods split into pre-training-, prompting-, fine-tuning-, compression-, and miscellaneous-based. These categories carry the survey's argument because they turn a scattered literature into a structured map from which gaps can be read off.

What would settle it

Checking the primary sources of the fourteen datasets would settle the accuracy of Table 4; for example, comparing LEDGAR's official release counts with the reported 60,540 contracts and 846,274 provisions, or ContractNLI's release with the reported 607 contracts. Finding a published survey focused exclusively on legal contract classification that predates this one would falsify the 'first comprehensive survey' claim directly.

Watch

Extended reading notes

Core claim

The paper's central claim is that legal contract classification is a distinct, maturing subfield of legal NLP whose work can be systematically organized. It names seven recurring tasks: topic classification, risky/unfair clause identification, deontic modality classification, contractual ambiguity identification, norm conflict identification, obligatory clause classification, and natural language inference for contracts. It inventories fourteen datasets for English-language contracts, grouped by those tasks, and introduces a methodology taxonomy with three families—traditional machine learning, classical deep learning, and transformer-based methods (subdivided into pre-training, prompting, fine-tuning, compression, and miscellaneous). Across the reviewed results, transformer-based models, especially domain-adapted ones, generally outperform classical baselines on topic classification and unfair-clause identification, while fine-tuned models beat zero- and few-shot prompting on nuanced label sets; the paper presents these as the field's current best-achieved results.

Load-bearing premise

The survey's usefulness depends on its literature search and inclusion criteria having recovered the complete set of relevant work, and on the reported dataset statistics and performance numbers faithfully reflecting the original papers.

Editorial extensions

If this is right

  • A researcher beginning legal contract classification can locate the relevant dataset, task definition, and strongest reported baseline directly from the survey's tables.
  • For topic and unfair-clause tasks, fine-tuned or domain-adapted transformer models are the reliable default; zero-shot and few-shot prompting with general-purpose LLMs remains weaker on fixed label sets.
  • The absence of a dedicated multi-task contractual benchmark is identified as a concrete gap; building one is the paper's proposed next step.
  • Deontic modality and obligatory clause classification cannot yet be compared across studies because datasets and metrics differ, so standardized evaluation is needed before progress in those tasks can be measured.
  • Small sample sizes in datasets such as the Contract Ambiguity corpus limit reliability, so larger, multi-jurisdictional corpora are required.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the survey's own claims, the fourteen datasets could be fused into a shared multi-task benchmark, allowing one model to be evaluated across topic, risk, deontic, ambiguity, norm, obligation, and NLI tasks at once.
  • The reported failures of general-purpose prompting on legal classification suggest a testable extension: prompt suites designed with lawyers and probed for hallucination (statutory misinterpretation, jurisprudential drift) could become a standalone evaluation benchmark.
  • Because most datasets come from U.S. or EU sources, cross-lingual transfer (for example, English-German clause classification) is a plausible low-cost way to test whether current methods generalize beyond common-law drafting conventions.
  • The result table shows best-achieved scores varying widely across tasks, so a user should treat cross-paper numbers as indicative rather than directly comparable, since datasets and metrics differ.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The manuscript is a survey of automatic legal contract classification (LCC). It identifies seven classification tasks, reviews fourteen English-language contract datasets, and organizes the methodology into Traditional Machine Learning, Deep Learning, and Transformer-based approaches. It also discusses evaluation metrics, reports best-achieved performance in a summary table, and outlines challenges and future research directions. The paper claims to be the first comprehensive survey focused exclusively on LCC.

Significance. If the survey's tables are accurate, it would be a valuable entry point for researchers and practitioners: it provides a documented snowballing methodology, a structured task taxonomy, a compilation of 35 studies, and a discussion of evaluation practices and open problems. These organizational contributions are genuinely useful. However, the paper's reference value hinges on the correctness of its summary tables and the consistency between the text and the tables; several numeric and internal inconsistencies currently undermine that reliability. The paper does not ship code or machine-checked proofs, so its contribution rests entirely on the accuracy and completeness of its literature synthesis.

major comments (5)
  1. [Table 4, Fine-grained Obligation row; §5.2.6] The table reports the size of the Fine-grained Obligation dataset as '50 contracts (16,538 sentences)', but §5.2.6 states that the corpus contains 50 contracts totaling 57,200 statements, of which 16,538 are obligation clauses. As printed, the table conflates the obligation-clause subset with the full corpus and uses 'sentences' instead of 'statements'. Because the tables are the advertised 'easy access' point of the survey, this is a load-bearing accuracy issue.
  2. [Table 13, Graham et al. row] The Graham et al. (2023) entry reports P=89, R=8, F1=89 for deontic modality classification. With precision 89 and recall 8, the F1-score would be approximately 14.7, not 89, so at least one of the three numbers is erroneous or the row mixes metrics from different settings. Since Table 13 is the summarizing reference for best-reported performance, this entry must be checked against the source and corrected or annotated.
  3. [§6.3.2 vs Table 13, Wang and Zhao (2024)] The text says that Wang and Zhao (2024) do not provide prompts for multi-label or multi-class tasks, and consequently the specific prompts for LEDGAR (multi-class) and UNFAIR-ToS (multi-label) 'remain unavailable'; yet Table 13 lists Wang and Zhao results for exactly those two tasks (Micro-F1 78.1/75.6, Macro-F1 62.8/55.8). This is internally inconsistent. Either the prompts are available and the results should be reported, or the table should not list those numbers without the underlying prompt definitions.
  4. [Table 4, LEGALBENCH and LexGLUE rows] The size entries for LEGALBENCH ('41 tasks: [2-8]') and LexGLUE ('2 tasks: [9,100]') are not interpretable as dataset sizes, and they do not match the accompanying text in §5.2.8, which describes LEGALBENCH as containing 162 tasks from 36 data sources and LexGLUE as containing seven datasets (two contract-related). These cells need to be rewritten with unambiguous values so that Table 4 can serve as a reliable reference.
  5. [§4 and Table 5; §6.2.2, §6.3.2] Table 5 is presented as the overview of task-specific methodology for the 35 reviewed articles, but the text discusses Neill et al. (2017) in §6.2.2 and Funaki et al. (2020) in §5.1/§6.3.2, and neither appears in Table 5. This makes it impossible to reconstruct the reviewed corpus from the summary table and weakens the survey's completeness claim. Please reconcile Table 5 with all works actually discussed, or mark background citations separately.
minor comments (5)
  1. [Section 9] 'Thirty-five approaches' should be 'thirty-five studies' (or 'approaches from thirty-five studies') to match the methodology section.
  2. [Table 3, Contractual Ambiguity row] The second example sentence contains a duplicated phrase ('Either Party may pledge this Agreement to Either Party may pledge this Agreement').
  3. [Section 7.2] The displayed Micro-F1 and Macro-F1 formulas have missing parentheses in the denominators; for example, P_micro should be (sum_i TP_i) / (sum_i (TP_i + FP_i)).
  4. [Section 7.3] The statement that an AUC-PR of 0.3 means the model performs no better than random guessing is not generally true; the random baseline for precision-recall curves depends on class prevalence.
  5. [Table 4, Norm row] '1193 and 111 sentences' is ambiguous; §5.2.5 describes 1,193 clauses and 111 conflicting norm pairs, so the table should distinguish these quantities.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; self-citation is descriptive and non-load-bearing, though summary tables contain numeric errors.

full rationale

The paper is a survey, not a derivation chain: it does not fit parameters and then predict a quantity that reduces to the fit. Its central claims—seven LCC tasks, fourteen datasets, a methodology taxonomy, and first-survey status—are descriptive claims about the reviewed literature. The only self-citations are to Singh et al. (2024), which appears as one reviewed dataset (Fine-grained Obligation), one reviewed method (DDH/BiLSTM-Att), and in background statements about long contracts and obligation tracking. These citations are descriptive rather than load-bearing: the survey does not invoke Singh et al. (2024) to justify the task taxonomy, to exclude alternative approaches, or to prove that its own choice is forced. The 'Obligatory Clause Classification' task category is supported by independent prior work (Sainani et al., 2020) as well, and the methodology tables aggregate many external studies. No equation or table entry is equivalent by construction to an input. Two numeric inconsistencies do appear—Table 4 lists Fine-grained Obligation as '50 contracts (16,538 sentences)' while Section 5.2.6 reports 57,200 statements of which 16,538 are obligation clauses, and Table 13 lists Graham et al. (2023) as 'P: 89, R: 8, F1: 89, Acc: 88', which is arithmetically impossible since F1 would be about 14.7 with P=89 and R=8—but these are accuracy/verification errors in summary tables, not circular reasoning. Weighted as minor self-citation with independent content, the circularity burden is low.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The survey has no free parameters or invented entities. It relies on assumptions about literature coverage and taxonomy validity.

assumptions (2)
  • domain assumption The 35 selected articles are representative of the LCC literature.
    Section 4 describes a search with inclusion/exclusion criteria and snowballing, but does not report a PRISMA-style flow diagram, per-database counts, or the list of excluded papers, so completeness is unverified.
  • ad hoc to paper The seven identified tasks form a valid partition of LCC.
    The task list is constructed by the authors from the reviewed papers; no formal justification is given that these categories are exhaustive or orthogonal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of Classification Tasks and Approaches for Legal Contracts." pith.science (2026). https://pith.science/paper/ZLY2LEEI

@misc{pith2026250721108,
  author       = {Pith},
  title        = {Pith review of: A Survey of Classification Tasks and Approaches for Legal Contracts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZLY2LEEI}},
  note         = {Machine review of arXiv:2507.21108}
}
read the original abstract

Given the large size and volumes of contracts and their underlying inherent complexity, manual reviews become inefficient and prone to errors, creating a clear need for automation. Automatic Legal Contract Classification (LCC) revolutionizes the way legal contracts are analyzed, offering substantial improvements in speed, accuracy, and accessibility. This survey delves into the challenges of automatic LCC and a detailed examination of key tasks, datasets, and methodologies. We identify seven classification tasks within LCC, and review fourteen datasets related to English-language contracts, including public, proprietary, and non-public sources. We also introduce a methodology taxonomy for LCC, categorized into Traditional Machine Learning, Deep Learning, and Transformer-based approaches. Additionally, the survey discusses evaluation techniques and highlights the best-performing results from the reviewed studies. By providing a thorough overview of current methods and their limitations, this survey suggests future research directions to improve the efficiency, accuracy, and scalability of LCC. As the first comprehensive survey on LCC, it aims to support legal NLP researchers and practitioners in improving legal processes, making legal information more accessible, and promoting a more informed and equitable society.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

127 extracted references · 77 canonical work pages

  1. [1]

    In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp 308--318

    Abadi M, Chu A, Goodfellow I, et al (2016) Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp 308--318

    2016

  2. [2]

    https://aclanthology.org/, accessed: 10 March 2025

    ACL Anthology (2025) Acl anthology. https://aclanthology.org/, accessed: 10 March 2025

    2025

  3. [3]

    http://portal.acm.org/, accessed: 10 March 2025

    ACM Digital Library (2025) Acm digital library. http://portal.acm.org/, accessed: 10 March 2025

    2025

  4. [4]

    In: Proceedings of Sixth International Congress on Information and Communication Technology: ICICT 2021, London, Volume 4, Springer, pp 763--771

    Aejas B, Bouras A, Belhi A, et al (2022) A review of contract entity extraction. In: Proceedings of Sixth International Congress on Information and Communication Technology: ICICT 2021, London, Volume 4, Springer, pp 763--771

    2022

  5. [5]

    Aires JP, Meneguzzi F (2021) Norm conflict identification using a convolutional neural network. In: Coordination, Organizations, Institutions, Norms, and Ethics for Governance of Multi-Agent Systems XIII: International Workshops COIN 2017 and COINE 2020, Sao Paulo, Brazil, May 8-9, 2017 and Virtual Event, May 9, 2020, Revised Selected Papers, Springer, pp 3--19

    2021

  6. [6]

    Artificial Intelligence and Law 25(4):397--428

    Aires JP, Pinheiro D, Lima VSd, et al (2017) Norm conflict identification in contracts. Artificial Intelligence and Law 25(4):397--428

    2017

  7. [7]

    In: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1--8

    Aires JP, Monteiro J, Granada R, et al (2018) Norm conflict identification using vector space offsets. In: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1--8

    2018

  8. [8]

    PhD thesis

    Amoah MO (2021) Effectiveness of evaluation processes to increase organisational transparency and efficiency in contracts management: Kma. PhD thesis

    2021

Show all 127 references
  1. [9]

    ACM Comput Surv 1(1)

    Ariai F, Demartini G (2024) Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. ACM Comput Surv 1(1)

    2024

  2. [10]

    Jones International University

    Armstead J (2015) Implementing Job Classification Procedures into the Human Resource Certification Institute Official PHR and SPHR Certification Guide . Jones International University

    2015

  3. [11]

    Artificial intelligence and law 32(3):839--862

    Braun D (2024) I beg to differ: how disagreement is handled in the annotation of legal machine learning data sets. Artificial intelligence and law 32(3):839--862

    2024

  4. [12]

    In: The 5th Workshop on e-Commerce and NLP, ECNLP 2022, Association for Computational Linguistics (ACL), pp 199--209

    Braun D, Matthes F (2022) Clause topic classification in german and english standard form contracts. In: The 5th Workshop on e-Commerce and NLP, ECNLP 2022, Association for Computational Linguistics (ACL), pp 199--209

    2022

  5. [13]

    Sustainability 16(11):4669

    Cardona LF, Guzm \'a n-Luna JA, Restrepo-Carmona JA (2024) Bibliometric analysis of intelligent systems for early anomaly detection in oil and gas contracts: Exploring recent progress and challenges. Sustainability 16(11):4669

    2024

  6. [14]

    Available at SSRN 4385460

    Chalkidis I (2023) Chatgpt may pass the bar exam soon, but has a long way to go for the lexglue benchmark. Available at SSRN 4385460

    2023

  7. [15]

    Artificial Intelligence and Law 27(2):171--198

    Chalkidis I, Kampas D (2019) Deep learning in law: early adaptation and legal word embeddings trained on large corpora. Artificial Intelligence and Law 27(2):171--198

    2019

  8. [16]

    In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp 254--259

    Chalkidis I, Androutsopoulos I, Michos A (2018) Obligation and prohibition extraction using hierarchical rnns. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp 254--259

    2018

  9. [17]

    In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp 2898--2904

    Chalkidis I, Fergadiotis M, Malakasiotis P, et al (2020) Legal-bert: The muppets straight out of law school. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp 2898--2904

    2020

  10. [18]

    Chalkidis I, Fergadiotis M, Tsarapatsanis D, et al (2021) Paragraph-level rationale extraction through regularization: A case study on E uropean court of human rights cases. In: Toutanova K, Rumshisky A, Zettlemoyer L, et al (eds) Proceedings of the 2021 Conference of the Nort...

    2021

  11. [19]

    In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 4310--4330

    Chalkidis I, Jana A, Hartung D, et al (2022) Lexglue: A benchmark dataset for legal language understanding in english. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 4310--4330

    2022

  12. [20]

    arXiv preprint arXiv:230507507

    Chalkidis I, Garneau N, Goanta C, et al (2023) Lexfiles and legallama: Facilitating english multinational legal language model development. arXiv preprint arXiv:230507507

    2023

  13. [21]

    arXiv preprint arXiv:231014735

    Chen B, Zhang Z, Langren \'e N, et al (2023) Unleashing the potential of prompt engineering in large language models: a comprehensive review. arXiv preprint arXiv:231014735

    2023

  14. [22]

    In: The Twelfth International Conference on Learning Representations

    Cheng D, Huang S, Wei F (2023) Adapting large language models via reading comprehension. In: The Twelfth International Conference on Learning Representations

    2023

  15. [23]

    Automation in Construction 154:105020

    Chung S, Moon S, Kim J, et al (2023) Comparing natural language processing (nlp) applications in construction and computer science using preferred reporting items for systematic reviews (prisma). Automation in Construction 154:105020

    2023

  16. [24]

    arXiv preprint arXiv:240303883

    Colombo P, Pires TP, Boudiaf M, et al (2024) Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:240303883

    2024

  17. [25]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 8440--8451

    Conneau A, Khandelwal K, Goyal N, et al (2020) Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 8440--8451

    2020

  18. [26]

    Cui J, Ning M, Li Z, et al (2024) Chatlaw: A multi-agent collaborative legal assistant with knowledge graph enhanced mixture-of-experts large language model

    2024

  19. [27]

    In: Proceedings of the Australasian Language Technology Association Workshop

    Curtotti M, Mccreath E (2010) Corpus based classification of text in australian contracts. In: Proceedings of the Australasian Language Technology Association Workshop

    2010

  20. [28]

    arXiv preprint arXiv:150103726

    Danezis G, Domingo-Ferrer J, Hansen M, et al (2015) Privacy and data protection by design-from policy to engineering. arXiv preprint arXiv:150103726

    2015

  21. [29]

    Devlin J, Chang MW, Lee K, et al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume ...

    2019

  22. [30]

    The modern law review 82(4):633--664

    Edmond G, Martire KA (2019) Just cognition: scientific research on bias and some implications for legal procedure and decision-making. The modern law review 82(4):633--664

    2019

  23. [31]

    In: Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol

    Freitas LJG (2024) Text clustering applied to unbalanced data in legal contexts. In: Proceedings of the 16th International Conference on Computational Processing of Portuguese-Vol. 1, pp 639--642

    2024

  24. [32]

    Oxford University Press, USA

    Fried C (2015) Contract as promise: A theory of contractual obligation. Oxford University Press, USA

    2015

  25. [33]

    In: Proceedings of the Twelfth Language Resources and Evaluation Conference, pp 2045--2053

    Funaki R, Nagata Y, Suenaga K, et al (2020) A contract corpus for recognizing rights and obligations. In: Proceedings of the Twelfth Language Resources and Evaluation Conference, pp 2045--2053

    2020

  26. [34]

    In: Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp 101--108

    Gao X, Singh MP (2014) Extracting normative relationships from business contracts. In: Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pp 101--108

    2014

  27. [35]

    In: International conference on artificial intelligence and statistics, PMLR, pp 1287--1296

    Garreau D, Luxburg U (2020) Explaining the explainer: A first theoretical analysis of lime. In: International conference on artificial intelligence and statistics, PMLR, pp 1287--1296

    2020

  28. [36]

    In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp 409--416

    Gee L, Zugarini A, Rigutini L, et al (2022) Fast vocabulary transfer for language model compression. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp 409--416

    2022

  29. [37]

    In: Wang M, Zitouni I (eds) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track

    Gee L, Rigutini L, Ernandes M, et al (2023) Multi-word tokenization for sequence compression. In: Wang M, Zitouni I (eds) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics, Singapor...

    2023

  30. [38]

    arXiv preprint arXiv:210906862

    Geng S, Lebret R, Aberer K (2021) Legal transformer models may not always help. arXiv preprint arXiv:210906862

    2021

  31. [39]

    In: Bouamor H, Pino J, Bali K (eds) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

    Ghosh S, Evuru CKR, Kumar S, et al (2023) DALE : Generative data augmentation for low-resource legal NLP . In: Bouamor H, Pino J, Bali K (eds) Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Sin...

    2023

  32. [40]

    https://scholar.google.com/, accessed: 10 March 2025

    Google Scholar (2025) Google scholar. https://scholar.google.com/, accessed: 10 March 2025

    2025

  33. [41]

    Artificial Intelligence and Law pp 1--22

    Graham SG, Soltani H, Isiaq O (2023) Natural language processing for legal document review: categorising deontic modalities in contracts. Artificial Intelligence and Law pp 1--22

    2023

  34. [42]

    In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp 9647--9676

    Gretz S, Halfon A, Shnayderman I, et al (2023) Zero-shot topical text classification with llms-an experimental study. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp 9647--9676

    2023

  35. [43]

    Neural Computing and Applications 33:17569--17587

    Guarino A, Lettieri N, Malandrino D, et al (2021) A machine learning-based approach to identify unlawful practices in online terms of service: analysis, implementation and evaluation. Neural Computing and Applications 33:17569--17587

    2021

  36. [44]

    Advances in Neural Information Processing Systems 36

    Guha N, Nyarko J, Ho D, et al (2024) Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems 36

    2024

  37. [45]

    Legal Informatics, Cambridge, UK: Cambridge University Press Published ahead of print as part of doctoral dissertation 37

    Haapio H, Passera S (2017) Contracts as interfaces: exploring visual representation patterns in contract design. Legal Informatics, Cambridge, UK: Cambridge University Press Published ahead of print as part of doctoral dissertation 37

    2017

  38. [46]

    Journal of Construction Engineering and Management 147(9):03121004

    Hassan Fu, Le T, Lv X (2021) Addressing legal and contractual matters in construction using natural language processing: A critical review. Journal of Construction Engineering and Management 147(9):03121004

    2021

  39. [47]

    Advances in Neural Information Processing Systems 35:29217--29234

    Henderson P, Krass M, Zheng L, et al (2022) Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances in Neural Information Processing Systems 35:29217--29234

    2022

  40. [48]

    Hendrycks D, Burns C, Chen A, et al (2021) Cuad: An expert-annotated nlp dataset for legal contract review. NeurIPS

    2021

  41. [49]

    arXiv preprint arXiv:230515062

    Huang Q, Tao M, Zhang C, et al (2023) Lawyer llama technical report. arXiv preprint arXiv:230515062

    2023

  42. [50]

    https://ieeexplore.ieee.org/, accessed: 10 March 2025

    IEEE Xplore (2025) Ieee xplore digital library. https://ieeexplore.ieee.org/, accessed: 10 March 2025

    2025

  43. [51]

    In: Proceedings of the third annual ACM Bangalore conference, pp 1--5

    Indukuri KV, Krishna PR (2010) Mining e-contract documents to classify clauses. In: Proceedings of the third annual ACM Bangalore conference, pp 1--5

    2010

  44. [52]

    In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 1275--1280

    Joshi V, Anish PR, Ghaisas S (2021) Domain adaptation for an automated classification of deontic modalities in software engineering contracts. In: Proceedings of the 29th ACM joint meeting on European software engineering conference and symposium on the foundations of software...

    2021

  45. [53]

    Legal Analytics pp 47--62

    Katrak M (2022) The role of language prediction models in contractual interpretation: the challenges and future prospects of gpt-3. Legal Analytics pp 47--62

    2022

  46. [54]

    PloS one 17(1):e0262710

    Khan GM, Khan SU, Khan HU, et al (2022) Challenges and practices identification in complex outsourcing relationships: A systematic literature review. PloS one 17(1):e0262710

    2022

  47. [55]

    https://kirasystems.com, accessed: 2025-06-02

    Kira Systems (2025) Kira Systems: AI Contract Analysis Software . https://kirasystems.com, accessed: 2025-06-02

    2025

  48. [56]

    Advances in neural information processing systems 35:22199--22213

    Kojima T, Gu SS, Reid M, et al (2022) Large language models are zero-shot reasoners. Advances in neural information processing systems 35:22199--22213

    2022

  49. [57]

    In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp 1907--1919

    Koreeda Y, Manning CD (2021) Contractnli: A dataset for document-level natural language inference for contracts. In: Findings of the Association for Computational Linguistics: EMNLP 2021, pp 1907--1919

    2021

  50. [58]

    Lee JS (2023) Lexgpt 0.1: pre-trained gpt-j models with pile of law. In: Proceedings of the Seventeenth International Workshop on Juris-Informatics 2023 (JURISIN 2023): in association with JSAI International Symposia on AI 2023 (IsAI-2023), pp 15--24

    2023

  51. [59]

    Lee K, He L, Zettlemoyer L (2018) Higher-order coreference resolution with coarse-to-fine inference. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp 687--692

    2018

  52. [60]

    arXiv preprint arXiv:201010386

    Leivaditi S, Rossi J, Kanoulas E (2020) A benchmark for lease contract review. arXiv preprint arXiv:201010386

    2020

  53. [61]

    Advances in neural information processing systems 33:9459--9474

    Lewis P, Perez E, Piktus A, et al (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33:9459--9474

    2020

  54. [62]

    In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp 1876--1888

    Lin YC, Chen SA, Liu JJ, et al (2023) Linear classifier: An often-forgotten baseline for text classification. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp 1876--1888

    2023

  55. [63]

    Artificial Intelligence and Law 27:117--139

    Lippi M, Pa ka P, Contissa G, et al (2019) Claudette: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law 27:117--139

    2019

  56. [64]

    In: International Conference on Learning Representations

    Madry A, Makelov A, Schmidt L, et al (2018) Towards deep learning models resistant to adversarial attacks. In: International Conference on Learning Representations

    2018

  57. [65]

    In: International conference on data intelligence and cognitive informatics, Springer, pp 387--402

    Marvin G, Hellen N, Jjingo D, et al (2023) Prompt engineering in large language models. In: International conference on data intelligence and cognitive informatics, Springer, pp 387--402

    2023

  58. [66]

    In: 2014 IEEE 22nd international requirements engineering conference (RE), IEEE, pp 83--92

    Massey AK, Rutledge RL, Ant \'o n AI, et al (2014) Identifying and classifying ambiguity for regulatory requirements. In: 2014 IEEE 22nd international requirements engineering conference (RE), IEEE, pp 83--92

    2014

  59. [67]

    In: Artificial intelligence and statistics, PMLR, pp 1273--1282

    McMahan B, Moore E, Ramage D, et al (2017) Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics, PMLR, pp 1273--1282

    2017

  60. [68]

    In: 2020 International Conference on Data Mining Workshops (ICDMW), IEEE, pp 775--781

    Montelongo A, Becker JL (2020) Tasks performed in the legal domain through deep learning: A bibliometric review (1987--2020). In: 2020 International Conference on Data Mining Workshops (ICDMW), IEEE, pp 775--781

    2020

  61. [69]

    Springer

    Montemagni S, Peters W, Tiscornia D (2010) Semantic Processing of Legal Texts. Springer

    2010

  62. [70]

    In: Proceedings of the 29th international conference on computational linguistics, pp 4593--4603

    Mosca E, Szigeti F, Tragianni S, et al (2022) Shap-based explanation methods: a review for nlp interpretability. In: Proceedings of the 29th international conference on computational linguistics, pp 4593--4603

    2022

  63. [71]

    In: Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law, pp 159--168

    Neill JO, Buitelaar P, Robin C, et al (2017) Classifying sentential modality in legal language: a use case in financial regulations, acts and directives. In: Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law, pp 159--168

    2017

  64. [72]

    In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp 3016--3054

    Niklaus J, Matoshi V, Rani P, et al (2023 a ) Lextreme: A multi-lingual and multi-task benchmark for the legal domain. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp 3016--3054

    2023

  65. [73]

    ArXiv abs/2306.02069

    Niklaus J, Matoshi V, Sturmer M, et al (2023 b ) Multilegalpile: A 689gb multilingual legal corpus. ArXiv abs/2306.02069

    2023 arXiv

  66. [74]

    World J Adv Res Rev 25:1315--1333

    Osifo EO, Omumu ES, Alozie M (2025) Evolving contractual obligations in construction law: Implications of regulatory changes on project delivery. World J Adv Res Rev 25:1315--1333

    2025

  67. [75]

    In: International Workshop on Rules and Rule Markup Languages for the Semantic Web, Springer, pp 298--312

    Palmirani M, Governatori G, Rotolo A, et al (2011) Legalruleml: Xml-based rules and norms. In: International Workshop on Rules and Rule Markup Languages for the Semantic Web, Springer, pp 298--312

    2011

  68. [76]

    In: Proceedings of the AAAI conference on artificial intelligence, pp 11139--11146

    Paul S, Goyal P, Ghosh S (2022) Lesicin: A heterogeneous graph-based approach for automatic legal statute identification from indian legal documents. In: Proceedings of the AAAI conference on artificial intelligence, pp 11139--11146

    2022

  69. [77]

    In: Proceedings of 19th International Conference on Artificial Intelligence and Law - ICAIL 2023

    Paul S, Mandal A, Goyal P, et al (2023) Pre-trained language models for the legal domain: A case study on indian law. In: Proceedings of 19th International Conference on Artificial Intelligence and Law - ICAIL 2023

    2023

  70. [78]

    Legal Studies in Digital Age 2(3):1--12

    Pazhohan H (2023) Global data protection standards: A comparative analysis of gdpr and other international privacy laws. Legal Studies in Digital Age 2(3):1--12

    2023

  71. [79]

    In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

    Rajpurkar P, Jia R, Liang P (2018) Know what you don ' t know: Unanswerable questions for SQ u AD . In: Gurevych I, Miyao Y (eds) Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Lin...

    2018

  72. [80]

    Intersentia

    Reich N, Micklitz HW, Rott P, et al (2014) European consumer law. Intersentia

    2014

  73. [81]

    Digital Society 3(1):1

    Richmond KM, Muddamsetty SM, Gammeltoft-Hansen T, et al (2024) Explainable ai and law: an evidential survey. Digital Society 3(1):1

    2024

  74. [82]

    https://www.rossintelligence.com, accessed: 2025-06-02

    ROSS Intelligence (2025) ROSS Intelligence: AI for Legal Research . https://www.rossintelligence.com, accessed: 2025-06-02

    2025

  75. [83]

    Artificial Intelligence and Law 30(1):59--92

    Ruggeri F, Lagioia F, Lippi M, et al (2022) Detecting and explaining unfairness in consumer contracts through memory networks. Artificial Intelligence and Law 30(1):59--92

    2022

  76. [84]

    In: 2020 IEEE 28th international requirements engineering conference (RE), IEEE, pp 147--157

    Sainani A, Anish PR, Joshi V, et al (2020) Extracting and classifying requirements from software engineering contracts. In: 2020 IEEE 28th international requirements engineering conference (RE), IEEE, pp 147--157

    2020

  77. [85]

    In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp 11563--11579

    Sancheti A, Garimella A, Srinivasan BV, et al (2022) Agent-specific deontic modality detection in legal language. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp 11563--11579

    2022

  78. [86]

    In: Natural Legal Language Processing Workshop 2024, pp 381--403

    Santosh T, Weiss C, Grabmair M (2024) Lexsumm and lext5: Benchmarking and modeling legal summarization tasks in english. In: Natural Legal Language Processing Workshop 2024, pp 381--403

    2024

  79. [87]

    Frontiers in Artificial Intelligence 6:1279794

    Savelka J, Ashley KD (2023) The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts. Frontiers in Artificial Intelligence 6:1279794

    2023

  80. [88]

    Harvard Business Press

    Schuh C, Schnellbacher W, Triplat A, et al (2022) Profit from the source: transforming your business by putting suppliers at the core. Harvard Business Press

    2022

  81. [89]

    In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 4211--4221

    Sen P, Danilevsky M, Li Y, et al (2020) Learning explainable linguistic expressions with neural inductive logic programming for sentence classification. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 4211--4221

    2020

  82. [90]

    In: International conference on construction engineering and project management, Korea Institute of Construction Engineering and Management, pp 57--66

    Seo W, Kang Y (2022) Systematic literature review for the application of artificial intelligence to the management of construction claims and disputes. In: International conference on construction engineering and project management, Korea Institute of Construction Engineering ...

    2022

  83. [91]

    Working Notes of CLEF

    Siino M, Tinnirello I (2024) Gpt hallucination detection through prompt engineering. Working Notes of CLEF

    2024

  84. [92]

    IEEE Access

    Siino M, Falco M, Croce D, et al (2025) Exploring llms applications in law: A literature review on current legal nlp approaches. IEEE Access

    2025

  85. [93]

    Scientific Reports 14(1):12755

    Singh A, Rose Anish P, Verma A, et al (2024) A data decomposition-based hierarchical classification method for multi-label classification of contractual obligations for the purpose of their governance. Scientific Reports 14(1):12755

    2024

  86. [94]

    In: Proceedings of the Natural Legal Language Processing Workshop 2023, pp 99--112

    Singhal A, Anish PR, Karande S, et al (2023) Towards mitigating perceived unfairness in contracts from a non-legal stakeholder’s perspective. In: Proceedings of the Natural Legal Language Processing Workshop 2023, pp 99--112

    2023

  87. [95]

    In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp 7611--7622

    Singhal A, Jain C, Anish PR, et al (2024) Generating clarification questions for disambiguating contracts. In: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp 7611--7622

    2024

  88. [96]

    Fla L Rev 77:1

    Solove DJ (2025) Artificial intelligence and privacy. Fla L Rev 77:1

    2025

  89. [97]

    https://link.springer.com/, accessed: 10 March 2025

    Springer (2025) Springer. https://link.springer.com/, accessed: 10 March 2025

    2025

  90. [98]

    PhD thesis, Wageningen University and Research

    Tauqeer A (2024) An infrastructure with semantic contracts and licenses for improving data sharing. PhD thesis, Wageningen University and Research

    2024

  91. [99]

    Journal of empirical legal studies 20(4):852--894

    Teichman D, Zamir E, Ritov I (2023) Biases in legal decision-making: Comparing prosecutors, defense attorneys, law students, and laypersons. Journal of empirical legal studies 20(4):852--894

    2023

  92. [100]

    In: Proceedings of the twelfth language resources and evaluation conference, pp 1235--1241

    Tuggener D, Von D \"a niken P, Peetz T, et al (2020) Ledgar: A large-scale multi-label corpus for text classification of legal provisions in contracts. In: Proceedings of the twelfth language resources and evaluation conference, pp 1235--1241

    2020

  93. [101]

    SEC’s EDGAR database (2025) Software license agreement

    U.S. SEC’s EDGAR database (2025) Software license agreement. https://www.sec.gov/Archives/edgar/data/786344/000119312507182563/dex1027.htm, accessed: 4 June 2025

    2025

  94. [102]

    Dyna 90(SPE230):9--17

    Vel \'a squez-Henao JD, Franco-Cardona CJ, Cadavid-Higuita L (2023) Prompt engineering: a methodology for optimizing interactions with ai-language models in the field of engineering. Dyna 90(SPE230):9--17

    2023

  95. [103]

    Artificial Intelligence and Law 30(4):561--591

    Villata S, Araszkiewicz M, Ashley K, et al (2022) Thirty years of artificial intelligence and law: the third decade. Artificial Intelligence and Law 30(4):561--591

    2022

  96. [104]

    Wang BT (2024) Prompts and large language models: A new tool for drafting, reviewing and interpreting contracts? Law, Technology and Humans 6(2):88--106

    2024

  97. [105]

    arXiv preprint arXiv:241103350

    Wang F, Zhang Z, Zhang X, et al (2024 a ) A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness. arXiv preprint arXiv:241103350

    2024

  98. [106]

    In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 2609--2634

    Wang L, Xu W, Lan Y, et al (2023) Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 2609--2634

    2023

  99. [107]

    NPJ digital medicine 7(1):41

    Wang L, Chen X, Deng X, et al (2024 b ) Prompt engineering in consistency and reliability with the evidence-based guideline for llms. NPJ digital medicine 7(1):41

    2024

  100. [108]

    In: Findings of the Association for Computational Linguistics: NAACL 2022, pp 2208--2221

    Wang Q, Zhao K, Amor R, et al (2022 a ) D2gclf: Document-to-graph classifier for legal document classification. In: Findings of the Association for Computational Linguistics: NAACL 2022, pp 2208--2221

    2022

  101. [109]

    In: The Eleventh International Conference on Learning Representations

    Wang X, Wei J, Schuurmans D, et al (2022 b ) Self-consistency improves chain of thought reasoning in language models. In: The Eleventh International Conference on Learning Representations

    2022

  102. [110]

    Wang Y, Zhao Y (2024) Metacognitive prompting improves understanding in large language models. In: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp 1914--1926

    2024

  103. [111]

    Advances in neural information processing systems 35:24824--24837

    Wei J, Wang X, Schuurmans D, et al (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35:24824--24837

    2022

  104. [112]

    In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, pp 1--10

    Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, pp 1--10

    2014

  105. [113]

    arXiv preprint arXiv:240102415

    Wu C, Gan Y, Ge Y, et al (2024) Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:240102415

    2024

  106. [114]

    In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 1480--1489

    Yang Z, Yang D, Dyer C, et al (2016) Hierarchical attention networks for document classification. In: Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pp 1480--1489

    2016

  107. [115]

    Advances in Neural Information Processing Systems 36

    Yao S, Yu D, Zhao J, et al (2024) Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36

    2024

  108. [116]

    In: Findings of the Association for Computational Linguistics ACL 2024, pp 355--385

    Ye Q, Ahmed M, Pryzant R, et al (2024) Prompt engineering a prompt engineer. In: Findings of the Association for Computational Linguistics ACL 2024, pp 355--385

    2024

  109. [117]

    Tehni c ki vjesnik 28(2):357--362

    Yu H, Li H, et al (2021) A knowledge graph construction approach for legal domain. Tehni c ki vjesnik 28(2):357--362

    2021

  110. [118]

    arXiv preprint arXiv:230911325

    Yue S, Chen W, Wang S, et al (2023) Disc-lawllm: Fine-tuning large language models for intelligent legal services. arXiv preprint arXiv:230911325

    2023

  111. [119]

    In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp 13617--13628

    Yun J, Kim M, Kim Y (2023) Focus on the core: Efficient attention via pruned token compression for document classification. In: Findings of the Association for Computational Linguistics: EMNLP 2023, pp 13617--13628

    2023

  112. [120]

    Journal of Construction Engineering and Management 150(6):03124002

    Zeberga MS, Haaskjold H, Hussein B (2024) Digital technologies for preventing, mitigating, and resolving contractual disagreements in the aec industry: A systematic literature review. Journal of Construction Engineering and Management 150(6):03124002

    2024

  113. [121]

    IEEE Network 30(2):62--66

    Zhang H, Shu Y, Cheng P, et al (2016) Privacy and performance trade-off in cyber-physical systems. IEEE Network 30(2):62--66

    2016

  114. [122]

    Journal of Management in Engineering 39(4):04023017

    Zhang L, Yao H, Fu Y, et al (2023) Comparing subjective and objective measurements of contract complexity in influencing construction project performance: Survey versus machine learning. Journal of Management in Engineering 39(4):04023017

    2023

  115. [123]

    In: Findings of the Association for Computational Linguistics: EMNLP 2022, pp 5671--5685

    Zhang Z, Wang S, Xu Y, et al (2022) Task compass: Scaling multi-task pre-training with task prefix. In: Findings of the Association for Computational Linguistics: EMNLP 2022, pp 5671--5685

    2022

  116. [124]

    In: Proceedings of the eighteenth international conference on artificial intelligence and law, pp 159--168

    Zheng L, Guha N, Anderson BR, et al (2021) When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In: Proceedings of the eighteenth international conference on artificial intelligence and law, pp 159--168

    2021

  117. [125]

    arXiv preprint arXiv:230615546

    Zhuang W, Chen C, Lyu L (2023) When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:230615546

    2023

  118. [126]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key keywords month note number organization pages publisher school series title type url volume year archivePrefix primaryClass adsurl adsnote version label extra.labe...

  119. [127]

    write newline

    " write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.