Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that explicit input formatting and a structure-aware system prompt can lift GPT-4.1's exact-match accuracy on contract question answering from about 48% to 79%.

desk verdict The 30-point formatting effect on GPT-4.1 is plausible and worth testing, but the exact-match metric is under-specified to the point that the magnitude is not yet established. read the letter →

arxiv 2505.12837 v1 pith:ICB6T36Z submitted 2025-05-19 cs.CL cs.AI

Christian Braun , Alexander Lilienbeck , Daniel Mentjukov This is my paper · ORCID
classification cs.CLcs.AI
keywords legaldocumentunderstandingcontractquestionansweringinputformattingMarkdownstructurepromptengineeringexact-matchaccuracyGPT-4.1CUADdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the exact formatting of a legal contract—whether it is cleanly structured, aggressively cleaned, or OCR-degraded—and where the task instructions sit in the prompt are first-order determinants of LLM performance on legal question answering, not cosmetic details. On a 928-question contract-review task derived from CUAD, GPT-4.1 scored around 48% exact-match on poorly structured text and 66–67% on well-structured text, then 77–79% when the system prompt explicitly announced the input structure. GPT-4o, by contrast, stayed nearly flat at 47–52% across all conditions. The authors conclude that document preprocessing and prompt architecture should be managed as core variables in high-stakes legal AI, and that vision-extracted Markdown is the best-performing input when the model is primed for it. This matters because practitioners commonly assume LLMs can infer structure from any text; the data say otherwise for at least one widely used model.

What carries the argument

The load-bearing object is a spectrum of five input formats that vary only in structural explicitness: original CUAD plain text, regex-cleaned text with line breaks removed, Azure OCR text, GPT-4o Vision plain text, and GPT-4o Vision Markdown. The second lever is prompt placement: strategy S1 keeps the system prompt generic and puts all task instructions in the user message, while strategy S2 moves the instructions to the system prompt and states the input type ('structured Markdown' or 'structured Plain Text'). The outcome measure that carries the argument is exact-match accuracy on a bipartite answer—a correct yes/no decision plus an exact, verbatim clause citation—which forces the model both to decide and to locate the evidence.

What would settle it

Pull the raw model outputs for the 928 S2 Vision-Markdown questions and apply a strict scorer that requires the cited span to be a character-for-character substring of the CUAD ground-truth clause; if the 79% figure falls toward the ~48% unstructured baseline, the headline result is a scoring artifact rather than a structure effect.

Watch

Extended reading notes

Core claim

The central empirical discovery is a large, model-specific sensitivity to input structure and prompt placement. With a generic system prompt, GPT-4.1's exact-match accuracy on the bipartite yes/no-plus-verbatim-citation task is about 48% when line breaks are removed or raw OCR is used, and rises to about 66–67% for well-structured plain text and for text extracted by GPT-4o Vision, including Markdown. Moving the full task instructions into the system prompt and adding an explicit 'you will receive structured Markdown / plain text' cue adds another 10–13 points, bringing GPT-4.1 to 77% on original CUAD text and 79% on GPT-4o Vision Markdown. GPT-4o is far less sensitive to formatting changes but also lower in absolute accuracy, improving only 2–5 points under the same prompt shift. Under the stronger prompt, Markdown outperforms well-structured plain text, whereas without the prompt the two formats are equivalent.

Load-bearing premise

The reported accuracy gains stand or fall on the scorer requiring both halves of each answer—a correct yes/no and an exact verbatim citation—in every condition; the paper's own definition of the metric is ambiguous about whether both halves were actually enforced.

Editorial extensions

If this is right

  • GPT-4.1-class models can lose roughly 20 points of exact-match accuracy when contracts arrive as cleaned text or raw OCR, so document preprocessing should be treated as a core performance lever.
  • A structure-aware system prompt ('you will receive structured Markdown') is worth 10–13 points on top of good formatting, so prompt architecture belongs in legal LLM evaluation.
  • Vision-model PDF-to-Markdown conversion can match or exceed human-formatted plain text, making it a practical path for scanned contracts.
  • Markdown beats well-structured plain text only when the system prompt primes the model; without priming the formats are equivalent.
  • Even at its best, the model reaches 79% exact-match, so legal workflows still require human verification rather than autonomous decision-making.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The results imply that legal NLP leaderboards may be inadvertently measuring input-format sensitivity rather than model reasoning; standardizing a canonical document format across models would make comparisons fairer.
  • A concrete extension would test whether the structure-awareness cue works by activating format-specific priors; varying the cue's specificity could isolate the mechanism.
  • The 20-point gap for GPT-4.1 suggests many retrieval-augmented pipelines that feed OCR text into modern models may be silently underperforming; re-running RAG evaluations with Markdown inputs is a cheap test.
  • The same input-format manipulation could be applied to open-weight models to see whether the sensitivity pattern tracks model family or training objective.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper reports an empirical study of how input text structure and prompt-placement design affect GPT-4o and GPT-4.1 performance on a legal question-answering task built from an excerpt of the CUAD dataset. Five input modalities are compared (original structured plain text, regex-cleaned plain text, Azure OCR text, GPT-4o Vision plain text, and GPT-4o Vision Markdown) under two prompting strategies (user-centric task definition and system-centric structure-aware prompting). The central finding is that GPT-4.1 is highly sensitive to input structure, improving by roughly 20 percentage points when given well-structured inputs, and by another 10-13 percentage points under the optimized system prompt, with the best condition (GPT-4o Vision Markdown, Strategy S2) reaching 79% exact-match accuracy. GPT-4o is reported as robust to format variation but lower in absolute accuracy. The paper argues that input formatting and prompt design are first-order variables for legal LLM deployment.

Significance. If the main effects are real, the paper has substantial practical significance for legal technology: it would demonstrate that document preprocessing and system-prompt design can change contract-review accuracy by tens of percentage points, and it would add an important, domain-specific data point to the growing literature on format sensitivity in LLMs. The work is also useful in its explicit focus on a realistic document-processing pipeline (original text, regex cleaning, OCR, vision-based extraction) rather than only synthetic perturbations. The paper is transparent about several limitations and does not overclaim generalizability beyond the tested models and task. However, the quantitative claims are currently supported only by a single point-estimate comparison with no uncertainty quantification, and the evaluation metric is not fully operationalized, so the size and even the direction of the headline effects are not yet securely established.

major comments (5)
  1. [§3.1.5, Table 1] The exact-match metric is defined inconsistently. The text first states that an answer is correct only if both the Yes/No component and, for Yes answers, the exact verbatim citation match, and that no partial credit is awarded. It then reports 'the percentage of test instances for which the model produced a fully correct Yes/No answer', which omits the citation requirement. Clarify which criterion produced the numbers in Table 1. In addition, the paper does not specify the string-comparison procedure: is whitespace normalized? Are line breaks collapsed in both the model output and the ground-truth span? This matters directly for the RegexClean condition, because those inputs have line breaks removed while the CUAD ground-truth citations are spans from the original text. Without a defined normalization rule, the ~20-point gap between RegexClean/OCR and well-structured inputs may confound citation-string fidelity with comprehension.
  2. [§3.2, Table 1] All results are reported as single point estimates with no confidence intervals, standard errors, or information about repeated runs. Because the headline comparisons (e.g., 48% vs. 67% for GPT-4.1 under S1; 66% vs. 79% for Markdown under S2) are the central evidence, the paper should report either multiple runs with different seeds/temperatures or bootstrap confidence intervals over the 928-question sample. Without this, the reader cannot assess whether the observed differences are statistically stable, especially given the strict, partly citation-based scoring rule.
  3. [§3.1.1, Appendix D] The description of the test set is internally inconsistent. The text says that after filtering to four topics ('Competitive Restriction Exception', 'Non-Compete', 'Exclusivity', 'No-Solicit of Customers') 928 questions were used. Appendix D, however, lists 35 question types with counts whose total far exceeds 928, and the named topics do not obviously correspond to those question types. The paper should state precisely how the 928 questions were derived from the CUAD excerpt, whether each question is paired with a uniquely associated contract, and how the four selected topics map to the question templates in Appendix A.
  4. [§3.1.3, §4.2] There is a generator-evaluator confound for the vision-extracted modalities. GPT-4o Vision produces both GPT4oVision_TXT and GPT4oVision_MD, and GPT-4o is then evaluated on those inputs. This creates an advantage for the GPT-4o condition and, more importantly, bundles input-format effects with the extraction source. To separate the format effect from the source effect, the paper should include a condition in which the same Markdown content is rendered as plain text (or the same text is re-extracted by an independent OCR engine) and evaluated under otherwise identical conditions.
  5. [§3.1.5, §4.3] The paper mentions that an LLM-as-a-judge and the answer_correctness metric were used for content-level evaluation but states the results are not included. To make the exact-match results independently verifiable, the evaluation script, the judge prompt, and the normalization logic should be released or, minimally, described in enough detail to replicate the comparison. As written, the only quantitative evidence is a black-box point estimate.
minor comments (5)
  1. [General] Several typos and formatting artifacts appear, such as 'Out goal' in footnote 4, 'theContract Understanding' in the abstract, and the line 'while modern while...' in the introduction. The paper would benefit from a careful proofreading pass.
  2. [Figure 1] The caption labels the subfigures as '(a).txt', '(b).txt', and '(c).md', but the subfigures themselves are not referenced clearly in the text. The figure could also state which contract section is shown and whether the same section is displayed across all three panels.
  3. [§3.1.4, Strategy 2] The system prompt shown contains the literal string 'Markdown // Plain Text' and the footnote explains that the segment was varied. It would be clearer to show the two actual variants (one for Markdown, one for plain text) as separate prompt blocks.
  4. [Appendix D] Table 2 reports unique question counts but does not explain the relationship between these counts and the 928 questions used in the study, nor how the four selected topics map to the templates in Appendix A. Adding a reconciliation table would resolve the apparent inconsistency.
  5. [References] Reference [43] appears both as 'Wang et al. (2023) MAUD' and as 'Wei et al. (2023) Chain-of-thought'; the numbering is inconsistent and should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical benchmark with external measurements; its confounds are validity concerns, not derivation steps.

full rationale

This paper contains no derivation chain that could collapse into its inputs. It fits no parameters, defines no theoretical quantity, and makes no prediction from a model fitted to the outcome being predicted; the reported accuracies are direct measurements of GPT-4o and GPT-4.1 on an external benchmark (CUAD). The load-bearing comparisons (S1 vs S2, TXT vs MD vs OCR) are external to the paper's own definitions: the inputs are generated by independent pipelines (CUAD originals, Azure OCR, GPT-4o Vision via vision-parse) and the outputs are scored against CUAD ground-truth annotations. No central claim rests on a self-citation: the cited prior work (CUAD, LegalBench, MDEVAL, OpenAI prompting guide, vision-parse) is external or machine-checkable, and the acknowledgements' mention of a coauthor-provided framework is not used as evidence for the empirical effect. The paper even discloses that its LLM-as-judge and answer_correctness content-level results were excluded, an admitted limitation rather than a hidden input. The main caveats—GPT-4o Vision generating inputs later evaluated on GPT-4o, and exact-match citation scoring possibly not being whitespace-normalized across RegexClean/OCR modalities—are measurement-validity risks that could affect the size of the reported gaps, but they do not make the observed accuracy differences true by definition. Accordingly, there is no circular step to report.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters or invented entities are used. The central claim depends on domain assumptions about ground-truth correctness, the isolation of formatting effects, and the validity of the exact-match scoring procedure.

assumptions (4)
  • domain assumption CUAD expert annotations and the derived question templates provide correct ground truth for exact-match scoring.
    The entire accuracy measure depends on CUAD labels and the Appendix A templates; if ground truth is wrong, all percentages shift.
  • domain assumption The five input modalities differ primarily in structural explicitness, not in content or difficulty; in particular the Regex-clean and Azure OCR variants introduce only format degradation, not new content errors.
    The H1 comparison attributes performance differences to structure; but Azure OCR also adds character errors, and Regex cleaning may merge words or change tokenization.
  • domain assumption The exact-match evaluation deterministically compares model output against CUAD gold citations, including verbatim matching, as described in Section 3.1.5.
    The main numeric claims depend on this scoring procedure, but the procedure is not fully specified and the metric description is internally inconsistent.
  • ad hoc to paper The 928-question run set is a representative sample of the four target CUAD topics.
    The sampling procedure is not described and Appendix D reports a much larger question distribution, so representativeness is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting." pith.science (2026). https://pith.science/paper/ICB6T36Z

@misc{pith2026250512837,
  author       = {Pith},
  title        = {Pith review of: The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ICB6T36Z}},
  note         = {Machine review of arXiv:2505.12837}
}
read the original abstract

Legal contracts possess an inherent, semantically vital structure (e.g., sections, clauses) that is crucial for human comprehension but whose impact on LLM processing remains under-explored. This paper investigates the effects of explicit input text structure and prompt engineering on the performance of GPT-4o and GPT-4.1 on a legal question-answering task using an excerpt of the CUAD. We compare model exact-match accuracy across various input formats: well-structured plain-text (human-generated from CUAD), plain-text cleaned of line breaks, extracted plain-text from Azure OCR, plain-text extracted by GPT-4o Vision, and extracted (and interpreted) Markdown (MD) from GPT-4o Vision. To give an indication of the impact of possible prompt engineering, we assess the impact of shifting task instructions to the system prompt and explicitly informing the model about the structured nature of the input. Our findings reveal that GPT-4o demonstrates considerable robustness to variations in input structure, but lacks in overall performance. Conversely, GPT-4.1's performance is markedly sensitive; poorly structured inputs yield suboptimal results (but identical with GPT-4o), while well-structured formats (original CUAD text, GPT-4o Vision text and GPT-4o MD) improve exact-match accuracy by ~20 percentage points. Optimizing the system prompt to include task details and an advisory about structured input further elevates GPT-4.1's accuracy by an additional ~10-13 percentage points, with Markdown ultimately achieving the highest performance under these conditions (79 percentage points overall exact-match accuracy). This research empirically demonstrates that while newer models exhibit greater resilience, careful input structuring and strategic prompt design remain critical for optimizing the performance of LLMs, and can significantly affect outcomes in high-stakes legal applications.

Figures

Figures reproduced from arXiv: 2505.12837 by the authors.

Figure 1
Figure 1. Visual comparison of file formats of the same contract section. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Results after answering the questions and extracting the relevant clauses for answering [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Top 20 questions distribution from the used excerpt of the CUAD dataset. [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Building ground truth from CUAD dataset 21 [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agent-Based Software Artifact Evaluation

    cs.SE 2026-02 reject novelty 6.0 of 10

    ArtifactCopilot automates artifact evaluation end-to-end with graph-based execution, claiming 85.42% agreement with human badge outcomes on 48 software engineering artifacts.

Reference graph

Works this paper leans on

48 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llm leaderboard — compare gpt-4o, llama 3, mistral, gemini & other models

    Artificial Analysis. Llm leaderboard — compare gpt-4o, llama 3, mistral, gemini & other models. https://artificialanalysis.ai/leaderboards/models, 2025. Accessed 2025-05-15

    work page 2025

  2. [2]

    Bender, Timnit Gebru, Angelina McMillan-Major, and S

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? InProceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, Virtual Event, Canada, March 2021. ACM

    work page 2021

  3. [3]

    Notes on applicability of gpt-4 to document understanding, May 2024

    Łukasz Borchmann. Notes on applicability of gpt-4 to document understanding, May 2024

    work page 2024

  4. [4]

    Vision-parse: Parse pdfs into markdown using vision llms

    Arun Brahma. Vision-parse: Parse pdfs into markdown using vision llms. https:// github.com/iamarunbrahma/vision-parse, December 2024. Python software, accessed 2025-05-14

    work page 2024

  5. [5]

    Brown et al

    Tom B. Brown et al. Language models are few-shot learners, July 2020

    work page 2020

  6. [6]

    Evaluating question answering evaluation

    Allen Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Evaluating question answering evaluation. InProceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 119–124, Hong Kong, China, 2019. Association for Computational Linguistics. 14

    work page 2019

  7. [7]

    Mdeval: Evaluating and enhancing markdown awareness in large language models, January 2025

    Zeyu Chen et al. Mdeval: Evaluating and enhancing markdown awareness in large language models, January 2025

    work page 2025

  8. [8]

    What is f1 score? an essential metric in llm evaluation

    Data Science Dojo. What is f1 score? an essential metric in llm evaluation. https:// datasciencedojo.com/blog/understanding-f1-score/, 2025. Accessed 2025-05-15

    work page 2025

Show all 48 references
  1. [9]

    Metrics to evaluate a question answering system

    deepset. Metrics to evaluate a question answering system. https://www.deepset.ai/blog/ metrics-to-evaluate-a-question-answering-system, 2025. Accessed 2025-05-15

    2025

  2. [10]

    David Freeman Engstrom and Jonah B. Gelbach. Legal tech, civil procedure, and the future of adversarialism, 2021. Unpublished manuscript

    2021

  3. [11]

    Ragas: Automated evaluation of retrieval augmented generation, April 2025

    Sreyan Es, Jack James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation, April 2025

    2025

  4. [12]

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, August 2023

    Nilay Guha et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, August 2023

    2023

  5. [13]

    A detailed analysis of optical character recognition tech- nology.International Journal of Applied Mathematics, Electronics and Computers, 4(Special Issue-1):244–244, December 2016

    Khaled Hamad and Mehmet Kaya. A detailed analysis of optical character recognition tech- nology.International Journal of Applied Mathematics, Electronics and Computers, 4(Special Issue-1):244–244, December 2016

    2016

  6. [14]

    Effect of selection format on llm performance, March 2025

    Yujia Han, Yan Wu, and Jacob Willard. Effect of selection format on llm performance, March 2025

    2025

  7. [15]

    Wang, and Samira Hasan

    Jinyuan He, Mansi Rungta, Dylan Koleczek, Amandeep Sekhon, Fangxiang X. Wang, and Samira Hasan. Does prompt formatting have any impact on llm performance?, November 2024

    2024

  8. [16]

    The law of large documents: Understanding the structure of legal contracts using visual cues, July 2021

    Alexander Hegel, Mohammed Shah, Graham Peaslee, Benjamin Roof, and Ehab Elwany. The law of large documents: Understanding the structure of legal contracts using visual cues, July 2021

    2021

  9. [17]

    Ocr with tesseract, amazon textract, and google document ai: a bench- marking experiment.Journal of Computational Social Science, 5(1):861–882, May 2022

    Thomas Hegghammer. Ocr with tesseract, amazon textract, and google document ai: a bench- marking experiment.Journal of Computational Social Science, 5(1):861–882, May 2022

    2022

  10. [18]

    Cuad: An expert-annotated nlp dataset for legal contract review, November 2021

    Dan Hendrycks, Collin Burns, Anya Chen, and Steven Ball. Cuad: An expert-annotated nlp dataset for legal contract review, November 2021

    2021

  11. [19]

    Jia Lai, Wei Gan, Jiahao Wu, Ziqi Qi, and Philip S. Yu. Large language models in law: A survey, November 2023

    2023

  12. [20]

    Holistic evaluation of language models, October 2023

    Percy Liang et al. Holistic evaluation of language models, October 2023

    2023

  13. [21]

    Let’s verify step by step, May 2023

    Howard Lightman et al. Let’s verify step by step, May 2023

    2023

  14. [22]

    Better call gpt, comparing large language models against lawyers, January 2024

    Lucy Martin, Neil Whitehouse, Sandy Yiu, Lewis Catterson, and Rishad Perera. Better call gpt, comparing large language models against lawyers, January 2024

    2024

  15. [23]

    Azure ai vision with ocr and ai

    Microsoft. Azure ai vision with ocr and ai. https://azure.microsoft.com/en-us/ products/ai-services/ai-vision, 2025. Accessed 2025-05-14

    2025

  16. [24]

    Hai, and Sufiyan Haq

    Anas Nawar, Md Rakib, Sadik A. Hai, and Sufiyan Haq. An open source contractual language understanding application using machine learning. InProceedings of the First Workshop on Language Technology and Resources for a Fair , Inclusive, and Safe Society within the 13th Language...

    2022

  17. [25]

    Truc T. H. Nguyen, Adam Jatowt, Matthieu Coustaty, and Antoine Doucet. Survey of post-ocr processing approaches.ACM Computing Surveys, 54(6):1–37, July 2022

    2022

  18. [26]

    Gpt-4.1 prompting guide | openai cookbook

    OpenAI. Gpt-4.1 prompting guide | openai cookbook. https://cookbook.openai.com/ examples/gpt4-1_prompting_guide, 2025. Accessed 2025-05-13

    2025

  19. [27]

    Gpt-4o system card

    OpenAI. Gpt-4o system card. https://openai.com/index/gpt-4o-system-card/ , 2025. Accessed 2025-05-14. 15

    2025

  20. [28]

    Introducing gpt-4.1 in the api

    OpenAI. Introducing gpt-4.1 in the api. https://openai.com/index/gpt-4-1/ , 2025. Accessed 2025-05-14

    2025

  21. [29]

    Gpt-4 technical report, March 2024

    OpenAI et al. Gpt-4 technical report, March 2024

    2024

  22. [30]

    Bleu: A method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, page 311, Philadelphia, Pennsylvania, 2002. Association for C...

    2002

  23. [31]

    Niccolò Pipitone and Ghadeer H. Alami. Legalbench-rag: A benchmark for retrieval-augmented generation in the legal domain, August 2024

    2024

  24. [32]

    Polo et al

    Francisco M. Polo et al. Efficient multi-prompt evaluation of llms, October 2024

    2024

  25. [33]

    Publications Office of the EU. 2. structure of a legal act — interinstitutional style guide. https://style-guide.europa.eu/en/content/-/isg/topic?identifier=2- structure-legal-act, 2025. Accessed 2025-05-14

    2025

  26. [34]

    No, ruth bader ginsburg did not dissent in obergefell — and other things chatgpt gets wrong about the supreme court

    Jordan Romoser. No, ruth bader ginsburg did not dissent in obergefell — and other things chatgpt gets wrong about the supreme court. https://www.scotusblog.com/2023/01/no- ruth-bader-ginsburg-did-not-dissent-in-obergefell-and-other-things- chatgpt-gets-wrong-about-the-supreme-...

    2023

  27. [35]

    Singh, Souvik Saha, Vaibhav Jain, Subhankar Mondal, and Aditya Chadha

    Priyabrata Sahoo, Ayush K. Singh, Souvik Saha, Vaibhav Jain, Subhankar Mondal, and Aditya Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications, March 2025

    2025

  28. [36]

    The prompt report: A systematic survey of prompt engineering techniques, February 2025

    Samuel Schulhoff et al. The prompt report: A systematic survey of prompt engineering techniques, February 2025

    2025

  29. [37]

    A law reasoning benchmark for llm with tree-organized structures including factum probandum, evidence and experiences, March 2025

    Jiawei Shen et al. A law reasoning benchmark for llm with tree-organized structures including factum probandum, evidence and experiences, March 2025

    2025

  30. [38]

    Structured clinical reasoning prompt enhances llm’s diagnostic capabilities in diagnosis please quiz cases.Radiology and Imaging, September 2024

    Yuji Sonoda et al. Structured clinical reasoning prompt enhances llm’s diagnostic capabilities in diagnosis please quiz cases.Radiology and Imaging, September 2024. Preprint

    2024

  31. [39]

    Ethics of ai in law: Basic questions

    Harry Surden. Ethics of ai in law: Basic questions. In Markus Dirk Dubber, Frank Pasquale, and Sunit Das, editors,The Oxford Handbook of Ethics of AI, pages 719–736. Oxford University Press, 2020

    2020

  32. [40]

    Struct-x: Enhancing large language models reasoning with structured data, July 2024

    Xiaoming Tan et al. Struct-x: Enhancing large language models reasoning with structured data, July 2024

    2024

  33. [41]

    Chatgpt coming to court, by way of self-represented liti- gants

    Eugene V olokh. Chatgpt coming to court, by way of self-represented liti- gants. https://reason.com/volokh/2023/05/27/chatgpt-coming-to-court-by- way-of-self-represented-litigants/, 2023. Reason.com, accessed 2025-05-13

    2023

  34. [42]

    Legal evaluations and challenges of large language models, November 2024

    Junlin Wang et al. Legal evaluations and challenges of large language models, November 2024

    2024

  35. [43]

    Maud: An expert-annotated legal nlp dataset for merger agreement under- standing, 2023

    Shuo Wang et al. Maud: An expert-annotated legal nlp dataset for merger agreement under- standing, 2023

    2023

  36. [44]

    Chain-of-thought prompting elicits reasoning in large language models, January 2023

    Jason Wei et al. Chain-of-thought prompting elicits reasoning in large language models, January 2023

    2023

  37. [45]

    Here’s what happens when your lawyer uses chatgpt

    Benjamin Weiser. Here’s what happens when your lawyer uses chatgpt. https:// www.nytimes.com/2023/05/27/nyregion/avianca-airline-lawsuit-chatgpt.html , May 2023. The New York Times, accessed 2025-05-13

    2023

  38. [46]

    Rouge-sem: Better evaluation of summarization using rouge combined with semantics.Expert Systems with Applications, 237:121364, March 2024

    Ming Zhang, Chengpeng Li, Min Wan, Xiaoxuan Zhang, and Qian Zhao. Rouge-sem: Better evaluation of summarization using rouge combined with semantics.Expert Systems with Applications, 237:121364, March 2024. 16

    2024

  39. [47]

    Judging llm-as-a-judge with mt-bench and chatbot arena, December 2023

    Liangliang Zheng et al. Judging llm-as-a-judge with mt-bench and chatbot arena, December 2023

    2023

  40. [48]

    Promptbench: A unified library for evaluation of large language models, August 2024

    Kai Zhu et al. Promptbench: A unified library for evaluation of large language models, August 2024. 17 A Used questions and question template as part of CUAD question_templates = [ "Does the {contract} contract contain an exception or carve-out to any ,→competitive restriction...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.