Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 25,600-case Supreme Court dataset shows automatic metrics and expert human rankings disagree on which AI summary is best.

desk verdict A genuinely useful legal summarization dataset, paired with an evaluation section whose headline human-vs-automatic discrepancy is not yet supported by the reported statistics. read the letter →

arxiv 2501.00097 v1 pith:3PDEGQYY submitted 2024-12-30 cs.CL cs.AIcs.CYcs.LG

Mourad Heddaya , Kyle MacMillan , Anup Malani , Hongyuan Mei , Chenhao Tan This is my paper · ORCID
classification cs.CLcs.AIcs.CYcs.LG
keywords legalsummarizationlong-contextSupremeCourtdatasethumanevaluationhallucinationROUGEG-Eval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CaseSumm pairs 25,600 U.S. Supreme Court opinions with their official syllabuses, the Court's own summaries written by an attorney and approved by the Justices, spanning 1815-2019. Using this resource, the paper claims that standard automatic summarization metrics and LLM-based judges systematically disagree with expert human judgment about what makes a good legal summary. Fine-tuned Mistral 7B wins on ROUGE and BERTScore, but law-student readers rank GPT-4 summaries as clearer, more sensitive, and more specific, and flag Mistral as the most hallucination-prone. The paper also reports that G-Eval, a GPT-4-based evaluation, does not correlate with human rankings better than traditional metrics, and that GPT-4 summaries often outrank human-written ones including the official syllabuses except on factual correctness. If correct, the lesson is that legal summarization evaluation cannot rely on lexical or LLM scores alone, and the very notion that human-written ground-truth summaries are inherently superior is questionable.

What carries the argument

The central object is the dataset itself: pairs of Supreme Court majority opinions and their official summaries, or syllabuses, which serve as gold-standard references because they are written by a Court-employed attorney and approved by the Justices. The argument's mechanism is a side-by-side evaluation in which five candidate summaries (Mistral Base, Mistral FT, GPT-4 Turbo, and human-written Oyez and Westlaw summaries) are scored against the official syllabus with ROUGE, BERTScore, and G-Eval, and are also ranked by law students who read the original opinion on sensitivity, specificity, clarity, style, and factual error. Comparing the rankings produced by these different methods is what exposes the discrepancy between automatic and human judgment.

What would settle it

Re-run the human evaluation with a cohort of practicing attorneys who are told only that the summaries 'may be written by humans or machines,' and compare their rankings to the law-student rankings from the paper; if GPT-4 no longer outranks Mistral FT and the official syllabuses, the claimed human-versus-automatic discrepancy does not generalize.

Watch

Extended reading notes

Core claim

The paper's central comparative claim is that evaluation method changes the winner: automatic metrics (ROUGE, BERTScore) and even LLM-based G-Eval disagree with expert human rankings when judging LLM-generated legal summaries. In the 622-case automatic evaluation, fine-tuned Mistral 7B leads on recall and F1, and its summaries are closest in length and compression to official syllabuses; but in a human evaluation where second- and third-year law students ranked five candidate summaries against the source opinion, GPT-4 most often ranks first on sensitivity, specificity, clarity, and style, while Mistral FT roughly matches the official syllabus on those dimensions yet is the only candidate with a conspicuous error rate (about 20% of Mistral FT summaries contain at least one factual error). The paper further finds that G-Eval, whether used with its default prompts or prompts adapted to the human rubric, does not correlate with human rankings better than traditional automatic metrics, and that GPT-4 summaries often outperformed human-written summaries including official syllabuses in several quality dimensions except factual correctness.

Load-bearing premise

The human rankings that drive the main discrepancy come from 33 unique cases read by second- and third-year law students, and the recruitment email told them they were grading summaries from the authors' AI tool, so the load-bearing premise is that these rankings are a valid, unbiased proxy for expert legal judgment.

Editorial extensions

If this is right

  • Legal summarization evaluation should include human reading; automatic metrics alone can pick a winner that human experts reject.
  • LLM-based judges such as G-Eval do not yet offer a better reference-free alternative to traditional lexical and semantic metrics.
  • Fine-tuning an open 7B model can match or beat much larger models on lexical overlap, but it can also increase the risk of confident factual hallucinations.
  • Human-written summaries, including official syllabuses and paid commercial services, are not inherently better than strong LLM summaries on clarity or style, though they remain more factually reliable.
  • The 200-year span of the dataset makes it possible to study how summary length, compression, and responsiveness to source length have changed over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The recruitment email shown in Figure 7 tells participants they are grading summaries from the authors' AI tool, so it is untested whether the reported preference for GPT-4 would survive a fully blinded protocol; that is an inference because the paper claims participants were not told the source of each summary.
  • A practical consequence is that legal research tools optimized toward ROUGE-like scores may be rewarded for stylistic mimicry while still producing citation errors and fact misrepresentations that require separate verification.
  • Because the human evaluation rests on only 33 unique cases read by a median of 5 cases per student, the effect sizes are uncertain; a larger replication with practicing attorneys could change the ranking even if the direction holds.
  • The dataset's temporal structure could support tests of whether modern LLMs reproduce the historical drift in syllabus style, such as the emergence of the 'Held:' section, although the paper does not run those tests.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces CaseSumm, a dataset of 25,642 U.S. Supreme Court opinion/syllabus pairs spanning 1815–2019, with the official Court syllabus serving as a gold-standard summary. The authors describe an extraction pipeline using PDFs from Public Resource Org and the Library of Congress, validate it on 100 randomly sampled cases (96 perfect extractions), and release the dataset publicly. They then benchmark GPT-4 Turbo, Mistral 7B (base and fine-tuned), and two human-written controls (Westlaw, Oyez) against the syllabuses using ROUGE, BERTScore, and human evaluation by second- and third-year law students. The central reported finding is a discrepancy: automatic metrics favor fine-tuned Mistral, while human evaluators most often rank GPT-4 higher, and G-Eval (an LLM-based metric) does not correlate better with human judgments than traditional automatic metrics. The paper also includes a hallucination error analysis with examples from both models.

Significance. The dataset contribution is substantial: CaseSumm is the largest open legal summarization dataset by document count, spans two centuries, and provides official syllabuses rather than crowd-sourced or machine-constructed summaries. The 96% manual extraction check and public release on HuggingFace are concrete strengths. If the evaluation findings hold, the paper would provide valuable evidence that automatic metrics and LLM-based judges diverge from human judgment in a high-stakes domain, which is an important result for summarization evaluation research. However, the human evaluation underpinning the headline discrepancy rests on a small sample (33 unique cases, 57 readings, ~11 students) and lacks inferential statistics, so the central comparative claims should be treated as preliminary rather than definitive.

major comments (2)
  1. [§4.2, §5.3, Figure 3] The central claim that expert humans most commonly rank GPT-4 over Mistral FT is not statistically supported. The human evaluation consists of 57 opinion readings across 33 unique cases by second- and third-year law students (median 5 readings per student). No confidence intervals, effect sizes, or pairwise significance tests are reported for the rank differences in Figure 3, so the observed GPT-4-vs-Mistral-FT gap cannot be distinguished from sampling noise; the authors' own minimum detectable effect of 0.52 rank points is computed but never applied to the actual results. In addition, the recruitment email in Figure 7 states that participants will "evaluate the quality of the summaries produced by our AI-based tool," which is in tension with the §4.2 statement that students were not told the source of each summary. This does not necessarily break the GPT-4/Mistral pairwise comparison, but it undermines the claimed neutrality of the evaluation and the comparisons against human-written controls (e.g., the conclusion that GPT-4 outperforms official syllabuses on some dimensions). Please report raw rank distributions, effect sizes with confidence intervals, and clarify exactly what participants were told about the provenance of the summaries.
  2. [§5.3, Table 4, Abstract] The conclusion that "LLM-based evaluation does not correlate with human judgments better than traditional automatic metrics" is too strong for the evidence presented. Table 4 shows that G-Eval correlates much more strongly with human specificity judgments (adapted G-Eval specificity 0.51 vs. ROUGE-1 -0.28) and clarity (0.23–0.28 vs. ~0.0 for ROUGE-2), which the paper's own bullet points acknowledge. No confidence intervals or tests for differences between dependent correlations are reported; with 33 unique cases (or 57 opinion readings), differences such as ROUGE-1 sensitivity 0.54 vs. adapted G-Eval sensitivity 0.43 are likely within sampling noise. The abstract's blanket claim should be qualified to state that traditional metrics perform comparably or better on sensitivity/style, while G-Eval performs better on specificity/clarity, and the correlation comparisons should include uncertainty quantification.
minor comments (6)
  1. [§4.1, Appendix A.1] GPT-4's prompt was optimized with DSPy using ROUGE-2 as the optimization metric, and the same ROUGE family is later used as an evaluation metric for GPT-4. This means GPT-4's ROUGE scores are partially fitted to the evaluation metric; the paper should acknowledge this when interpreting the automatic evaluation comparison, even though the prompt optimization likely biases in GPT-4's favor.
  2. [Table 1] The row for "GovReport*" has an asterisk but no footnote or citation identifying the dataset source; please add the reference.
  3. [§5.2] The statement "doubling opinion length increases syllabus length by nearly 2/3" appears to be based on an unreported regression; the correlation of 0.676 reported in Table 3 does not by itself imply this elasticity. Please report the underlying regression or rephrase the claim.
  4. [References] In the reference for Kapoor et al., "Peter Henderon" should be "Peter Henderson."
  5. [Appendix C.1] The sentence "For all measures where the outcome is rank, we mark the mean rank identically 3) with a red dashed line" is garbled and should be rewritten (e.g., "we mark the mean rank, 3, with a red dashed line").
  6. [Abstract, §4.2] The evaluators are described as "expert human annotators" in the abstract and as "expert humans" in §5, but the participants are second- and third-year law students. Please either adjust the terminology to "law student evaluators" or justify the "expert" label with information about their legal training and experience.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor fitted-component in GPT-4 ROUGE evaluation; no central circularity.

  1. fitted input called prediction [§4.1 Modeling; §4.2 Automatic Evaluation]
    "For GPT-4, we optimize prompt-selection using DSPy (Khattab et al., 2023) with 10 opinion/syllabus pairs from the training set and ROUGE-2 as the optimization metric. ... Following recent work on summarization (Koh et al., 2022), we use ROUGE and BERTScore (Lin, 2004; Zhang et al., 2019) as our automated metrics for evaluating generated summaries against the reference syllabuses."

    The paper fits GPT-4's prompt by maximizing ROUGE-2 on a small training subsample, then presents ROUGE and BERTScore scores as automatic evaluations of GPT-4 in Table 2. Because the same metric family used for optimization is later used for evaluation, the GPT-4 ROUGE results are partly a product of that fit rather than an independent measurement of an unoptimized model. This is a mild fitted-component in the reported automatic scores. It does not force the paper's central metric-vs-human discrepancy, since Mistral FT still beats the ROUGE-optimized GPT-4 on ROUGE-2 (23.0 vs 19.2), and the human evaluation and G-Eval correlation analyses are independent of this optimization.

full rationale

The paper's central claims are empirical comparisons rather than derivations, and no load-bearing step reduces to its own inputs by construction. The only partial fitted-component is the DSPy prompt optimization for GPT-4 using ROUGE-2, after which ROUGE-family metrics are used in the automatic evaluation; this contaminates GPT-4's absolute ROUGE scores but does not manufacture the headline discrepancy, since Mistral FT still wins on ROUGE despite GPT-4 being optimized for it. The human evaluation, though potentially affected by the recruitment email in Figure 7 that tells participants they are evaluating 'summaries produced by our AI-based tool,' is independent of the automatic metrics and is a validity concern rather than a circularity. Self-citations (e.g., Cao et al. 2024) are related-work extensions and not load-bearing. The dataset construction is externally validated by manual inspection of 100 sampled extractions. Overall, no significant circularity is present; the fitted ROUGE component is minor and localized.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No free parameters in a derivation sense because the paper is an empirical dataset and evaluation study. The fitted artifacts are the DSPy-optimized GPT-4 prompt and the fine-tuning and generation hyperparameters. The evaluation rests on domain assumptions: syllabuses are gold-standard summaries, law students are acceptable expert raters, the 96% extraction accuracy generalizes, and the 33-case sample is sufficient. No invented entities are introduced.

free parameters (3)
  • DSPy-optimized GPT-4 prompt = Not stated; optimized prompt in Listing 2
    The prompt used for GPT-4 summaries was selected by optimizing ROUGE-2 on 10 opinion/syllabus pairs, so it is a fitted input to the model.
  • LoRA fine-tuning learning rate = Selected from {2e-5, 2e-4, 2e-3}
    The authors chose the learning rate by dev loss convergence; this is a model selection choice, not a scientific parameter.
  • Mistral generation fallback hyperparameters = repetition_penalty=1.3, top_p=0.9
    Used when length-limit truncation occurs in roughly 10% of Mistral generations; chosen ad hoc to ensure complete summaries.
assumptions (5)
  • domain assumption The official syllabus is a gold-standard summary of the majority opinion.
    Stated in §1 and §3: syllabuses are written by a Court attorney, approved by Justices, and therefore ideal references. The paper's own human evaluation later shows syllabuses underperform GPT-4 on several dimensions, complicating this assumption.
  • domain assumption Second- and third-year law students provide valid expert human judgments of legal summary quality.
    Section 4.2 recruits law students, not practicing attorneys or Court reporters, yet calls the evaluation 'expert'. If student judgments do not match expert legal assessment, the headline human rankings are not authoritative.
  • domain assumption The 100-case manual extraction check (96% perfect) generalizes to the full 25.6K dataset.
    Section 3 reports 96/100 perfect extractions; Limitations admits early-1800s low-quality scans may yield incorrect syllabuses. A 4% error rate on early scans could inject noise into the dataset.
  • domain assumption The 33-case human evaluation sample has sufficient power to detect meaningful ranking differences.
    Section 4.2 computes a minimum detectable effect of 0.52 rank points, but with 33 unique cases, Spearman correlations in Table 4 are estimated with wide uncertainty and no confidence intervals are reported.
  • domain assumption ROUGE and BERTScore are meaningful proxies for legal summarization relevance.
    The paper asserts these metrics assess relevance in §4.2, but the paper's central finding is that they align poorly with human judgments, which weakens this assumption for the legal domain.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions." pith.science (2026). https://pith.science/paper/3PDEGQYY

@misc{pith2026250100097,
  author       = {Pith},
  title        = {Pith review of: CaseSumm: A Large-Scale Dataset for Long-Context Summarization from U.S. Supreme Court Opinions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3PDEGQYY}},
  note         = {Machine review of arXiv:2501.00097}
}
read the original abstract

This paper introduces CaseSumm, a novel dataset for long-context summarization in the legal domain that addresses the need for longer and more complex datasets for summarization evaluation. We collect 25.6K U.S. Supreme Court (SCOTUS) opinions and their official summaries, known as "syllabuses." Our dataset is the largest open legal case summarization dataset, and is the first to include summaries of SCOTUS decisions dating back to 1815. We also present a comprehensive evaluation of LLM-generated summaries using both automatic metrics and expert human evaluation, revealing discrepancies between these assessment methods. Our evaluation shows Mistral 7b, a smaller open-source model, outperforms larger models on most automatic metrics and successfully generates syllabus-like summaries. In contrast, human expert annotators indicate that Mistral summaries contain hallucinations. The annotators consistently rank GPT-4 summaries as clearer and exhibiting greater sensitivity and specificity. Further, we find that LLM-based evaluations are not more correlated with human evaluations than traditional automatic metrics. Furthermore, our analysis identifies specific hallucinations in generated summaries, including precedent citation errors and misrepresentations of case facts. These findings demonstrate the limitations of current automatic evaluation methods for legal summarization and highlight the critical role of human evaluation in assessing summary quality, particularly in complex, high-stakes domains. CaseSumm is available at https://huggingface.co/datasets/ChicagoHAI/CaseSumm

Figures

Figures reproduced from arXiv: 2501.00097 by the authors.

Figure 1
Figure 1. Opinion and syllabus lengths, compression rates by syllabuses, and correlations between opinion and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. ROUGE-2 evaluation of model-generated and human summaries, by Chief Justice of SCOTUS when the opinion was written. Markers are means and whiskers are 95% confidence intervals. metadata can explain variation in summarization quality. While we do not find notable variation across most of these features, we observe one ex￾ception: the divergence between recall and preci￾sion across all summaries increases over time. F… view at source ↗
Figure 3
Figure 3. Human evaluation of model-generated and human summaries. x-axis is a rank, where 1 is best and 5 is [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Lexical Variation. Measures the fraction of [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Labelstudio Annotation Interface [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Consent form for research participation. [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Email with instructions sent to participants. [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    100-LongBench and LongScore evaluate LLMs at controlled context lengths and rank models by relative performance drop from a short-context baseline, not by raw accuracy.

Reference graph

Works this paper leans on

47 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Emmanuel Bauer, Dominik Stammbach, Nianlong Gu, and Elliott Ash. 2023. Legal extractive summarization of us court opinions. arXiv preprint arXiv:2305.08428

    work page Pith review arXiv 2023

  4. [4]

    Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.751 Re-evaluating evaluation in text summarization . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347--9359, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.emnlp-main.751 2020

  5. [5]

    G. Bradski. 2000. The OpenCV Library . Dr. Dobb's Journal of Software Tools

    work page 2000

  6. [6]

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. https://arxiv.org/abs/2303.12712 Sparks of artificial general intelligence: Early experiments with gpt-4 . Preprint, arXiv:2303.12712

    arXiv 2023

  7. [7]

    Tianyu Cao, Natraj Raman, Danial Dervovic, and Chenhao Tan. 2024. https://arxiv.org/abs/2404.06162 Characterizing multimodal long-form summarization: A case study on financial reports . Preprint, arXiv:2404.06162

    arXiv 2024

  8. [8]

    Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. https://doi.org/10.18653/v1/2022.acl-long.297 L ex GLUE : A benchmark dataset for legal language understanding in E nglish . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    doi:10.18653/v1/2022.acl-long.297 2022

Show all 47 references
  1. [9]

    Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. https://arxiv.org/abs/2310.00785 Booookscore: A systematic exploration of book-length summarization in the era of llms . Preprint, arXiv:2310.00785

    2024 arXiv

  2. [10]

    Claude Team . 2024. https://www.anthropic.com/news/claude-3-family Introducing the next generation of claude

    2024

  3. [11]

    Arman Cohan and Nazli Goharian. 2016. https://aclanthology.org/L16-1130 Revisiting summarization evaluation for scientific articles . In Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC '16) , pages 806--813, Portoro z , Slovenia. E...

    2016

  4. [12]

    Junyun Cui, Xiaoyu Shen, and Shaochun Wen. 2023. A survey on legal judgment prediction: Datasets, metrics, models and challenges. IEEE Access

    2023

  5. [13]

    Vladimir Eidelman. 2019. https://doi.org/10.18653/v1/d19-5406 Billsum: A corpus for automatic summarization of us legislation . In Proceedings of the 2nd Workshop on New Frontiers in Summarization, page 48–56. Association for Computational Linguistics

    2019 doi

  6. [14]

    Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. https://doi.org/10.18653/v1/2022.naacl-main.187 QAF act E val: Improved QA -based factual consistency evaluation for summarization . In Proceedings of the 2022 Conference of the North American Chapter of th...

    2022 doi

  7. [15]

    Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev

    Alexander R. Fabbri, Wojciech Kryściński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. https://doi.org/10.1162/tacl_a_00373 SummEval: Re-evaluating Summarization Evaluation . Transactions of the Association for Computational Linguistics, 9:391--409

    2021 doi

  8. [16]

    Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/P19-1213 Ranking generated summaries by correctness: An interesting but challenging application for natural language inference . In Proceedings of the 57...

    2019 doi

  9. [17]

    Biaoyan Fang, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2023. Super-scotus: A multi-sourced dataset for the supreme court of the us. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 202--214

    2023

  10. [18]

    Diego Garat and Dina Wonsever. 2022. Automatic curation of court documents: Anonymizing personal data. Information, 13(1):27

    2022

  11. [19]

    GPT-4 Team . 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  12. [20]

    Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances in Neural Information Processing Systems, 35:29217--29234

    2022

  13. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  14. [22]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. https://arxiv.org/abs/2311.05232 A survey on hallucination in large language models: Principles, taxonomy, challenges, and ...

    2023 arXiv

  15. [23]

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. https://arxiv.org/abs/2104.02112 Efficient attentions for long document summarization . Preprint, arXiv:2104.02112

    2021 arXiv

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  17. [25]

    Sayash Kapoor, Peter Henderon, and Arvind Narayanan. 2024. Promises and pitfalls of artificial intelligence for legal applications. Journal of Cross-Disciplinary Research in Computational Law

    2024

  18. [26]

    Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2023 a . Gpt-4 passes the bar exam. Available at SSRN 4389233

    2023

  19. [27]

    Daniel Martin Katz, Dirk Hartung, Lauritz Gerlach, Abhik Jana, and Michael J Bommarito II. 2023 b . Natural language processing in the legal domain. arXiv preprint arXiv:2302.12039

    2023 arXiv

  20. [28]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. https://arxiv.org/abs/2310.03714 Dspy: Compiling declarati...

    2023 arXiv

  21. [29]

    Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. 2022. https://doi.org/10.1145/3545176 An empirical survey on long document summarization: Datasets, models, and metrics . ACM Computing Surveys, 55(8):1–35

    2022 doi

  22. [30]

    Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. 2023. https://doi.org/10.18653/v1/2023.eacl-main.121 L ong E val: Guidelines for human evaluation of faithfulness in long-form summarization . In Proceedings of the 17th Confere...

    2023 doi

  23. [31]

    Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2022. https://arxiv.org/abs/2105.08209 Booksum: A collection of datasets for long-form narrative summarization . Preprint, arXiv:2105.08209

    2022 arXiv

  24. [32]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. https://doi.org/10.1162/tacl_a_00453 SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization . Transactions of the Association for Computational Linguistics, 10:163--177

    2022 doi

  25. [33]

    Washington University Law. 2024. http://scdb.wustl.edu/documentation.php The supreme court database

    2024

  26. [34]

    Library of Congress . 2024. https://www.loc.gov/collections/united-states-reports/about-this-collection/ United states reports (official opinions of the u.s. supreme court)

    2024

  27. [35]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  28. [36]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment . Preprint, arXiv:2303.16634

    2023 arXiv

  29. [37]

    Masha Medvedeva, Martijn Wieling, and Michel Vols. 2023. Rethinking the field of automatic prediction of court decisions. Artificial Intelligence and Law, 31(1):195--212

    2023

  30. [38]

    Public Resource Org . 2024. https://law.resource.org/pub/us/case/reporter/US/ United states reports

    2024

  31. [39]

    Natalie Schluter. 2017. https://aclanthology.org/E17-2007 The limits of automatic summarisation according to ROUGE . In Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume 2, Short Papers , pages 41--45, Valencia...

    2017

  32. [40]

    Eva Sharma, Chen Li, and Lu Wang. 2019. https://doi.org/10.18653/v1/P19-1212 BIGPATENT : A large-scale dataset for abstractive and coherent summarization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204--2213, Florence, I...

    2019 doi

  33. [41]

    Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. https://doi.org/10.18653/v1/2024.acl-long.51 F ine S ur E : Fine-grained summarization evaluation using LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics...

    2024 doi

  34. [42]

    Pawan Trivedi, Digha Jain, Shilpa Gite, Ketan Kotecha, Anant Bhatt, and Nithesh Naik. 2024. https://doi.org/10.30919/es1022 Indian legal corpus (ilc): A dataset for summarizing indian legal proceeding using natural language . Engineered Science, 27:1022

    2024 doi

  35. [43]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. https://doi.org/10.18653/v1/2020.acl-main.450 Asking and answering questions to evaluate the factual consistency of summaries . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 500...

    2020 doi

  36. [44]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. https://arxiv.org/abs/2109.01652 Finetuned language models are zero-shot learners . CoRR, abs/2109.01652

    2021 arXiv

  37. [45]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263--27277

    2021

  38. [46]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675

    2019 arXiv

  39. [47]

    Jie Zou and Evangelos Kanoulas. 2020. Towards question-based high-recall information retrieval: Locating the last few relevant documents for technology-assisted reviews. ACM Transactions on Information Systems (TOIS), 38(3):1--35

    2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.