pith. sign in

arxiv: 2607.01256 · v1 · pith:X5TY5TE4new · submitted 2026-06-04 · 💻 cs.CY · cs.AI

AI Assistance for Human Review of Default Judgments

Theodora Worledge , Othman Bensouda Koraichi , Daniel Bernal , Aviv Caspi , Tatsunori Hashimoto , Carlos Guestrin , David Freeman Engstrom This is my paper

Pith reviewed 2026-07-04 00:14 UTC · model grok-4.3

classification 💻 cs.CY cs.AI
keywords default judgmentsAI assistancelegal document reviewcourt efficiencylarge language modelshuman-AI collaborationerror reductionstatutory compliance
0
0 comments X

The pith

An AI assistant using cited case excerpts raises default judgment review accuracy by 6% and speed by 26%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that large language models can assist expert reviewers in checking default judgments against fixed legal requirements by generating recommendations anchored in direct quotes and tables from the filings. An initial audit of 188 Los Angeles debt-collection cases found 4% with major defects that should have blocked judgment, 10% with inconsistencies needing reduced awards, and 32% with errors needing amendment. In a controlled test, 66 law students using the tool reviewed requirements 6.0% more accurately and 25.9% faster than unaided peers, with the biggest improvements on statutory rules that demand extensive document search. If these gains transfer, courts could handle their millions of annual default cases with fewer wrongful judgments and less reviewer time.

Core claim

The Default Assistant employs large language models to evaluate a case with respect to predetermined legal requirements and provide cited recommendations for an expert user's review. Explanations are grounded in cited quotes and tables from the original case filings. In a controlled study with 66 law students that conservatively simulates court review, aided users were 6.0% more accurate on the average requirement and 25.9% faster, with error reductions up to 62% and time savings up to 34% on statutory requirements.

What carries the argument

The Default Assistant, an LLM system that scores cases against legal requirements and supplies verifiable excerpts from the source documents for human checking.

If this is right

  • Statutory requirements see the largest error drops (up to 62%) and time savings (up to 34%).
  • The 4% rate of major defects found in the audit could be lowered through systematic AI-supported checks.
  • Review time per requirement falls enough to let the same staff handle more of the millions of annual cases.
  • Citation grounding lets users verify outputs directly against filings rather than trusting model text alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same citation-grounded pattern could be applied to other high-volume court tasks such as motion review or discovery disputes.
  • Real deployment would need separate measurement of how often reviewers actually follow, override, or ignore the AI suggestions under time pressure.
  • If the tool reduces amendment rates, it could shorten the overall case lifecycle and lower costs for both courts and litigants.

Load-bearing premise

Gains measured with law students who had more time and resources will appear when the same tool is used by actual court staff working under real constraints.

What would settle it

A field trial with sitting court reviewers processing live default cases that shows no accuracy gain or no time savings, or shows losses on either measure.

Figures

Figures reproduced from arXiv: 2607.01256 by Aviv Caspi, Carlos Guestrin, Daniel Bernal, David Freeman Engstrom, Othman Bensouda Koraichi, Tatsunori Hashimoto, Theodora Worledge.

Figure 1
Figure 1. Figure 1: The Default Assistant processes PDF case documents to provide cited recommendations for case requirements. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: For each recommendation, the Default Assistant provides a binary recommendation and a free-form explanation cited [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of the number of incorrect AI recommendations that the assisted humans corrected (blue) to the number [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of the number of correct AI recommendations that the assisted humans agreed with (blue) to the number [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: From case management system data, the number of collections cases filed at the Superior Court of Los Angeles has [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The Default Assistant gathers general case information before reviewing the case for each of the seven requirements. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Participants review each case using general case information, case files, and the sub-requirements under the drop [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Each citation in the “Cited Quotes and Sources” section includes a pop-over that links to the original case-file PDF [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
read the original abstract

Overwhelmed courts in the United States review millions of default judgments each year. Unfortunately, such manual reviews are time-consuming and prone to error. In an audit of 188 debt collection cases granted default judgment by the Superior Court of Los Angeles, we find that 4% contained major defects that should have entirely prevented default judgment, 10% contained inconsistencies requiring reduced judgments, and 32% contained errors requiring amendment prior to judgment. To support courthouses in default judgment review, we collaborated with courthouse attorneys and judges in designing a Default Assistant. The Default Assistant employs large language models to evaluate a case with respect to predetermined legal requirements and provide cited recommendations for an expert user's review. We equip users to verify these recommendations by grounding the assistant's explanations in cited quotes and tables from the original case filings. We conduct a controlled study with 66 law students that conservatively simulates court review, with more time and resources than court staff. We nevertheless find users aided by the Default Assistant were 6.0% more accurate on the average requirement than unaided reviewers (p < 1.0e-4). Simultaneously, users were 25.9% faster in reviewing the average requirement than unaided reviewers (p < 2.5e-10). Statutory requirements demanding extensive document search realized the largest gains, with error reductions and time savings from AI assistance up to 62% and 34%, respectively, relative to unassisted user performance and with differences statistically significant (p < 0.05). Our work provides a proof-of-concept that AI assistants with citations have the potential to help resource-constrained courts conduct default judgment review more accurately and efficiently.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript audits 188 debt collection default judgment cases from the Los Angeles Superior Court, reporting 4% with major defects that should have prevented judgment, 10% with inconsistencies requiring reduced judgments, and 32% with errors requiring amendment. It describes the collaborative design of a Default Assistant that uses LLMs to evaluate cases against predetermined legal requirements and generates cited recommendations grounded in quotes and tables from the filings. A controlled user study with 66 law students (explicitly described as providing more time and resources than actual court staff) finds that AI-aided reviewers achieve 6.0% higher accuracy on the average requirement (p < 1.0e-4) and 25.9% faster review times (p < 2.5e-10), with larger gains (up to 62% error reduction) on statutory requirements. The paper concludes that the assistant has potential to help resource-constrained courts.

Significance. If the measured accuracy and speed gains hold under real courthouse conditions, the work could support meaningful improvements in error rates and throughput for the millions of default judgments reviewed annually. Strengths include the real-world audit of actual court cases, direct collaboration with courthouse attorneys and judges, statistically significant results from a controlled experiment, and the emphasis on citation-grounded explanations to support expert verification. These elements provide a concrete proof-of-concept for LLM assistance in legal review tasks.

major comments (1)
  1. [Abstract] Abstract (final sentence) and the description of the controlled study: the central claim that the Default Assistant 'has the potential to help resource-constrained courts' rests on the reported +6.0% accuracy and +25.9% speed gains. However, these results were obtained with 66 law students who, per the abstract, had 'more time and resources than court staff.' No data, sensitivity analysis, or discussion is provided on whether comparable benefits would be observed under the actual time pressure, case volume, and domain-expertise profile of courthouse personnel. This external-validity gap directly affects whether the measured deltas support the policy-oriented conclusion.
minor comments (2)
  1. [Evaluation / User Study] The manuscript lacks sufficient detail on LLM prompting strategies, the exact operational definitions of each legal requirement, participant training protocols, and the precise mapping between the simulation conditions and real court constraints. These omissions make it difficult to assess reproducibility and the robustness of the reported p-values.
  2. [Abstract] The abstract notes post-hoc selection of 'statutory requirements' for the larger-gain analysis; clarification is needed on whether this selection was pre-registered or could introduce selection bias in the error-reduction claims.

Simulated Author's Rebuttal

1 responses · 1 unresolved

We thank the referee for the constructive feedback emphasizing external validity. We address the major comment below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract (final sentence) and the description of the controlled study: the central claim that the Default Assistant 'has the potential to help resource-constrained courts' rests on the reported +6.0% accuracy and +25.9% speed gains. However, these results were obtained with 66 law students who, per the abstract, had 'more time and resources than court staff.' No data, sensitivity analysis, or discussion is provided on whether comparable benefits would be observed under the actual time pressure, case volume, and domain-expertise profile of courthouse personnel. This external-validity gap directly affects whether the measured deltas support the policy-oriented conclusion.

    Authors: We agree that the controlled study used law students rather than actual courthouse personnel and that no sensitivity analysis or direct data under real courthouse conditions (time pressure, case volume, expertise) is provided. The manuscript already states that the study 'conservatively simulates court review, with more time and resources than court staff,' and the statistically significant gains were observed even under these more favorable conditions for reviewers. We view this as supporting the 'potential' claim in a proof-of-concept sense, but we acknowledge the inference to real-world courthouse settings is not directly tested. We will revise the manuscript to add an expanded limitations discussion that explicitly addresses this external-validity gap, including differences in domain expertise, time constraints, and the need for future field validation with court staff. revision: yes

standing simulated objections not resolved
  • Direct empirical data or sensitivity analysis from actual courthouse personnel under operational time pressure, case volume, and expertise constraints (would require a separate field study).

Circularity Check

0 steps flagged

Empirical user study with direct performance measurements; no derivations or self-referential reductions

full rationale

The paper reports an audit of 188 cases and a controlled user study with 66 law students measuring accuracy (+6.0%) and speed (+25.9%) gains from the Default Assistant. These are direct empirical outcomes from randomized comparison, not derived from equations, fitted parameters, or prior self-citations. No load-bearing steps reduce to inputs by construction. The external-validity concern (law students vs. real court staff) is a limitation of generalizability, not circularity in the derivation chain. The work is self-contained against its stated benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on standard assumptions about legal document review and LLM capability to parse requirements; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Predetermined legal requirements for default judgments can be reliably checked via document review and are suitable for LLM evaluation.
    The Default Assistant is built around evaluating cases with respect to these requirements.

pith-pipeline@v0.9.1-grok · 5860 in / 1210 out tokens · 21306 ms · 2026-07-04T00:14:27.095394+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages · 3 internal anchors

  1. [1]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) , year =

    Slobodkin, Andrew and Chen, Yuhui and Zhang, Yiming and Khashabi, Daniel and Durrett, Greg , title =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024) , year =

    work page 2024

  2. [2]

    2024 , eprint=

    The Extractive-Abstractive Spectrum: Uncovering Verifiability Trade-offs in LLM Generations , author=. 2024 , eprint=

    work page 2024

  3. [3]

    AI Magazine , volume=

    In search of verifiability: Explanations rarely enable complementary performance in AI-advised decision making , author=. AI Magazine , volume=. 2024 , publisher=

    work page 2024

  4. [4]

    Proceedings of the 2021 CHI conference on human factors in computing systems , pages=

    Does the whole exceed its parts? the effect of ai explanations on complementary team performance , author=. Proceedings of the 2021 CHI conference on human factors in computing systems , pages=

    work page 2021

  5. [5]

    Model-Agnostic Interpretability of Machine Learning

    Model-agnostic interpretability of machine learning , author=. arXiv preprint arXiv:1606.05386 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Advances in neural information processing systems , volume=

    A unified approach to interpreting model predictions , author=. Advances in neural information processing systems , volume=

    work page

  7. [7]

    Cyborg Justice

    “Cyborg Justice” and the Risk of Technological--Legal Lock-In , author=. Columbia law review , volume=. 2019 , publisher=

    work page 2019

  8. [8]

    A rule of persons, not machines: the limits of legal automation , author=. Geo. Wash. L. Rev. , volume=. 2019 , publisher=

    work page 2019

  9. [9]

    Cardozo L

    The paradox of automation as anti-bias intervention , author=. Cardozo L. Rev. , volume=. 2019 , publisher=

    work page 2019

  10. [10]

    IEEE Access , year=

    Exploring llms applications in law: A literature review on current legal nlp approaches , author=. IEEE Access , year=

    work page

  11. [11]

    Ethics of data and analytics , pages=

    Machine bias , author=. Ethics of data and analytics , pages=. 2022 , publisher=

    work page 2022

  12. [12]

    Government Information Quarterly , volume=

    Automation bias in public administration--an interdisciplinary perspective from law and psychology , author=. Government Information Quarterly , volume=. 2024 , publisher=

    work page 2024

  13. [13]

    World tax journal , volume=

    Towards eXplainable Artificial Intelligence (XAI) in tax law: the need for a minimum legal standard , author=. World tax journal , volume=. 2022 , publisher=

    work page 2022

  14. [14]

    Proceedings of the ACM on Human-Computer Interaction , volume=

    Explanations can reduce overreliance on ai systems during decision-making , author=. Proceedings of the ACM on Human-Computer Interaction , volume=. 2023 , publisher=

    work page 2023

  15. [15]

    Default Procedures , author=. U. Pa. L. Rev. , volume=. 2024 , publisher=

    work page 2024

  16. [16]

    Dirty Debts Sold Dirt Cheap , author=. Harv. J. on Legis. , volume=. 2015 , publisher=

    work page 2015

  17. [17]

    Available at SSRN 2700745 , year=

    The Landscape of Civil Litigation in State Courts , author=. Available at SSRN 2700745 , year=

    work page

  18. [18]

    University of Illinois Chicago School of Law July , year=

    One-Sided Litigation: Lessons from Civil Docket Data in California Debt Collection Lawsuits , author=. University of Illinois Chicago School of Law July , year=

    work page

  19. [19]

    Eviction Lab , volume=

    New data release shows that 3.6 million eviction cases were filed in the United States in 2018 , author=. Eviction Lab , volume=

    work page 2018

  20. [20]

    Social forces , volume=

    Eviction's fallout: housing, hardship, and health , author=. Social forces , volume=. 2015 , publisher=

    work page 2015

  21. [21]

    Mitchell Hamline L

    Fifty years after the Consumer Credit Protection Act: The high price of wage garnishments , author=. Mitchell Hamline L. Rev. , volume=. 2019 , publisher=

    work page 2019

  22. [22]

    arXiv preprint arXiv:2504.01349 , year=

    Tasks and Roles in Legal AI: Data Curation, Annotation, and Verification , author=. arXiv preprint arXiv:2504.01349 , year=

    work page arXiv

  23. [23]

    New England journal of medicine , volume=

    A surgical safety checklist to reduce morbidity and mortality in a global population , author=. New England journal of medicine , volume=. 2009 , publisher=

    work page 2009

  24. [24]

    Kim, Sunnie S. Y. and Vaughan, Jennifer Wortman and Liao, Q. Vera and Lombrozo, Tania and Russakovsky, Olga , title =. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems , articleno =. 2025 , isbn =. doi:10.1145/3706598.3714020 , abstract =

    work page doi:10.1145/3706598.3714020 2025

  25. [25]

    Bloomberg , year =

    Yin, Leon and Alba, Davey and Nicoletti, Leonardo , title =. Bloomberg , year =

    work page

  26. [26]

    Educational and psychological measurement , volume=

    A coefficient of agreement for nominal scales , author=. Educational and psychological measurement , volume=. 1960 , publisher=

    work page 1960

  27. [27]

    Systematic content analysis of judicial opinions , author=. Calif. L. Rev. , volume=. 2008 , publisher=

    work page 2008

  28. [28]

    Journal of Empirical Legal Studies , volume =

    Nielsen, Aileen and Skylaki, Stavroula and Norkute, Milda and Stremitzer, Alexander , title =. Journal of Empirical Legal Studies , volume =. doi:https://doi.org/10.1111/jels.12396 , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1111/jels.12396 , abstract =

    work page doi:10.1111/jels.12396

  29. [29]

    Lawyering in the age of artificial intelligence , author=. Minn. L. Rev. , volume=. 2024 , publisher=

    work page 2024

  30. [30]

    Proceedings of the ACM on Human-computer Interaction , volume=

    To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making , author=. Proceedings of the ACM on Human-computer Interaction , volume=. 2021 , publisher=

    work page 2021

  31. [31]

    2026 , institution=

    Access to Justice in the Age of AI: Evidence from US Federal Courts , author=. 2026 , institution=

    work page 2026

  32. [32]

    Translational psychiatry , volume=

    How machine-learning recommendations influence clinician treatment selections: the example of antidepressant selection , author=. Translational psychiatry , volume=. 2021 , publisher=

    work page 2021

  33. [33]

    WebGPT: Browser-assisted question-answering with human feedback

    Webgpt: Browser-assisted question-answering with human feedback , author=. arXiv preprint arXiv:2112.09332 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  34. [34]

    Teaching language models to support answers with verified quotes

    Teaching language models to support answers with verified quotes , author=. arXiv preprint arXiv:2203.11147 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  35. [35]

    Assisting in Writing W ikipedia-like Articles From Scratch with Large Language Models

    Shao, Yijia and Jiang, Yucheng and Kanell, Theodore and Xu, Peter and Khattab, Omar and Lam, Monica. Assisting in Writing W ikipedia-like Articles From Scratch with Large Language Models. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2...

    work page doi:10.18653/v1/2024.naacl-long.347 2024

  36. [36]

    Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations

    Jiang, Yucheng and Shao, Yijia and Ma, Dekun and Semnani, Sina and Lam, Monica. Into the Unknown Unknowns: Engaged Human Learning through Participation in Language Model Agent Conversations. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024

    work page 2024

  37. [37]

    L ong C ite: Enabling LLM s to Generate Fine-grained Citations in Long-Context QA

    Zhang, Jiajie and Bai, Yushi and Lv, Xin and Gu, Wanjun and Liu, Danqing and Zou, Minhao and Cao, Shulin and Hou, Lei and Dong, Yuxiao and Feng, Ling and Li, Juanzi. L ong C ite: Enabling LLM s to Generate Fine-grained Citations in Long-Context QA. Findings of the Association for Computational Linguistics: ACL 2025. 2025. doi:10.18653/v1/2025.findings-acl.264

    work page doi:10.18653/v1/2025.findings-acl.264 2025

  38. [38]

    arXiv preprint arXiv:2210.08726 , year=

    Rarr: Researching and revising what language models say, using language models , author=. arXiv preprint arXiv:2210.08726 , year=

    work page arXiv

  39. [39]

    Citation-Enhanced Generation for LLM -based Chatbots

    Li, Weitao and Li, Junkai and Ma, Weizhi and Liu, Yang. Citation-Enhanced Generation for LLM -based Chatbots. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.79

    work page doi:10.18653/v1/2024.acl-long.79 2024

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting , author=. Advances in Neural Information Processing Systems , volume=

    work page

  41. [41]

    Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering , pages=

    LicenseGPT: A fine-tuned foundation model for publicly available dataset license compliance , author=. Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering , pages=

    work page