Pith. sign in

REVIEW 3 major objections 4 minor 40 references

LAW: Legal Agentic Workflows for Custody and Fund Services Contracts

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LAW, a legal agentic workflow that orchestrates reusable domain-specific tools and text agents, outperforms direct LLM prompting on custody-contract queries, with a 95.4% hit rate on termination-date retrieval versus 2.5% for the baseline.

desk verdict LAW is a genuinely engineered system for custody contracts, but its headline claims are inflated by a circular evaluation that scores orchestration against ground truth produced by the same tools. read the letter →

arxiv 2412.11063 v1 pith:6KKWYGJO submitted 2024-12-15 cs.AI cs.CLcs.SE

William Watson , Nicole Cho , Nishan Srishankar , Zhen Zeng , Lucas Cecchi , Daniel Scott , Suchetha Siddagangappa , Rachneet Kaur
show 2 more authors
Tucker Balch Manuela Veloso
This is my paper · ORCID
classification cs.AIcs.CLcs.SE
keywords legalagenticworkflowLLMorchestrationcontractanalysiscustodyservicesretrieval-augmentedgenerationmulti-hopreasoningcodeagent40Actfunds
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces LAW, a system that answers questions about custodial and fund-services contracts by having a code-generating language model orchestrate reusable, domain-specific tools and text agents. The authors claim this design significantly outperforms an off-the-shelf LLM prompted directly, with the largest gap on multi-hop tasks such as computing a contract's termination date, where LAW achieves a 95.4% retrieval hit rate against 2.5% for the baseline. The motivation is practical: fine-tuned legal LLMs are expensive, static, and need labeled data, whereas tool-based orchestration is cheaper, updatable, and works with open or closed LLMs. If correct, LAW would automate contract retrieval and comparison work currently done by hand across 23 years of SEC filings.

What carries the argument

The load-bearing mechanism is the code generation agent with its three-tier validation loop: the LLM is prompted, with tool names and examples, to write Python code that calls LAW's APIs; the code is checked for syntax and security, scanned to ensure only existing tools with valid signatures are invoked (hallucination detection), and executed with error feedback routed back to the agent for correction. Around this agent sit the tools: RoBERTa-based span detection plus regex and HTML parsing for dates; fuzzy matching for parties; a date-and-party matcher that links amendments to master contracts; a fine-tuned T5-large classifier that labels paragraphs into 20 section types so clauses can be retrieved by BM25 from an OpenSearch index; and parallel sub-agent summarization and comparison for text that exceeds the context window. This combination lets the system decompose a query like 'find the termination date' into retrieve-effective-date, read-duration, add, and return steps.

What would settle it

Have independent legal experts, blind to LAW's design, manually label a random sample of the evaluation queries (for example, the actual termination date for 100 contracts); if LAW's accuracy against these human labels falls substantially below its accuracy against the script-generated ground truth, the headline gains are artifacts of evaluation circularity rather than genuine legal competence.

Watch

Extended reading notes

Core claim

The paper's central claim is that an agentic workflow—replacing direct prompting with a code-generation agent that writes Python calls to a suite of legal-specific tools (date extraction, party matching, master-contract lookup, section labeling) and text agents (summarize, compare)—can handle both simple retrieval and multi-hop analytical queries over thousands of long legal contracts. On a labeled dataset of 720 queries drawn from 17,831 custody contracts, LAW reports near-perfect hit rates for finding master agreements and parties, 95.4% for termination dates, and BERTScore F1 of 89.5 for clause summarization, each well above the gpt-3.5-turbo baseline. The authors also claim the system is cost-effective and extensible because tools are reusable and can be added without retraining the underlying model.

Load-bearing premise

The ground truth answers for the 720-question evaluation are generated by hand-coded scripts that use the same tools and text agents LAW itself calls, so the reported scores measure how faithfully LAW reproduces its own components' outputs; if those components are inaccurate, the high numbers reflect self-consistency, not legal understanding.

Editorial extensions

If this is right

  • A custody bank could answer retrieval and comparison questions across the full 485BPOS corpus, covering contracts that govern trillions in retail and retirement assets, without manual document review.
  • Because the code agent composes tools at query time, new analytical questions can be fielded without retraining the model, as long as existing tools cover the needed operations.
  • The system runs on either open-source or closed LLMs, making it a lower-cost alternative to fine-tuned legal LLMs for institutions that already maintain contract databases.
  • The same orchestration pattern extends beyond custody contracts to other legal document families, such as non-English contracts or similar regulatory filings, by adding new tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation may overstate real-world accuracy because the ground-truth scripts share components with LAW; a human-labeled test set is the natural next check to separate orchestration skill from tool accuracy.
  • The simulated noisy-RAG baseline gives gpt-3.5-turbo only four relevant contracts per query, so the reported gap may shrink against a stronger RAG pipeline with better chunking or reranking, especially on the 'explore all contracts' task.
  • If the toolset were released with its unit-test failure modes documented, other financial institutions could adopt the same legal primitives, converting this single system into a shared library of contract-analysis tools.
  • The cost argument predicts that maintenance cost stays roughly flat as new contract types are added, whereas fine-tuned LLMs require fresh labels and retraining; this is a testable claim for a follow-up study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces LAW, a modular agentic workflow that combines a code-generation LLM orchestrator with domain-specific tools (date and party extraction, master-contract retrieval, section-title labeling, lifecycle calculation) and text agents (summarization, comparison), applied to custody and fund services contracts sourced from SEC EDGAR. The authors report experiments on retrieval and analytical queries, claiming that LAW significantly outperforms a gpt-3.5-turbo baseline, with the strongest result being a 92.9-percentage-point advantage for termination-date retrieval (95.4% vs 2.5%, Table 1). The central claim is that orchestration of reusable, guardrailed tools yields accurate and cost-effective automated analysis of long legal contracts.

Significance. If the empirical claims were valid, LAW would be a practical contribution: it demonstrates an agentic orchestration approach to a real domain with a large corpus (17,831 contracts covering 23 years of filings), reusable tools, and a cost argument against fine-tuned legal LLMs. The system design is coherent and the engineering infrastructure is described in reasonable detail. However, the evaluation as presented does not support the headline claims. The ground truth is generated by the same tools and text agents LAW uses, making the reported accuracy measures reflect self-consistency rather than legal correctness. The baseline is weak, not a full retrieval system, and is not tested on all tasks. The absence of independent human validation and statistical testing further limits the conclusions. The paper is therefore best read as a system description with an unsupported comparative evaluation, rather than as a validated empirical study.

major comments (3)
  1. [Section 7, Dataset Curation] The ground truth answers are generated using hand-coded scripts that leverage the same tools and text agents that the proposed system has access to, as explicitly stated in Section 7. This makes the evaluation circular: Table 1 compares LAW's outputs against labels produced by LAW's own components. The headline result, a 92.9-point advantage for termination dates (95.4% vs 2.5%), therefore measures whether the code-generation agent selects the correct tool, not whether the extracted termination dates are legally correct. The paper even states that the procedure "focuses exclusively on LAW's ability to generate code that correctly orchestrates the tools," which contradicts the abstract's claim that LAW "significantly outperforms the baseline" in legal tasks. To support the claim, the authors need independent human-verified labels, or a validation study showing that the hand-coded scripts agree with expert annotations; the small pilot study in Appendix D.1, which reports no agreement rates, does not suffice.
  2. [Section 7, Baseline setup; Table 1] The baseline is not a comparable system. For retrieval queries, the baseline is given a small set of four relevant and four distractor contracts and asked True/False or extraction questions; it does not perform corpus retrieval, unlike LAW. For "Compare clause X" no baseline number is reported, so the general claim of outperformance is not tested for that task. The evaluation also lacks significance tests or confidence intervals: with the stated 20 retrieval queries per combination, reported percentages such as 94.4% and 71.8% are not multiples of 5%, which suggests either a different number of queries or a reporting error. Please report the exact per-cell query counts, variance, and statistical tests before making "significantly outperforms" claims.
  3. [Appendix D.1 and Appendix F] The human validation is limited to a "small pilot study" with no details on the number of participants, contracts, queries, or agreement metrics, so it cannot validate the tool-generated ground truth. Additionally, the section-title classification model used for retrieval has only 46% accuracy (Appendix F, Table 3), yet the end-to-end retrieval hit rates in Table 1 are near or above 90%; the paper should analyze whether retrieval is actually driven by the noisy titles or by other signals, and how title errors propagate to the reported results.
minor comments (4)
  1. [Section 4.2] The text contains a typo: "force majure" should be "force majeure."
  2. [Table 1] Please define "Retrieval Hit Rate" precisely (e.g., exact string match, partial match, or manual inspection); the caption's "hit rate/recall" is ambiguous.
  3. [Section 3] The claim that Form 485BPOS filings "account for a non-trivial (14%) of EDGAR filings" is ambiguous; please clarify whether this is 14% of all EDGAR filings or 14% of a specific subset.
  4. [Section 4] The statement that tools are "rigorously guardrailed through unit tests that map their failure modes" is not supported by any unit-test documentation or failure-mode analysis in the paper; please include at least a summary or a reference.

Circularity Check

2 steps flagged · score 8.0 of 10

Ground-truth labels are generated by LAW's own tools, so the reported 92.9-point lead measures self-consistency, not legal accuracy.

  1. self definitional [Section 7, 'Dataset Curation']
    "The ground truth answers are generated using hand-coded scripts that leverage the same tools and text agents that the proposed system has access to. This procedure makes the evaluation agnostic to the implementation of the tools and focuses exclusively on LAW's ability to generate code that correctly orchestrates the tools and the text agents."

    The gold labels are produced by the same tools that LAW orchestrates, so a correct LAW run reproduces the gold label by construction. The reported retrieval hit rates, such as 95.4% for termination dates in Table 1, measure agreement between LAW's tool calls and the hand-coded scripts' tool calls, not agreement with any independent legal fact. The headline 92.9-point gain over the baseline is therefore an artifact of the evaluation protocol: the baseline has no access to the tools that define correctness, making the comparison a test of tool availability rather than legal understanding.

  2. self definitional [Section 4.2, 'Tool to Calculate Contract Lifecycle'; Section 7, 'Dataset Curation']
    "It uses our existing tool for dates to extract the effective date of the contracts. Next, it searches for the contract's duration or the termination date. If the contract mentions the duration (e.g. 3 years), the tool translates the text into a numerical value. Finally, this numerical value is added to the effective date to generate a termination date."

    The termination-date gold standard is literally the output of LAW's 'Tool to Calculate Contract Lifecycle,' since Section 7 states that the ground-truth scripts use the same tools available to LAW. Consequently, LAW's 95.4% hit rate on termination dates means the system is calling the very function whose output defines the correct answer. Any error in the date-extraction or duration-parsing logic appears identically in both the ground truth and LAW's output, so the 92.9-point advantage over the baseline is a measure of self-consistency, not of correctness against independent legal annotations.

full rationale

The paper's central claim, that LAW significantly outperforms the baseline and excels at termination-date retrieval by 92.9 percentage points, depends entirely on the ground-truth labels used in Table 1. Section 7 states explicitly that these labels were generated by 'hand-coded scripts that leverage the same tools and text agents that the proposed system has access to.' A correct LAW execution therefore reproduces the gold label by construction, and the comparison reduces to whether the code-generation agent selects the right tool, not whether the extracted legal facts are correct. The termination-date case is the clearest instance: the tool described in Section 4.2 computes effective date plus duration, and the gold standard is generated by the same tool. The paper reports no independent human validation with quantified agreement; Appendix D.1 mentions only a 'small pilot study' without reporting metrics. Because the evaluation is self-referential and the baseline is denied access to the tools that define the ground truth, the headline result is forced by the evaluation design rather than by demonstrated legal accuracy. The system may still have engineering value, but the paper's central empirical claim is circular as presented.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the correctness of the hand-coded tool outputs used as ground truth, on the completeness of the 485BPOS corpus, and on several arbitrary engineering constants (top-20 retrieval, 8K chunks). No independent human validation of the tools is reported.

free parameters (4)
  • Top-k retrieved sections = 20
    Section search retrieves top 20 relevant sections per contract; no sensitivity analysis is reported.
  • Text chunk size = 8000 tokens
    Long texts are split into 8K token chunks for the summarization agent; the choice is not justified.
  • Ground truth script logic
    Hand-coded scripts using the same tools define the gold labels; this is a free design choice that determines the reported accuracy.
  • Section title model training samples = 1500 paragraphs per title
    Training set size for the T5-large classifier, which reaches only 46% accuracy.
assumptions (4)
  • domain assumption Form 485BPOS captures the full universe of '40Act funds
    The dataset is built from this SEC form; if other funds are missing, the claim about 'entire scope' fails.
  • domain assumption Hand-coded scripts are correct ground truth
    The evaluation assumes these scripts produce accurate answers, though they are not validated against human legal experts.
  • domain assumption BERTScore F1 is a valid similarity measure for clause summaries
    Analytical tasks are scored via BERTScore; this metric does not verify factual content and can be gamed.
  • domain assumption Tool implementations are correct
    The paper claims tools are guardrailed with unit tests, but no test results are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LAW: Legal Agentic Workflows for Custody and Fund Services Contracts." pith.science (2026). https://pith.science/paper/6KKWYGJO

@misc{pith2026241211063,
  author       = {Pith},
  title        = {Pith review of: LAW: Legal Agentic Workflows for Custody and Fund Services Contracts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KKWYGJO}},
  note         = {Machine review of arXiv:2412.11063}
}
read the original abstract

Legal contracts in the custody and fund services domain govern critical aspects such as key provider responsibilities, fee schedules, and indemnification rights. However, it is challenging for an off-the-shelf Large Language Model (LLM) to ingest these contracts due to the lengthy unstructured streams of text, limited LLM context windows, and complex legal jargon. To address these challenges, we introduce LAW (Legal Agentic Workflows for Custody and Fund Services Contracts). LAW features a modular design that responds to user queries by orchestrating a suite of domain-specific tools and text agents. Our experiments demonstrate that LAW, by integrating multiple specialized agents and tools, significantly outperforms the baseline. LAW excels particularly in complex tasks such as calculating a contract's termination date, surpassing the baseline by 92.9% points. Furthermore, LAW offers a cost-effective alternative to traditional fine-tuned legal LLMs by leveraging reusable, domain-specific tools.

Figures

Figures reproduced from arXiv: 2412.11063 by the authors.

Figure 1
Figure 1. Comparing the traditional method of fine-tuning a legal LLM vs. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. System Overview of LAW. (1) User query input on front-end (§6); (2) Query manipulation and custom modification added to prompt and sent to the code generation agent (§6); (3) Chat completion return from the code generation agent; (4) Execution of backend API tools (§4); (5) Tool retrieval of information from internal cache (§6); (6) Calls to text agents (§5); (7) Calls to multi-node OpenSearch cluster for text retri… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 7 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  6. [6]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. https://arxiv.org/abs/2211.12588 Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks . Preprint, arXiv:2211.12588

    arXiv 2023

  7. [7]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. https://lmsys.org/blog/2023-03-30-vicuna/ Vicuna: An open-source chatbot impressing gpt-4 with 90\

    2023

  8. [8]

    Nicole Cho, Nishan Srishankar, Lucas Cecchi, and William Watson. 2024. https://doi.org/10.1145/3677052.3698597 Fishnet: Financial intelligence from sub-querying, harmonizing, neural-conditioning, expert swarms, and task planning . In Proceedings of the 5th ACM International Conference on AI in Finance, ICAIF '24, page 591–599, New York, NY, USA. Associati...

    work page arXiv 2024

Show all 40 references
  1. [9]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  2. [10]

    Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. 2024. https://arxiv.org/abs/2403.03883 Saullm-7b: A pioneering large language model for law . Pr...

    2024 arXiv

  3. [11]

    Zhiwei Fei, Songyang Zhang, Xiaoyu Shen, Dawei Zhu, Xiao Wang, Maosong Cao, Fengzhe Zhou, Yining Li, Wenwei Zhang, Dahua Lin, Kai Chen, and Jidong Ge. 2024. https://arxiv.org/abs/2406.14887 Internlm-law: An open source chinese legal large language model . Preprint, arXiv:2406.14887

    2024 arXiv

  4. [12]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023 a . https://arxiv.org/abs/2211.10435 Pal: Program-aided language models . Preprint, arXiv:2211.10435

    2023 arXiv

  5. [13]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023 b . Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  6. [14]

    Thanmay Jayakumar, Fauzan Farooqui, and Luqman Farooqui. 2023. https://doi.org/10.18653/v1/2023.nllp-1.22 Large language models are legal but they are not: Making the case for a powerful L egal LLM . In Proceedings of the Natural Legal Language Processing Workshop 2023, pages ...

    2023 doi

  7. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  8. [16]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  9. [17]

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. https://arxiv.org/abs/2209.07753 Code as policies: Language model programs for embodied control . Preprint, arXiv:2209.07753

    2023 arXiv

  10. [18]

    Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352

    2023 arXiv

  11. [19]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  12. [20]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://openrev...

    2023

  13. [21]

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. 2024. https://arxiv.org/abs/2402.06196 Large language models: A survey . Preprint, arXiv:2402.06196

    2024 arXiv

  14. [22]

    Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. https://arxiv.org/abs/2112.00114 Show your work: Scratchpads for intermediate c...

    2021 arXiv

  15. [23]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  16. [24]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  17. [25]

    Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. https://arxiv.org/abs/2204.00498 Evaluating the text-to-sql capabilities of large language models . Preprint, arXiv:2204.00498

    2022 arXiv

  18. [26]

    Adam Roegiest, Radha Chitta, Jonathan Donnelly, Maya Lash, Alexandra Vtyurina, and Francois Longtin. 2023. https://doi.org/10.18653/v1/2023.nllp-1.8 Questions about contracts: Prompt templates for structured answer generation . In Proceedings of the Natural Legal Language Proc...

    2023 doi

  19. [27]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. https://arxiv.org/abs/2302.04761 Toolformer: Language models can teach themselves to use tools . Preprint, arXiv:2302.04761

    2023 arXiv

  20. [28]

    Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. 2023. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Comp...

    2023

  21. [29]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  22. [30]

    Dietrich Trautmann, Alina Petrova, and Frank Schilder. 2022. https://arxiv.org/abs/2212.02199 Legal prompt engineering for multilingual legal judgement prediction . Preprint, arXiv:2212.02199

    2022 arXiv

  23. [31]

    Securities and Exchange Commission

    U.S. Securities and Exchange Commission . 1984. Electronic Data Gathering, Analysis, and Retrieval (EDGAR) System . https://www.sec.gov/edgar

    1984

  24. [32]

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. https://arxiv.org/abs/2402.01030 Executable code actions elicit better llm agents . Preprint, arXiv:2402.01030

    2024 arXiv

  25. [33]

    William Watson, Nicole Cho, Tucker Balch, and Manuela Veloso. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.442 H idden T ables and P y QT ax: A cooperative game and dataset for T able QA to ensure scale and data privacy across a myriad of taxonomies . In Proceedings of th...

    2023 doi

  26. [34]

    William Watson, Nicole Cho, and Nishan Srishankar. 2024. https://arxiv.org/abs/2404.12535 Is there no such thing as a bad question? h4r: Hallucibot for ratiocination, rewriting, ranking, and routing . Preprint, arXiv:2404.12535

    2024 arXiv

  27. [35]

    William Watson and Bo Liu. 2021. https://doi.org/10.1145/3383455.3422520 Financial table extraction in image documents . In Proceedings of the First ACM International Conference on AI in Finance, ICAIF '20, New York, NY, USA. Association for Computing Machinery

    2021

  28. [36]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  29. [37]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. https://arxiv.org/abs/2210.03629 React: Synergizing reasoning and acting in language models . Preprint, arXiv:2210.03629

    2023 arXiv

  30. [38]

    Linan Yue, Qi Liu, Yichao Du, Weibo Gao, Ye Liu, and Fangzhou Yao. 2024. https://arxiv.org/abs/2309.08173 Fedjudge: Federated legal large language model . Preprint, arXiv:2309.08173

    2024 arXiv

  31. [39]

    Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, and Manuela Veloso. 2023. https://doi.org/10.1145/3604237.3626908 Flowmind: Automatic workflow generation with llms . In Proceedings of the Fourth ACM International Conference on AI in Finance...

    2023

  32. [40]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.