Pith. sign in

REVIEW 1 major objections 5 minor 44 references

Hell or High Water: Evaluating Agentic Recovery from External Failures

T0 review · 1 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Language agents fail to recover when the first tool in their plan breaks, even with a valid alternative.

desk verdict A genuinely reusable benchmark for a real failure mode, but the headline accuracy drop rests on a solvability guarantee the paper asserts more than it verifies. read the letter →

arxiv 2508.11027 v1 pith:VFGABQMB submitted 2025-08-14 cs.CL

Andrew Wang , Sophia Hager , Adi Asija , Daniel Khashabi , Nicholas Andrews This is my paper · ORCID
classification cs.CL
keywords agenticplanningbackupplansexternalfailuresfunctioncallingLLMagentsfaulttolerancetoolusebenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to measure whether language-model agents can recover when an external failure breaks the first step of their plan. It introduces a benchmark, Hell-or-High-Water, in which every problem has two distinct, non-overlapping sequences of function calls that yield the correct answer, and then automatically disables the first function the agent attempts so that completing the task requires discovering and executing a backup plan. Across commercial and open-weight models, accuracy falls by roughly a third to nearly half when this failure is injected, even though every problem remains solvable. The paper argues that current agents can often identify the right function in context but cannot reliably adapt to environment feedback by reformulating their approach.

What carries the argument

The load-bearing construction is the pairing of two guaranteed solution paths per question, derived from parameterized SQL queries containing subqueries: the full query becomes one Python function giving a direct solution, while the subquery and the outer query become separate functions whose outputs chain together as the backup path. At run time the benchmark takes the first function from the correct set that the agent attempts and replaces it with a version that always raises an error, such as 'function 987 is currently unavailable. Please try a different function.' This makes every episode a forced test of backup planning while keeping the task solvable. The second ingredient is a large function search space of 4,450 tools with deliberately uninformative names, so the agent must consult tool documentation and search results instead of guessing purposes from names.

What would settle it

After the failure injection, run an exhaustive or oracle search over all 830 test problems using the same documentation the agents see, enumerating every one-step and chained call sequence and checking whether at least one produces the gold answer. If a nontrivial fraction of problems have no executable alternative, the measured accuracy drop partly reflects unsolvable tasks rather than agent inflexibility; if all alternatives work, the drop measures recovery ability directly.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that language agents are not fault-tolerant planners: when the environment reports that a chosen function is unavailable, they frequently fail to switch to an alternative route even when a correct alternative is guaranteed to exist. The benchmark is built so that each of 830 test questions can be solved either by a single function call or by composing two or more subquery functions, and both routes are known to be valid. In the open-world setting, where agents must search among 4,450 functions with uninformative names, injecting external errors lowers accuracy from 60.5–71.4 percent to 23.3–41.1 percent across the models tested; Gemini 2.0 Flash falls from 71.4 to 41.1 percent and GPT-4o from 60.5 to 38.4 percent. Error analysis attributes the majority of failures to search, meaning the correct tool never appears in results, followed by failure to select the correct tool once it appears. The paper concludes that scaling model size improves absolute performance but does not make agents resilient to external failures.

Load-bearing premise

That every one of the 830 test problems still has a valid alternative sequence of function calls after the first solution function is replaced with an error-raising version, and that this alternative can be carried out using only the provided documentation; the paper asserts this from the SQL-subquery construction but reports no per-problem verification.

Editorial extensions

If this is right

  • If the finding holds, deployed agents should be treated as brittle under API outages and service changes: a plan that was sound a moment ago can fail through no fault of the agent, and the models tested here do not reliably route around it.
  • The two-path construction gives the field a reusable way to isolate recovery ability from basic tool-calling skill, because the same problem is measured with and without an injected failure.
  • Because search failures account for more than half of all errors in the open-world setting, improving how agents formulate queries over large tool sets may yield larger gains than improving low-level reasoning.
  • Scaling model size alone will not deliver fault tolerance; interventions such as explicit retry instructions, persistent memory of past failures, or training on recovery trajectories are the plausible levers.
  • Underspecification is separable from recovery: adding disambiguating hints helped some models substantially but did not close the failure gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to give agents an oracle search tool that always surfaces the correct backup functions; if accuracy still drops, the bottleneck is plan reformulation rather than retrieval, and if it recovers, retrieval is the lever.
  • Because the error message is explicit and a valid backup is guaranteed, the benchmark likely measures a lower bound on real-world difficulty: in production, backups may not exist, may be undocumented, or may fail silently, so the recovery gap could be wider outside the benchmark.
  • The failure taxonomy points to an architectural division of labor the paper does not test: a retrieval module or a planner that pre-computes fallback paths could offload recovery from the LLM.
  • Since the problems are synthesized from a text-to-SQL corpus, the benchmark inherits that source's underspecification and data quirks; rerunning the pipeline on cleaner or more complex instruction sets could reveal whether the recovery gap changes with task ambiguity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. This paper introduces Hell-or-High-Water, a benchmark for studying whether language-model agents can recover from external failures such as a function being disabled. The benchmark is built from Spider text-to-SQL queries by parameterizing SQL, wrapping queries as Python functions, and decomposing queries containing subqueries into two alternative solution paths: a direct single-function call and a multi-call backup plan. Agents operate in a CodeAct environment with a large function library and a search tool. The main experiment (Table 1) measures functional correctness on 830 test problems with and without forced external errors, reporting accuracy drops of 36–45% across five models. Additional experiments analyze error types, the effect of search-space size in a closed world, model scaling, the ability to recognize unsolvable tasks, and question underspecification. The paper concludes that language agents struggle to formulate and execute backup plans in response to environment feedback.

Significance. If the benchmark's solvability guarantee is sound, this is a valuable contribution. It provides a reproducible, deterministic evaluation of agentic recovery from external failures, avoiding LLM-based verification and using bootstrapped standard errors. The open-source code and data, the closed-world ablation, and the error taxonomy are useful for the community. The negative result that resilience to external errors does not improve with model size is an important and practically relevant finding. The benchmark design also isolates backup-planning behavior from basic tool-calling skill more cleanly than prior benchmarks, especially in the closed-world condition.

major comments (1)
  1. [§3.2, Appendix C.1] The central claim of Table 1 rests on the assertion in §3.2 that 'SQL containing subqueries can be solved in at least two ways,' so that after the first solution function is disabled a valid backup plan always exists. Appendix C.1 demonstrates this construction only for an uncorrelated scalar subquery in a WHERE '=' clause (function_985 + function_986 versus function_987). Correlated subqueries, and subqueries in SELECT, FROM, or EXISTS positions, cannot in general be precomputed by one argument-free function call and then piped into a second call through the scratchpad mechanism; the backup path would either return wrong results or require additional orchestration that the two-call decomposition does not provide. The quality-control section filters empty results and outputs over 100 rows but does not filter or validate subquery types, and the Limitations section does not mention this caveat. Because unsolvable problems in the external-error condition would inflate the measured accuracy drop and thus overstate agent inflexibility, please report per-problem verification that, after disabling the first attempted solution function, the known backup sequence is executable and produces the reference answer. Filter or fix any violating problems and rerun the main analysis, or report the fraction of items with invalid backups and quantify the sensitivity of the Table 1 results.
minor comments (5)
  1. [§A] The description of error injection states that the first function in the solution set that the agent tries is replaced with an error-throwing version. It is unclear whether, in the 'External errors (Y)' condition, every run actually experiences an injected error or only the runs in which the agent attempts at least one solution-set function. If the latter, runs without any injected error are included in the Y condition, which would dilute the measured drop; please clarify and, if necessary, condition the analysis on trajectories that attempted a solution function.
  2. [§3.2] There is a typo in the quality-control paragraph: 'correpsond' should be 'correspond.'
  3. [Appendix D/E] The search_tools documentation in Appendix D and the prompt in Appendix E.2 list 'query' as the property name but specify 'required': ['search_query'], which does not match. Although the example traces call the tool with 'query', this inconsistency in the provided documentation could confuse models and should be corrected to match the actual API.
  4. [§A, footnote 10] The footnote states that 'there will always exist multiple subsets of Fs that lead to a correct solution'; this is exactly the unverified claim that underlies the benchmark. It should be either supported with a per-problem check or qualified with the verification results requested in the major comment.
  5. [Figure 5] The left panel's legend labels 'Distractors only (Y)' and 'Distractors only (N)' are not defined in the text or caption; please explain what the Y/N distinction refers to.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy results are direct empirical measurements against deterministic SQL-derived reference answers, with no fitted inputs and no load-bearing self-citation.

full rationale

The paper's central claim (language agents struggle to formulate and execute backup plans in response to external failures) is an empirical measurement on a benchmark built by a deterministic pipeline: Spider SQL queries containing subqueries are parameterized and wrapped into Python functions, with the gold answer fixed as the deterministic output of the original SQL query. The external-errors condition replaces the first attempted solution function with an error-throwing variant, and the reported accuracy drops in Table 1 (e.g., Gemini 2.0 Flash from 71.4% to 41.1%) come from comparing agent outputs against that SQL-derived reference. No parameter is fitted and then re-predicted: the backup path (function_985 followed by function_986) is shown in Appendix C.1 to be equivalent by construction to the direct function (function_987), and this equivalence is a property of the benchmark construction, not an assumption used to score the agents. The only self-citation (Jiang et al., 2025, in the future-work discussion) is not load-bearing: the central results rest on the paper's own Table 1 measurements and its Table 2 error taxonomy, not on the cited work. The assertion that every one of the 830 test problems remains solvable after error injection is argued from the SQL-subquery construction (Section 3.2 and Appendix C.1) rather than per-problem verification, and correlated or non-scalar subqueries may not decompose as claimed; however, that is a validity and correctness risk about benchmark solvability, not circularity, because no equation or fitted parameter in the paper reduces the reported accuracy to an input of the construction. Consequently, no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted and no new entities are postulated; the benchmark is a measurement instrument. Its validity rests on construction guarantees, backup-path solvability, Spider ground truth, and the accuracy of generated function documentation. The novelty-to-pretraining assumption is asserted but not tested, making it the most fragile entry.

assumptions (4)
  • domain assumption Every SQL query with a subquery yields at least two valid, non-overlapping function-call solutions after parameterization and wrapping.
    Core construction guarantee in Section 3.2 and Appendix A; backs the claim that all forced-failure tasks remain solvable.
  • domain assumption The original Spider SQL query output is the correct reference answer for each task.
    Used for automatic verification in Section 4.1; assumes Spider ground truth is correct apart from manually filtered empty or oversized results.
  • domain assumption LLM-generated documentation for 4,450 functions is complete and accurate enough to solve tasks without prior knowledge of the un-descriptive function names.
    The agent must rely on search_tools and get_info documentation in Appendix D; documentation is unit-tested but not verified per function.
  • ad hoc to paper Benchmark tasks are novel relative to the LLMs' pre-training data.
    Asserted in Section 1 ('our tasks are novel relative to the LLM pre-training data') without a contamination check; built from the public Spider dataset, which may appear in training corpora.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hell or High Water: Evaluating Agentic Recovery from External Failures." pith.science (2026). https://pith.science/paper/VFGABQMB

@misc{pith2026250811027,
  author       = {Pith},
  title        = {Pith review of: Hell or High Water: Evaluating Agentic Recovery from External Failures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VFGABQMB}},
  note         = {Machine review of arXiv:2508.11027}
}
read the original abstract

As language model agents are applied to real world problems of increasing complexity, they will be expected to formulate plans across large search spaces. If those plans fail for reasons beyond their control, how well do language agents search for alternative ways to achieve their goals? We devise a specialized agentic planning benchmark to study this question. Each planning problem is solved via combinations of function calls. The agent searches for relevant functions from a set of over four thousand possibilities, and observes environmental feedback in the form of function outputs or error messages. Our benchmark confronts the agent with external failures in its workflow, such as functions that suddenly become unavailable. At the same time, even with the introduction of these failures, we guarantee that the task remains solvable. Ideally, an agent's performance on the planning task should not be affected by the presence of external failures. Overall, we find that language agents struggle to formulate and execute backup plans in response to environment feedback. While state-of-the-art models are often able to identify the correct function to use in the right context, they struggle to adapt to feedback from the environment and often fail to pursue alternate courses of action, even when the search space is artificially restricted. We provide a systematic analysis of the failures of both open-source and commercial models, examining the effects of search space size, as well as the benefits of scaling model size in our setting. Our analysis identifies key challenges for current generative models as well as promising directions for future work.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 6 canonical work pages

  1. [1]

    Evaluating large language models trained on code, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, and Michael Petrov et al. Evaluating large language models trained on code, 2021. URL https://arxiv.org/abs/2107.03374

    arXiv 2021

  2. [2]

    Teaching large language models to self-debug, 2023 a

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, 2023 a . URL https://arxiv.org/abs/2304.05128

    arXiv 2023

  3. [3]

    Chatcot: Tool-augmented chain-of-thought reasoning on chat-based large language models

    Zhipeng Chen, Kun Zhou, Beichen Zhang, Zheng Gong, Wayne Xin Zhao, and Ji-Rong Wen. Chatcot: Tool-augmented chain-of-thought reasoning on chat-based large language models. arXiv preprint arXiv:2305.14323, 2023 b

    arXiv 2023

  4. [4]

    Anytool: Self-reflective, hierarchical agents for large-scale api calls, 2024

    Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls, 2024. URL https://arxiv.org/abs/2402.04253

    arXiv 2024

  5. [5]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, ..., Zhiwei Zhao, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    arXiv 2024

  6. [6]

    Critic: Large language models can self-correct with tool-interactive critiquing, 2024

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing, 2024. URL https://arxiv.org/abs/2305.11738

    arXiv 2024

  7. [7]

    Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings

    Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305.11554, 2023. URL https://arxiv.org/abs/2305.13252

    arXiv 2023

  8. [8]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    arXiv 2021

Show all 44 references
  1. [9]

    Not all llm reasoners are created equal, 2024

    Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, and Rishabh Agarwal. Not all llm reasoners are created equal, 2024. URL https://arxiv.org/abs/2410.01748

    2024 arXiv

  2. [10]

    Creativity in ai: Progresses and challenges, 2024

    Mete Ismayilzada, Debjit Paul, Antoine Bosselut, and Lonneke van der Plas. Creativity in ai: Progresses and challenges, 2024. URL https://arxiv.org/abs/2410.17218

    2024 arXiv

  3. [11]

    Feedback friction: Llms struggle to fully incorporate external feedback, 2025

    Dongwei Jiang, Alvin Zhang, Andrew Wang, Nicholas Andrews, and Daniel Khashabi. Feedback friction: Llms struggle to fully incorporate external feedback, 2025. URL https://arxiv.org/abs/2506.11930

    2025

  4. [12]

    Chain of code: Reasoning with a language model-augmented code emulator, 2024

    Chengshu Li, Jacky Liang, Andy Zeng, Xinyun Chen, Karol Hausman, Dorsa Sadigh, Sergey Levine, Li Fei-Fei, Fei Xia, and Brian Ichter. Chain of code: Reasoning with a language model-augmented code emulator, 2024. URL https://arxiv.org/abs/2312.04474

    2024 arXiv

  5. [13]

    Api-bank: A benchmark for tool-augmented llms

    Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A benchmark for tool-augmented llms. Conference on Empirical Methods in Natural Language Processing EMNLP , 2023. URL https://aclanthology.org/2023.emnlp-main.187/

    2023

  6. [14]

    Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool use capabilities. arXiv preprint arXiv:2408.04682, 2024 a . UR...

    2024 arXiv

  7. [15]

    GEAR: augmenting language models with generalizable and efficient tool resolution

    Yining Lu, Haoping Yu, and Daniel Khashabi. GEAR: augmenting language models with generalizable and efficient tool resolution. In Conference of the European Chapter of the Association for Computational Linguistics EACL , 2024 b . URL https://arxiv.org/abs/2307.08775

    2024 arXiv

  8. [16]

    Benchmarking language model creativity: A case study on code generation

    Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, and Daniel Khashabi. Benchmarking language model creativity: A case study on code generation. In Conference of the North American Chapter of the Association for Computational Linguistics NAACL , 2025. URL https://arxiv.org/ab...

    2025 arXiv

  9. [17]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterativ...

    2023

  10. [18]

    Talm: Tool augmented language models

    Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models. arXiv preprint arXiv:2205.12255, 2022

    2022 arXiv

  11. [19]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. URL https://arxiv.org/abs/2305.15334

    2023 arXiv

  12. [20]

    Escapebench: Pushing language models to think outside the box, 2024

    Cheng Qian, Peixuan Han, Qinyu Luo, Bingxiang He, Xiusi Chen, Yuji Zhang, Hongyi Du, Jiarui Yao, Xiaocheng Yang, Denghui Zhang, Yunzhu Li, and Heng Ji. Escapebench: Pushing language models to think outside the box, 2024. URL https://arxiv.org/abs/2412.13549

    2024 arXiv

  13. [21]

    Making language models better tool learners with execution feedback

    Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. Making language models better tool learners with execution feedback. arXiv preprint arXiv:2305.13068, 2023. URL https://arxiv.org/abs/2305.13252

    2023 arXiv

  14. [22]

    Tool learning with foundation models

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023 a . URL https://arxiv.org/abs/2304.08354

    2023 arXiv

  15. [23]

    Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023 b

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to m...

    2023 arXiv

  16. [24]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  17. [25]

    Self-critiquing models for assisting human evaluators, 2022

    William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. URL https://arxiv.org/abs/2206.05802

    2022 arXiv

  18. [26]

    ToolFormer: language models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. ToolFormer: language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. URL https://arxiv.org/abs/2302.04761

    2023 arXiv

  19. [27]

    Reflexion: an autonomous agent with dynamic memory and self-reflection

    Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. In Advances in Neural Information Processing Systems NeurIPS , 2023. URL https://arxiv.org/abs/2303.11366

    2023 arXiv

  20. [28]

    Tools fail: Detecting silent errors in faulty tools

    Jimin Sun, So Yeon Min, Yingshan Chang, and Yonatan Bisk. Tools fail: Detecting silent errors in faulty tools. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 14272--14289...

    2024 doi

  21. [29]

    Toward self-improvement of llms via imagination, searching, and criticizing, 2024 a

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. Toward self-improvement of llms via imagination, searching, and criticizing, 2024 a . URL https://arxiv.org/abs/2404.12253

    2024 arXiv

  22. [30]

    Griffiths, and Faeze Brahman

    Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas L. Griffiths, and Faeze Brahman. Macgyver: Are large language models creative problem solvers?, 2024 b . URL https://arxiv.org/abs/2311.09682

    2024 arXiv

  23. [31]

    AppWorld: a controllable world of apps and people for benchmarking interactive coding agents

    Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: a controllable world of apps and people for benchmarking interactive coding agents. arXiv preprint arXiv:2407.18901, 202...

    2024 arXiv

  24. [32]

    Llms in the imaginarium: tool learning through simulated trial and error

    Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. Llms in the imaginarium: tool learning through simulated trial and error. arXiv preprint arXiv:2403.04746, 2024 a

    2024 arXiv

  25. [33]

    Executable code actions elicit better llm agents, 2024 b

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents, 2024 b . URL https://arxiv.org/abs/2402.01030

    2024 arXiv

  26. [34]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. T...

    2022

  27. [35]

    Travelplanner: A benchmark for real-world planning with language agents

    Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In International Conference on Machine Learning ICML , 2024. URL https://arxiv.org/pdf/2402.01622

    2024 arXiv

  28. [36]

    Gpt4tools: Teaching large language model to use tools via self-instruction

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2305.18752, 2023. URL https://arxiv.org/abs/2305.18752

    2023 arXiv

  29. [37]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations ICLR , 2023. URL https://openreview.net/pdf?id=WE\_vluYUL-X

    2023

  30. [38]

    Toolhop: A query-driven benchmark for evaluating large language models in multi-hop tool use, 2025

    Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, and Jiecao Chen. Toolhop: A query-driven benchmark for evaluating large language models in multi-hop tool use, 2025. URL ...

    2025 arXiv

  31. [39]

    S pider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to- SQL task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. S pider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to- SQL task. In Ellen...

    2018

  32. [40]

    Instruction-following evaluation for large language models

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023

    2023 arXiv

  33. [41]

    Toolqa: A dataset for llm question answering with external tools

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools. Advances in Neural Information Processing Systems, 36: 0 50117--50143, 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/9cb2a749...

    2023

  34. [42]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  35. [43]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  36. [44]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.