Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Language models that call external tools are often unable to notice and correct their own calling errors, and CRITICTOOL is the first benchmark built specifically to measure this recovery ability.

desk verdict A genuinely new self-critique benchmark for tool-calling errors, with a real artifact and a broad model sweep, but the FINISH evaluator scores any FinishAction as success, so the paper's headline external-error results may be inflated. read the letter →

arxiv 2506.13977 v1 pith:QFDEQQ3Y submitted 2025-06-11 cs.SE cs.CL

Shiting Huang , Zhen Fang , Zehui Chen , Siyu Yuan , Junjie Ye , Yu Zeng , Lin Chen , Qi Mao
show 1 more author
Feng Zhao
This is my paper
classification cs.SEcs.CL
keywords self-critiqueevaluationtoollearningLLMbenchmarkfunctioncallingerrorserrorrecoverytool-callingagentsdataevolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CRITICTOOL aims to establish that the main weakness in current tool-using language models is not whether they can call an API correctly, but whether they can notice and recover when a call goes wrong. Existing tool benchmarks score final outputs or compare against golden answers, and mostly filter out error cases, so they never measure this recovery behavior. The paper builds a dataset of 1,490 base and 1,250 evolved test cases from BFCL v3 and T-Eval, injects five kinds of errors using a GPT-4o simulator, and scores models on four self-critique skills: reflecting on the error, correcting it, retrying external failures, and skipping or finishing when retries fail. On this benchmark GPT-4o reaches an overall 69.01, the best open models come close, and tool-use-finetuned models nearly collapse, with ToolLLaMA2-7B at 0.13 overall. If the benchmark is right, then accuracy on existing tool-use leaderboards overstates how well models will survive real, error-prone tool interactions.

What carries the argument

The load-bearing object is the CRITICTOOL construction pipeline plus its scoring scheme. Starting from 733 ground-truth tool-calling trajectories drawn from BFCL v3 and T-Eval, the pipeline uses a GPT-4o error simulator prompted with few-shot examples to generate internal errors across a wider set of tools, collects external environment errors from real repeated API calls and a GPT-4o API simulator, and produces tool responses by cache lookup, live API execution, or simulated response. The SRM (Scalable and Robust Mixed Self-Evolution) strategy then mutates each task along four axes—Long Context, Extra Tools, Noisy Query, Harder Tools—followed by GPT-4o equivalence verification and manual filtering at an 18.63% pass rate. Evaluation is step-level: a trajectory is cut at a random step k, an error may be inserted at that step, and the model is scored separately for REFLECT, CORRECT, RETRY, and SKIP/FINISH, with overall score weighted 0.2/0.3/0.05/0.45.

What would settle it

Run CRITICTOOL with errors injected by several different models, such as GPT-3.5 and LLaMA-3.1, instead of GPT-4o and compare model rankings; if rankings change substantially, the benchmark measures style matching rather than critique. Separately, present a model's valid but non-golden recovery actions to human annotators; if a large share are judged acceptable, the single-golden-trajectory scoring inflates failure rates.

Watch

Extended reading notes

Core claim

The paper's central claim is that self-critique during tool calls is a distinct, measurable capability that current models largely lack and current benchmarks ignore. CRITICTOOL operationalizes self-critique as four behaviors: REFLECT (detect the error and name its category), CORRECT (issue the fixed tool call), RETRY (repeat a failed call, at most three times), and SKIP/FINISH (move on or ask the user once the environment error persists). Errors are divided into internal model-driven ones—tool selection, tool hallucination, parameter key, parameter value—and external environment ones such as permission and connection failures. The paper reports a wide but consistent ordering: GPT-4o leads at 69.01, LLaMA3.1-70B and Qwen2.5-72B are close, and tool-use-finetuned models (ToolLLaMA2-7B, ToolACE-8B, AgentLM-7B) perform worst on the very skill their training should support. It also finds that making the benchmark harder with longer context, extra tools, noisy queries, and degraded API documentation lowers every model's score, and that tool-use accuracy and self-critique performance move together across models.

Load-bearing premise

The whole measurement stands on the assumption that the errors GPT-4o injects into otherwise correct tool calls are indistinguishable from errors models naturally make, and that each error has exactly one correct recovery—the original ground-truth next step.

Editorial extensions

If this is right

  • Existing tool-use leaderboards should be read as measuring happy-path accuracy; a model with high BFCL or T-Eval accuracy can still be unable to recover from errors on CRITICTOOL.
  • Tool-use fine-tuning can trade away self-critique: ToolLLaMA2-7B, ToolACE-8B, and AgentLM-7B score far below their base-ability peers despite being trained for function calling.
  • Adding realistic distractions—long contexts, noisy queries, extra tools, degraded API docs—uniformly lowers self-critique scores, so robustness degrades exactly where real deployments are hardest.
  • Since tool-use and self-critique scores track each other (Fig. 4), improving the latter is a plausible lever for improving the former, not just a diagnostic.
  • CoT prompting raises self-critique scores for most models, notably GPT-4o to 78.0 overall, so the measured gap is partly a prompting problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If error style matters, CRITICTOOL's numbers could partly reward models that mimic GPT-4o's error patterns, since the same model generates the injected errors and verifies the data; regenerating the benchmark with two or three different error generators and checking rank stability would separate critique skill from style matching.
  • The single-golden-trajectory scoring assumes exactly one valid recovery; in many real tasks the next subtask could be done in several orders or parameter choices, so human-acceptance trials on non-golden completions would estimate how much of the reported failure is real versus an artifact of scoring strictness.
  • A direct training corollary is testable: fine-tune a model on CRITICTOOL-style error traces and measure whether its success on BFCL/T-Eval and on this benchmark rise together; the paper's correlation result predicts they should.
  • The benchmark's 18.63% manual pass rate means the released examples are heavily curated; whether the same ranking appears with less aggressive filtering is an open question about how robust the phenomenon is.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper analyzes LLM error patterns in existing tool-calling benchmarks and introduces CRITICTOOL, a benchmark for evaluating self-critique capabilities in tool-use error scenarios. It defines five error patterns (tool selection, tool hallucination, parameter key, parameter value, environment) and evaluates four dimensions: reflect, correct, retry, and skip/finish. The dataset is built from BFCL and T-Eval, augmented by GPT-4o-based error simulation, API simulation, and four evolution strategies, with manual filtering at an 18.63% pass rate. Experiments on 14 LLMs find that GPT-4o achieves the highest Overall score (69.01), while tool-use-finetuned models such as ToolLLaMA2-7B score near zero, and the paper discusses relationships between tool-use and self-critique abilities.

Significance. If valid, CRITICTOOL fills a real gap: existing tool-use benchmarks largely ignore error recovery, and the observation that most models fail to recover (Table 1) is plausible and valuable. The benchmark provides a fine-grained, multi-dimensional evaluation, is grounded in an initial observation study, and the code is publicly released. The manual filtering and evolutionary augmentation are strengths. However, the validity of the benchmark as a measure of self-critique depends on two fragile links: whether the FINISH evaluator actually checks the required content, and whether GPT-4o-injected errors are representative of natural tool-calling errors. Because the external-error branch is small and heavily weighted in the Overall score, weaknesses in the FINISH metric and the weight selection can materially affect the leaderboard and the paper's central conclusions.

major comments (4)
  1. [Appendix C.3.5; §2.3.1–2.3.2] The FINISH evaluator (Appendix C.3.5) sets the tool score to 1 for any action whose name is FinishAction, without verifying that the content asks the user for further guidance. This contradicts the specification in §2.3.1, where finish means to 'inform the user that further guidance is required,' and the paper's own qualitative failure example in Fig. 28 labels a FinishAction claiming 'I have fetched the meta information...' as a hallucination. Because Skip/Finish is weighted 0.45 in the Overall score and the external-error branch comprises only 174 base cases (Appendix B.2), a model that hallucinates a final answer and terminates can receive the same credit as a model that honestly reports the error. This threatens the leaderboard and the conclusion about which models properly handle environment errors. The evaluator should be revised to require that the FinishAction content matches the golden request for guidance (e.g., via semantic comparison or exact matching), or to explicitly classify hallucinated completions as failures.
  2. [§2.2.3, §2.2.6, Limitations] The error simulator, API simulator, evolution strategies, and verification are all performed by GPT-4o, and GPT-4o tops the resulting leaderboard. While the error taxonomy is grounded in an independent observation study (Table 3) and the 18.63% manual pass rate (§2.2.6) helps quality, the benchmark may still reflect GPT-4o's error style rather than a model-independent notion of tool-calling self-critique. The paper should validate that simulated errors are indistinguishable from naturally occurring ones—for example, by human annotation of error naturalness or by comparing model performance on real observed errors versus synthetic errors. Without such evidence, the claim of a general self-critique benchmark is overstated, and GPT-4o's top position may be partly an artifact of the construction pipeline.
  3. [§2.3.2] The Overall score weights (reflect 0.2, correct 0.3, retry 0.05, skip/finish 0.45) are introduced without justification or sensitivity analysis. Given the small external-error branch (174 base cases) and the FINISH metric issue above, the weights can materially affect the rankings used in Q2 and Q4 (e.g., which model leads, and the correlation with BFCL overall accuracy). The paper should either justify the weights with an explicit task-completion cost model or report results under a plausible range of weightings to show that the central conclusions are robust.
  4. [Appendix C.3 (golden solution definitions)] The evaluator compares predicted recovery sequences to the single ground-truth trajectory. For external environment errors, multiple acceptable behaviors exist: a model might retry twice and then finish, while the golden answer retries three times and skips to the next subtask; a model that immediately skips after one retry receives a retry score of 0 even though the paper only specifies 'no more than three times.' This unique-path assumption penalizes valid strategies and may overstate differences between models. The paper should allow a set of acceptable golden sequences (e.g., human-annotated alternative recoveries) or use a more flexible partial-credit scheme.
minor comments (6)
  1. [Table 2 header] The header 'Retry Skip/Finish Break Tool Args' is ambiguous; clarify which columns belong to Retry and which to Skip/Finish, and specify how the Overall score is computed from the sub-scores.
  2. [§2.2.5] The text contains a duplicated phrase: 'we randomly we randomly select 2-4 evolution strategies' (Appendix C.2.3). Please fix the typo.
  3. [Figure 1] Figure 1 is difficult to read; the boxes and arrows are small and the labels become garbled in the PDF extraction. A higher-resolution figure with larger fonts would help.
  4. [Appendix C.3.5] The description of the FINISH evaluator only mentions the tool score and the break score; it is not clear how the Args column in Table 2 is computed for FINISH cases. Please specify whether the FinishAction arguments are compared to the golden request for guidance, or whether the Args score is simply ignored.
  5. [Related Work] The claim that CRITICTOOL is 'the first' self-critique benchmark for tool utilization should be tempered given existing robustness-oriented works such as RoTBench and ToolEyes; please add a more careful positioning discussion.
  6. [Appendix B.2] The external-error branch has only 174 base cases; consider reporting confidence intervals or bootstrap estimates for the main metrics to assess the stability of the leaderboard.

Circularity Check

1 steps flagged · score 4.0 of 10

The FINISH evaluator scores any FinishAction as success, conflating hallucinated terminal answers with honest requests for guidance in the 45%-weighted Skip/Finish dimension; GPT-4o also generates the error data it tops.

  1. other [Appendix C.3.5 (FINISH); cf. §2.3.1 and Fig. 28]
    "The finish evaluator first evaluates the break score in the same manner as the skip evaluator. It then checks whether the break-free action is FinishAction. If so, the tool score is set to 1. ... the assistant should ... finish the tool-calling process and inform the user that further guidance is required."

    The task definition requires the assistant to report the persistent environment error and request user guidance, but the FINISH evaluator assigns full credit for any FinishAction without checking the args content. Under this rule, the paper's own Fig. 28 'Hallucination' example—a FinishAction claiming 'I have fetched the meta information...'—receives tool score 1, identical to a genuine request for guidance. Because skip/finish carries weight 0.45 in Overall, the leaderboard's external-error scores and overall scores reduce, by construction, to a token-name check rather than to the self-critique behavior the paper claims to measure. This is a metric-level circularity, not a fit of parameters, but it is a direct reduction of the claimed construct to the output format.

full rationale

The remainder of the benchmark is largely non-circular. The error taxonomy comes from an independent observation study of BFCL, T-Eval, API-Bank, and NESTFUL (§2.1, Appendix A); the base data are ground-truth trajectories from BFCL and T-Eval that were manually filtered; and 18.63% of synthetic examples passed manual review (§2.2.6), so the internal error data are not merely GPT-4o's self-consistency. The most serious residual circularity stress is acknowledged by the authors in the Limitations: 'The construction of our benchmark relies on GPT-4o for error generation, evolution, and verification. The synthetic data may inevitably introduce biases inherent to GPT-4o.' GPT-4o therefore both writes and takes the exam; that is a benchmark-contamination concern, but it is not an equation-level reduction because GPT-4o's 69.01 score is an empirical outcome, not a consequence of the construction equations. I do not count the upstream self-citation of T-Eval as load-bearing circularity: T-Eval is an independently published benchmark and supplies only the correct trajectories, not the error labels or the evaluation rule. The concrete circular step is the FINISH scorer, which is explicitly defined to accept any FinishAction and thus cannot distinguish honest guidance-seeking from hallucinated termination; since this dimension is 45% of Overall, the headline result is materially affected. Score 4 reflects one genuine by-construction reduction plus a substantial self-generation limitation, while acknowledging the benchmark's independent external grounding.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The benchmark's claims rest on four domain assumptions and two hand-chosen design constants. There are no fitted numerical parameters in the usual sense; the free parameters are evaluation-design choices (weights, retry limit) that directly shape the headline numbers. The heaviest burden is the GPT-4o-in-the-loop assumption: error generation, API response simulation, and evolution verification are all performed by the same model family that then ranks first on the benchmark.

free parameters (2)
  • Overall score weights = reflect 0.2, correct 0.3, retry 0.05, skip/finish 0.45
    Hand-chosen weights in §2.3.2 sum to 1 and split internal vs external contributions equally (0.5/0.5) even though internal errors are 88% of the dataset; all headline 'best model' statements depend on this choice, which is presented without justification.
  • Retry limit for environment errors = 3
    Assistants are instructed 'NO MORE THAN THREE TIMES' (§2.3.1, Fig. 18); a different limit would change the trade-off between retry scoring and skip/finish scoring.
assumptions (4)
  • domain assumption GPT-4o-generated error trajectories and API responses, after the 18.63% manual pass filter, faithfully represent real-world tool-use errors and responses.
    The entire injected-error dataset and much of the tool feedback is produced by GPT-4o (§2.2.3, §2.2.4). If simulated errors are not representative, benchmark rankings become artifacts of the simulator.
  • domain assumption The original ground-truth trajectory is the unique correct recovery; deviation is scored as failure.
    Evaluation compares predicted actions to golden answers (Appendix C.3.2, C.3.3). This assumes tool use has a single correct trace, which is contestable for open-ended multi-step tasks.
  • domain assumption All 'error-free' filtered trajectories contain no errors, so over-reflection is always a mistake.
    Error-free data comes from manual filtering of BFCL and T-Eval trajectories (§2.2.2); if any residual error exists, the over-reflection penalty punishes correct detection.
  • standard math Cosine similarity of Sentence-BERT embeddings is a valid measure of parameter-value correctness.
    Used as the args score for string/any parameters (Appendix C.3.2); the similarity value is treated directly as a correctness score without calibration.
invented entities (1)
  • Five-pattern error taxonomy (tool selection, tool hallucination, parameter key, parameter value, environment) independent evidence
    purpose: Organizes the error categories on which the benchmark's reflect/correct/retry tasks are built.
    A classification scheme, not a physical entity, so the graviton problem does not apply; it is grounded in the §2.1 observation study of real LLM outputs across four benchmarks and matches categories in prior error analyses (RoTBench, Tools Fail).

how reviews work

0 comments
Cite this review

Pith. "Pith review of CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios." pith.science (2026). https://pith.science/paper/QFDEQQ3Y

@misc{pith2026250613977,
  author       = {Pith},
  title        = {Pith review of: CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFDEQQ3Y}},
  note         = {Machine review of arXiv:2506.13977}
}
read the original abstract

The ability of large language models (LLMs) to utilize external tools has enabled them to tackle an increasingly diverse range of tasks. However, as the tasks become more complex and long-horizon, the intricate tool utilization process may trigger various unexpected errors. Therefore, how to effectively handle such errors, including identifying, diagnosing, and recovering from them, has emerged as a key research direction for advancing tool learning. In this work, we first extensively analyze the types of errors encountered during the function-calling process on several competitive tool evaluation benchmarks. Based on it, we introduce CRITICTOOL, a comprehensive critique evaluation benchmark specialized for tool learning. Building upon a novel evolutionary strategy for dataset construction, CRITICTOOL holds diverse tool-use errors with varying complexities, which better reflects real-world scenarios. We conduct extensive experiments on CRITICTOOL, and validate the generalization and effectiveness of our constructed benchmark strategy. We also provide an in-depth analysis of the tool reflection ability on various LLMs, offering a new perspective on the field of tool learning in LLMs. The code is available at \href{https://github.com/Shellorley0513/CriticTool}{https://github.com/Shellorley0513/CriticTool}.

Figures

Figures reproduced from arXiv: 2506.13977 by the authors.

Figure 1
Figure 1. Overview of CRITICTOOL construction pipeline. The pipeline begins with collecting and testing tool-use benchmarks to obtain a variety of correct and incorrect tool-calling trajectories. GPT-based simulators and repeated API calls are employed to diversify internal and external error patterns. And responses to internal errors are generated via cache retrieval, API execution, and API simulator. Finally, the error data… view at source ↗
Figure 2
Figure 2. Examples of Errors in multi-step tool call tasks. Multi-step tool call errors are categorized into five patterns based on the source and characteristics of the errors: Tool Selection Errors, Tool Hallucination Errors, Parameters Key Errors, Parameters Value Errors and Environment Errors. process, resulting in eventual task failure. This is￾sue becomes particularly pronounced as tasks grow more complex and long-horiz… view at source ↗
Figure 3
Figure 3. Comparison of the performance of five mod [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Comparison between BFCL Overall Accuracy [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Error distribution for Base data in CRITIC￾TOOL [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Length distribution for Base and Evolution [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: The framework of Scalable and Robust Mixed Self-Evolution (SRM) [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Comparison of CRITICTOOL Overall Scores with tool-use benchmarks’ Overall Accuracy across several models. C.3.2 CORRECT The correct evaluator assesses the model’s ability to correct its actions after making a mistake. For trajectories containing errors, the evaluator f…
Figure 9
Figure 9. Figure 9: The framework of Evaluation Process. score are determined using the same comparison method as in the correct evaluation. C.3.5 FINISH The finish evaluator first evaluates the break score in the same manner as the skip evaluator. It then checks whether the break-free ac…
Figure 10
Figure 10. Figure 10: An example of standardized test data [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Examples of LLMs’ performance in the presence of internal errors. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Examples of LLMs’ performance in the presence of external errors. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: An example of collected tool-calling trajectories. [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: An example refined API documentation: TravelAPI. [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: An example prompt of Error Diversification. [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Prompt of API simulator [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: An example of internal error critique task in C [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: An example of external error critique task in C [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: An example of Noisy Query Evolution task in C [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: An example prompt of Noisy Query Evolution. [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: An example prompt of Harder Tools Evolution. [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: An example prompt of the verification of Long Context. [PITH_FULL_IMAGE:figures/full_fig_p028_22.png]
Figure 23
Figure 23. Figure 23: An example prompt of the verification of Noisy Query. [PITH_FULL_IMAGE:figures/full_fig_p028_23.png]
Figure 24
Figure 24. Figure 24: An example prompt of the verification of Extra Tools. [PITH_FULL_IMAGE:figures/full_fig_p029_24.png]
Figure 25
Figure 25. Figure 25: An example prompt of the verification of Harder Tools. [PITH_FULL_IMAGE:figures/full_fig_p030_25.png]
Figure 26
Figure 26. Figure 26: Qualitative examples of self-critique performance in internal error-injected task. [PITH_FULL_IMAGE:figures/full_fig_p031_26.png]
Figure 27
Figure 27. Figure 27: Qualitative examples of self-critique performance in internal error-free task. [PITH_FULL_IMAGE:figures/full_fig_p031_27.png]
Figure 28
Figure 28. Figure 28: Qualitative examples of self-critique performance in external error task. [PITH_FULL_IMAGE:figures/full_fig_p032_28.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video-DeepResearch trains video agents with a two-stage SFT plus GRPO pipeline to do visual web searches before text searches, and introduces a 200-question benchmark requiring both video grounding and open-web knowledge.

Reference graph

Works this paper leans on

53 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. 2024. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. In EMNLP, pages 1131--1139

    work page 2024

  2. [2]

    Mistral AI. 2024. https://mistral.ai/news/ministraux/ Un ministral, des ministraux

    work page 2024

  3. [3]

    AI@Meta. 2024. https://github.comf/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    work page 2024

  4. [4]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023. Learning from mistakes makes llm better reasoner. arXiv preprint arXiv:2310.20689

    arXiv 2023

  5. [5]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

    2024

  6. [6]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  7. [7]

    Kinjal Basu, Ibrahim Abdelaziz, Kelsey Bradford, Maxwell Crouse, Kiran Kate, Sadhana Kumaravel, Saurabh Goyal, Asim Munawar, Yara Rizk, Xin Wang, et al. 2024. Nestful: A benchmark for evaluating llms on nested sequences of api calls. arXiv preprint arXiv:2409.03797

    arXiv 2024

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS

    2020

Show all 53 references
  1. [9]

    Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2023. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793

    2023 arXiv

  2. [10]

    Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. 2024 a . Advancing tool-augmented large language models: Integrating insights from errors in inference trees. arXiv preprint arXiv:2406.07115

    2024 arXiv

  3. [11]

    Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. 2024 b . T -eval: Evaluating the tool utilization capability of large language models step by step. In ACL, pages 9510--9529

    2024

  4. [12]

    Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. 2024 c . Agent- FLAN : Designing data and methods of effective agent tuning for large language models. In ACL, pages 9354--9366

    2024

  5. [13]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

    2024 arXiv

  6. [14]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738

    2023 arXiv

  7. [15]

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024 a . S table T ool B ench: Towards stable large-scale benchmarking on tool learning of large language models. In ACL, pages 11143--11156

    2024

  8. [16]

    Zishan Guo, Yufei Huang, and Deyi Xiong. 2024 b . CT ool E val: A C hinese benchmark for LLM -powered agent evaluation in real-world API interactions. In ACL, pages 15711--15724

    2024

  9. [17]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  10. [18]

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515

    2024 arXiv

  11. [19]

    Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. 2024. C ritique LLM : Towards an informative critique generation model for evaluation of large language model generation. In AC...

    2024

  12. [20]

    Yilun Kong, Jingqing Ruan, YiHong Chen, Bin Zhang, Tianpeng Bao, Shi Shiwei, du Guo Qing, Xiaoru Hu, Hangyu Mao, Ziyue Li, Xingyu Zeng, Rui Zhao, and Xueqian Wang. 2024. TPTU -v2: Boosting task planning and tool usage of large language model-based agents in real-world industry...

    2024

  13. [21]

    Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-ling Mao. 2024. Criticeval: Evaluating large language models as critic. arXiv preprint arXiv:2402.13764

    2024 arXiv

  14. [22]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API -bank: A comprehensive benchmark for tool-augmented LLM s. In EMNLP, pages 3102--3116

    2023

  15. [23]

    Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. 2024. C ritic B ench: Benchmarking LLM s for critique-correct reasoning. In ACL, pages 1552--1587

    2024

  16. [24]

    Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. 2024. Toolace: Winning the points of llm function calling. arXiv preprint arXiv:2409.00920

    2024 arXiv

  17. [25]

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. 2024. Llm critics help catch llm bugs. arXiv preprint arXiv:2407.00215

    2024 arXiv

  18. [26]

    OpenAI. 2022. https://openai.com/blog/chatgpt Introducing chatgpt

    2022

  19. [27]

    OpenAI. 2024. https://openai.com/o1/ Introducing openai o1

    2024

  20. [28]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334

    2023 arXiv

  21. [29]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789

    2023 arXiv

  22. [30]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. From exploration to mastery: Enabling llms to master tools via self-driven interactions. arXiv preprint arXiv:2410.08197

    2024 arXiv

  23. [31]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    2019 arXiv

  24. [32]

    Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2023. Taskbench: Benchmarking large language models for task automation. arXiv preprint arXiv:2311.18760

    2023 arXiv

  25. [33]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In NeurIPS

    2023

  26. [34]

    Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization of LLM agents. In ACL, pages 7584--7600

    2024

  27. [35]

    Jimin Sun, So Yeon Min, Yingshan Chang, and Yonatan Bisk. 2024. Tools fail: Detecting silent errors in faulty tools. In EMNLP, pages 14272--14289

    2024

  28. [36]

    Qwen Team. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671

    2024 arXiv

  29. [37]

    Qwen Team. 2024 b . https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  30. [38]

    Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Haitao Mi, and Dong Yu. 2024. Toward self-improvement of llms via imagination, searching, and criticizing. arXiv preprint arXiv:2404.12253

    2024 arXiv

  31. [39]

    Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su. 2024 a . LLM s in the imaginarium: Tool learning through simulated trial and error. In ACL, pages 10583--10604

    2024

  32. [40]

    Siyuan Wang, Zhuohan Long, Zhihao Fan, Zhongyu Wei, and Xuanjing Huang. 2024 b . Benchmark self-evolving: A multi-agent framework for dynamic llm evaluation. arXiv preprint arXiv:2402.11443

    2024 arXiv

  33. [41]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS

    2022

  34. [42]

    Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504

    2023 arXiv

  35. [43]

    Patil, Ion Stoica, and Joseph E

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard

    2024

  36. [44]

    Jihan Yao, Wenxuan Ding, Shangbin Feng, Lucy Lu Wang, and Yulia Tsvetkov. 2024. Varying shades of wrong: Aligning llms with wrong answers only. arXiv preprint arXiv:2410.11055

    2024 arXiv

  37. [45]

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, et al. 2024 a . Tooleyes: Fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprint arXiv:2401.00741

    2024 arXiv

  38. [46]

    Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024 b . R o TB ench: A multi-level benchmark for evaluating the robustness of large language models in tool learning. In EMNLP, pages 313--333

    2024

  39. [47]

    Jiahao Ying, Mingbao Lin, Yixin Cao, Wei Tang, Bo Wang, Qianru Sun, Xuanjing Huang, and Shuicheng Yan. 2024. LLM s-as-instructors: Learning from errors toward automating model improvement. In EMNLP, pages 11185--11208

    2024

  40. [48]

    Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. 2024. Easytool: Enhancing llm-based agents with concise tool instruction. arXiv preprint arXiv:2401.06201

    2024 arXiv

  41. [49]

    Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. Agenttuning: Enabling generalized agent abilities for llms. arXiv preprint arXiv:2310.12823

    2023 arXiv

  42. [50]

    Wenqi Zhang, Yongliang Shen, Linjuan Wu, Qiuying Peng, Jun Wang, Yueting Zhuang, and Weiming Lu. 2024. Self-contrast: Better reflection through inconsistent solving perspectives. In ACL, pages 3602--3622

    2024

  43. [51]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong W...

    2023 arXiv

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.