Pith. sign in

REVIEW 3 major objections 5 minor 41 references

When2Call: When (not) to Call Tools

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Knowing when not to call a tool is a distinct, under-measured capability, this paper argues: its When2Call benchmark shows even GPT-4-class models far from ceiling, and preference optimization on the data nearly doubles open-model scores.

desk verdict When2Call is a genuinely new four-way benchmark for tool-calling decisions, but its own appendix undercuts the central 'direct answer is always hallucination' assumption, so the headline numbers need a grain of salt. read the letter →

arxiv 2504.18851 v1 pith:BCBKPMDX submitted 2025-04-26 cs.CL

Hayley Ross , Ameya Sunil Mahabaleshwarkar , Yoshi Suhara This is my paper · ORCID
classification cs.CL
keywords toolcallingwhen-to-calldecisionsabstentionhallucinationbenchmarkpreferenceoptimizationmultiple-choiceevaluationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When2Call is a new benchmark for a capability that existing tool-calling benchmarks skip: not whether a language model calls the right tool with the right arguments, but whether it should call a tool at all. Each question is cast as a forced choice among four behaviors — answer directly, call the tool, ask a follow-up question, or admit it cannot answer — and because every question genuinely requires a tool, the direct answer is a hallucination by construction. The paper reports that open and closed tool-calling models, including GPT-4-class systems, perform far from ceiling on this choice, and that performance does not reliably improve with model size. It then shows that a preference-optimization training regime on the When2Call data improves the when-to-call decision far more than supervised fine-tuning, while keeping accuracy on the standard tool-calling leaderboard broadly intact. If the result holds, developers gain a way to measure and fix a failure mode — calling tools, or hallucinating answers, when the right move is to ask or abstain — that current evaluation pipelines largely miss.

What carries the argument

The central object is a four-way multiple-choice formulation of tool-calling as a behavior decision: direct text answer, tool call, follow-up question, or unable to answer. Every question is constructed to require tool access, which makes the four choices mutually exclusive and lets three distinct errors be read off a confusion matrix: choosing the direct answer is answer hallucination, choosing a tool call when no tool specification is provided is tool hallucination, and choosing a tool call when a required parameter is missing is parameter hallucination. Evaluation takes log-probabilities over the four options rather than parsing generated calls, and the same four-way structure doubles as the training signal for preference optimization by pairing the correct answer as the chosen response and an incorrect answer as the rejected one.

What would settle it

The central claim would be undercut if a current frontier tool-calling model scored near ceiling on the released When2Call evaluation without any When2Call-specific training; a reader can check directly by running the released evaluation script on any new model and inspecting its macro F1 and tool-hallucination rate. The claim that the RPO gains are a general skill rather than memorized phrasings can be tested by training on When2Call and evaluating on mismatches built from tool specifications never seen in training: if the improvement vanishes, part of the benchmark's headroom is a phrasing artifact.

Watch

Extended reading notes

Core claim

The paper's central claim is that deciding whether to call a tool, ask for clarification, or refuse is a distinct tool-calling capability, separable from the call-accuracy skills measured by benchmarks like BFCL, and that When2Call is a benchmark that isolates it. It reports that across the Llama, Qwen, and xLAM families, macro F1 on this four-way decision sits between roughly 16 and 38, that GPT-4-class closed models reach only about 52 to 65 F1, and that scores do not reliably rise with model size. It further claims that the failure is trainable but not by naive instruction tuning: supervised fine-tuning on When2Call data improves the decision at the cost of over-conservatism and lower BFCL accuracy, while reward-aware preference optimization (RPO) on the same multiple-choice data raises When2Call macro F1 from about 32 to 52 on an 8B model, cuts tool hallucination from 19 percent to 1.2 percent, and lifts BFCL Irrelevance accuracy from 36.3 to 78.1 percent, showing the two pressures can be balanced.

Load-bearing premise

The load-bearing premise is that every When2Call question is truly unanswerable without a tool, making the direct answer always a hallucination; the paper's own quality audit found that 18 percent of sampled 'unable to answer' questions had issues, including questions partially answerable without a tool, which makes the ground-truth label for those items ambiguous.

Editorial extensions

If this is right

  • Because the direct answer is always a hallucination by construction, When2Call turns abstention quality into a number: answer, tool, and parameter hallucination rates can be read directly off the model's confusion matrix.
  • A high score on BFCL Irrelevance does not imply a high score on When2Call, so the benchmark measures a finer-grained and harder skill than existing when-not-to-call evaluations.
  • Naive negative-example training makes models over-conservative and hurts BFCL accuracy; RPO, with the correct answer as the chosen response and an incorrect answer as rejected, is the paper's demonstrated way to improve when-to-call decisions while preserving tool-call accuracy.
  • The reported scores show no consistent scaling trend within the Qwen family (3B, 7B, 72B), indicating that training larger versions of current recipes will not necessarily fix when-to-call decisions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A held-out transfer test the paper does not run would separate a general abstention skill from memorized phrasings: train RPO on When2Call, then evaluate on same-domain mismatches built from tool sets never seen in training; if the gains vanish, part of the measured headroom is surface-level.
  • The four-way choice format transplants naturally to neighboring abstention problems — retrieval-augmented systems deciding whether to answer from retrieved context, or agents deciding whether to escalate to a human — wherever 'no answer' is a legitimate action rather than a failure.
  • The confusion-matrix patterns suggest a deployment recipe the paper leaves implicit: before shipping a tool-calling model against a fixed API set, probe it with same-domain mismatches like the student-records-versus-grades case, and tune the RPO rejection pairs toward the specific error signature observed.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces When2Call, a multiple-choice benchmark for evaluating tool-calling decision-making: whether a model should call a tool, ask a follow-up question, admit it cannot answer, or answer directly. The benchmark is constructed by synthetically modifying BFCL and APIGen questions using Mixtral 8x22B, and it is evaluated via log-probability multiple-choice scoring plus an LLM-as-judge alternative. The authors evaluate a range of community models (Llama, Qwen, xLAM) and GPT-4 family models, finding that all models perform far from ceiling on When2Call despite reasonable accuracy on BFCL. They also construct a training set and compare supervised fine-tuning (SFT) with reward-aware preference optimization (RPO) on Mistral-NeMo-Minitron 4B and 8B models, reporting that RPO yields larger When2Call gains than SFT while largely preserving BFCL accuracy and improving BFCL Irrelevance.

Significance. If the measurements are sound, When2Call addresses a real gap: existing tool-calling benchmarks mainly check whether the correct tool is called with the correct parameters, not whether calling a tool is the right decision at all. The consistently low F1 scores of community models in Table 3, together with the qualitative error patterns in the confusion matrices, make a credible case that 'when to call' is a distinct and under-trained capability. The training comparison is also practically valuable: RPO provides a concrete recipe for improving when-to-not-call behavior without sacrificing BFCL accuracy, and the public release of the benchmark, training data, and evaluation scripts supports reproducibility and follow-up work. The main qualification is that the benchmark's ground-truth labeling, and therefore the quantitative claims that depend on it, have unresolved quality issues.

major comments (3)
  1. [Section 2.1 and Appendix A]
  2. [Section 3.4 and Table 3]
  3. [Sections 3.2, 4.1, and Table 5]
minor comments (5)
  1. [Appendix A, Table 10]
  2. [Limitations and Appendix A]
  3. [Figure 2]
  4. [Table 3 caption]
  5. [Abstract and Section 1]

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: When2Call is an external measurement, and the central claims are supported by held-out evaluation and BFCL transfer checks.

full rationale

This paper does not contain a derivation chain in which a prediction reduces to its inputs by construction. The benchmark is an independently constructed measurement: test items are generated from BFCL v2 Live, training items from APIGen, and community models are evaluated with no When2Call training, so the headline result that current tool-calling LMs perform far from ceiling is not forced by the paper's construction. The RPO and SFT comparisons are controlled training experiments evaluated on the When2Call test split and on BFCL, including the Irrelevance category, which provides an external transfer check rather than a self-referential target. The paper's self-citations (to the NVIDIA Nemotron technical report for RPO and to the Minitron work for base models) supply methods and base checkpoints, but neither is invoked as a uniqueness theorem or as evidence that When2Call measures what it claims; they are not load-bearing in a way that makes the evaluation circular. The quality-audit finding in Appendix A, where 18% of sampled 'unable to answer' questions are flagged and some direct answers may be 'not entirely incorrect,' is a validity and labeling-quality concern about the benchmark's central assumption, not a circularity in the paper's derivations. The Limitations section honestly acknowledges the simplifying assumption that a direct answer is always wrong. Accordingly, no circular step meets the evidentiary bar of quoting a specific reduction, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No invented entities. The central claims rest on three classes of assumptions: (1) dataset labeling assumptions (all questions require tools; four answer types are exhaustive), (2) synthetic generation reliability via Mixtral 8x22B and judge reliability via GPT-4-Turbo, and (3) hand-chosen training-distribution hyperparameters. These are disclosed in the paper but are load-bearing for the reported gains.

free parameters (4)
  • SFT data blend ratio = 2:1 (tool-calling to non-tool-calling)
    Chosen in experiments as best overall; influences all When2Call-SFT results (Section 3.3.1).
  • RPO preference dataset composition = 1:1 combination of tool-call and non-tool-call preference subsets; preference training variant uses 4,500 tool-call /…
    Set to avoid over-conservativeness while providing negative signals; directly shapes RPO gains in Table 3 (Sections 2.3 and 3.3.2).
  • KL penalty for RPO = 0.05
    Reported as best on tool-calling benchmarks (Section 3.3.2); low penalty is central to the RPO-vs-SFT comparison.
  • Learning rates = 5e-6 and 4e-6 for SFT; 9e-7 and 7e-7 for RPO (4B and 8B)
    Standard hyperparameters; chosen for training stability, not a conceptual contribution.
assumptions (4)
  • domain assumption All When2Call questions require tool access to answer, so the direct text answer (a) is always a hallucination.
    Stated in Section 2.1 and Limitations; load-bearing for labeling 'direct answer' as incorrect.
  • domain assumption The four behavior categories (direct answer, tool call, follow-up, unable) are exhaustive and mutually exclusive.
    Section 2.1 structures the multiple-choice evaluation; if a model produces a mixture (e.g., tool call plus follow-up) the evaluation may misclassify.
  • domain assumption Mixtral 8x22B's classification and generation reliably produce valid benchmark questions and answers.
    The whole pipeline in Section 2.2 depends on Mixtral 8x22B; quality is manually checked on 10% with 82% question quality and 94% answer quality (Appendix A).
  • domain assumption GPT-4-Turbo as LLM-as-judge accurately classifies free-form model outputs into the four categories.
    Section 4 relies on judge classification; no validation of judge agreement with human labels is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When2Call: When (not) to Call Tools." pith.science (2026). https://pith.science/paper/BCBKPMDX

@misc{pith2026250418851,
  author       = {Pith},
  title        = {Pith review of: When2Call: When (not) to Call Tools},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCBKPMDX}},
  note         = {Machine review of arXiv:2504.18851}
}
read the original abstract

Leveraging external tools is a key feature for modern Language Models (LMs) to expand their capabilities and integrate them into existing systems. However, existing benchmarks primarily focus on the accuracy of tool calling -- whether the correct tool is called with the correct parameters -- and less on evaluating when LMs should (not) call tools. We develop a new benchmark, When2Call, which evaluates tool-calling decision-making: when to generate a tool call, when to ask follow-up questions and when to admit the question can't be answered with the tools provided. We find that state-of-the-art tool-calling LMs show significant room for improvement on When2Call, indicating the importance of this benchmark. We also develop a training set for When2Call and leverage the multiple-choice nature of the benchmark to develop a preference optimization training regime, which shows considerably more improvement than traditional fine-tuning. We release the benchmark and training data as well as evaluation scripts at https://github.com/NVIDIA/When2Call.

Figures

Figures reproduced from arXiv: 2504.18851 by the authors.

Figure 1
Figure 1. Example of the type of question in When2Call. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. When2Call measures more complex capabili [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 12 canonical work pages

  1. [1]

    Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G. P. Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachin Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Meza Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David C...

    arXiv doi:10.48550/arxiv.2407.00121 2024

  2. [2]

    Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Vernon Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis Lastras. 2024. https://doi.org/10.18653/v1/2024.acl-long.694 API - BLEND : A comprehensive corpora for training and benchmarking API LLM s . In Proceedings of the 62nd Annual Meeting of...

    doi:10.18653/v1/2024.acl-long.694 2024

  3. [3]

    Matthew Carrigan. 2024. https://huggingface.co/blog/unified-tool-use Tool Use , Unified

    work page 2024

  4. [4]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  5. [5]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. https...

    doi:10.5281/zenodo.12608602 2024

  6. [6]

    Glaive AI . 2024. https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2 glaiveai/glaive-function-calling-v2 · Datasets at Hugging Face

    work page 2024

  7. [7]

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. 2024. https://doi.org/10.48550/arXiv.2403.07714 StableToolBench : Towards Stable Large - Scale Benchmarking on Tool Learning of Large Language Models . arXiv preprint

    arXiv doi:10.48550/arxiv.2403.07714 2024

  8. [8]

    Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, and Megan Ung. 2024. https://arxiv.org/abs/2406.19470 Changing answer order can decrease mmlu accuracy . Preprint, arXiv:2406.19470

    arXiv 2024

Show all 41 references
  1. [9]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring Massive Multitask Language Understanding

    2020

  2. [10]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024. https://doi.org/10.48550/arXiv.2310.03128 MetaTool Benchmark for Large Language Models : Deciding Whether to Use Tools and Which to Use...

    2024 doi arXiv

  3. [11]

    interstellarninja and Teknium . 2024. https://huggingface.co/NousResearch/hermes-function-calling-v1 Hermes-function-calling-dataset-v1

    2024

  4. [12]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.187 API -bank: A comprehensive benchmark for tool-augmented LLM s . In Proceedings of the 2023 Conference on Empiric...

    2023 doi

  5. [13]

    Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024. https://doi.org/10.48550/arXiv.24...

    2024 doi arXiv

  6. [14]

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2024. https://doi.org/10.48550/arXiv.2408.04682 ToolSandbox : A Stateful , Conversational , Interactive Evaluation Benchmark f...

    2024 doi arXiv

  7. [15]

    Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Tianjun Zhang, and Shishir G. Patil. 2024. https://gorilla.cs.berkeley.edu/blogs/12_bfcl_v2_live.html BFCL V2 • Live Dataset

    2024

  8. [16]

    Meta. 2024. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Llama 3.2

    2024

  9. [17]

    Mistral AI Team . 2024. https://mistral.ai/news/mixtral-8x22b/ Cheaper, Better , Faster , Stronger : Mixtral 8x22B

    2024

  10. [18]

    Nexusflow. 2023. https://huggingface.co/spaces/Nexusflow/Nexus_Function_Calling_Leaderboard Nexus Function Calling Leaderboard

    2023

  11. [19]

    Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander F...

    2024 arXiv

  12. [20]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. https://doi.org/10.48550/arXiv.2305.15334 Gorilla: Large Language Model Connected with Massive APIs . arXiv preprint

    2023 doi arXiv

  13. [21]

    Pouya Pezeshkpour and Estevam Hruschka. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.130 Large language models sensitivity to the order of options in multiple-choice questions . In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006--2017...

    2024 doi

  14. [22]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. https://doi.org/10.48550/arXiv.2307.16789 ...

    2023 doi arXiv

  15. [23]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. https://doi.org/10.48550/arXiv.2405.17935 Tool Learning with Large Language Models : A Survey . arXiv preprint

    2024 doi arXiv

  16. [24]

    Qwen Team . 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  17. [25]

    Michael C Rodriguez. 2005. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational measurement: issues and practice, 24(2):3--13

    2005

  18. [26]

    Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, and Oleksii Kuchaiev. 2024. https://arxiv.org/abs/2405.01481 NeMo-Aligner : Scalable toolkit for efficie...

    2024 arXiv

  19. [27]

    Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, Shizhe Diao, Chenhan Yu, Wei-Chun Chen, Hayley Ross, Oluwatobi Olabiyi, Ashwath Aithal, Oleksii Kuchaiev, Danie...

    2024 arXiv

  20. [28]

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. https://doi.org/10.48550/arXiv.2306.05301 ToolAlpaca : Generalized Tool Learning for Language Models with 3000 Simulated Cases . arXiv preprint

    2023 doi arXiv

  21. [29]

    Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. 2024. https://doi.org/10.48550/arXiv.2407.18418 The Art of Refusal : A Survey of Abstention in Large Language Models . arXiv preprint

    2024 doi arXiv

  22. [30]

    Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. https://doi.org/10.48550/arXiv.2305.16504 On the Tool Manipulation Capability of Open -source Large Language Models . arXiv preprint

    2023 doi arXiv

  23. [31]

    Gonzalez, Tianjun Zhang, and Shishir G

    Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Ion Stoica, Joseph E. Gonzalez, Tianjun Zhang, and Shishir G. Patil. 2024. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html Berkeley Function Calling Leaderboard

    2024

  24. [32]

    Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/e393677793767624f2821cec8bdd02f1-Abstract-Conference.html GPT4Tools : Teaching Large Language Model to Use Tools via Self -instruction ...

    2023

  25. [33]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. https://doi.org/10.48550/arXiv.2406.12045 \ tau\ -bench: A Benchmark for Tool - Agent - User Interaction in Real - World Domains . arXiv preprint

    2024 doi arXiv

  26. [34]

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. https://doi.org/10.48550/arXiv.2401.00741 ToolEyes : Fine - Grained Evaluation for Tool Learning Capabilities of Large Language Mode...

    2024 doi arXiv

  27. [35]

    Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Silvio S...

    2024 arXiv

  28. [36]

    Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, and Hayato Yamana. 2024 b . https://doi.org/10.48550/arXiv.2406.20015 ToolBeHonest : A Multi -level Hallucination Diagnostic Benchm...

    2024 doi arXiv

  29. [37]

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024 a . https://openreview.net/forum?id=shr9PXz7T0 Large language models are not robust multiple choice selectors . In The Twelfth International Conference on Learning Representations

    2024

  30. [38]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2024 b . Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th Inter...

    2024

  31. [39]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/9cb2a7495900f8b602cb10159246a016-Abstract-Datasets_and_Benchmarks.html ToolQA : A Dataset for LLM Question Answering with External Tools . Advances i...

    2023

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.