{"id":"e6203db1-f0b5-44b2-ab4a-549d1cd05b65","arxiv_id":"2504.18851","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"When2Call measures when language models should call tools versus ask questions or refuse, and shows that RPO training substantially improves this decision-making.","lead":"This paper introduces a benchmark, When2Call, that tests whether AI assistants know when to call external tools, when to ask follow-up questions, and when to admit they cannot answer. It finds that current tool-using models often call tools inappropriately or hallucinate, and shows that preference-optimization training markedly improves this judgment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix A's own audit contradicts the linchpin assumption that direct answers are always hallucinations: 18% of sampled 'unable to answer' questions are flagged, including 6% partially answerable without tools, so headline F1 and hallucination rates may be overstated.","rationale":"The paper is a careful, well-scoped benchmark contribution: it releases data and evaluation scripts, evaluates both multiple-choice and free-form judging, and reports BFCL transfer. I read the central claim as a measurement claim: When2Call isolates a distinct decision-making capability, and current tool-calling LMs are far from ceiling. That requires the gold labels to be unassailable, especially for the (d) category, because (a) is treated as a hallucination by construction. The paper's own Appendix A shows the labels are not unassailable: 18% of sampled (d) questions had issues, the most damaging being 6% partially answerable without a tool. This directly affects the headline metrics, and because the same labels are used as chosen/rejected pairs in RPO training, noisy preferences can be learned, not just measured imprecisely. I also noticed the Limitations text (92% question quality) contradicts Appendix A (82%), making even the paper's own quality estimate unclear. The pipeline-overlap and single-seed training concerns raised by the reader are real but secondary: they bear on the RPO-vs-SFT comparison rather than on the benchmark's validity as a measurement instrument. If the relabeling check shows the ranking and the large gap remain, the CONDITIONAL verdict should stand; if not, the benchmark's central claim would need to be weakened. Because the reader already conditioned on this concern, I do not move the verdict.","tokens_in":18308,"tokens_out":10489,"duration_ms":110593,"concrete_test":"Manually re-audit a stratified sample of 200 'unable to answer' test items (plus 50 each of tool-call and follow-up items) with three independent annotators using Table 6's checklist plus a 'would a direct answer be acceptable?' judgment. Compute inter-annotator agreement; then recompute Table 3's macro F1, accuracy, and hallucination rates on only the items where all annotators agree the gold label is unambiguous. If the 'far from ceiling' gap and model ranking persist within about 2 F1 points, the concern is bounded; if excluding ambiguous items shifts any headline number by more than about 5 points or changes the (d) recall pattern, the benchmark's central interpretation requires revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 makes the benchmark's central identification assumption: every question requires tool use, so choosing (a) direct text answer is always a hallucination. This assumption underwrites the answer-hallucination metric, the interpretation of low recall on (d) as unwillingness to admit inability, and the preference labels used for RPO in Section 3.3.2. The paper's own quality audit (Appendix A, Table 6) weakens it: among 50 sampled questions whose gold answer is (d), 9 (18%) are flagged, including 3 (6%) 'partially answerable without a tool' and 2 (4%) 'generic/vague terms (no specific values)'; the text concedes the pipeline can generate a vague direct answer that is 'not entirely incorrect.' Since (d) items are 1,295 of 3,652 test examples, even a 6-18% dispute rate puts well over a hundred items in a category where the gold label and the hallucination definition are contestable. Macro F1 and the tool/answer hallucination rates in Table 3 are therefore not clean measurements. The Limitations section's '92% for questions' also contradicts Appendix A's 82%, so the quality estimate is itself uncertain, and the 18% figure is based on only 50 samples. The same noisy labels feed RPO chosen/rejected pairs, so training may reinforce the same ambiguity rather than only mis-measuring it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces When2Call, a multiple-choice benchmark for evaluating tool-calling decision-making: whether a model should call a tool, ask a follow-up question, admit it cannot answer, or answer directly. The benchmark is constructed by synthetically modifying BFCL and APIGen questions using Mixtral 8x22B, and it is evaluated via log-probability multiple-choice scoring plus an LLM-as-judge alternative. The authors evaluate a range of community models (Llama, Qwen, xLAM) and GPT-4 family models, finding that all models perform far from ceiling on When2Call despite reasonable accuracy on BFCL. They also construct a training set and compare supervised fine-tuning (SFT) with reward-aware preference optimization (RPO) on Mistral-NeMo-Minitron 4B and 8B models, reporting that RPO yields larger When2Call gains than SFT while largely preserving BFCL accuracy and improving BFCL Irrelevance.","tokens_in":18602,"tokens_out":3741,"duration_ms":38326,"significance":"If the measurements are sound, When2Call addresses a real gap: existing tool-calling benchmarks mainly check whether the correct tool is called with the correct parameters, not whether calling a tool is the right decision at all. The consistently low F1 scores of community models in Table 3, together with the qualitative error patterns in the confusion matrices, make a credible case that 'when to call' is a distinct and under-trained capability. The training comparison is also practically valuable: RPO provides a concrete recipe for improving when-to-not-call behavior without sacrificing BFCL accuracy, and the public release of the benchmark, training data, and evaluation scripts supports reproducibility and follow-up work. The main qualification is that the benchmark's ground-truth labeling, and therefore the quantitative claims that depend on it, have unresolved quality issues.","major_comments":[{"comment":"","section":"Section 2.1 and Appendix A"},{"comment":"","section":"Section 3.4 and Table 3"},{"comment":"","section":"Sections 3.2, 4.1, and Table 5"}],"minor_comments":[{"comment":"","section":"Appendix A, Table 10"},{"comment":"","section":"Limitations and Appendix A"},{"comment":"","section":"Figure 2"},{"comment":"","section":"Table 3 caption"},{"comment":"","section":"Abstract and Section 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution that fits the scope of a top NLP venue, but the quality-control inconsistency between the Limitations section and Appendix A, plus the evaluation-protocol mismatch between the multiple-choice and LLM-as-judge settings, need to be resolved before the quantitative claims are trustworthy. The benchmark and training data release are valuable, but the paper should be more careful about distinguishing the benchmark's conceptual contribution from the noisy labels that currently back it. No concerns about novelty or citation practices beyond the standard expectation that the BFCL, APIGen, and ToolBeHonest related work be accurately positioned."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nThe one thing you should know: When2Call is the first benchmark I've seen that explicitly tests the four-way decision — direct answer, tool call, follow-up question, or 'unable to answer' — in a multiple-choice format, and it shows that current tool-calling LMs are far from ceiling. The follow-up and refusal categories are genuinely new relative to BFCL Irrelevance and ToolSandbox, which only check whether a tool call happens. The paper also does something smart: it uses the multiple-choice structure to build a preference dataset for RPO, and the RPO-trained 8B model improves on When2Call and BFCL Irrelevance while roughly holding BFCL AST. That's a real result, and the confusion matrices are a nice diagnostic.\n\nThe soft spots are proportionate. The largest gains are measured on a test set generated by the same pipeline as the training data, so in-domain improvement is partly a pipeline-overfitting effect. The transfer to BFCL mitigates this, but not fully. There are no error bars or multiple seeds; the RPO-vs-SFT comparison is single-run. And the stress-test note is right: Appendix A's own audit found 9/50 (18%) of sampled 'unable to answer' questions flagged, including 3 'partially answerable without a tool', and the text concedes the direct answer option can be 'not entirely incorrect'. That directly undercuts the Section 2.1 assumption that (a) is always a hallucination. The Limitations section says 92% question quality while Appendix A says 82% — an internal contradiction that should be fixed. Over 1,295 test items in category (d), even a 6% dispute rate is over 70 items where the gold label is contestable, so the hallucination rates in Table 3 are not clean.\n\nThat said, the paper is transparent about these issues in Appendix A, and the core contribution — measuring the decision to call vs. not call, and showing RPO helps — survives. The benchmark is useful for anyone building or evaluating tool-calling agents. I'd bring it to reading group and would cite it. It deserves peer review, but I'd push for a revision that addresses the quality discrepancy, adds a seed/error-bar analysis, and ideally validates on a small independently constructed set. The trained weights are released, which helps.\n\nRecommendation: send to review, with a request for the above revisions.","headline":"When2Call is a genuinely new four-way benchmark for tool-calling decisions, but its own appendix undercuts the central 'direct answer is always hallucination' assumption, so the headline numbers need a grain of salt.","tokens_in":19142,"tokens_out":3861,"would_cite":true,"duration_ms":33298,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Knowing when not to call a tool is a distinct, under-measured capability, this paper argues: its When2Call benchmark shows even GPT-4-class models far from ceiling, and preference optimization on the data nearly doubles open-model scores.","keywords":["tool calling","when-to-call decisions","abstention","hallucination","benchmark","preference optimization","multiple-choice evaluation","large language models"],"falsifier":"The central claim would be undercut if a current frontier tool-calling model scored near ceiling on the released When2Call evaluation without any When2Call-specific training; a reader can check directly by running the released evaluation script on any new model and inspecting its macro F1 and tool-hallucination rate. The claim that the RPO gains are a general skill rather than memorized phrasings can be tested by training on When2Call and evaluating on mismatches built from tool specifications never seen in training: if the improvement vanishes, part of the benchmark's headroom is a phrasing artifact.","tokens_in":18105,"feed_emoji":"📞","tokens_out":14279,"duration_ms":114484,"temperature":0.7,"pith_summary":"When2Call is a new benchmark for a capability that existing tool-calling benchmarks skip: not whether a language model calls the right tool with the right arguments, but whether it should call a tool at all. Each question is cast as a forced choice among four behaviors — answer directly, call the tool, ask a follow-up question, or admit it cannot answer — and because every question genuinely requires a tool, the direct answer is a hallucination by construction. The paper reports that open and closed tool-calling models, including GPT-4-class systems, perform far from ceiling on this choice, and that performance does not reliably improve with model size. It then shows that a preference-optimization training regime on the When2Call data improves the when-to-call decision far more than supervised fine-tuning, while keeping accuracy on the standard tool-calling leaderboard broadly intact. If the result holds, developers gain a way to measure and fix a failure mode — calling tools, or hallucinating answers, when the right move is to ask or abstain — that current evaluation pipelines largely miss.","feed_headline":"Tool-calling AI models can't tell when not to call","feed_subtitle":"Even GPT-4-class agents hallucinate tools when they should abstain; new benchmark lets developers measure and fix it.","key_machinery":"The central object is a four-way multiple-choice formulation of tool-calling as a behavior decision: direct text answer, tool call, follow-up question, or unable to answer. Every question is constructed to require tool access, which makes the four choices mutually exclusive and lets three distinct errors be read off a confusion matrix: choosing the direct answer is answer hallucination, choosing a tool call when no tool specification is provided is tool hallucination, and choosing a tool call when a required parameter is missing is parameter hallucination. Evaluation takes log-probabilities over the four options rather than parsing generated calls, and the same four-way structure doubles as the training signal for preference optimization by pairing the correct answer as the chosen response and an incorrect answer as the rejected one.","core_discovery":"The paper's central claim is that deciding whether to call a tool, ask for clarification, or refuse is a distinct tool-calling capability, separable from the call-accuracy skills measured by benchmarks like BFCL, and that When2Call is a benchmark that isolates it. It reports that across the Llama, Qwen, and xLAM families, macro F1 on this four-way decision sits between roughly 16 and 38, that GPT-4-class closed models reach only about 52 to 65 F1, and that scores do not reliably rise with model size. It further claims that the failure is trainable but not by naive instruction tuning: supervised fine-tuning on When2Call data improves the decision at the cost of over-conservatism and lower BFCL accuracy, while reward-aware preference optimization (RPO) on the same multiple-choice data raises When2Call macro F1 from about 32 to 52 on an 8B model, cuts tool hallucination from 19 percent to 1.2 percent, and lifts BFCL Irrelevance accuracy from 36.3 to 78.1 percent, showing the two pressures can be balanced.","pith_inferences":["A held-out transfer test the paper does not run would separate a general abstention skill from memorized phrasings: train RPO on When2Call, then evaluate on same-domain mismatches built from tool sets never seen in training; if the gains vanish, part of the measured headroom is surface-level.","The four-way choice format transplants naturally to neighboring abstention problems — retrieval-augmented systems deciding whether to answer from retrieved context, or agents deciding whether to escalate to a human — wherever 'no answer' is a legitimate action rather than a failure.","The confusion-matrix patterns suggest a deployment recipe the paper leaves implicit: before shipping a tool-calling model against a fixed API set, probe it with same-domain mismatches like the student-records-versus-grades case, and tune the RPO rejection pairs toward the specific error signature observed."],"forward_implications":["Because the direct answer is always a hallucination by construction, When2Call turns abstention quality into a number: answer, tool, and parameter hallucination rates can be read directly off the model's confusion matrix.","A high score on BFCL Irrelevance does not imply a high score on When2Call, so the benchmark measures a finer-grained and harder skill than existing when-not-to-call evaluations.","Naive negative-example training makes models over-conservative and hurts BFCL accuracy; RPO, with the correct answer as the chosen response and an incorrect answer as rejected, is the paper's demonstrated way to improve when-to-call decisions while preserving tool-call accuracy.","The reported scores show no consistent scaling trend within the Qwen family (3B, 7B, 72B), indicating that training larger versions of current recipes will not necessarily fix when-to-call decisions."],"supporting_citations":[{"why":"Defines BFCL, the standard tool-calling benchmark When2Call complements and measures itself against; its Live subset supplies base questions and its AST and Irrelevance categories anchor the comparisons.","marker":"Yan et al., 2024"},{"why":"Supplies the human-generated BFCL v2 Live questions and correct tool calls that When2Call inherits and rewrites into the other three behavior types.","marker":"Mao et al., 2024"},{"why":"Supplies APIGen, the source of the When2Call training set, converted into the four-answer format by the same pipeline.","marker":"Liu et al., 2024"},{"why":"The Mixtral 8x22B model used for all classification and synthetic generation of When2Call questions and answer options.","marker":"Mistral AI Team, 2024"},{"why":"Provides reward-aware preference optimization (RPO), the training method behind the paper's main improvement claim.","marker":"Nvidia et al., 2024"},{"why":"ToolSandbox, a prior benchmark with an insufficient-information category that When2Call goes beyond by evaluating what the model does instead of calling a tool.","marker":"Lu et al., 2024"},{"why":"ToolBeHonest, the closest prior measure of tool hallucination via solvability judgments, which When2Call makes explicit and quantifiable.","marker":"Zhang et al., 2024b"},{"why":"Defines the Mistral-NeMo-Minitron 4B and 8B base models that the paper's SFT and RPO experiments fine-tune and align.","marker":"Sreenivas et al., 2024"}],"fun_headline_variants":["Tool-calling AI fails abstention: new benchmark and fix","When2Call: teaching models when not to call tools","New benchmark measures AI's tool-call timing decisions","AI tool-calling needs 'when not to call' training","When not to call: AI tool-calling decision benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every When2Call question is truly unanswerable without a tool, making the direct answer always a hallucination; the paper's own quality audit found that 18 percent of sampled 'unable to answer' questions had issues, including questions partially answerable without a tool, which makes the ground-truth label for those items ambiguous.","fun_headline_variants_meta":{"raw":{"variants":["Tool-calling AI fails abstention: new benchmark and fix","When2Call: teaching models when not to call tools","New benchmark measures AI's tool-call timing decisions","AI tool-calling needs 'when not to call' training","When not to call: AI tool-calling decision benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1583,"prompt_tokens":950,"completion_tokens":633,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":551}},"tokens_in":566,"tokens_out":633,"duration_ms":6254,"temperature":1.0,"reasoning_tokens":551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:07:23.445152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The central claim would be undercut if a current frontier tool-calling model scored near ceiling on the released When2Call evaluation without any When2Call-specific training; a reader can check directly by running the released evaluation script on any new model and inspecting its macro F1 and tool-hallucination rate. The claim that the RPO gains are a general skill rather than memorized phrasings can be tested by training on When2Call and evaluating on mismatches built from tool specifications never seen in training: if the improvement vanishes, part of the benchmark's headroom is a phrasing artifact.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Mixtral 8x22B model used for all classification and synthetic generation of When2Call questions and answer options."}],"review_version":1}