The reviewed record of science sign in
Pith

arxiv: 2606.12451 · v1 · pith:W7ZRYPQF · submitted 2026-06-04 · cs.AI · cs.IR· cs.LG

ToolSense: A Diagnostic Framework for Auditing Parametric Tool Knowledge in LLMs

Ashutosh Hathidara , Sai Shruthi Sistla , Sebastian Schreiber , Sahil Bansal This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-28 00:54 UTCgrok-4.3pith:W7ZRYPQFrecord.jsonopen to challenge →

classification cs.AI cs.IRcs.LG
keywords tool retrievalparametric modelsLLM agentsbenchmark generationknowledge dissociationrealistic queriesfactual probingToolBench
0
0 comments X

The pith

Parametric LLMs trained for tool retrieval collapse on realistic ambiguous queries and score near-random on factual probes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents ToolSense as a diagnostic framework that ingests any tool catalog and automatically produces three new benchmarks: a Realistic Retrieval Benchmark with queries at three ambiguity levels, plus MCQ and QA probes for factual tool knowledge. When applied to a large tool set, five different parametric training setups that had looked strong on standard benchmarks show drops of 50-64 percentage points on the new realistic queries, falling below embedding-model performance. Several models also perform near chance on the factual probes even when retrieval looks solid. A sympathetic reader would care because this points to a gap between apparent retrieval skill and actual tool understanding in agent systems that must handle imperfect user requests.

Core claim

Applying ToolSense to ToolBench reveals a knowledge-retrieval dissociation: parametric model configurations that perform well on fully-specified ToolBench benchmarks with constrained decoding drop sharply on RRB queries at three ambiguity tiers and score near-random on the generated factual probes, indicating that strong retrieval does not imply genuine tool knowledge.

What carries the argument

ToolSense, the LLM-powered framework that takes a tool catalog as input and generates the Realistic Retrieval Benchmark at three ambiguity tiers together with MCQ and QA probing benchmarks.

If this is right

  • Standard ToolBench-style benchmarks with verbose queries and constrained decoding overestimate the reliability of parametric tool retrieval.
  • Some parametric training configurations produce models whose retrieval succeeds without corresponding factual tool knowledge.
  • Embedding-based retrieval can outperform certain parametric setups once queries become realistic rather than fully specified.
  • Any new parametric training run should be audited with ambiguity-tiered retrieval tests and factual probes before deployment.
  • The same dissociation pattern can be checked on other tool catalogs by running the generated benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Agent builders may need to combine parametric retrieval with embedding methods or add runtime fact-checking to handle ambiguous inputs.
  • Training objectives could be extended to include explicit factual consistency losses so retrieval and knowledge stay aligned.
  • The three-tier ambiguity structure offers a way to measure how much specification a model actually requires before its performance falls apart.
  • Repeated application of the framework across catalogs could reveal whether the dissociation is architecture-specific or training-stage-specific.

Load-bearing premise

The LLM-generated benchmarks at different ambiguity tiers and the probing sets measure genuine tool understanding and factual knowledge without systematic artifacts that affect parametric models differently from embedding models.

What would settle it

Human-authored realistic queries at matching ambiguity levels and human-authored factual probes that show the same performance gaps between parametric configurations and embedding baselines would support the dissociation; the absence of those gaps would undermine it.

Figures

Figures reproduced from arXiv: 2606.12451 by Ashutosh Hathidara, Sahil Bansal, Sai Shruthi Sistla, Sebastian Schreiber.

Figure 1
Figure 1. Figure 1: The ToolSense diagnostic framework generates [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Human annotation study (N = 100 items × 3 annotators). RRB agreement decreases gracefully with difficulty while majority-vote accuracy remains high (85–94%). retrieval SFT, Stage 1 memorization alone already reaches 38–47% on G1 and 27–41% on RRB for flat configurations, denoting how well the model has absorbed tool semantics during memorization. Stage 2 retrieval training then lifts G1 sharply into a 90–9… view at source ↗
Figure 3
Figure 3. Figure 3: IS@50 across Stage 2 training steps for Gemma3-4B. Shaded bands are 95% bootstrap CIs. Full results across all splits & models are in Appendix G. configurations achieve RRB IS in the range 0.75– 0.85, while hierarchical configurations fall to 0.33– 0.79, a gap of Cohen’s d = 1.45 (Cohen, 1988), indicating the two groups occupy fundamentally different regions of trie-dependency. The gap is larger on RRB tha… view at source ↗
Figure 5
Figure 5. Figure 5: shows the gap persists across all beam widths: TG-H plateaus below 9% Rf@k regardless of beam budget, while flat configs scale to 37%. 1 5 10 20 30 50 Beam width (k) 0 10 20 30 40 50 RRB Recall@k (%) Constrained 1 5 10 20 30 50 Beam width (k) Free-form TG TG-3FM TG-3FM (LoRA) TG-H TG-5FM [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Correlation between Stage 1 MCQ accuracy [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 7
Figure 7. Figure 7: Relative L2 drift (drel) of virtual vs. randomly sampled vocabulary tokens across Stage 1→Stage 2 for Gemma3-4B configurations. (|∆cosim| < 0.002), suggesting tokens shift as coherent clusters rather than diverging to en￾code distinct knowledge content. Hierarchical to￾kens enter Stage 2 near-orthogonal to each other (cosimS1≈0.038 vs. 0.37 for flat tokens) and re￾quire 7–24× less repositioning, yet still … view at source ↗
Figure 8
Figure 8. Figure 8: Side-by-side comparison of ToolBench standard evaluation split queries (G1/G2/G3) and RRB queries for [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: IS@50 over Stage 2 training for Gemma3-4B across all four evaluation splits. Shaded bands are 95% [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: IS@50 over Stage 2 training for Qwen3.5-4B across all four evaluation splits. Shaded bands are 95% [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: desc→tok prompt (all configurations). You will be provided with a tool representa￾tion tokens. Your task is to describe what this tool does in plain language. Please don’t include any additional text in your response other than the description. The tool represen￾tation is provided below: {tool_token} [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: tok→desc prompt (TG-3FM, TG-5FM). configurations use it consistently across Stage 2 training and inference. You will be provided with the user query. Your task is to predict the tool that can best fulfill the user’s request. You only need to predict the tool token. Please don’t include any additional text in your response. The user query is provided below: {query} [PITH_FULL_IMAGE:figures/full_fig_p018_12.png] view at source ↗
Figure 16
Figure 16. Figure 16: Stage 2 retrieval training and inference [PITH_FULL_IMAGE:figures/full_fig_p018_16.png] view at source ↗
Figure 13
Figure 13. Figure 13: desc→api_tok prompt (TG-5FM only). You will be provided with the information about the tool in the JSON format along with its API. Your task is to identify the Endpoint this tool belongs to within that API and pro￾duce its corresponding Endpoint token. Only respond with the Endpoint token and noth￾ing else. The tool information and API are provided below: Tool Information: {tool_info} API: {api_token} [P… view at source ↗
Figure 15
Figure 15. Figure 15: MCTS discriminative prompt (TG-3FM, TG￾5FM). You are validating a yes/no Q&A entry for an AI tool probing benchmark. Tool description: {description} Generated entry: Question : {question} Answer : {answer} Check ALL of the following: 1. The question is specific to THIS tool — not generi￾cally answerable for any API. 2. The answer (“{answer}”) is directly and unambigu￾ously supported by the description. 3.… view at source ↗
Figure 18
Figure 18. Figure 18: QA probing benchmark: judge prompt (tem [PITH_FULL_IMAGE:figures/full_fig_p019_18.png] view at source ↗
Figure 17
Figure 17. Figure 17: QA probing benchmark: generation prompt. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png] view at source ↗
Figure 19
Figure 19. Figure 19: MCQ probing benchmark: generation prompt. You are validating a multiple-choice question entry for an AI tool probing benchmark. Tool description: {description} Generated entry: Question : {question} Correct answer: {correct_answer} Wrong answers : {wrong_answers} Check ALL of the following: 1. The question is specific to THIS tool — not generi￾cally answerable for any API. 2. The correct answer is directl… view at source ↗
Figure 20
Figure 20. Figure 20: MCQ probing benchmark: judge prompt (temperature = 0.0). You are a data generation expert creating an evaluation benchmark for a tool retrieval model. Your task is to generate {{n_samples}} evaluation sam￾ples, each consisting of a concise enterprise user query that points to EXACTLY ONE specific tool. [START OF TASK DESCRIPTION] The goal is to simulate how real enterprise users phrase requests — NOT how … view at source ↗
read the original abstract

Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck. As embedding-based retrieval approaches rely on compact encoders that may under-capture specialized tool semantics, parametric tool retrieval addresses this by encoding each tool as a virtual token appended to the LLM vocabulary, fine-tuned in two stages (memorization then retrieval SFT) to use the LLM as a retriever, achieving strong performance on standard ToolBench retrieval benchmarks. Yet these benchmarks use verbose, fully-specified queries, and their evaluation applies constrained decoding that restricts outputs to valid token paths, neither reveals whether the model actually understands its tools. We introduce \textbf{ToolSense}, an open-source LLM-powered diagnostic framework that takes any tool catalog as input and automatically generates three benchmarks: a Realistic Retrieval Benchmark (RRB) with queries at three ambiguity tiers, an MCQ probing benchmark, and a QA probing benchmark. Applying ToolSense to ToolBench (~47k tools) and evaluating five parametric model training configurations reveals a knowledge-retrieval dissociation: on RRB queries, several configurations collapse by ~50-64 percentage points compared to fully-specified ToolBench benchmarks, falling below the embedding-model baseline. Additionally, despite strong retrieval performance, some models score near-random on factual probes, suggesting a knowledge-retrieval dissociation. We open-source the ToolSense framework and the ToolBench diagnostic benchmarks at https://github.com/SAP/toolsense.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces ToolSense, an LLM-powered framework that ingests a tool catalog and automatically generates three diagnostic benchmarks: a Realistic Retrieval Benchmark (RRB) at three ambiguity tiers, an MCQ probing benchmark, and a QA probing benchmark. When applied to the ToolBench catalog of ~47k tools, evaluation of five parametric model training configurations shows large performance collapses (50-64 pp) on RRB relative to standard ToolBench retrieval numbers, with some models falling below an embedding baseline, plus near-random scores on factual probes despite strong retrieval, indicating a knowledge-retrieval dissociation.

Significance. If the dissociation is shown to be robust, the result would be significant for LLM agent research: it would demonstrate that strong performance on fully-specified, constrained-decoding benchmarks does not imply usable tool knowledge under realistic conditions, motivating changes in how parametric retrievers are trained and evaluated. The open-sourcing of both the framework and the generated ToolBench diagnostics is a clear strength that enables follow-up work.

major comments (2)
  1. [Abstract] Abstract (performance comparison paragraph): the claimed 50-64 pp collapse on RRB is obtained by comparing open-generation RRB queries directly against ToolBench numbers that were produced under constrained decoding restricting outputs to valid token paths. Because the decoding regime differs, the drop cannot be unambiguously attributed to query realism or lack of parametric tool knowledge; an ablation that re-evaluates the same models on ToolBench-style queries without constrained decoding is required to isolate the effect.
  2. [Abstract] Abstract (benchmark generation description): the LLM-powered construction of the three benchmarks (RRB at three ambiguity tiers, MCQ, and QA probes) is presented without reported validation (human review of query naturalness, inter-rater reliability, or controls for generation artifacts that might systematically disadvantage parametric models relative to the embedding baseline). This leaves open whether the observed dissociation reflects genuine tool understanding deficits or artifacts of the synthetic test construction.
minor comments (1)
  1. The abstract states that five parametric configurations were evaluated but does not list their training details or hyper-parameters; a short table or explicit enumeration in the main text would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address each major comment below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract (performance comparison paragraph): the claimed 50-64 pp collapse on RRB is obtained by comparing open-generation RRB queries directly against ToolBench numbers that were produced under constrained decoding restricting outputs to valid token paths. Because the decoding regime differs, the drop cannot be unambiguously attributed to query realism or lack of parametric tool knowledge; an ablation that re-evaluates the same models on ToolBench-style queries without constrained decoding is required to isolate the effect.

    Authors: We agree that the reported performance collapse mixes open-generation evaluation on RRB with constrained-decoding results on ToolBench, preventing unambiguous attribution to query realism. In the revised manuscript we will add an ablation that re-evaluates all five parametric configurations on the original ToolBench queries under open generation (no constrained decoding) and report the resulting scores alongside the existing numbers. revision: yes

  2. Referee: [Abstract] Abstract (benchmark generation description): the LLM-powered construction of the three benchmarks (RRB at three ambiguity tiers, MCQ, and QA probes) is presented without reported validation (human review of query naturalness, inter-rater reliability, or controls for generation artifacts that might systematically disadvantage parametric models relative to the embedding baseline). This leaves open whether the observed dissociation reflects genuine tool understanding deficits or artifacts of the synthetic test construction.

    Authors: We acknowledge that the manuscript does not report human validation of the generated benchmarks. In the revision we will add a validation subsection that describes human review of a stratified sample of RRB queries and probes, reports naturalness ratings, inter-annotator agreement, and explicit checks for generation artifacts that could favor or disfavor parametric models relative to the embedding baseline. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical benchmarks generated independently of model parameters

full rationale

The paper introduces ToolSense as an external framework that takes any tool catalog and generates RRB, MCQ, and QA benchmarks via LLM prompting; these are applied to ToolBench to produce raw performance numbers on parametric models versus an embedding baseline. No equations, fitted parameters, or self-citations are used to define the reported collapse or dissociation; the metrics are direct accuracies on separately generated test sets. The central claim rests on empirical comparison rather than any reduction to inputs by construction, self-definition, or load-bearing self-citation. Minor self-citation risk is absent from the provided text.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on the validity of automatically generated benchmarks that assume LLM generation captures genuine ambiguity and factual knowledge without circular dependence on the models being evaluated.

axioms (2)
  • domain assumption Embedding-based retrieval may under-capture specialized tool semantics
    Stated as motivation for parametric approaches in the abstract.
  • domain assumption LLM-powered generation produces unbiased and realistic diagnostic benchmarks
    Core premise of the ToolSense framework described in the abstract.

pith-pipeline@v0.9.1-grok · 5798 in / 1462 out tokens · 38879 ms · 2026-06-28T00:54:34.728702+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references

  1. [1]

    {tar- get_answer}

    The answer to your question must be exactly “{tar- get_answer}” based on the description

  2. [2]

    this tool

    Use “this tool” in the question — never include the actual tool name or service name. (The model will see only a token at inference time, so the name must not be a hint.)

  3. [3]

    Does this tool provide an API?

    The question must be specific to THIS tool. “Does this tool provide an API?” is too generic. Good examples for Yes: “Does this tool process image inputs?”, “Is this tool designed for financial data?” Good examples for No: “Does this tool support voice/audio input?”, “Does this tool return results in XML?” For No questions: ask about a plausible capability...

  4. [4]

    {format_instructions} Figure 17: QA probing benchmark: generation prompt

    Set skip=true if you cannot form a specific, unam- biguous question with the required answer. {format_instructions} Figure 17: QA probing benchmark: generation prompt. You will be provided with the description of a tool along with line-separated multiple choice options for the tool tokens. Your task is to select the correct tool token that corresponds to ...

  5. [6]

    {answer}

    The answer (“{answer}”) is directly and unambigu- ously supported by the description

  6. [7]

    this tool

    The question uses “this tool” as a placeholder — the actual tool name does not appear

  7. [8]

    Set accept=true only if ALL four checks pass

    The question tests a verifiable property (domain, capability, input/output type, format, etc.). Set accept=true only if ALL four checks pass. Other- wise set accept=false and state which check failed. {format_instructions} Figure 18: QA probing benchmark: judge prompt (tem- perature= 0.0). You are building a multiple-choice probing benchmark for AI tool r...

  8. [9]

    this tool

    Use “this tool” in the question — never include the actual tool name or service name. (The model will see only a virtual token at inference time, so the name must not be a hint.)

  9. [10]

    Does this tool provide an API?

    The question must be specific to THIS tool — not answerable for a generic API. Good topics: primary output type, domain/industry, key input, core capability, supported format. Bad question: “Does this tool provide an API?” (too generic)

  10. [11]

    Each answer option must be a short phrase (2–8 words), not a full sentence

  11. [12]

    The correct_answer must be directly supported by the description above

  12. [13]

    The three wrong answers must be plausible alterna- tives — the kind of answer a user might expect from a tool in the same domain — but clearly incorrect for THIS specific tool

  13. [14]

    All four options (correct + wrong) must be meaning- fully distinct from each other

  14. [15]

    {format_instructions} Figure 19: MCQ probing benchmark: generation prompt

    Set skip=true if you cannot form a specific, unam- biguous factual question from this description. {format_instructions} Figure 19: MCQ probing benchmark: generation prompt. You are validating a multiple-choice question entry for an AI tool probing benchmark. Tool description: {description} Generated entry: Question : {question} Correct answer: {correct_a...

  15. [16]

    The question is specific to THIS tool — not generi- cally answerable for any API

  16. [17]

    The correct answer is directly and unambiguously supported by the description

  17. [18]

    Each of the three wrong answers is plausible for a tool in the same domain but clearly incorrect for THIS tool

  18. [19]

    All four options are meaningfully distinct from each other

  19. [20]

    this tool

    The question uses “this tool” as a placeholder — the actual tool name does not appear. Set accept=true only if ALL five checks pass. Otherwise set accept=false and state which check failed. {format_instructions} Figure 20: MCQ probing benchmark: judge prompt (temperature= 0.0). You are a data generation expert creating an evaluation benchmark for a tool r...

  20. [21]

    A query — concise, business-language question pointing to the target tool

  21. [22]

    An answer — a list containing exactly ONE tool name (the target tool) [END OF TASK DESCRIPTION] [START OF GENERATION RULES]

  22. [23]

    Queries must be CONCISE — avoid verbose, over- specified phrasing

  23. [24]

    Use BUSINESS LANGUAGE — never include API method names, OData syntax, or system-specific tech- nical identifiers in the query

  24. [25]

    The query should naturally lead to the target tool without explicitly naming it

  25. [26]

    The query should NOT trivially match every other tool in the pool — it must be specific enough to distin- guish the target

  26. [27]

    Do NOT use OData query syntax ($filter, $select, $expand, etc.) in queries

  27. [28]

    Do NOT hallucinate tool names — only use names exactly as they appear in the tool list below

  28. [29]

    Show me all open purchase orders

    Every query must be distinct — vary phrasing and angle of attack GOOD query examples (concise, business language): - “Show me all open purchase orders” - “Which customers have overdue invoices?” - “I need to track employee time-off requests” - “List products that are low on stock” BAD query examples (too technical or verbose): - “Retrieve all PurchaseOrde...

  29. [30]

    Is the query CONCISE and written in BUSINESS LANGUAGE (not technical API language)?

  30. [31]

    Does the query sound like something a real enterprise user would actually ask?

  31. [32]

    For {{complexity}} tier: is the number of answer tools and the ambiguity level appropriate? - easy: query should be specific enough that one tool is the clear answer; the phrasing should not be technical - medium: query should be genuinely ambiguous be- tween 2–3 tools; ambiguity must arise from overlapping domains, not vague phrasing - hard: query should...

  32. [33]

    get data

    Are all answer tools plausibly relevant to the query? [END OF TASK DESCRIPTION] [START OF V ALIDATION RULES] A sample PASSES (validation_result: true) if ALL of the following hold: - The query is CONCISE (1–3 sentences, not verbose or over-specified) - The query uses BUSINESS LANGUAGE — no API method names, technical identifiers, OData operators ($filter,...

    2025