arxiv: 2605.12521 · v1 · submitted 2026-04-03 · 💻 cs.CL · cs.AI

Recognition: unknown

ToolWeave: Structured Synthesis of Complex Multi-Turn Tool-Calling Dialogues

Dinesh Khandelwal , Gnana Prakash Punnavajhala , GPS Bhargav , Gaurav Pandey , Sachin Joshi , Hima Karanam , Dinesh Raghu

Authors on Pith no claims yet

Pith reviewed 2026-05-14 21:16 UTC · model grok-4.3

classification 💻 cs.CL cs.AI

keywords multi-turn tool callingsynthetic dialogue generationLLM fine-tuningtool dependenciesparameter provenanceautonomous agentsdialogue synthesisbenchmark evaluation

0 comments

The pith

ToolWeave synthesizes multi-turn tool-calling dialogues with tracked parameter sources and goal-aligned workflows, raising fine-tuned model scores on benchmarks.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that prior synthetic data pipelines for training LLMs on multi-turn tool calling create unrealistic dialogues by chaining superficially related tools and inventing parameters without user input or prior results. ToolWeave counters this through a structured process that first builds tools with explicit dependencies, then filters complete workflows for alignment with plausible user goals, and finally applies a planning stage that records the exact origin of every parameter. The resulting dialogues show a higher share of genuine multi-step sequences and fewer invented details. Models fine-tuned on this data then record higher accuracy on existing evaluation sets than models trained on earlier collections. A reader would care because improved synthetic data could let LLMs develop reliable autonomous tool use without access to large volumes of private user logs.

Core claim

ToolWeave is a structured framework for synthesizing realistic multi-turn tool-calling dialogues by constructing tools with built-in dependencies, filtering workflows according to alignment with user goals, and using a fine-grained planning stage that explicitly tracks parameter provenance, which produces dialogues containing 45 percent multi-step tool interactions and fewer hallucinations in parameters and tool names than prior methods.

What carries the argument

The ToolWeave synthesis pipeline that combines dependency-built tools, goal-alignment filtering of workflows, and provenance-aware planning to control argument generation.

If this is right

LLMs fine-tuned on ToolWeave data achieve higher accuracy on multi-turn tool-calling benchmarks such as BFCL-V3.
The generated dialogues contain 45 percent multi-step tool interactions.
Hallucinations in tool parameters and names are reduced relative to one-shot generation methods.
The same fine-tuned models outperform those trained on earlier state-of-the-art synthetic datasets across three public benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The provenance-tracking step could be reused to generate explanations of why an agent chose each tool call.
Similar dependency and provenance machinery might improve synthetic data quality for other structured generation tasks such as multi-step planning or code repair dialogues.
If the realism gains hold, organizations could train capable tool-using agents without collecting and storing large volumes of real user interaction traces.
Testing the same synthesis pipeline on tool sets drawn from additional domains would show whether the performance lift generalizes beyond the original benchmarks.

Load-bearing premise

That the synthetic dialogues match the realism and distribution of actual user-tool interactions closely enough to produce transferable capability gains rather than benchmark-specific patterns.

What would settle it

A new evaluation set drawn from genuine multi-turn user-tool interaction logs, collected independently of the three public benchmarks, on which ToolWeave-trained models show no accuracy advantage over models trained on earlier synthetic collections.

Figures

Figures reproduced from arXiv: 2605.12521 by Dinesh Khandelwal, Dinesh Raghu, Gaurav Pandey, Gnana Prakash Punnavajhala, GPS Bhargav, Hima Karanam, Sachin Joshi.

**Figure 2.** Figure 2: Input schema comparison for generate_maintenance_schedule(). One-shot generation yields a flat parameter list, while ToolWeave produces a richer schema with nested objects. Prior tool graph construction typically relies on either real-world APIs from marketplaces like RapidAPI (RapidAPI, 2025) or one-shot synthetic tool generation via LLMs. The RapidAPI-based approaches provide realistic tools but have two… view at source ↗

**Figure 3.** Figure 3: E-commerce tool graph demonstrating linear, fan-in-fan-out (dashed), and conditional (green) dependency [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

read the original abstract

Multi-turn tool calling is essential for LLMs to function as autonomous agents, yet synthesizing the training data required for these capabilities remains a fundamental challenge. Existing synthetic data generation pipelines often produce unrealistic dialogues for two reasons: they chain tools that are only superficially compatible rather than aligned with meaningful user tasks, and they generate dialogues in one shot, which often introduces arguments that were neither provided by the user nor produced by prior tool calls. These issues also lead to a severe underrepresentation of multi-step tool interactions. We introduce ToolWeave, a structured framework for synthesizing realistic multi-turn tool-calling dialogues. ToolWeave support realistic multi-step workflows (or tool sequences) by constructing tools with built-in dependencies and filters the workflows based on alignment with user goals. It reduces parameter hallucination by using a fine-grained planning stage that explicitly tracks parameter provenance. As a result, ToolWeave-generated synthetic dialogues contain more multi-step tool interactions (45%) and fewer hallucinations in parameters and tool names. Consequently, LLMs fine-tuned on ToolWeave consistently outperform those fine-tuned on prior datasets across three public benchmarks. Notably, Llama-3.1-70B fine-tuned on ToolWeave achieves 39.75% on BFCL-V3 multi-turn, compared to 23.50% when fine-tuned on SOTA ToolFlow data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ToolWeave's pipeline for building dependent multi-turn tool dialogues produces data that lifts fine-tuned model scores on benchmarks, but the gains could stem from reduced mismatch with test distributions rather than the claimed structural fixes.

read the letter

ToolWeave gives a synthesis pipeline that generates more multi-turn tool dialogues with fewer parameter errors, resulting in stronger fine-tuned models on public benchmarks, though the source of the improvement is not fully pinned down. The new elements are explicit construction of tool dependencies, goal-based workflow filtering, and a planning stage that tracks parameter provenance to avoid hallucinations. These steps address the common failures of one-shot generation, where tools get chained without real task alignment and arguments appear from nowhere. The output data shows 45% multi-step interactions and leads to clear lifts, such as Llama-3.1-70B reaching 39.75% on BFCL-V3 multi-turn versus 23.50% when trained on ToolFlow data. The paper does a solid job describing these mechanisms and reporting the benchmark numbers across three datasets. The focus on provenance tracking is a direct, practical response to a known problem in tool-calling data. The soft spot is the missing experimental detail. The abstract does not report data volumes, whether the training tools overlap with the evaluation sets, or any controls for distribution shift. Without those, the 16-point gap could reflect incidental alignment with the test distributions instead of the dependency and provenance features. No statistical tests or out-of-distribution checks are mentioned either. This work is aimed at researchers building training data for tool-using agents. Anyone scaling multi-turn capabilities would want to examine the pipeline. I would send it to peer review. The method is concrete enough and the reported gains large enough to justify referee time, even if the authors need to add controls and volume comparisons to strengthen the central claim.

Referee Report

2 major / 1 minor

Summary. The paper introduces ToolWeave, a structured synthesis framework for multi-turn tool-calling dialogues. It constructs dependency-aware tool workflows, applies goal-alignment filters, and uses fine-grained planning with explicit parameter provenance tracking to reduce hallucinations and increase multi-step interactions (reported at 45%). The central empirical claim is that LLMs fine-tuned on ToolWeave data outperform those fine-tuned on prior datasets (including SOTA ToolFlow) across three public benchmarks, with a specific result of Llama-3.1-70B reaching 39.75% on BFCL-V3 multi-turn versus 23.50% on ToolFlow data.

Significance. If the reported gains are shown to stem from the claimed structural properties rather than incidental alignment with benchmark distributions, the work would provide a concrete advance in synthetic data generation for agentic LLMs, addressing under-representation of realistic multi-turn tool use and enabling more reliable fine-tuning for autonomous tool-calling systems.

major comments (2)

[Experimental results] The abstract and evaluation results report concrete benchmark improvements (e.g., 39.75% vs. 23.50% on BFCL-V3 multi-turn) but supply no details on whether the training tool sets overlap with those in the evaluation benchmarks, whether total data volumes or dialogue lengths were matched across compared datasets, or any statistical significance tests. Without these controls, it is impossible to attribute the gap specifically to ToolWeave's dependency tracking and provenance mechanisms rather than reduced distribution shift.
[Data synthesis pipeline] The claim that ToolWeave produces 45% multi-step interactions and fewer parameter hallucinations is presented as a direct outcome of the workflow construction and planning stages, yet the manuscript provides no quantitative validation (e.g., inter-annotator agreement, comparison to real user logs, or out-of-distribution test sets) that these synthetic dialogues are realistic proxies rather than artifacts tuned to the evaluation distributions.

minor comments (1)

[Abstract] The abstract refers to 'three public benchmarks' but names only BFCL-V3 explicitly; the other two should be listed for clarity.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below, providing clarifications and committing to revisions that strengthen the empirical claims without altering the core contributions.

read point-by-point responses

Referee: The abstract and evaluation results report concrete benchmark improvements (e.g., 39.75% vs. 23.50% on BFCL-V3 multi-turn) but supply no details on whether the training tool sets overlap with those in the evaluation benchmarks, whether total data volumes or dialogue lengths were matched across compared datasets, or any statistical significance tests. Without these controls, it is impossible to attribute the gap specifically to ToolWeave's dependency tracking and provenance mechanisms rather than reduced distribution shift.

Authors: We agree these controls are essential for rigorous attribution. In the revised manuscript we will add an explicit experimental controls subsection documenting: (1) tool-set overlap analysis showing <5% shared tools between ToolWeave training data and BFCL-V3/ToolFlow evaluation sets, (2) confirmation that all compared datasets were subsampled to identical total volume (approximately 50k dialogues) and matched average dialogue length (4.2 turns), and (3) statistical significance via 1,000-iteration bootstrap resampling yielding p<0.01 for the reported gains. These internal checks support that improvements derive from dependency tracking rather than distribution alignment. revision: yes
Referee: The claim that ToolWeave produces 45% multi-step interactions and fewer parameter hallucinations is presented as a direct outcome of the workflow construction and planning stages, yet the manuscript provides no quantitative validation (e.g., inter-annotator agreement, comparison to real user logs, or out-of-distribution test sets) that these synthetic dialogues are realistic proxies rather than artifacts tuned to the evaluation distributions.

Authors: The 45% multi-step rate and hallucination reductions were obtained via automated parsing of provenance logs that explicitly record parameter sources. We will expand the data synthesis section with the precise measurement protocol, inter-annotator agreement scores (targeting >0.85) from human review of 500 sampled dialogues, and additional results on out-of-distribution tool sets. Direct comparison to real user logs is not possible without access to proprietary corpora; however, the structural constraints of dependent workflows and provenance tracking provide an intrinsic guarantee of realism that is corroborated by the downstream benchmark improvements. revision: partial

standing simulated objections not resolved

Direct comparison against real user logs, as no public or accessible proprietary multi-turn tool-calling corpora were available for quantitative matching.

Circularity Check

0 steps flagged

No circularity: empirical gains measured on external benchmarks

full rationale

The paper's central claim is an empirical comparison: LLMs fine-tuned on ToolWeave synthetic dialogues achieve higher scores (e.g., 39.75% vs 23.50% on BFCL-V3 multi-turn) than those fine-tuned on prior datasets. This result is obtained by training on the generated data and evaluating on independent public benchmarks. No equations, fitted parameters, or self-citations are used to derive the performance numbers; the synthesis process (dependency-aware workflows, provenance tracking) is described procedurally without reducing the outcome to quantities defined by the method's own inputs. The derivation chain is therefore self-contained against external evaluation.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The framework rests on the domain assumption that procedurally generated dialogues with enforced dependencies will transfer to real user interactions; no free parameters or invented entities are introduced in the abstract description.

axioms (1)

domain assumption Synthetic dialogues generated via dependency construction and provenance tracking serve as realistic proxies for real multi-turn user-tool interactions.
This premise underpins the claim that fine-tuning on ToolWeave data produces genuine capability improvements.

pith-pipeline@v0.9.0 · 5572 in / 1187 out tokens · 41678 ms · 2026-05-14T21:16:40.573501+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages

[1]

Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay

Toolace: Winning the points of LLM function calling. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh R. N., Liangwei Yang, Silvio Savarese, Juan Ca...

work page arXiv 2025
[2]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao

OpenReview.net. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). Junjie Ye, Zhengyin Du, Xuesong Yao, Weĳian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan,...

work page 2023
[3]

Magnet: Multi-turn tool-use data synthesis and distillation via graph translation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages32600–32616,Vienna,Austria.Associationfor Computational Linguistics. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A datase...

work page 2023
[4]

entry-point

Seed Generation:This initial stage generates a small batch of fundamental, “entry-point” APIs for the domain (e.g., search_product in e- commerce)

work page
[5]

Entity Expansion:This stage uses the entities extracted from Wikidata (see Appendix B.5.1) to generate new APIs that specifically cover domain-relevant entities, ensuring breadth

work page
[6]

It takes existing, simple APIs and itera- Table 9: Comparison of multi-turn synthetic data pipelines across different dimensions

Schema Enrichment: This stage focuses on depth. It takes existing, simple APIs and itera- Table 9: Comparison of multi-turn synthetic data pipelines across different dimensions. Framework Synthesize APIs Code d Data Released, License Friendly License Friendly Synthesis Model Fine-Grained Plan Tooldial (Shim et al.) ✗ ✓ ✓ , ✗ ✗ ✗ ToolAce (Liu et al.) ✓ ✗ ✓...

work page
[7]

Connection Discovery: It generates new APIs that plausibly connect existing APIs, creating data-flow paths (e.g., a get_product_details API that takes a product_id from search_product)

work page
[8]

steps": [ {

Pattern Expansion:This final stage diversifies the API pool by generating parallel variations of existing APIs (e.g., different ways to search, like search_by_name, search_by_category, etc.). For reproducibility, we include the synthesis plan used in our experiments below. Prompt paths have been abstracted for clarity. B.5 Detailed Sub-Algorithms Themaing...

work page
[9]

This measures the data-flow potential

Interconnectivity (𝐼𝐶 ) The average number of input parameters per API that can be filled by an output from another API in the graph. This measures the data-flow potential. 𝐼𝐶 = Í𝑛 𝑖=1 𝐼𝑎𝑖 ∩ 𝑂 𝑛 Where: • 𝑂 = Ð𝑛 𝑖=1 𝑂 𝑎𝑖: The set of all unique output parameters from all APIs. • 𝐼𝑎𝑖: The set of input parameters for API𝑎𝑖. • |𝐼𝑎𝑖 ∩ 𝑂|: The count of input par...

work page
[10]

object”or“array

Complex API Use(𝐶 𝐴𝑈) TheproportionofAPIsthatuseatleastonecomplex parametertype(i.e.,“object”or“array”),measuring schema depth. 𝐶 𝐴𝑈 = Í𝑛 𝑖=1 𝐶 (𝑎𝑖) 𝑛 Where the indicator function𝐶 (𝑎𝑖) is defined as: 𝐶 (𝑎𝑖) =    1, if 𝑎𝑖 has an object or array parameter , 0, otherwise

work page
[11]

required

Required Parameter Ratio (RPR) TheaverageproportionofanAPI’sinputparameters that are marked as “required”, measuring schema strictness. 𝑅𝑃𝑅 = 1 𝑛 𝑛∑︁ 𝑖=1 |𝐼req(𝑎𝑖)| |𝐼total(𝑎𝑖)| Where 𝐼req(𝑎𝑖) is the set of required input parame- ters for API𝑎𝑖, and𝐼total(𝑎𝑖) is the set of all input parameters for𝑎𝑖

work page
[12]

This met- ric quantifies the maximum number of sequential, multi-step operations possible in a single dialogue

Longest Chain Length The length of the longest simple path (a path with no repeated nodes) in theTool Graph. This met- ric quantifies the maximum number of sequential, multi-step operations possible in a single dialogue. C Goal Generation Implementation Details This section provides the low-level implementation details for theGoal Generationstage (Section...

work page
[13]

The search explores paths from a start node, expand- ing a “beam” of the top-𝑘 candidates at each depth

LinearPaths: Thesearefoundusingabounded beam search(Norvig and Russel, 2002). The search explores paths from a start node, expand- ing a “beam” of the top-𝑘 candidates at each depth

work page 2002
[14]

common children

Fan-in / Fan-out: These patterns are found by analyzing node connectivity. For a given start_node, we find its successors. We then find the “common children” of combinations of these successors using set intersection. This allows us to identify points where parallel data flows (fan-out) later merge (fan-in)

work page
[15]

Call”) and Level 2 (“Retrieve+Call

Conditional Branches: These are identified by scanning an API’s output schema. We scan for output fields that can act as a logical predi- cate, specifically those typed asboolean, enum, or other simple types, while filtering out non- conditional fields like IDs. C.2 Path Scoring and Ranking Torankthepathsfoundbyouralgorithms,weusea combination of a hybrid...

work page 1999
[16]

Impact of Structured Sampler:Replacing our motif-based sampler with a random walk (Row 2) degrades accuracy from 13.62% to 12.25%

work page
[17]

I.3 LLM-as-Judge Evaluation Details This appendix provides full details of the LLM-as- judge protocol used in Section 4.4 to assess the semantic quality of synthetic dialogues

Impact of Fine-Grained Planner: Row 3 shows that replacing our planner with ToolFlow’s monolithic planner collapses per- formanceto7.50%,demonstratingtheplanner is the most critical component for enabling non-frontier models. I.3 LLM-as-Judge Evaluation Details This appendix provides full details of the LLM-as- judge protocol used in Section 4.4 to assess...

work page 2025
[18]

To ensure deterministic and reproducible scoring, the model is run with temperature 0.0, greedy decoding, and a fixed maximumtokenbudget

as the judge model. To ensure deterministic and reproducible scoring, the model is run with temperature 0.0, greedy decoding, and a fixed maximumtokenbudget. Eachdialogueisevaluated in a single forward pass; no score aggregation or self-consistency sampling is applied. Evaluation Dimensions Each dialogue is inde- pendently rated along four dimensions on a...

work page 2025
[19]

Can you tell me about the weather?

Naturalness: - User: "Can you tell me about the weather?" - Assistant: "The weather is nice." - User: "What about tomorrow?" - Assistant: "Tomorrow will be nice too."

work page
[20]

What’s the capital of France?

Coherence: - User: "What’s the capital of France?" - Assistant: "The capital of France is Paris." - User: "And the capital of Germany?" - Assistant: "The capital of Germany is Berlin."

work page
[21]

I need help with my homework

Helpfulness: - User: "I need help with my homework." - Assistant: "What subject is it?" - User: "Math." - Assistant: "Okay."

work page
[22]

What’s the capital of Italy?

Accuracy: - User: "What’s the capital of Italy?" - Assistant: "The capital of Italy is Rome." - User: "What’s the capital of Spain?" - Assistant: "The capital of Spain is Barcelona." Some more examples specifically catered to tool usage:

work page
[23]

Can you raise a ticket for my internet issue?

Coherence: (Tool simulated before assistant provides tool to be called along with its input parameters) - User: "Can you raise a ticket for my internet issue?" - Tool: {"ticket_id": "12345"}

work page
[24]

Please book a flight for me

Coherence: (Assistant goes on rambling trying to correct erroneous tool output but never calls the tool) - User: "Please book a flight for me." - Assistant: "Sure, I can help you with that. Can you please provide me with the details of your flight?" - User: "I need a flight from New York to San Francisco on March 10th." - Assistant: book_flight({"from": "...

work page
[25]

Can you check the weather for me?

Coherence: (Assistant talks about calling tool but does not use the required tool-calling format) - User: "Can you check the weather for me?" - Assistant: "Sure, I can help you with that. Let me check the weather for you." - Tool: {"temperature": "72°F", "condition": "Sunny"} - Assistant: "The weather is 72°F and sunny."

work page
[26]

Can you reserve a table for two at an Italian restaurant in New York City for tomorrow night?

Accuracy: (Assistant hallucinates information) - User: "Can you reserve a table for two at an Italian restaurant in New York City for tomorrow night?" - Assistant: "Sure, I have reserved a table for two at ’Luigi’s Italian Bistro’ in New York City for tomorrow night at 7 PM." All such examples must be heavily penalized, and the scores they receive should ...

work page
[27]

Naturalness: [Score] / 5 - Comments: [Brief comments or feedback]

work page
[28]

Coherence: [Score] / 5 - Comments: [Brief comments or feedback]

work page
[29]

Helpfulness: [Score] / 5 - Comments: [Brief comments or feedback]

work page
[30]

name": "create_support_ticket

Accuracy: [Score] / 5 - Comments: [Brief comments or feedback] J ToolWeave Sample Data - Goal, Plan, Tools, and Dialogue We present a detailed example from the customer support domain that illustrates a complex, multi-step dialogue. The example showcases the five available tools, the high-level goal constructed using these tools, the 15-step fine-grained ...

work page 2025