REVIEW 8 cited by
ToolTalk: Evaluating Tool-Usage in a Conversational Setting
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have displayed massive improvements in reasoning and decision-making skills and can hold natural conversations with users. Many recent works seek to augment LLM-based assistants with external tools so they can access private or up-to-date information and carry out actions on behalf of users. To better measure the performance of these assistants, this paper introduces ToolTalk, a benchmark consisting of complex user intents requiring multi-step tool usage specified through dialogue. ToolTalk contains 28 tools grouped into 7 plugins, and includes a complete simulated implementation of each tool, allowing for fully automated evaluation of assistants that rely on execution feedback. ToolTalk also emphasizes tools that externally affect the world rather than only tools for referencing or searching information. We evaluate GPT-3.5 and GPT-4 on ToolTalk resulting in success rates of 26% and 50% respectively. Our analysis of the errors reveals three major categories and suggests some future directions for improvement. We release ToolTalk at https://github.com/microsoft/ToolTalk.
Forward citations
Cited by 8 Pith papers
-
DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
A new benchmark and metric show that large language models still struggle to call tools when the needed details are scattered across multi-party, multi-round group dialogues.
-
Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agents
Under controlled authority invalidation, agents reach visible goals in 262/270 runs but only 55 remain authorized; CommitGuard blocks stale durable effects when runtimes emit boundary signals.
-
CallBench: A Benchmark for Dual-Goal Coordination in Phone Call Assistants
CALLBENCH is a 50k-dialogue Chinese benchmark showing current dialogue methods achieve only ~0.61-0.77 overall scores and about 10.6% safety violations on dual-goal phone-call assistant decisions.
-
UserBench: An Interactive Gym Environment for User-Centric Agents
A new multi-turn agent benchmark shows that current LLMs elicit fewer than 30% of user preferences and reach full intent alignment only about 20% of the time.
-
LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback
Self-generated agent trajectories from LAM SIMULATOR improved fine-tuned model pass rates by up to 49.3% on ToolBench and CRMArena.
-
AgentOmnia: Scaling Agentic Models for Full-Scenario Applications
Post-training an open Qwen3-30B agent on 52,361 verifiable tasks in 5,018 synthesized stateful environments lifts its average across four agent benchmarks from 22.9% to 41.7%.
-
CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactions
A new 109-conversation, 86-API benchmark for LLM function-calling in multi-turn dialogue shows top models at about 40% accuracy and near-zero performance on chains of 4+ calls.
-
Large Language Models for Planning: A Comprehensive and Systematic Survey
A structured survey of LLM planning methods, benchmarks, and interpretability work, organized around a three-way taxonomy.
Discussion (0). Continue with ORCID to comment.