REVIEW 5 cited by
ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Existing evaluations of tool learning primarily focus on validating the alignment of selected tools for large language models (LLMs) with expected outcomes. However, these approaches rely on a limited set of scenarios where answers can be pre-determined, diverging from genuine needs. Furthermore, a sole emphasis on outcomes disregards the complex capabilities required for LLMs to effectively use tools. To tackle this issue, we propose ToolEyes, a fine-grained system tailored for the evaluation of the LLMs' tool learning capabilities in authentic scenarios. The system meticulously examines seven real-world scenarios, analyzing five dimensions crucial to LLMs in tool learning: format alignment, intent comprehension, behavior planning, tool selection, and answer organization. Additionally, ToolEyes incorporates a tool library boasting approximately 600 tools, serving as an intermediary between LLMs and the physical world. Evaluations involving ten LLMs across three categories reveal a preference for specific scenarios and limited cognitive abilities in tool learning. Intriguingly, expanding the model size even exacerbates the hindrance to tool learning. The code and data are available at https://github.com/Junjie-Ye/ToolEyes.
Forward citations
Cited by 5 Pith papers
-
DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
A new benchmark and metric show that large language models still struggle to call tools when the needed details are scattered across multi-party, multi-round group dialogues.
-
Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems
LLM tool agents fail at parameter filling in five recurring ways; perturbing tool documents and user queries drives most failures, and invented parameter names are tied to the model rather than the input.
-
RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving
RepoMaster, a repository-aware code agent, lifts the task pass rate from 40.7% to 62.9% and cuts token use by about 95% versus OpenHands on the new GitTaskBench benchmark.
-
Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training
Continual pre-training on a large agent-focused corpus of API docs and tool trajectories improves an 8B LLM's function-calling and planning, but the reported generalization is undermined by benchmark data appearing in...
-
CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
CRITICTOOL, a benchmark of 2,740 tool-calling error scenarios built from BFCL and T-Eval with GPT-4o-based error injection, finds most LLMs rarely recover from tool-use errors, with GPT-4o best at 69.01 overall and to...
Discussion (0). Continue with ORCID to comment.