REVIEW 9 cited by
CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) are often augmented with tools to solve complex tasks. By generating code snippets and executing them through task-specific Application Programming Interfaces (APIs), they can offload certain functions to dedicated external modules, such as image encoding and performing calculations. However, most existing approaches to augment LLMs with tools are constrained by general-purpose APIs and lack the flexibility for tailoring them to specific tasks. In this work, we present CRAFT, a general tool creation and retrieval framework for LLMs. It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. For each task, we collect specific code solutions by prompting GPT-4 to solve the training examples. Following a validation step ensuring the correctness, these solutions are abstracted into code snippets to enhance reusability, and deduplicated for higher quality. At inference time, the language model retrieves snippets from the toolsets and then executes them or generates the output conditioning on the retrieved snippets. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning. Experiments on vision-language, tabular processing, and mathematical reasoning tasks show that our approach achieves substantial improvements compared to strong baselines. In addition, our in-depth analysis reveals that: (1) consistent performance improvement can be achieved by scaling up the number of tools and the capability of the backbone models; (2) each component of our approach contributes to the performance gains; (3) the created tools are well-structured and reliable with low complexity and atomicity. The code is available at https://github.com/lifan-yuan/CRAFT.
Forward citations
Cited by 9 Pith papers
-
GraspMAS: Zero-Shot Language-driven Grasp Detection with Multi-Agent System
A multi-agent system with planner, coder, and observer agents achieves zero-shot language-driven grasp detection that outperforms existing baselines on benchmarks and robots.
-
Retrieval-Augmented Generation for Service Discovery: Chunking Strategies and Benchmarking
Endpoint-based chunking outperforms naive splitting for RAG-based endpoint discovery, and an LLM agent raises precision at the cost of recall, evaluated on a new multi-domain benchmark and RestBench.
-
FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills
FlowEvo compiles successful agent workflows into executable skill records and reuses them at inference time, reporting the best accuracy-cost tradeoff across ALFWorld, HumanEval, and GSM8K among tested baselines.
-
AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.
-
MemTool: Optimizing Short-Term Memory Management for Dynamic Tool Calling in LLM Agent Multi-Turn Conversations
MemTool is a short-term memory framework with three modes (autonomous, workflow, hybrid) that lets LLM agents add and remove tools across multi-turn conversations, evaluated over 100 turns on 13+ models.
-
Augmented Vision-Language Models: A Systematic Review
A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.
-
A Compute-Matched Re-Evaluation of TroVE on MATH
After matching computational budget, TroVE's toolbox mechanism yields only a marginal, statistically non-significant 1% accuracy gain over a plain sampling baseline on MATH.
-
Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolution
Alita achieves 75.15% pass@1 on GAIA validation by dynamically generating and reusing MCP-based tools, but the evaluation confounds model choice with the self-evolution mechanism.
-
Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.
Discussion (0). Sign in to comment.