REVIEW 4 major objections 4 minor 2 cited by
ToolRegistry: A Protocol-Agnostic Tool Management Library for Function-Calling LLMs
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Every LLM tool call is structurally an RPC, and ToolRegistry builds one universal stub to serve them all.
desk verdict Real open-source library with a sensible RPC framing, but the evaluation's headline numbers are internally inconsistent and the protocol-agnostic claim skips error semantics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Tool stub, a validated data model holding name, description, JSON parameter schema, callable reference, and metadata such as tags, concurrency safety, timeout, and defer flags. It stands in for an RPC client stub, with ToolRegistry as the client runtime that normalizes calls into a generic ToolCall, dispatches them through the ExecutionBackend protocol, and converts results back into provider-compatible messages. The two built-in backends—ThreadBackend and ProcessPoolBackend—carry the throughput comparison, with cloudpickle handling cross-process serialization and automatic thread fallback when serialization fails. An event-driven callback system propagates registration changes to the server's MCP and OpenAPI adapters, keeping serving interfaces synchronized.
What would settle it
Take a stateful MCP or OpenAPI tool that streams incremental results and requires a persistent session, register it through ToolRegistry, and invoke it with the generic execute_tool_calls() path; if session state is lost between calls, streaming output is truncated, or errors are re-typed, the universal stub has not preserved the protocol's semantics and the protocol-agnostic claim fails for that class of tools.
Extended reading notes
Core claim
The central claim is that every LLM tool call is structurally an RPC, and therefore a single Tool object can act as a universal stub regardless of transport. ToolRegistry operationalizes this by normalizing provider-specific calls into a generic ToolCall, dispatching them through pluggable thread or process backends, and formatting results back into provider-specific messages, with schema generation playing the role of an interface definition language. The paper reports that this cuts integration code by 60–80% (79–86% in its LOC table) and that the right concurrency mode delivers up to 3.1× throughput over the alternative in the abstract, while the conclusion cites up to 4.5×, both figures visible in the benchmark table. It also claims supporting mechanisms: tag-based permissions, BM25F-powered progressive tool disclosure, think-augmented calling, multi-provider schema support, and a near-zero-dependency stdlib-only core.
Load-bearing premise
The load-bearing premise is that the universal Tool object and the generic ToolCall representation capture all behavior of every supported protocol—native Python, MCP, OpenAPI, and LangChain—so that no protocol-specific semantics such as stateful connections, streaming, custom error handling, or security handshakes are lost during registration, dispatch, and result formatting.
Editorial extensions
If this is right
- Developers can register native Python functions, MCP servers, OpenAPI specs, and LangChain tools in one registry and invoke all of them through a single execute_tool_calls() path, with the paper's measurements showing 79–86% less glue code.
- Execution mode becomes a workload-level decision rather than an architectural one: thread mode is reported faster for CPU-bound native tools (2.4–4.5× over process), while process mode is reported faster for I/O-bound network tools (1.8–3.1× over thread).
- Hand-written JSON schemas disappear for registered tools, since schemas are generated from type hints and docstrings and then translated to OpenAI, Anthropic, and Gemini formats.
- Large tool registries can keep prompt size and token costs down through the defer flag and BM25F-based ToolDiscoveryTool, and sensitive tools can be filtered out of prompts or gated by tag-based permission policies.
- Because ToolRegistry is positioned as a helper library rather than an orchestration framework, agent stacks can adopt it incrementally without replacing their existing orchestration layer.
Reading between the lines
- If the generic ToolCall representation is truly lossless, the same registry pattern should extend to other RPC-style protocols such as gRPC and A2A without changing the execution engine, which is the direction the authors list as future work.
- The benchmark spread implies a simple decision rule for practitioners: classify a workload as CPU-bound or I/O-bound, set the executor accordingly, and the up-to-3.1×/4.5× gap is the measurable cost of choosing wrong; this rule could be tested on other hardware and larger call batches.
- The universal-stub premise would be strained by stateful or streaming tools, so the strongest test of the claim is not code reduction but whether session state, incremental output, custom error semantics, and authentication handshakes survive a round trip through the normalized ToolCall path.
- The 60–80% code reduction likely reflects the fact that the benchmarked scenarios are stateless request–response tools; the reduction may be smaller for tools whose integration work lies in complex auth, streaming, or long-running state rather than in schema and dispatch boilerplate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents ToolRegistry, an open-source library that treats every LLM tool call as an RPC and provides a single Tool object to wrap native Python functions, MCP servers, OpenAPI endpoints, and LangChain tools. A central registry handles schema generation, dispatch, concurrency (thread/process backends), permissions, progressive disclosure, and multi-provider API compatibility. The paper claims a 60–80% reduction in integration code and up to 3.1x (abstract) or 4.5x (conclusion) throughput differences between execution modes, and includes case studies and benchmarks.
Significance. If the central claims hold, ToolRegistry addresses a real pain point in LLM tool integration by unifying protocols behind one interface, and the open-source release and documented architecture are strengths. The RPC framing is a useful conceptual contribution. However, the evaluation as presented does not yet substantiate the headline numbers: the semantic-loss concern for protocol error/status fields, the lack of variance reporting and baselines, and the in-house code-reduction measurement all need to be addressed before the claims can be assessed. With those fixes, the paper could be a valuable systems contribution.
major comments (4)
- [§3.5, §3.6.2] The universal ToolCall and normalized result structure omit protocol-level error semantics. MCP's CallToolResult carries an isError flag and typed content parts, and OpenAPI responses carry HTTP status codes that distinguish success from failure; the paper does not state how these are encoded in the normalized result (Sections 3.5.1 and 3.6.2). The evaluation (Section 5.1) exercises only calculator-style tools that always return success, so the claim that a single Tool object acts as a universal stub regardless of transport (§3.1) is not established for error-bearing tools. Section 6.1's limitations cover serialization, error recovery, and schema validation, but not protocol error semantics. Please specify the normalization of error/status information and demonstrate it with failing or partial-failure tool calls.
- [§5.1, Table 3] The throughput comparison reports only means of 10 runs with no standard deviations, confidence intervals, or statistical tests, so it is impossible to tell whether the 1.8x–4.5x differences are significant given run-to-run variance. There is also no comparison against a baseline of using the protocol SDKs directly (e.g., a raw MCP client or plain HTTP calls for OpenAPI), which would be needed to support the claim that ToolRegistry adds negligible overhead. Without these, the headline 'up to 3.1x' (abstract) / 'up to 4.5x' (conclusion) figures are not reproducible.
- [§5.2, Table 4] The code-reduction numbers are measured on in-house 'manual' baselines written by the same authors who wrote ToolRegistry, with no external benchmark or articulated LOC-counting methodology. The 79–86% figures in Table 4 are thus the authors' own estimate rather than an independent measure, and the abstract/conclusion state 60–80% without reconciliation. Please provide the exact code used for both manual and ToolRegistry implementations, a precise definition of what counts as a line, and ideally a comparison with existing frameworks (e.g., LangChain, FastMCP, or direct OpenAPI SDK integration).
- [Abstract vs. Conclusion] The abstract reports 'cuts integration code by 60–80%' and 'up to 3.1x throughput', while the conclusion repeats the code figure and says 'up to 4.5x higher throughput'. Table 4 reports 79–86% reduction and Table 3's maximum margin is 4.5x. These are mutually inconsistent as written; the manuscript must state a single, defensible set of headline numbers derived from the evaluation.
minor comments (4)
- [§3.2.2] The sentence 'provides both synchronous and asynchronous execution throughrun()andarun()methods' lacks spaces around the method names; it should read 'through run() and arun() methods'.
- [§3.1] The full text contains a typo 'JSONC/Y AML' where the abstract and sections elsewhere use 'JSONC/YAML'; please correct this for consistency.
- [§4.1.2] The case study states 'cutting integration effort by roughly 70%', which is inconsistent with Table 4's 79–86% for multi-protocol setup; please reconcile the quoted percentage with the measured table values.
- [§5.1.1] The statement 'All configurations achieved 100% success rates' is ambiguous given that Section 3.5.4 describes automatic fallback from process mode to thread mode on serialization failure; please define what counts as success and clarify whether fallback is included in the reported throughput.
Circularity Check
No circularity: the RPC-unification claim is a design position, not a derived prediction; the code-reduction and throughput numbers are self-reported measurements, not fitted parameters renamed as predictions, and the self-citations are not load-bearing.
full rationale
The paper's central claim is that every LLM tool call is structurally an RPC and that a Tool object can act as a universal stub. This is presented as a design principle in Sections 3.1 and 3.3, then implemented via adapters for native Python, MCP, OpenAPI, and LangChain. It is not derived from a fitted model or from a self-citation; it is an architectural commitment. The code-reduction figures (Table 4) are the authors' own comparisons of hand-written adapters against ToolRegistry code, and the throughput figures (Table 3) are internal measurements. These are self-reported benchmarks, which raises reproducibility and bias concerns, but they are not cases where a parameter is fit to one quantity and then the same quantity is reported as a prediction. The same-author citations, llm-rosetta [Ding, 2025] and zerodep [Ding and Stevens, 2026], are used to attribute implementation details: multi-provider schema conversion and vendored stdlib modules. These are peripheral to the RPC-unification argument and are not invoked to forbid alternatives or to prove uniqueness. The abstract's '3.1x' and conclusion's '4.5x' refer to different workload classes in Table 3 and are internally inconsistent in presentation, but that is a consistency issue, not circularity. No step in the paper reduces, by equation or by self-citation, to its own input, so the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Every LLM tool call is structurally an RPC: a function name, JSON arguments, a serialized result.
- ad hoc to paper A generic Tool object can wrap any native function, MCP server, OpenAPI endpoint, or LangChain tool without losing protocol-specific semantics.
- domain assumption Automatic JSON Schema generation from type hints and docstrings is correct and complete for all supported providers.
- domain assumption Thread and process pools, with cloudpickle serialization, preserve correctness for all supported tools.
Cite this review
Pith. "Pith review of ToolRegistry: A Protocol-Agnostic Tool Management Library for Function-Calling LLMs." pith.science (2026). https://pith.science/paper/FLVOZMTO
@misc{pith2026250710593,
author = {Pith},
title = {Pith review of: ToolRegistry: A Protocol-Agnostic Tool Management Library for Function-Calling LLMs},
year = {2026},
howpublished = {\url{https://pith.science/paper/FLVOZMTO}},
note = {Machine review of arXiv:2507.10593}
}
read the original abstract
Every LLM tool call is structurally an RPC -- a function name, JSON arguments, and a serialized result -- yet each protocol (native Python, MCP, OpenAPI, LangChain) is integrated from scratch. We present ToolRegistry, a system that makes this RPC nature explicit: a single Tool object acts as a universal stub regardless of transport, while the registry serves as the RPC client runtime for dispatch, schema generation, and execution. The system ships as three packages -- a core registry, a server exposing tools over MCP and OpenAPI, and a hub of production-ready implementations -- and invokes tools through pluggable thread or process backends. The system now also provides tag-based permission policies, BM25F-powered progressive tool disclosure for large registries, think-augmented function calling, multi-provider schema support (OpenAI, Anthropic, Gemini), declarative JSONC/YAML configuration, and a near-zero-dependency core built on stdlib-only vendored modules. In our benchmarks the library cuts integration code by 60-80%, and choosing the right concurrency mode (thread vs. process) yields up to 3.1x throughput over the alternative for a given workload. ToolRegistry is open-source at https://github.com/Oaklight/ToolRegistry; documentation lives at https://toolregistry.readthedocs.io/.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 2 Pith papers
-
A Large-Scale Dataset of MCP Implementations on GitHub
A multi-stage GitHub mining pipeline yields 2,297 verified MCP client, server, and gateway repositories with role labels, evidence tags, and 83% precision on a manual sample.
-
AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
AutoTool's two-phase SFT/RL plus ranking training lets 8B LLM agents beat larger fixed-tool agents across math, search, code, and vision benchmarks, though unseen-tool gains are asserted, not isolated.
Reference graph
Works this paper leans on
-
[2]
Yu Du, Fangyun Wei, and Hongyang Zhang. Anytool: Self-reflective, hierarchical agents for large-scale api calls.arXiv preprint arXiv:2402.04253,
-
[5]
URL https://modelcontextprotocol. io/specification/2025-03-26. OpenAI. Introducing the responses api, Mar
work page 2025
-
[6]
URL https://community.openai.com/t/ introducing-the-responses-api/1140929. Demis Hassabis. Post on x about mcp support for gemini models, Apr
-
[7]
URL https://x.com/demishassabis/ status/1910107859041271977. OpenWebUI. Mcpo: A simple, secure mcp-to-openapi proxy server, Apr 2025a. URL https://github.com/ open-webui/mcpo. OpenWebUI. Openapi tool servers, Mar 2025b. URLhttps://docs.openwebui.com/openapi-servers/. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun...
-
[8]
Arash Ahmadi, Sarah Sharif, and Yaser M Banad. Mcp bridge: A lightweight, llm-agnostic restful proxy for model context protocol servers.arXiv preprint arXiv:2504.08999,
-
[9]
A survey of ai agent protocols.arXiv preprint arXiv:2504.16736,
Yingxuan Yang, Huacan Chai, Yuanyi Song, Siyuan Qi, Muning Wen, Ning Li, Junwei Liao, Haoyi Hu, Jianghao Lin, Gaowei Chang, et al. A survey of ai agent protocols.arXiv preprint arXiv:2504.16736,
-
[10]
Abul Ehtesham, Aditi Singh, Gaurav Kumar Gupta, and Saket Kumar. A survey of agent interoperability protocols: Model context protocol (mcp), agent communication protocol (acp), agent-to-agent protocol (a2a), and agent network protocol (anp).arXiv preprint arXiv:2505.02279,
-
[11]
Lei Wei, Xiao Peng, Jinpeng Ou, and Bin Wang. Think-augmented function calling: Improving llm parameter accuracy through embedded reasoning.arXiv preprint arXiv:2601.18282,
Show all 12 references
-
[2023]
Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789,
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789,
-
[2024]
Retrieval models aren’t tool-savvy: Benchmarking tool retrieval for large language models
Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, and Zhaochun Ren. Retrieval models aren’t tool-savvy: Benchmarking tool retrieval for large language models. InFindings of the Association for Computational Linguistics: ACL 2025,
2025
-
[2025]
Llm with tools: A survey.arXiv preprint arXiv:2409.18807,
Zhuocheng Shen. Llm with tools: A survey.arXiv preprint arXiv:2409.18807,
-
[2026]
Stdlib or third-party? empirical performance and correctness of llm-assisted zero- dependency python libraries.arXiv preprint arXiv:2605.21405,
Peng Ding and Rick Stevens. Stdlib or third-party? empirical performance and correctness of llm-assisted zero- dependency python libraries.arXiv preprint arXiv:2605.21405,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.