{"id":"319f841f-0e2d-44c0-8c41-b3503dedc765","arxiv_id":"2411.15399","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Selectively reducing the number of tools presented to an LLM, using embedding similarity over individual tools or clusters, improves function-calling success and efficiency on edge devices.","lead":"This paper tests whether giving an LLM fewer, better-chosen tools improves its function-calling accuracy and speed on edge hardware. The authors find that their embedding-based tool selector cuts execution time and power while raising success rates across several small models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gains conflate tool-count reduction, retrieval selection, and context shortening; no same-context or non-selective subset baseline isolates the claimed mechanism.","rationale":"The reader's weakest_assumption focuses on the Tool Recommender's ability to generate ideal descriptions without seeing the APIs. That is a genuine point of failure: if the recommender produces poor descriptions, similarity search retrieves irrelevant tools and performance collapses. However, I see a more fundamental gap in the experimental design that makes the central claim underdetermined even assuming a perfect recommender. Figures 2 and 3 compare default (16K, all tools) with LiS (8K, k tools), so tool count, relevance selection, and context length all change simultaneously. The abstract claims 'selectively reducing the number of tools available to LLMs significantly improves their function-calling performance, execution time, and power efficiency.' To support the word 'selectively,' the authors must show that a non-selective reduction of the same size at the same context performs worse. Without that baseline, the reported efficiency gains could simply reflect shorter prompts, and the success-rate gains could reflect fewer options rather than the recommender's choices. The Recommender concern is a component of this larger issue: a recommender failure would indeed break the pipeline, but even a successful recommender does not establish the causal role of selection unless the context/tool-count confound is removed. I therefore agree with the CONDITIONAL verdict, but the condition should be a matched-context, non-selective-subset ablation rather than only a recommender-quality check. The paper is otherwise a plausible systems contribution with a clear method and reasonable hardware evaluation; the missing control is the load-bearing gap.","tokens_in":10414,"tokens_out":4020,"duration_ms":40956,"concrete_test":"Run LiS (k=3 and k=5, 8K context) against two new baselines on the same 230-query BFCL and GeoEngine subsets: (a) a non-selective subset containing the first k tools in the API list at 8K context, and (b) the full tool set at the smallest context that fits all tools, plus a random-k subset. If the non-selective or random subset matches LiS on success rate and tool accuracy, the selective-retrieval mechanism is not what drives the gains; if the full-tool-set 8K baseline matches LiS on execution time and power, context shortening is the main efficiency driver.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV compares the default configuration (16K context, all tools) against LiS (8K context, k=3 or k=5 tools), changing three variables at once: the number of tools, the selection mechanism, and the context-window length. Since inference latency and energy scale with prompt length, the reported execution-time reductions (up to 70–80%) and power reductions are likely dominated by the 16K→8K context cut rather than by tool selection. The success-rate claim is similarly underdetermined: no baseline presents the same number of tools at the same context without the recommender, so the paper cannot distinguish \"fewer tools help\" from \"more relevant tools help\" from \"shorter prompts help.\" Table II's 46-vs-19 demonstration at fixed 16K is one anecdotal query, not a statistical comparison, and tool accuracy is mechanically inflated when the candidate set shrinks. The Recommender assumption in Section III-B is one fragile link, but even a perfect recommender would not rescue the missing control; the central causal claim requires showing that LiS outperforms a non-selective k-tool subset at the same context.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Less-is-More, a fine-tuning-free dynamic tool-selection scheme for LLM function calling on edge devices. API descriptions and GPT-4-augmented benchmark queries are embedded offline into latent spaces at three granularities (individual tools, tool clusters, full tool set). At runtime, an LLM is first asked to describe the 'ideal' tools for the user query without seeing any API definitions; a FAISS k-NN search then retrieves the most similar real tools or clusters, and the agent is invoked with only that reduced tool set. Experiments on BFCL and GeoEngine with six quantized LLMs on an NVIDIA Jetson AGX Orin report improved success rates and tool accuracy, with execution time reduced by up to 70-80% and power by up to 40%.","tokens_in":10769,"tokens_out":7755,"duration_ms":70866,"significance":"The problem is timely: deploying capable function-calling agents on resource-constrained edge hardware is an active research area, and a plug-and-play, fine-tuning-free tool-selection method would be practically valuable. The paper has concrete strengths: direct measurements on edge hardware, multiple LLMs and quantization variants, comparison against a prior similarity-based baseline (Gorilla), and a hierarchical search-level design that is simple to implement. However, the central empirical claim is not currently established because the experiments confound tool selection with context-window shortening, the universal-improvement conclusion is contradicted by the paper's own reported Mistral results, and part of the evaluation is built from benchmark-derived clusters and selective baselines. If these issues are addressed, the contribution would be solid; in its present form, the significance is uncertain.","major_comments":[{"comment":"The comparison varies three factors at once. The default baseline uses the full tool set at a 16k context window, while Less-is-More and Gorilla use k=3 or k=5 tools at an 8k context window. Consequently, the reported execution-time reductions (up to 70-80%) and power reductions cannot be attributed to tool selection: prefill cost scales with prompt length, so the 16k-to-8k context cut is a plausible dominant factor. The success-rate claim is likewise underdetermined. Without a same-context non-selective subset baseline (e.g., random or fixed k-tool subset at 8k context) or a full-tool-set condition at 8k context, the paper cannot distinguish 'fewer tools help', 'more relevant tools help', and 'shorter prompts help'. The Table II 16K 46-tool vs 16K 19-tool illustration is a single query, not a statistical comparison. I request at least two additional conditions: (a) all tools at 8k context and (b) an arbitrary k-tool subset at 8k context, with absolute execution time and power reported.","section":"§IV, paragraph beginning 'We determined the minimum context window...'"},{"comment":"The text states for Mistral-8b on BFCL that 'the optimizations did not result in any gain in success rate and tool accuracy', yet the next paragraphs conclude that 'our method improved all four metrics—success rate, tool accuracy, execution time, and power consumption—across all the LLMs'. On GeoEngine, Mistral-8b is reported as having 'Execution time was 10% higher for some variations', which further contradicts a universal-improvement claim. The authors should restrict the conclusions to the metrics and models for which gains are actually observed, or present the negative cases with equal prominence.","section":"§IV, Mistral-8b BFCL paragraph and the 'Overall' paragraph"},{"comment":"Phi3 and Qwen2-1.5b are excluded from the GeoEngine analysis because their default success rate was approximately 10% and their execution-time/power measurements were deemed unreliable. Excluding the two models with the weakest default performance from the more complex benchmark removes the cases where the method is most likely to fail and makes the cross-model claim unrepresentative. Please include these models in a secondary analysis, or explicitly limit the GeoEngine claim to the reported subset of models.","section":"§IV, GeoEngine paragraph"},{"comment":"The Tool Recommender is prompted to produce 'ideal' tool descriptions without being shown any API definitions. The entire retrieval chain depends on this step: if the base quantized LLM cannot infer plausible tool descriptions for a domain such as GeoEngine's remote-sensing tools, the similarity search cannot retrieve relevant tools. This premise is not validated. The paper reports no ablation comparing the LLM-generated descriptions against ground-truth tool descriptions, no comparison against directly embedding the user query, and no oracle-retrieval upper bound. In addition, the free parameters k and the 0.5 fallback threshold are not subjected to sensitivity analysis. These experiments are needed before the success-rate gains can be attributed to the recommender.","section":"§III-B"},{"comment":"The augmented latent space is constructed from GPT-4-augmented queries sampled from the same GeoEngine and BFCL categories used later for evaluation. Although the paper says these generated queries 'are not used for training or runtime decisions', they are used to form the clusters that the Tool Controller selects between at runtime, so the disclaimer is misleading. This is an evaluation-contamination risk: the cluster structure may encode benchmark-specific query-tool associations, and the reported success rates may not transfer to unseen tool spaces. I request a leave-one-benchmark-out or held-out-category experiment, or a demonstration that clusters built without the benchmark queries yield similar performance.","section":"§III-A, Search Level 2"}],"minor_comments":[{"comment":"The latent space is described as 768-dimensional in one place and '728-dimensional' in another; please reconcile the inconsistency.","section":"§III-A and §III-B"},{"comment":"Only normalized execution time and normalized power are reported; without absolute values, units, or confidence intervals/k-fold repetitions, the magnitude of the efficiency gains is hard to assess.","section":"§IV, Figures 2 and 3"},{"comment":"The sentence 'For the default models, we also tested context windows larger than 16k' is not backed by any presented result; either include the results or remove the claim.","section":"§IV"},{"comment":"The last row 'Max drop ↓ 43% ↓ 19%' is ambiguous: it is unclear what quantity is maximized and why the row is labeled a drop; please define the comparison explicitly.","section":"Table II"},{"comment":"The statement that for simple queries the LLM 'would likely recommend just one ideal tool description' is informal; the paper does not analyze the distribution of the number of recommended tools, which is relevant for choosing k.","section":"§III-C"}],"recommendation":"major_revision","confidential_remarks":"The main barrier is the confounded comparison: the current evidence does not isolate the proposed tool-selection mechanism from the 8k-vs-16k context change. I would not accept the current version without at least a same-context non-selective subset baseline and a correction of the universal-improvement claim. The benchmark-derived cluster contamination is a secondary but real concern that should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about arXiv:2411.15399 is that it has a genuinely simple and plausible idea—dynamically prune the tool list an LLM sees, using hierarchical retrieval over tool embeddings, and fall back to all tools when uncertain—but the evaluation as written doesn't isolate the mechanism behind the reported gains. The default baseline uses a 16k context window with all tools; the method uses 8k with a small selected subset. So latency, power, and even success-rate improvements could come from shorter prompts, from fewer tools, or from better tool relevance. There is no control that gives the model the same number of tools at the same context without the recommender, so the causal claim 'fewer tools help' is underdetermined. Gorilla is a useful comparison but it also runs at 8k, so it doesn't fill that gap.\n\nWhat is genuinely new: the three search levels (individual tools, tool clusters, full set), the LLM-generated 'ideal tool' descriptions as a query representation, and the runtime controller with a fallback. That combination does not appear in the cited prior work. The authors also deserve credit for testing six quantized LLMs on a real edge device (Jetson AGX Orin) and reporting execution time and power, not just accuracy. The motivation in Table I and II is telling, even if anecdotal.\n\nSoft spots are real but fixable. The Mistral results show no success-rate gain, yet the conclusion claims all four metrics improved across all LLMs; that needs to be corrected. On GeoEngine, excluding Phi3 and Qwen2-1.5b because their default success rate was around 10% removes exactly the models that might show whether the method rescues weak base models; the exclusion should be justified or the results shown. There are no error bars or per-query variance, despite only 230 queries per benchmark. The clustering space is built from GPT-4-augmented queries drawn from the same benchmark categories used for evaluation, which risks mild contamination; an ablation on held-out tool sets would help. The Recommender's premise—that an LLM can describe needed tools without seeing any APIs—is a genuinely fragile link and deserves an ablation where the recommender is given tool names (or a random subset).\n\nWho this is for: researchers working on edge LLM inference, agent tool selection, or function calling with small quantized models. They will find the method worth trying and the system-level measurements useful. The paper should go to peer review; it is not a desk reject, but it needs a matched-context ablation (same context length, same number of tools, with and without selection) before the central claim is credible. I'd encourage a serious referee to accept it as a systems paper with major revisions.","headline":"A practical tool-selection scheme with plausible results, but the evaluation conflates tool pruning with context shortening and doesn't isolate why it helps.","tokens_in":11180,"tokens_out":3239,"would_cite":false,"duration_ms":28876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Providing an LLM with fewer, better-chosen API tools improves function-calling accuracy while cutting execution time and power on edge devices.","keywords":["function calling","edge AI","dynamic tool selection","LLM agents","latent-space retrieval","quantized LLMs","FAISS","tool recommendation"],"falsifier":"Run the same edge models on the same benchmarks with the same reduced context window but replace the LLM-generated ideal descriptions with the user query's own embedding as the retrieval key; if success rate stays the same, the Recommender step adds nothing and the gains come purely from fewer tools. Alternatively, measure success rate when the reduced tool set is chosen at random instead of by similarity: random selection performing equally would show that tool-space size, not relevance, drives the result.","tokens_in":10232,"feed_emoji":"⚡","tokens_out":4363,"duration_ms":37821,"temperature":0.7,"pith_summary":"The paper argues that handing an LLM agent every available API at once is self-defeating on edge devices: with 46 tools in context, a quantized 8B model fails a simple mapping query, and succeeds once the tool list is cut to 19. It introduces Less-is-More, a fine-tuning-free pipeline that at runtime asks the LLM to describe the tools it would need, embeds those descriptions, retrieves the closest real tools from offline-built search levels, and only then calls the function with the reduced tool set. The reported effect is that success rate and tool accuracy rise across several quantized LLMs while execution time drops by up to 70-80% and power consumption by up to 40-45%. If the effect holds, small on-device models can act as capable agents without retraining.","feed_headline":"Fewer tools make edge AI agents faster and smarter","feed_subtitle":"Cutting the API list an LLM sees lifts success rates and slashes run time and power on edge hardware.","key_machinery":"The load-bearing object is a three-level latent tool space built offline with the MPNet sentence encoder: Search Level 1 embeds each API description individually, Search Level 2 augments the space with GPT-generated variations of benchmark queries and agglomerative clusters the results so that tools used in the same task lie close together, and Search Level 3 is simply the full JSON tool set used as fallback. At runtime the same encoder embeds the LLM's self-generated ideal tool descriptions plus the user query, and a FAISS k-NN search retrieves the top-k items from Levels 1 and 2; the Controller picks the level with the higher average similarity and passes only that reduced tool set to the agent. This lets the method trade tool granularity against query complexity without fine-tuning.","core_discovery":"Less-is-More claims that selectively reducing the number of tools available to an LLM is itself a performance optimization: fewer options reduce the model's confusion, let it select and invoke the right function more often, and shrink the context window enough to cut latency and energy. The discovery is a mechanism, not just a heuristic: a runtime Recommender generates ideal tool descriptions from the user query alone, and a Controller matches them through k-NN similarity against three offline-constructed representations of the real tool space, namely individual tools, clusters of tools that tend to be used together, and the full set as fallback. Across six LLMs and two benchmarks, the reduced-tool protocol improves success rate and tool accuracy over both the default all-tools setup and a similarity-selection baseline, with the largest gains on the sequential GeoEngine tasks.","pith_inferences":["The less-is-more effect likely generalizes beyond tool calling: any LLM task with a large enumerated option set, such as multiple-choice, routing, or retrieval, may suffer similar confusion, and the same reduction-by-retrieval pattern could be tested there.","A cheaper variant can be tested that skips the LLM Recommender and embeds the raw query directly; if accuracy holds, the entire Recommender call is removable, saving its latency and power.","Search Level 2 clusters are built from GPT-generated query variations, so generalization to new tool sets could depend on how well those synthetic queries cover real usage patterns; usage logs from deployed agents would be a stronger clustering signal.","The 0.5 similarity threshold for falling back to all tools is a tunable knob; lowering it trades more Recommender mistakes for a smaller context, and mapping that tradeoff could produce an adaptive per-model policy."],"forward_implications":["Off-the-shelf quantized LLMs can serve as function-calling agents on edge boards without fine-tuning, widening the range of deployable models.","Smaller context windows become sufficient, which compounds the latency and power savings beyond the tool-count reduction alone.","The scheme is plug-and-play: any existing LLM with function calling can be wrapped by the Recommender and Controller without retraining or weight changes.","Applications that must keep user data on-device for privacy gain a practical path to cloud-free agent execution.","Even when accuracy gains are small, as with Mistral on BFCL, the method still delivers large execution-time and power reductions."],"supporting_citations":[{"why":"Gorilla: the similarity-based tool-selection baseline that Less-is-More is compared against and must beat.","marker":"[1]"},{"why":"BFCL: the general function-calling benchmark used to measure success rate and tool accuracy.","marker":"[11]"},{"why":"GeoEngine: the sequential multi-tool benchmark used for application-specific evaluation.","marker":"[12]"},{"why":"Anthropic contextual retrieval: the RAG-style inspiration for building latent-space tool search.","marker":"[36]"},{"why":"MPNet: the pretrained embedding model that maps tool and query descriptions into a shared latent space.","marker":"[37]"},{"why":"ToolQA: the benchmark-augmentation recipe whose GPT-generated queries are reused to build tool clusters for Search Level 2.","marker":"[38]"},{"why":"FAISS: the library used for the k-NN similarity search that retrieves top-k tools and clusters.","marker":"[40]"}],"fun_headline_variants":["Cutting tool options boosts edge LLM success","Fewer APIs on edge AI: better and faster","Simplify tool calls, speed up edge AI","Tool pruning lifts edge LLM accuracy and speed","Less is more: fewer tools, faster edge LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an LLM prompted with no API list can still describe the tools a task needs accurately enough that embedding-based similarity retrieves the real tools; if the ideal descriptions are off, the reduced set is wrong and the claimed accuracy gains vanish.","fun_headline_variants_meta":{"raw":{"variants":["Cutting tool options boosts edge LLM success","Fewer APIs on edge AI: better and faster","Simplify tool calls, speed up edge AI","Tool pruning lifts edge LLM accuracy and speed","Less is more: fewer tools, faster edge LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3561,"prompt_tokens":871,"completion_tokens":2690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":2617}},"tokens_in":487,"tokens_out":2690,"duration_ms":17359,"temperature":1.0,"reasoning_tokens":2617,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:20:23.535387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same edge models on the same benchmarks with the same reduced context window but replace the LLM-generated ideal descriptions with the user query's own embedding as the retrieval key; if success rate stays the same, the Recommender step adds nothing and the gains come purely from fewer tools. Alternatively, measure success rate when the reduced tool set is chosen at random instead of by similarity: random selection performing equally would show that tool-space size, not relevance, drives the result.","supporting_citations":[{"cited_title":"Berkeley function calling leaderboard,","cited_arxiv_id":null,"evidence_quote":"BFCL: the general function-calling benchmark used to measure success rate and tool accuracy."},{"cited_title":"Geollm-engine: A realistic environ- ment for building geospatial copilots,","cited_arxiv_id":null,"evidence_quote":"GeoEngine: the sequential multi-tool benchmark used for application-specific evaluation."},{"cited_title":"Contextual retrieval,","cited_arxiv_id":null,"evidence_quote":"Anthropic contextual retrieval: the RAG-style inspiration for building latent-space tool search."},{"cited_title":"The faiss library,","cited_arxiv_id":null,"evidence_quote":"FAISS: the library used for the k-NN similarity search that retrieves top-k tools and clusters."}],"review_version":1}