{"id":"efbd50b2-a35a-4939-a7ad-f17929dbf359","arxiv_id":"2505.16303","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A routing framework that scores LLMs by their historical performance on capability and knowledge tags, then picks the best model per query, beating the best single model by 1.28 points on average across four benchmarks.","lead":"This paper proposes InferenceDynamics, a system that automatically chooses which large language model should answer a query by profiling each model's strengths in knowledge areas and reasoning skills. If it works as claimed, it would let applications mix and match AI models to get better answers at lower cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed OOD routing gain is not validated for truly novel knowledge: Eq. (1) relies on Index Set knowledge matches, while unseen knowledge is silently bucketed as 'Other', so the 1.28-point edge may reflect near-duplicate topics, not generalization.","rationale":"The reader's weakest assumption is the transfer from Index Set scores to OOD queries, and my analysis of §3.3–§4.2 makes this concrete. Equation (1) only has predictive value if the knowledge labels assigned to an unseen query correspond to knowledge elements for which the Index Set contains reliable, sufficiently frequent performance statistics. The paper's own preprocessing step (§4.2) removes infrequent and novel knowledge into 'Other', which is a silent failure mode: if a query is assigned to 'Other', the router cannot use the structured score at all. Since the paper never specifies what score 'Other' receives, the reader cannot determine whether the reported results are due to the proposed mechanism or to a generic fallback. The evaluation benchmarks are not cleanly OOD relative to the Index Set (MMLU-Pro vs MMLU; LiveBench math/coding vs MATH/BigCodeBench), so the observed edge could be explained by near-duplicate knowledge labels. This is the single most load-bearing concern because the paper's contribution is specifically 'generalization capabilities on previously unseen queries' (Section 1). If the router only works when queries share knowledge elements with the Index Set, the central claim is overstated. The proposed test—computing router accuracy on the 'Other' subset—settles this directly. I keep the reader's conditional verdict because the issue is testable and may be addressable with additional analysis, but the current evidence does not support a stronger claim.","tokens_in":17125,"tokens_out":7768,"duration_ms":62936,"concrete_test":"Partition each evaluation set by the provenance of the query's top-ranked knowledge label: 'matched' if any k_i has cosine similarity >0.6 to an Index Set knowledge entry, and 'Other' otherwise. Recompute Table 1 separately on the 'Other' subset. If InferenceDynamics does not outperform the best fixed single model on 'Other' queries, the transfer assumption fails and the headline 1.28 gain is attributable to near-duplicate knowledge rather than OOD generalization. Also report the size of the 'Other' subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract and §4.4) is that InferenceDynamics beats the best single LLM by 1.28 points on OOD benchmarks. This depends on Eq. (1): for an unseen query x, the knowledge score uses aggregate scores S^α_β(M_t, D, k_i) computed on Index Set queries containing knowledge element k_i. But §4.2 states that entries with frequency below 10 or previously unseen knowledge are classified as 'Other', and the paper never defines the score assigned to 'Other' or reports how often routing decisions use it. If a large share of evaluation queries have knowledge elements that do not match any Index Set entry above the 0.6 cosine threshold, the router degenerates to comparing generic averaged scores, and the reported advantage is not evidence of capability-based routing. The OOD claim is also weakened by benchmark overlap: MMLU-Pro is derived from MMLU, and LiveBench contains math/coding tasks similar to MATH and BigCodeBench in the Index Set. No validation on genuinely novel knowledge elements appears anywhere, and the Limitations section does not acknowledge this gap. Combined with the absence of any variance or significance estimate, the 1.28-point margin is not yet established as robust.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes InferenceDynamics, an LLM routing framework that profiles candidate models along capability and knowledge dimensions extracted by an auxiliary LLM (GPT-4o-mini). The method computes per-query, per-element performance scores on an Index Set of 20 benchmarks, aggregates them into knowledge and capability scores for each model, and routes each unseen query by the weighted combination in Eq. (2). The authors introduce the RouteMix dataset (Index Set plus four OOD evaluation benchmarks) and report that their Mixed Routing strategy beats the best single LLM by 1.28 points on average while using about 80% of the cost (or about half under a cost penalty). They also present a dynamic-pool scalability experiment and an analysis of router-selected models.","tokens_in":17481,"tokens_out":5969,"duration_ms":44893,"significance":"If fully substantiated, InferenceDynamics would be a useful addition to LLM routing: it is training-free, naturally accommodates new models, and its knowledge/capability decomposition is well motivated. The paper contributes RouteMix, a broad 24-dataset benchmark, and provides model-selection and dynamic-pool analyses that are informative. However, the current evaluation does not compare against any existing routing algorithm, lacks error bars or significance tests, and leaves a central mechanism (the 'Other' knowledge bucket) undefined. These gaps mean that the headline 1.28-point improvement over the best single model is not yet established as robust or as evidence of knowledge-based OOD generalization.","major_comments":[{"comment":"The treatment of 'Other' knowledge is never specified. §4.2 states that knowledge entries with frequency below 10 and previously unseen knowledge are classified as 'Other', but the paper does not define how Eq. (1) accounts for 'Other' nor report how often evaluation queries fall into this bucket. If 'Other' is simply dropped, then queries whose extracted knowledge is all unseen yield zero knowledge scores for every model, and routing is decided by the capability term alone. In that case the reported 1.28-point gain could reflect generic capability routing rather than knowledge-aware routing. Please define the score for 'Other' and report the coverage of matched knowledge elements on each evaluation benchmark.","section":"§4.2 / Eq. (1)"},{"comment":"The experiments compare InferenceDynamics only against single LLMs and random routing; no existing routing algorithm (RouteLLM, RouterDC, C2MAB-V, P2L, Model-SAT, or EmbedLLM) is used as a baseline. The related work motivates the novelty by contrast with these methods, but without a numerical comparison the claimed advantage over the state of the art in routing is unquantified. Additionally, every result appears to come from a single run, with no error bars, confidence intervals, or significance tests. The 1.28-point margin over Gemini-1.5-Pro could be within sampling variability, particularly on LiveBench where the reported score is an average over six categories.","section":"§4.4 / Table 1"},{"comment":"The label 'parameter-free' is inaccurate. The routing decision depends on the hyperparameters α, β, γ, δ, on the cosine similarity threshold 0.6 for knowledge consolidation, on the frequency cutoff 10 for the 'Other' bucket, and on the performance thresholds for translation/summarization listed in Appendix A. The paper does not describe how these values were chosen or whether a held-out validation set was used. If they were tuned with knowledge of the evaluation benchmarks, the out-of-distribution claim is weakened; if they are arbitrary defaults, their sensitivity should be shown through ablations.","section":"§4.1.1 / §4.2"},{"comment":"The claim of generalization to out-of-distribution queries is not validated for genuinely novel knowledge elements. The Index Set contains MMLU, MATH-500, GSM8K, and BigCodeBench, while the evaluation set includes MMLU-Pro and LiveBench with largely overlapping categories; GPQA and BigGenBench may also share topical structure. The paper provides no analysis of the fraction of evaluation queries whose extracted knowledge elements match the Index Set only through the 'Other' bucket or fall below the cosine threshold. Without such an analysis, the reported improvement could be driven by near-duplicate topics instead of transfer to new knowledge domains.","section":"§3.4 / §4.1.2"},{"comment":"The cost-constrained results are not reproducible as reported. The Cost Ratio is defined as total cost relative to the best-performing single LLM, but the paper does not specify the cost model, token counts, or API pricing used to compute it. The reader cannot determine whether the 'approximately half the budget' figure includes the router's knowledge/capability generation overhead or only the inference cost of the selected model. Please provide the exact cost computation and the numeric performance/cost values behind Fig. 3.","section":"§4.5 / Fig. 3"}],"minor_comments":[{"comment":"The list of capabilities in §3.2 includes seven items, but the generation prompt in Appendix D lists eight, adding 'In-context Learning' between Coding and Multilingual; these should be made consistent.","section":"§3.2 / Appx. D"},{"comment":"The x-axis of Figure 3 appears to show the β penalty coefficient, but the axis label and caption mention 'Increasing Tasks'; please correct the label to match the actual experimental variable.","section":"Fig. 3"},{"comment":"The 'Random' routing baseline is given as a single point estimate; reporting the mean and standard deviation over multiple random seeds would provide a reference scale for interpreting the 1.28-point improvement.","section":"Table 1"},{"comment":"The performance threshold for translation and summarization tasks is mentioned but never quantified; please state the actual threshold values or the rule used to determine correctness.","section":"Appx. A"},{"comment":"The figure caption contains 'inRouteMix' (missing space) and the label 'IDOOD' in the figure is unclear; rephrase the caption and clarify the ID/OOD split.","section":"Fig. 4"},{"comment":"The code availability statement says the source code 'will be made publicly available'; for a journal submission, an anonymous repository link or a review-stage artifact would allow the evaluation to be verified.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and proposes a conceptually clean, training-free routing mechanism, but the empirical evaluation falls short of the standard for routing papers in this venue. The undefined 'Other' knowledge bucket and the absence of any learned-router baseline are the most serious technical gaps. I would require the authors to (1) specify and analyze the 'Other' handling, (2) add at least standard routing baselines, (3) report variance or significance, and (4) clarify hyperparameter selection, before the central claim can be accepted. The manuscript's scope is appropriate for the journal, and the RouteMix dataset is a potentially useful contribution if the evaluation is brought up to standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate extension of the capability-profiling line of work (Model-SAT, Re-Task) with a useful new dataset, but the evaluation as written cannot support the headline 1.28-point margin. The core idea is simple: use an auxiliary LLM to tag queries with ranked knowledge and capability elements, aggregate per-model scores over an Index Set, and route by weighted sum. That is sensible, and the dynamic-pool result is a nice property.\n\nThe RouteMix dataset—20 benchmarks for indexing plus four for evaluation—is a real asset for the community. The rank-weighted aggregation formula is a reasonable way to combine multiple knowledge elements, and the paper is refreshingly clear about how new models can be added by evaluating on the Index Set.\n\nThe problems are in the experimental validation. Most importantly, there is no comparison to any existing router—not RouteLLM, not Model-SAT, not RouterDC. Without those baselines, we cannot tell whether this is better than prior work or just a new heuristic. Second, there are no error bars or significance tests. The average gain of 1.28 points comes from winning on GPQA and LiveBench and losing on MMLU-Pro and BigGenBench; with no variance estimates, that margin could easily be noise. Third, the paper calls the method parameter-free, but alpha, beta, gamma, delta, the cosine threshold, and the frequency cutoff are all hand-chosen. They should be tuned on a validation split, or at least acknowledged as hyperparameters.\n\nThe stress-test concern about the 'Other' bucket is valid. The paper says low-frequency and unseen knowledge elements become 'Other', but never defines the score assigned to 'Other'. If many evaluation queries hit this bucket, the knowledge score becomes a generic average and the routing advantage has little to do with knowledge profiling. The authors need to either report the fraction of queries routed through 'Other' or show that the knowledge scores are still discriminative. The OOD claim is also weakened by the fact that MMLU-Pro and LiveBench overlap with Index Set benchmarks.\n\nOverall, the idea is sound and the dataset is valuable, but the evidence needs a substantial upgrade. I would send it out to a serious referee, with the expectation of major revisions. If the authors add baselines, uncertainty quantification, and clarify the 'Other' handling, this could be a useful contribution to the routing literature.","headline":"Useful routing extension with a new dataset, but the evaluation lacks baselines and error bars, so the 1.28-point claim is not yet established.","tokens_in":17950,"tokens_out":4622,"would_cite":false,"duration_ms":34336,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A router that profiles each model's knowledge and capability beats every single candidate by 1.28 points on average across four held-out benchmarks while spending about 80% of the best model's budget.","keywords":["LLM routing","model selection","capability profiling","knowledge profiling","RouteMix","parameter-free routing","cost-performance trade-off","out-of-distribution generalization"],"falsifier":"Take a new out-of-distribution benchmark whose knowledge domains are absent from RouteMix, so that most queries fall into the 'Other' bucket; if InferenceDynamics's average score then drops to the level of random routing or below the best single model, the transfer-from-index-set assumption would be refuted.","tokens_in":16950,"feed_emoji":"🧠","tokens_out":8398,"duration_ms":60399,"temperature":0.7,"pith_summary":"InferenceDynamics is an LLM routing framework that claims to choose the best model for a query by profiling each model's knowledge and capability on a fixed 'Index Set', then matching unseen queries to those profiles. The paper's central result is that this profile-based router beats the strongest single model by 1.28 average points on MMLU-Pro, GPQA, BigGenBench, and LiveBench, without any router training, and at roughly 80% of the best model's cost under default settings. The authors' motivating concern is that the LLM ecosystem now contains many specialized models; if routing can be rebuilt cheaply as new models and new knowledge domains appear, users can exploit that specialization without retraining a router.","feed_headline":"No-training LLM router beats best single model by 1.28 points","feed_subtitle":"Profiling knowledge and capability on an index set lets it route unseen queries well at about 80% of the best model's cost.","key_machinery":"The central object is the ranked knowledge-capability score pair. For a model $M_t$ and an index query $x_i$, each element $e$ (a knowledge item or capability) receives a per-query score $S^\\alpha_\\beta(M_t, x_i, e) = \\sum_{j=1}^{|K_{x_i}|} (s^i_t - \\beta c^i_t)\\, \\mathbb{1}(e = k_j)\\, \\alpha^{j-1} / \\sum_{m=1}^{|K_{x_i}|} \\alpha^{m-1}$, where $s^i_t$ is the model's average evaluation score over $K$ trials and $c^i_t$ its average cost. These per-query scores are averaged over all Index Set queries containing $e$ to give $S^\\alpha_\\beta(M_t, D, e)$. For a new query $x$, the router forms a knowledge score $KS^\\alpha(M_t, x)$ and an analogous capability score $CS^\\alpha(M_t, x)$ by weighting the per-element aggregates with the query's ranked labels, then returns $\\arg\\max_{M_t} (\\gamma\\, KS^\\alpha(M_t, x) + \\delta\\, CS^\\alpha(M_t, x))$. The machinery is deliberately parameter-free: no router weights are learned, and the only tuned numbers are the hyperparameters $\\alpha$, $\\beta$, $\\gamma$, and $\\delta$, together with the label-generation, embedding-consolidation, and frequency-threshold steps.","core_discovery":"On its own terms, the paper claims that routing can be decided by a simple aggregation of historical scores. For each query, an auxiliary LLM labels the capabilities and knowledge domains required; each candidate model's scores on index queries carrying those labels are converted into per-label proficiency scores; and the router simply picks the model that maximizes a weighted sum of knowledge and capability scores. The claim is that this transfer works on out-of-distribution queries: across MMLU-Pro, GPQA, BigGenBench, and LiveBench, the mixed router averaged 74.55, which is 1.28 points above the best single candidate, and under a cost penalty it stayed near the best model's performance while spending about half as much.","pith_inferences":["An implication the paper leaves implicit is that the same Index Set profiling could be published once per model, letting many downstream routers reuse a shared capability map without re-running inference.","Because rare knowledge is folded into an 'Other' bucket, the claim is really about label reuse; a stress test with adversarial queries whose knowledge is genuinely novel would reveal how much of the 1.28-point margin survives without any matching index entries.","The router's sensitivity to the auxiliary labeler could be probed cheaply by corrupting a fraction of generated labels; the margin should shrink smoothly with label noise if the transfer assumption holds.","The ranked-weighting step is a soft attention over capability and knowledge labels; a hybrid that combines this profile-based score with a small learned router could preserve fast adaptation while closing any remaining gap to trained baselines."],"forward_implications":["A new LLM can be added to the candidate pool by scoring it on the Index Set; the router itself needs no retraining.","New knowledge domains can be incorporated by computing a model's aggregate score for that domain and folding it into future routing decisions.","Routing by knowledge alone performs best on knowledge-intensive benchmarks (top on GPQA), while routing by capability alone performs best on capability-driven tasks (top on BigGenBench); the mixed strategy combines both advantages.","Applying a cost penalty barely degrades average performance while cutting spending to roughly half the best single model's budget, with text-generation tasks more sensitive to the penalty than question-answering tasks."],"supporting_citations":[{"why":"Supplies the fixed capability taxonomy (reasoning, comprehension, instruction following, agentic, knowledge retrieval, coding, multilingual) used to label each query.","marker":"(Minaee et al., 2024)"},{"why":"Provides MiniLM-L6 embeddings, used to consolidate semantically similar knowledge entries with cosine similarity above 0.6.","marker":"(Wang et al., 2020)"},{"why":"RouteLLM is the binary-classifier routing approach the paper positions against when motivating multi-model scalability.","marker":"(Ong et al., 2025)"},{"why":"RouterDC is a learned multi-LLM router whose retraining requirement InferenceDynamics seeks to remove.","marker":"(Chen et al., 2024)"},{"why":"Model-SAT's human-predefined capability sets are the limitation that motivates InferenceDynamics's open knowledge space.","marker":"(Zhang et al., 2025)"},{"why":"Supplies the GPT-4o-mini LLM-as-judge protocol and threshold (score above 4) used to score BigGenBench.","marker":"(Sprague et al., 2025)"},{"why":"GPQA is one of the four out-of-distribution evaluation benchmarks used to measure routing performance.","marker":"(Rein et al., 2023)"},{"why":"MMLU-Pro is one of the four out-of-distribution evaluation benchmarks used to measure routing performance.","marker":"(Wang et al., 2024b)"},{"why":"BigGenBench is one of the four out-of-distribution evaluation benchmarks used to measure routing performance.","marker":"(Kim et al., 2024)"},{"why":"LiveBench is one of the four out-of-distribution evaluation benchmarks used to measure routing performance.","marker":"(White et al., 2025)"}],"fun_headline_variants":["No-training LLM router beats best single model by 1.28 points","Profile-based LLM routing outdoes top model without extra training","LLM router scores 74.55, +1.28 over best single model","Cheap LLM routing via capability profiling beats all single models","Routing LLMs without retraining gains 1.28 points over best"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The router assumes that a model's past scores on index queries carrying the same predicted knowledge or capability labels predict its score on a brand-new query, even when those labels come from a smaller auxiliary model and rare knowledge is lumped into an 'Other' bucket.","fun_headline_variants_meta":{"raw":{"variants":["No-training LLM router beats best single model by 1.28 points","Profile-based LLM routing outdoes top model without extra training","LLM router scores 74.55, +1.28 over best single model","Cheap LLM routing via capability profiling beats all single models","Routing LLMs without retraining gains 1.28 points over best"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00085,"raw_usage":{"total_tokens":3662,"prompt_tokens":876,"completion_tokens":2786,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":2690}},"tokens_in":492,"tokens_out":2786,"duration_ms":15766,"temperature":1.0,"reasoning_tokens":2690,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:03:33.456797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new out-of-distribution benchmark whose knowledge domains are absent from RouteMix, so that most queries fall into the 'Other' bucket; if InferenceDynamics's average score then drops to the level of random routing or below the best single model, the transfer-from-index-set assumption would be refuted.","supporting_citations":[],"review_version":1}