{"id":"936fa7a1-ce7d-48f6-b2ba-7fc3af25332c","arxiv_id":"2607.18098","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"VDAR-Router improves LLM routing by retrieving past queries with similar verbalized difficulty descriptions and ranking models by performance minus cost.","lead":"VDAR-Router asks a language model to write a short “difficulty analysis” of each user query, then retrieves historical queries with similar difficulty descriptions to pick which LLM should answer. In tests on three routing benchmarks, it reports better cost-performance trade-offs than current routing methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cost-performance trade-off claim omits VDAR's own difficulty-analysis overhead, so the reported Reward may not reflect end-to-end deployment cost.","rationale":"After reading the paper and the reader's verdict, I found the most load-bearing issue to be not the ranking-transfer assumption (which the reader identified) but the exclusion of VDAR's own router-side cost from the reported cost-performance trade-off. The paper's Section 4.3 explicitly defines Total Cost as the cost of the selected model only. VDAR's test-time pipeline includes an LLM-based Difficulty Analyst; this consumes tokens and time. Baselines like RouteLLM and RouterDC have tiny router overhead, so comparing only selected-model costs is unfair. This directly threatens the central empirical claim. The reader's ranking-transfer concern is important but partially addressed by the ablation (Table 4) showing difficulty retrieval helps; even if the mechanism is not fully proven, the empirical routing improvement could still hold. In contrast, the cost omission affects the primary metric on all three datasets. I propose a concrete test: add the difficulty-analysis token cost to the Total Cost and recompute Reward. If the conclusion changes, the paper must be revised or rejected; if not, the claim stands. Therefore, I agree with the reader's CONDITIONAL verdict, but for a different, more concrete reason.","tokens_in":19534,"tokens_out":12654,"duration_ms":118088,"concrete_test":"Instrument the released code to log the number of prompt and completion tokens consumed by the Difficulty Analyst (system prompt + query + generated analysis) for every test query in RouterBench, LLMRouterBench, and ArenaExpert5K. Price these tokens using the same OpenRouter per-token rates used for candidate models (Appendix B.3), sum them with the selected model's cost, and recompute the Total Cost and Reward (with the same min-max normalization) in Tables 1–3 for VDAR and for baselines (including any router-side costs they incur). If VDAR no longer achieves the best Reward/ranking on all three datasets, the central claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that VDAR-Router consistently achieves better cost-performance trade-offs than baselines. However, the cost measured in Section 4.3 and Tables 1–3 is defined as 'total inference cost incurred by the selected model for each routing decision' — explicitly excluding the router's own overhead. At test time, VDAR must run a Difficulty Analyst (Gemma4-31B or Qwen3.5-2B) to generate a lengthy verbalized analysis (see Appendix F) and then embed/retrieve over it. This overhead is non-zero and is not included in Eq. (2)'s cost term, nor in the reported Total Cost. In contrast, baselines such as RouteLLM, RouterDC, and KNN have negligible router-side inference. The omission is most consequential on ArenaExpert5K, where VDAR's selected-model cost is 0.0006 per query; a single call to a 31B analyst could cost more than that. The paper's Limitations mention additional latency but not additional cost, so this measurement gap is unaddressed. If the analyst's token cost is added, VDAR's Reward and Total Cost could shift enough to overturn the claimed advantage on one or more datasets.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VDAR-Router, a training-free retrieval-based LLM routing framework. For each input query, a Difficulty Analyst LLM generates a verbalized difficulty analysis over seven capability dimensions; the analysis is embedded and used to retrieve top-k historical queries with similar difficulty profiles. The router then estimates candidate-model suitability from the retrieved queries' performance and cost and selects the model maximizing a linear reward R = (1/k) Σ (α·p − β·c). The method is evaluated on RouterBench, LLMRouterBench, and ArenaExpert5K, reporting improved Reward, Pairwise Accuracy, and Spearman Correlation over KNN, RouteLLM, RouterDC, ICL-Router, and IRT-Router, together with ablations and a Rasch-based difficulty-alignment analysis.","tokens_in":19809,"tokens_out":6708,"duration_ms":78745,"significance":"The core idea — retrieving on verbalized difficulty analysis rather than raw query embeddings — is original and practically relevant, and the paper ships a reproducible experimental setup (seed 42, greedy decoding, public code), ablations that isolate reward ranking and difficulty retrieval, and an external Rasch check of difficulty alignment. If the reported cost-performance advantage holds end-to-end, the method would be a useful plug-and-play router. However, the central empirical claim is currently not supported as stated because the reported cost excludes the router's own inference overhead, and the mechanistic assumption that difficulty-similar queries induce similar model capability rankings is not directly validated. Both concerns are addressable with additional analysis, so the contribution has clear potential but needs substantive revision.","major_comments":[{"comment":"The cost term in Eq. (2) and the reported Total Cost measure only 'total inference cost incurred by the selected model for each routing decision', explicitly excluding the router's own overhead. At test time VDAR-Router runs a Difficulty Analyst LLM (Gemma4-31B for RouterBench/LLMRouterBench; Qwen3.5-4B/2B for ArenaExpert5K) and an embedding model for every query. On ArenaExpert5K, Table 2 reports a selected-model Total Cost of 0.0006 for VDAR-Router (Qwen3.5-4B); a single analyst call is likely to cost more than this, so the reported Reward and Total Cost do not reflect true deployment cost. The Limitations section acknowledges additional latency but not additional cost. The authors should either include analyst+embedding cost in Total Cost, or report it separately and recompute the cost-performance comparison. This is load-bearing for the headline 'better cost-performance trade-off' cl","section":"§4.3 and Eq. (2)"},{"comment":"The method's mechanism rests on the assumption, stated in §5.2 and repeated in the Limitations, that 'queries with similar difficulty profiles tend to induce similar model capability rankings.' RQ4 (Figure 5) validates only that retrieved queries have numerically closer Rasch difficulty than query-embedding or random retrieval; it never tests whether model performance rankings actually transfer from retrieved neighbors to the test query. Because RouterBench and LLMRouterBench contain full per-query performance labels for all candidate models, the authors can compute, for each test query, the true model ranking and compare it with the ranking predicted from retrieved neighbors' performance (e.g., Spearman correlation), and contrast this against query-embedding retrieval. Without such a test, the retrieved reward estimates could be misleading when two queries share difficulty but require d","section":"§5.2, §5.4, Limitations"},{"comment":"All experiments appear to be a single run: one 80/20 split, temperature 0, seed 42. No error bars, confidence intervals, or significance tests are reported. Several margins are small — for example, in Table 1 LLMRouterBench at α=0.8, VDAR Reward is 42.58 vs. RouteLLM's 41.11, and at α=0.6 it is 31.14 vs. 30.34. The claim that VDAR-Router 'consistently achieves' better trade-offs would be strengthened by multiple seeds/splits or bootstrap CIs on reward differences, especially given the datasets are sampled subsets.","section":"Tables 1–3"}],"minor_comments":[{"comment":"The phrase 'cost ... linearly normalized to [0,1]' is ambiguous. Please specify whether the normalization is per-query across candidate models, per-model across queries, or global min-max, and whether the bounds are computed on the training set. This affects the interpretation of the reported Reward values.","section":"Eq. (2)"},{"comment":"The Cost column for ArenaExpert5K is not clearly labeled — state whether these are total costs across the test set or per-query averages, and specify the currency/unit (e.g., USD per 1K tokens).","section":"Table 2"},{"comment":"The fallback price of gpt-4o-mini is used whenever an ArenaExpert5K model name cannot be mapped to OpenRouter or has no price. Report how many of the 105 candidate models rely on this fallback; if the fraction is large, the Total Cost comparisons may be skewed.","section":"Appendix B.1"},{"comment":"With six values of k plotted on the same axes, the lines are difficult to distinguish. A small-multiples layout or added error bars would improve readability; also note that the y-axis of panel (c) starts at 45, which visually exaggerates differences.","section":"Figure 3"},{"comment":"The phrase 'one of the oracle models for this query' is undefined. Clarify what 'oracle' means here (e.g., a model with the highest true performance on that query in the dataset).","section":"§5.3 / Appendix F"},{"comment":"The sentence 'future research will be further explored to mitigate this test-time inference cost' is ungrammatical. More substantively, the Limitations should explicitly acknowledge that test-time analyst cost is excluded from the reported Total Cost metric, not just latency.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The cost-accounting gap is the main risk to the paper's central claim. If, after including the Difficulty Analyst and embedding overhead, VDAR-Router no longer lies on the Pareto frontier of cost vs. performance on the three benchmarks, the contribution would be substantially weakened. The missing ranking-transfer analysis is also important but is directly testable with the datasets already used. I recommend major revision rather than rejection because both issues are fixable within the manuscript's scope and the core retrieval idea has merit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely new and useful: instead of retrieving by query embedding, VDAR-Router has an LLM write a verbalized difficulty analysis and retrieves historical examples by that analysis's embedding. That is a real departure from IRT-Router and KNN, and the case study shows it finds queries that share the underlying skill demand rather than surface form. Second, the paper's central cost-performance claim is undermined by a measurement choice: the reported Total Cost excludes the difficulty analyst's own inference cost. The paper's Limitations mention extra latency but not extra cost. That's a problem because on the datasets where VDAR beats baselines on Reward, the analyst call is part of the deployed system's cost. On ArenaExpert5K, the selected model costs are tiny (0.0006 per query), and a 4B analyst call likely costs more than that; on RouterBench the analyst is Gemma4-31B. The comparison is therefore not end-to-end.\n\nWhat's good: the experiments are on three public benchmarks, the ablations isolate the difficulty-analysis retrieval and reward reranking, and the Rasch-based validation shows retrieved queries are closer in estimated difficulty than query-embedding retrieval. The ArenaExpert5K results, especially Spearman 0.51, are strong and not cost-dependent. The paper is honest about limitations.\n\nSoft spots beyond cost: no error bars or significance tests (single run, temperature 0), so \"consistently\" is doing a lot of work. The reward's cost term uses a linear normalization to [0,1] but the bounds are not fully specified, and the fallback price for missing ArenaExpert5K models is a modeling choice. The load-bearing assumption - difficulty-similar queries induce similar model rankings - is only indirectly validated via difficulty alignment, not directly via ranking transfer. The authors acknowledge this. These are addressable.\n\nVerdict: for a reader working on LLM routing, this is worth reading and discussing. The difficulty-analysis retrieval innovation deserves to be cited. It should go to peer review, but the authors should be asked to report end-to-end cost and ideally some variance estimates. If they can't, the cost-performance claim should be softened.","headline":"The difficulty-analysis retrieval idea is real and the routing results look good, but the reported costs omit VDAR's own analyst overhead, so the headline cost-performance claim needs a caveat.","tokens_in":20300,"tokens_out":2472,"would_cite":true,"duration_ms":362633,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM routing should be driven by a verbalized analysis of query difficulty rather than by surface similarity, and shows that a training-free router built on this idea achieves better cost-performance trade-offs than ex","keywords":["LLM routing","query difficulty","verbalized difficulty analysis","retrieval-based routing","cost-performance trade-off","model selection","training-free router","Rasch difficulty"],"falsifier":"Collect pairs of queries whose difficulty analyses are near-identical but which require disjoint skills (for instance, one coding, one factual recall). If the Spearman correlation between model accuracy rankings within those pairs is at or below zero, then difficulty-similar retrieval is carrying no routing-relevant signal.","tokens_in":19410,"feed_emoji":"🧭","tokens_out":5725,"duration_ms":62958,"temperature":0.7,"pith_summary":"The paper tries to establish that the right representation for routing a query to an LLM is not the query's surface text but a verbalized analysis of what makes it difficult. The proposed router first prompts an LLM to write a structured difficulty analysis across seven capability dimensions, embeds that analysis, retrieves historical queries with similar difficulty profiles, and then ranks candidate models by a reward that mixes retrieved performance and cost. Across three routing benchmarks, this difficulty-aware retrieval achieves better cost-performance trade-offs than query-embedding retrieval, learned routers, and IRT-based difficulty scoring, and it improves pairwise preference prediction on a 105-model leaderboard-style set. The paper supports its core assumption by showing that retrieved queries are closer in Rasch-estimated difficulty than query-embedding neighbors, and by a case study in which difficulty retrieval selects a correct oracle model while surface retrieval fails. If correct, the result implies that interpretable difficulty metadata can replace some amount of training and supervision in model selection.","feed_headline":"Analyzed difficulty beats raw text for routing LLM queries","feed_subtitle":"A training-free router retrieves past queries with similar difficulty profiles, cutting cost while keeping accuracy.","key_machinery":"The Difficulty Analyst—a prompted LLM that writes a structured prose analysis of the capabilities required and their difficulty levels across seven dimensions (reasoning, comprehension, instruction following, agentic, knowledge retrieval, coding, multilingual)—is the central object. Its output is embedded and used for top-k retrieval over a database of historical analyses; the retrieved examples' performance records are averaged and combined with normalized cost through a weighted reward. The same analyst is also used at test time, so retrieval and evidence live in the same representation space.","core_discovery":"VDAR-Router's central claim is that queries that are difficult in the same way should be answered with similar relative quality by a given set of LLMs, so historical performance on difficulty-similar queries is the right evidence for routing. The paper operationalizes this by having an LLM 'Difficulty Analyst' produce a free-text analysis of the required capabilities (reasoning, comprehension, instruction following, agentic, knowledge retrieval, coding, multilingual), embedding that analysis, and retrieving the top-k most similar analyses from a training set. It then scores each candidate model by the average retrieved performance penalized by normalized cost, with the α/β weight controlling","pith_inferences":["If the difficulty-analysis representation is the true carrier of the signal, then the same database should transfer across tasks and benchmarks without rebuilding, which would make routing infrastructure much cheaper to maintain.","A clean test of the core assumption would be to compare retrieved neighbors against neighbors selected by a multidimensional IRT model with separate ability parameters per skill dimension; if the verbal analysis adds nothing beyond the scalar Rasch difficulty, the routing gains should vanish.","The router inherits any blind spots of the analyst model: if the analyst systematically misjudges a category of queries, retrieval will retrieve similarly misjudged examples, so the framework's ceiling is set by the analyst's own calibration.","For preference-based data, adding a response-style or preference embedding alongside the difficulty analysis could fix the non-monotonic behavior as retrieval size grows, since the paper notes that difficulty-aligned neighborhoods are preference-heterogeneous."],"forward_implications":["A router can be deployed with no additional training: only a small language model that writes difficulty analyses and a database of past analyses and outcomes.","Queries that look different but are difficult in the same way are grouped together, while queries that look alike but need different capabilities are separated, so routing evidence transfers across topical boundaries.","The alpha/beta knob gives a direct, interpretable control over the performance-cost trade-off, so operators can tune a single routing policy instead of retraining.","Relative model rankings can be recovered even when supervision is only pairwise human preference, as in ArenaExpert5K, not full per-model scores.","Difficulty-similar neighbors stay aligned even when the training pool is halved, meaning the approach degrades gracefully with less historical data."],"fun_headline_variants":["Difficulty-based retrieval outperforms text similarity for LLM routing","Route LLMs by explicit difficulty analysis, not raw text","VDAR-Router: difficulty profiles drive cost-aware model selection","Explicit difficulty analysis yields better cost-performance in routing","Retrieval over difficulty analyses cuts LLM routing costs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes that two queries described as equally difficult will have similar relative quality rankings across candidate models; the paper validates difficulty closeness but never directly tests that rankings transfer.","fun_headline_variants_meta":{"raw":{"variants":["Difficulty-based retrieval outperforms text similarity for LLM routing","Route LLMs by explicit difficulty analysis, not raw text","VDAR-Router: difficulty profiles drive cost-aware model selection","Explicit difficulty analysis yields better cost-performance in routing","Retrieval over difficulty analyses cuts LLM routing costs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000654,"raw_usage":{"total_tokens":2814,"prompt_tokens":704,"completion_tokens":2110,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":2029}},"tokens_in":448,"tokens_out":2110,"duration_ms":13499,"temperature":1.0,"reasoning_tokens":2029,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T15:58:57.467389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect pairs of queries whose difficulty analyses are near-identical but which require disjoint skills (for instance, one coding, one factual recall). If the Spearman correlation between model accuracy rankings within those pairs is at or below zero, then difficulty-similar retrieval is carrying no routing-relevant signal.","supporting_citations":[],"review_version":1}