{"id":"4e75d610-0746-4282-90c7-4c2f86cbb07e","arxiv_id":"2509.09782","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A single-head cross-attention router that jointly encodes the query and each candidate model improves cost-quality trade-offs on RouterBench by a few percent over KNN, MLP, and SVM baselines, though the headline gains are smaller than the abstract suggests.","lead":"This paper builds a router that reads a user question, scores each available large language model with a cross-attention step, and sends the question to the model that gives the best predicted quality for its price. On the RouterBench benchmark it reports modest gains over simple routers, but it does not compare against the strongest published routers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attention router's advantage is not statistically supported: it beats the embedding-augmented regression baseline by 0.66% AIQ with no error bars, and Pool 2's gain over MLP is 0.00022.","rationale":"The reader's conditional verdict is appropriate. The weakest point is not the absence of SOTA baselines per se, but that the uniquely claimed component—cross-attention—is evaluated against a regression baseline that already uses the same model embeddings, and the gap is tiny. If that gap is within noise, the paper's main architectural contribution is not supported, regardless of KNN/SVM margins. The missing comparisons with Universal Model Routing and RouteLLM (Limitations F) make the headline claim unverifiable at the level of 'new standard.' I therefore keep the conditional verdict and ask for a statistical significance test as a precondition. Agreement with the reader is partial: the reader emphasizes response uncertainty, while I emphasize the small, unquantified attention-versus-Reg-emb gap, which the same repeated-sampling experiment would also address.","tokens_in":9599,"tokens_out":8630,"duration_ms":76436,"concrete_test":"Reproduce the R2 ablation on LLM Pool 1 and Pool 2 using the same 75/5/20 splits and the exact Reg-emb and Attn predictors, over 10 random seeds; if API budget permits, sample at least 3 responses per model per prompt, otherwise bootstrap over test prompts. Report the 95% confidence interval for Δ = AIQ(Attn) − AIQ(Reg-emb) and for AIQ(Attn) − AIQ(MLP) in Pool 2. If either interval includes zero, or the pooled mean Δ is below 0.5 percentage points, the claim that cross-attention specifically drives the improvement is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To establish the central claim, it must be shown that the single-head cross-attention mechanism, not the augmented model embeddings or random noise, produces the reported routing gains. Under the paper's own ablation (Appendix C, Table 5, R2 rewards, LLM Pool 1), the full attention model achieves AIQ 0.72737 while the regression-with-embeddings baseline achieves 0.72258, a 0.66% difference. In Table 2, Pool 2 attention is 0.66586 versus MLP at 0.66564, a 0.03% difference. No confidence intervals, seeds, or repeated trials are reported, so these margins are indistinguishable from noise. The large margins over KNN and SVM are not evidence in favor of cross-attention because those baselines lack model embeddings and underperform on Pools 2 and 3. The abstract's 'up to 6.6%' and '2.9%' claims are not directly traceable to the tables, and Limitations F concedes that no comparison with Universal Model Routing or other recent routers is made, so 'new standard' is unsupported. The same Limitation F acknowledges response uncertainty, which could further shrink the measured advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a cost-aware LLM routing framework in which a single-head cross-attention block jointly represents a query embedding and fixed-size model embeddings, predicts per-model response quality and generation cost, and then selects a model by maximizing an exponential reward R2 = s*exp(-c/lambda). The method is evaluated on RouterBench LLM pools 1-3 against KNN, MLP, SVM, and a self-implemented LLM-Blender baseline, with additional ablations in Appendix C comparing regression, fully connected, and embedding-augmented predictors. The paper claims up to 6.6% AIQ improvement and 2.9% improvement in maximum performance, and argues that the exponential reward is less sensitive to the user preference parameter lambda than the linear reward R1 = s - c/lambda.","tokens_in":9882,"tokens_out":5465,"duration_ms":427467,"significance":"If the reported gains are real and reproducible, the architecture is a useful lightweight addition to LLM routing: it decouples model representation construction from predictor training, uses a public benchmark, and includes a broad ablation study. The paper is also transparent in stating its major limitations in Appendix F. However, the empirical evidence for the central claim is not yet sufficient: the gain over the strongest relevant baseline is small, no statistical uncertainty is reported, the headline improvement numbers are not traceable to the tables, and recent state-of-the-art routers are not compared. As presented, the paper supports only a narrow claim that a particular attention configuration performs well on RouterBench pools, not the broader 'new standard' claim in the abstract.","major_comments":[{"comment":"The claimed improvements are not reproducible from the reported numbers. The abstract and §6 state 'up to 6.6% in AIQ' and '2.9% in maximum performance' improvements. In Appendix C, Table 5 (R2, LLM Pool 1), the full attention model achieves AIQ 0.72737 while the regression-with-embeddings baseline achieves 0.72258, which is only a 0.66% relative gain. Table 6 shows attention reaching maximum performance 0.78082, which is not above the 0.78337 achieved by Reg-emb. I could not locate any table where a 6.6% AIQ or 2.9% PerfMax advantage appears. Please identify the exact comparison behind these numbers and recompute them, or correct the abstract and Section 6.","section":"Abstract; §6; Appendix C Tables 5–6"},{"comment":"The central empirical claim is not statistically supported. The paper reports a single 75%-5%-20% split with no confidence intervals, no multiple seeds, and no repeated trials. The margin of the attention router over the best comparable baseline is tiny: in Table 2, Pool 2 attention AIQ 0.66586 versus MLP 0.66564 is a 0.00022 difference, and in Table 5 the attention router beats Reg-emb by 0.00479 AIQ in Pool 1. Given the noise in LLM response quality, these differences are indistinguishable from random variation. To support the claim that cross-attention improves routing, the authors should report mean and standard deviation over at least several training seeds and, ideally, a paired significance test.","section":"§5 Training and Test Details; Table 2; Appendix C Table 5"},{"comment":"The comparison does not isolate the contribution of the cross-attention mechanism. The main baselines (KNN, MLP, SVM) do not receive model embeddings, so their large performance gaps are not evidence for cross-attention over a similarly expressive embedding-based predictor. The relevant comparison is attention versus embedding-augmented regression and MLP in Appendix C, where the advantages are much smaller. I therefore ask for an ablation that removes or replaces only the attention module while keeping model embeddings and model capacity fixed, and also for an attention variant without model embeddings, so that the reader can attribute the reported gains to query-model interaction rather than to the augmented input representation.","section":"§3 Model Representations Augmentation; §6; Appendix C"},{"comment":"The paper explicitly acknowledges that it does not compare with Universal Model Routing or other recent multi-LLM routing methods on RouterBench, yet the abstract claims that the approach 'establishes a new standard for cost-aware LLM routing.' This claim is unsupported without comparison to state-of-the-art routers such as UMR, RouteLLM, or CARROT. In addition, the acknowledged single-response uncertainty in RouterBench directly affects the construction of the model embeddings (cluster-mean performance in §5) and the reported cost-performance trade-offs. The authors should either add comparisons and a sensitivity analysis on response sampling and cluster count, or substantially weaken the novelty and generality claims.","section":"Appendix F Limitations; Abstract"}],"minor_comments":[{"comment":"The fragment 'QX1 =S∈R n×q QX2 =C∈R n×q' appears to be a formatting artifact and should be removed or rewritten as a proper definition of the regression output.","section":"§3 Predictor Variants"},{"comment":"The definition of lambda-sensitivity should specify the base of the logarithm and clarify why only adjacent lambda values are weighted, since the text calls this a weighted average over the log-scale difference.","section":"Eq. (2)"},{"comment":"The row label 'Attn-eval' in Tables 5 and 6 is inconsistent with the column label 'Attn' and with the label 'Attn' used in Tables 3 and 4. Please unify the notation and explicitly state which predictor is the quality predictor and which is the cost predictor for the reported 'attention router' results.","section":"Appendix C Tables 3–6"},{"comment":"The LLM-Blender baseline is reported only for AIQ and not for PerfMax, and its cost is computed as the sum over all candidate models. This makes the comparison difficult to interpret; please report the same metrics for all baselines or explain the omission.","section":"Table 2 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision rather than reject: the method is plausible and the ablation study is extensive, but the empirical evidence for the cross-attention advantage must be strengthened with statistical significance testing, reproducible numbers, and at least one comparison to a recent routing method. I would also ask the editor to require the authors to reconcile the abstract claims with the actual table values."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline on this one is not the router; it's the reward function. The exponential R2 = s*exp(-c/λ) is a genuine idea, and the oracle analysis in Table 1 shows it is clearly less sensitive to λ than the linear R1. That part is well done. The cross-attention router itself is a modest architectural step: query embeddings attend over cluster-derived model embeddings, which extends Universal Model Routing's embedding construction. It is cheap to train and the ablation across regression, MLP, and attention variants is thorough.\n\nWhat I can't defend is the empirical claim that the attention mechanism is what's working. Table 2 shows the attention router beats MLP by 0.03% on pool 2 and by a few percent on pools 1 and 3. The larger margins over KNN and SVM don't help because those baselines don't get model embeddings at all. The ablation in Table 5 gives the cleaner comparison: attention at 0.72737 versus regression-with-embeddings at 0.72258, a 0.66% gap. With no confidence intervals, seeds, or repeated trials, that gap is indistinguishable from noise. The abstract's 'up to 6.6%' and '2.9%' claims also do not line up with the tables as reported; the numbers appear to come from a selective reading of the ablation.\n\nThe paper is honest about its biggest missing piece: Limitation F concedes no comparison with Universal Model Routing or other recent routers, and it acknowledges response uncertainty. Given that, 'establishing a new standard' in the abstract is overreach. There is also no code release, which makes the small-effect claims even harder to verify.\n\nWhere does this leave the paper? It is a reasonable incremental contribution with one solid idea (R2) and a plausible but unproven architecture. For a practitioner who wants a lightweight router and is willing to re-implement, the method is simple enough to try. I would cite it for the exponential reward, not for the routing result.\n\nMy recommendation: send it to peer review, but set the bar for revision high. The authors need to add strong baselines with the same embeddings, report variance across seeds, fix the headline numbers to match the tables, and likely release code. If those revisions come, the paper is a solid workshop-or-conference-level contribution. As it stands, the central claim is not statistically supported.","headline":"The exponential reward is the real contribution; the cross-attention router's advantage over same-embedding regression is not statistically demonstrated.","tokens_in":10417,"tokens_out":2608,"would_cite":true,"duration_ms":21487,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single-head cross-attention router, which scores each query against each candidate model, improves cost-quality trade-offs over existing routers on RouterBench.","keywords":["LLM routing","cross-attention","cost-aware model selection","RouterBench","exponential reward","query-model interaction","predictor-based routing"],"falsifier":"Re-run the RouterBench comparison with several sampled responses per model per prompt, scoring quality by expected or majority performance: if the attention router's AIQ advantage over KNN, MLP, and SVM shrinks or reverses, the reported gains are an artifact of single-response noise.","tokens_in":9430,"feed_emoji":"🤖","tokens_out":8537,"duration_ms":74130,"temperature":0.7,"pith_summary":"This paper tries to establish that the best way to route a query to one of several language models is to score the query against each model, rather than classify the query on its own. It proposes a lightweight router whose two predictors, one for response quality and one for generation cost, are built from a single-head cross-attention layer; the layer uses the query embedding as the attention query and a fixed per-model embedding as key and value. Evaluated on RouterBench, this attention router achieves higher cost-quality area (AIQ) than KNN, MLP, and SVM routers on all three model pools, and the paper argues that an exponential reward $s\\exp(-c/\\lambda)$ balances quality and cost more stably than the linear reward $s-c/\\lambda$. If the claim holds, it offers a cheap, adaptable component for LLM deployment that can absorb new models without retraining the whole predictor.","feed_headline":"Attention router beats KNN, MLP, and SVM at cost-aware LLM routing","feed_subtitle":"A single-head cross-attention scorer for query–model pairs lifts cost-quality trade-offs on RouterBench.","key_machinery":"The central object is a single-head cross-attention block $$\\mathrm{Attention}(q,k,v)=\\mathrm{softmax}\\!\\left(q k^\\top/\\sqrt{d_v}\\right)v,$$ applied with the prompt embedding as query $q$ and a model embedding as both key and value, producing a scalar estimate of that model's quality or cost for that prompt. The prompt side is a fixed 768-dimensional encoder output; the model side is a 20-dimensional 'expertise profile' formed by K-means clustering training prompts and averaging each model's score within each cluster. The same attention block is trained twice, once for quality and once for cost, and routing combines the two predictions through $R_2=s\\exp(-c/\\lambda)$, where $\\lambda$ is the user's willingness to pay. Decoupling the model embedding from training is what lets new models enter the pool by computing their embedding rather than retraining.","core_discovery":"The paper's central claim is that explicitly modeling query–model interaction with cross-attention improves cost-aware routing. In the reported experiments, the attention router reaches AIQ 0.72737, 0.66586, and 0.74439 on pools 1 through 3, beating KNN (0.70608, 0.49338, 0.55727), MLP (0.67598, 0.66564, 0.72655), and SVM (0.70220, 0.51452, 0.49760). The reason offered is that attention is a second-order similarity: it lets the router estimate how well a specific model's expertise profile matches a specific prompt's difficulty, something query-only classifiers cannot see. The paper also proposes $R_2=s\\exp(-c/\\lambda)$ as the reward for combining predicted quality and cost; an oracle analysis shows this reward has much lower $\\lambda$-sensitivity in both performance and cost than $R_1=s-c/\\lambda$, meaning the chosen trade-off stays stable under small changes in user preference. Together, these two pieces, interaction-based prediction and a bounded exponential reward, are what the paper claims make routing both more accurate and more robust.","pith_inferences":["The paper leaves implicit that its single-response quality labels are the main uncertainty: an obvious extension is to train the same cross-attention predictors on multiple sampled responses per model and prompt, and re-run the headline AIQ comparison under that uncertainty.","The formulation does not depend on RouterBench specifically; the same cross-attention block could route based on latency or token-level pricing in self-hosted serving, where generation cost is not a fixed API price.","A testable consequence of the mechanism's value is whether the learned attention scores themselves track per-domain model strength; if they do, the embeddings could double as a diagnostic for when to add a model to the pool."],"forward_implications":["Because the model embedding is separated from predictor training, adding or removing a model from the pool requires only computing the new embedding, not retraining the router.","The lower $\\lambda$-sensitivity of the exponential reward means the router's performance and cost stay nearly flat when the user's willingness-to-pay parameter is adjusted slightly, which simplifies tuning in deployment.","The attention router's largest gains over KNN and SVM appear in pools whose models are similar in performance, suggesting interaction-based scoring helps most when cheap models can handle many prompts.","Because quality and cost are predicted before generation, the router can enforce a spending cap while choosing the best model under that cap at request time."],"supporting_citations":[{"why":"It supplies the RouterBench dataset, the oracle router, the KNN/MLP/SVM baseline implementations, and the AIQ metric.","marker":"[15]"},{"why":"It supplies the clustering-based model embedding construction used to create fixed-size LLM expertise profiles.","marker":"[13]"},{"why":"It supplies the ranker used to implement the LLM-Blender baseline at test time.","marker":"[16]"}],"fun_headline_variants":["Attention router beats KNN, MLP, SVM on cost-quality LLM routing","Second-order query-model interaction improves LLM routing quality","Cross-attention router pairs queries with optimal LLMs per cost","Exponential reward stabilizes cost-quality trade-off in LLM routing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The router's ground-truth quality signal is a single RouterBench response per model per prompt, and each model's embedding is the cluster-mean performance of those responses; if those responses are noisy or the clusters miss prompt difficulty, the measured trade-offs may not reflect real deployment.","fun_headline_variants_meta":{"raw":{"variants":["Attention router beats KNN, MLP, SVM on cost-quality LLM routing","Second-order query-model interaction improves LLM routing quality","Cross-attention router pairs queries with optimal LLMs per cost","Exponential reward stabilizes cost-quality trade-off in LLM routing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00044,"raw_usage":{"total_tokens":2245,"prompt_tokens":971,"completion_tokens":1274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":1200}},"tokens_in":587,"tokens_out":1274,"duration_ms":9057,"temperature":1.0,"reasoning_tokens":1200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:58:33.402394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the RouterBench comparison with several sampled responses per model per prompt, scoring quality by expected or majority performance: if the attention router's AIQ advantage over KNN, MLP, and SVM shrinks or reverses, the reported gains are an artifact of single-response noise.","supporting_citations":[],"review_version":2}