{"id":"3f1bb84d-00b2-4b8a-bb93-eac3b8f1376b","arxiv_id":"2501.01818","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Adversarially optimized, query-independent token prefixes can reroute nearly all queries to the expensive strong model in both open-source and commercial LLM routers.","lead":"This paper shows that LLM routers, systems that send easy questions to cheap models and hard ones to expensive models, can be defeated by prepending a short nonsense token string to any query, forcing the expensive model to be used. The attack works in black-box settings against commercial routers, so routing-based cost control is exposed to cost inflation and premium-model arbitrage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Commercial-router black-box claim rests on unverified surrogate data-similarity; Section 7 results are mixed and lack error bars, so the broad 'commercial routers' claim is not established.","rationale":"The white-box attack is convincingly demonstrated: Table 1 shows near-universal rerouting across four router algorithms, and the response-quality tables (Tables 2–4 and 10) show no systematic degradation, with genuine gains when the weak model is much worse. The low-perplexity evasion in Section 8 is also a useful, reproducible result. The load-bearing weak point is black-box and commercial generalization. Section 4's transfer argument is conditional on surrogate-target data similarity, and for the four open-source routers that condition is trivially satisfied because they share the same RouteLLM training data. Commercial routers are the only test of genuinely different training distributions, and Section 7's numbers are mixed: Unify is strong, NotDiamond is weak (13–21% upgrade from an already 82% strong baseline), and OpenRouter is gadget-dependent. The paper reports no error bars for these commercial runs and uses only 72 MT-Bench queries. Because the title and abstract claim a general attack on LLM routers, the commercial evidence is the least secure support for that generality. This is not an internal inconsistency, but an empirical overreach; a conditional verdict remains appropriate. The concrete test above would settle whether the surrogate-similarity assumption is actually load-bearing by removing the shared-data confound.","tokens_in":27695,"tokens_out":7530,"duration_ms":86394,"concrete_test":"Reproduce the Section 7 commercial evaluation using a surrogate router trained from scratch on a preference dataset disjoint from RouteLLM's (e.g., a newly collected pairwise-ranking corpus), across at least 300 queries per commercial router with per-gadget 95% confidence intervals. If the disjoint-surrogate upgrade rates on Unify, NotDiamond, and OpenRouter fall to baseline, the 'similar data' assumption is the load-bearing condition and the black-box commercial claim fails; if they remain high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central black-box claim (Section 4, 'The black-box setting') assumes the adversary has a surrogate R' 'trained on data similar to that used for the target router.' For the four open-source routers this assumption is automatic: they all come from the same RouteLLM codebase and training preference data, so transfer among them is weak evidence of cross-distribution transfer. The only cross-family evidence is Section 7, but the results are mixed: Unify shows 79–91% upgrade, NotDiamond only 13–21% (with 1–3% downgrades and an already high 82% strong-model baseline), and OpenRouter only 3–4 of 10 gadgets produced full rerouting while none of the RCLS gadgets had any impact. No confidence intervals or per-gadget breakdowns are given for the commercial runs, and the evaluation uses only 72 MT-Bench queries. If a deployed router's scoring function or training distribution differs from the surrogate's, transfer can fail; the paper does not measure or bound that divergence. Thus the abstract's claim that the attack is 'successful ... against a variety of commercial routers' overstates the evidence: it is well-supported only for Unify and for same-family open-source transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the notion of LLM control plane integrity and studies the adversarial robustness of LLM routers. The attack, called a confounder gadget, is a short token sequence optimized by hill-climbing that, when prepended to any query, causes a router to send the query to the strong model. The authors evaluate the attack in white-box and black-box transfer settings against four open-source routers (RSW, RMF, RCLS, RLLM) on MT-Bench, MMLU, and GSM8K, and against three commercial routers (Unify, NotDiamond, OpenRouter). They further show that adding a perplexity penalty to the optimization evades GPT-2-based perplexity filtering, and they discuss alternative defenses.","tokens_in":27933,"tokens_out":5921,"duration_ms":59027,"significance":"If the core white-box and same-family transfer results hold, the paper identifies a practically relevant, economically motivated vulnerability in a class of systems that is growing in deployment. The formalization of control plane integrity is a useful framing, and the open-source-router evaluation is reasonably thorough: it covers four router algorithms, several benchmarks, multiple LLM pairs, and reports standard errors. The paper also honestly reports mixed commercial results rather than selecting favorable cases. The main contributions are the attack methodology and the demonstration that perplexity-based filtering is evadable, though the strength of the commercial-router and response-quality claims is not fully matched by the evidence.","major_comments":[{"comment":"The abstract claims the attack is successful \"against a variety of commercial routers,\" but the Section 7 evidence is mixed: Unify shows 79–91% upgrade rates, NotDiamond only 13–21% upgrades with a high 82% strong-model baseline and 1–3% downgrades, and OpenRouter only 3–4 of 10 gadgets produce full rerouting, while no RCLS gadget has any effect. No confidence intervals or per-gadget breakdowns are given for these commercial runs, and the evaluation uses only 72 MT-Bench queries. The commercial claim should be narrowed to Unify (and partially OpenRouter) or supported with additional experiments and error bars.","section":"Abstract and Section 7"},{"comment":"The black-box transfer evaluation is dominated by same-family transfer: all four open-source routers come from the same RouteLLM codebase and preference data, so transfer among them does not test cross-distribution transfer. The only cross-family evidence is the commercial-router results, which are mixed as noted above. The paper should either temper the \"black-box successful\" claim or provide additional evidence, for example by training surrogate routers on different data distributions and measuring transfer.","section":"Section 4 and Section 7"},{"comment":"The conclusion that \"NotDiamond is more robust than Unify\" is not supported by the reported upgrade rates. With GPT-4o/Mixtral, 82% of original queries already go to the strong model, so the maximum possible upgrade rate over the remaining weak queries is 18%; upgrade rates of 15–21% are therefore near or at the ceiling, meaning almost all weak queries are being rerouted. The comparison should account for the baseline strong-model fraction and the downgrade rate, or use a controlled setting with matched baselines.","section":"Section 7, NotDiamond paragraph"},{"comment":"The claim that confounding does not affect response quality is partly based on perplexity comparisons after filtering out responses with perplexity greater than 100 (footnote 1). The number of filtered responses is not reported separately for benign and confounded settings. If the attack increases the rate of high-perplexity responses, this filtering could hide a quality degradation. The benchmark-score evidence in Tables 3 and 4 is more convincing, but the authors should either report the filter counts or drop the perplexity-based quality claim.","section":"Section 6, footnotes on perplexity filtering"},{"comment":"The conclusion that \"perplexity-based filtering is not an effective defense\" is too broad. The evasive gadgets are optimized using the same GPT-2 perplexity function that the defender is assumed to use, and the paper tests no other perplexity estimators or ensembles. The claim should be restricted to GPT-2-based perplexity filtering, or additional experiments with other perplexity models should be reported.","section":"Section 8"}],"minor_comments":[{"comment":"The symbol B is used both for the batch size and for the candidate set B = {c~0, ..., c~B}, which is confusing; consider renaming one of them.","section":"Section 4, Eq. (1)"},{"comment":"The column header \"SF M\" appears to be a typo for \"RM F\"; please correct it.","section":"Tables 5, 6, 7, 9"},{"comment":"The phrase \"against against rerouting\" contains a duplicated word.","section":"Section 8"},{"comment":"The word \"withing\" should be \"within\" in the sentence about perplexity values being \"close and withing the margin of error.\"","section":"Appendix D"},{"comment":"The phrase \"we observe a a small improvement\" contains a duplicated article.","section":"Appendix B"},{"comment":"The paper does not state whether code or artifacts are available. Providing the implementation and trained gadget examples would aid reproducibility, which would strengthen the paper given the empirical nature of the contributions.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful and timely contribution, and the open-source-router results are convincing. The path to acceptance lies in aligning the claims with the evidence, particularly by tempering the commercial-router and response-quality statements and by clarifying the NotDiamond analysis. I do not see a fundamental flaw that would require rejection, but several load-bearing claims need revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this paper convincingly shows that query-independent token prefixes (\"confounder gadgets\") can force LLM routers to send almost any query to the expensive model, at least in white-box settings. The evaluation against four open-source routers from RouteLLM across three benchmarks is careful, with standard errors reported and no cherry-picking I can see. The framing of \"LLM control plane integrity\" as a distinct safety property is genuinely useful—it separates attacks on orchestration from attacks on the underlying LLM.\n\nWhat's actually new: applying adversarial token optimization to cost-based routers, and showing it transfers across router algorithms and evades perplexity filtering when the optimization is given a low-perplexity objective. The black-box transfer among the open-source routers is solid, as all four share the same preference data and codebase, so the same-family transfer is not surprising but still demonstrates the phenomenon.\n\nThe soft spots are in scope, not in execution. The abstract's \"successful against a variety of commercial routers\" overstates Section 7. Only Unify is convincingly attacked (79-91% upgrade rates). NotDiamond shows 13-21% with a high 82% strong-model baseline, and OpenRouter only 3-4 of 10 gadgets fully rerouted while none of the RCLS gadgets did anything. There are no error bars for the commercial runs and the evaluation is 72 MT-Bench queries. The black-box setting's surrogate-similarity assumption—that the surrogate is trained on data similar to the target—is stated but never verified or bounded. For proprietary routers the transfer is an empirical bet that the paper doesn't close. Also, no code or artifacts are released, which is a real gap for an attack paper others will want to build on. The novelty is modest: the hill-climbing over token replacements closely resembles universal trigger search in the text-attack literature (e.g., Wallace et al.), which isn't cited.\n\nThe core result holds up despite these issues. This deserves a serious referee, and I'd bring it to a reading group. My recommendation: send it to peer review, with requests to release code, add per-gadget and error-bar detail for the commercial runs, and temper the abstract's scope to what the evidence supports.","headline":"Read this for a well-run white-box attack on LLM routers and a useful new safety framing; don't take the commercial-router claims at face value.","tokens_in":28495,"tokens_out":2987,"would_cite":true,"duration_ms":29390,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simple token prefix can hijack LLM routers and force every query to the expensive model.","keywords":["LLM router","control plane integrity","confounder gadget","adversarial robustness","cost inflation","routing attack","black-box transfer","perplexity-based defense"],"falsifier":"A concrete test would be to take a commercial router like Unify or NotDiamond, train a surrogate router on a deliberately different distribution (e.g., only code queries instead of general chat), and check whether gadgets optimized on that surrogate still reroute queries on the target at the reported upgrade rates; if the upgrade rate drops to near zero, the transfer assumption fails.","tokens_in":27459,"feed_emoji":"🛠️","tokens_out":1379,"duration_ms":15240,"temperature":0.7,"pith_summary":"This paper defines a new safety property, LLM control plane integrity, and shows that current LLM routers fail it. The authors construct query-independent token sequences called confounder gadgets that, when prepended to any query, cause a router to send that query to the strong, expensive LLM. The attack works in white-box and black-box settings against both open-source and commercial routers, and the confounded queries still get high-quality responses. The paper also shows that perplexity-based filtering is not a reliable defense, since gadgets can be optimized to have low perplexity. If routers are widely deployed for cost control, this attack enables cost inflation and premium-model arbitrage.","feed_headline":"A fixed token prefix hijacks LLM routers to the pricey model","feed_subtitle":"One optimized gadget forces nearly all queries to the strong LLM, enabling cost inflation and premium-model arbitrage.","key_machinery":"The confounder gadget is a short sequence of tokens found by hill-climbing: starting from a fixed token repeated n times, each iteration replaces one token with a random candidate and keeps the replacement that maximizes the router's scoring function applied to the gadget (or to the gadget concatenated with the query). For query-independent attacks, the optimization maximizes the score of the gadget alone, and the resulting prefix transfers across queries and, in the black-box setting, across routers trained on similar data.","core_discovery":"The central claim is that an adversary can generate a single fixed token sequence, a confounder gadget, that when prepended to arbitrary user queries makes an LLM router classify those queries as complex and route them to the strong model. The paper demonstrates this against four open-source routing algorithms (similarity-weighted ranking, matrix factorization, a BERT classifier, and a fine-tuned LLM classifier) and against commercial routers including Unify, NotDiamond, and OpenRouter. In white-box tests the gadgets reroute nearly all weak-model queries to the strong model, often reaching 100% upgrade rates; in black-box transfer tests, gadgets optimized on one router still transfer to others with high upgrade rates. The paper further shows that the attack does not degrade response quality and can even improve it when the strong model is substantially better than the weak one.","pith_inferences":["The attack likely generalizes to other routing architectures that use a learned scalar scoring function, including quality-focused routers and cascading systems, if those scoring functions are differentiable or can be queried.","The same mechanism could be extended to indirect settings, where the confounder is embedded in third-party content that a victim application retrieves, enabling control-plane integrity attacks without direct user input.","A stronger defense would need to combine perplexity filtering with per-user routing statistics or with an oracle LLM that judges naturalness, but each of these adds cost and can be evaded by more sophisticated gadget optimization.","The commercial-router results are an empirical bet on transfer: for proprietary routers whose scoring functions and training data are unknown, the attack's success depends on the surrogate being trained on data similar to the target's, which is not verified."],"forward_implications":["Cost-control mechanisms built on learned routers can be bypassed by a static prefix, so any application relying on routing to keep per-query costs down is exposed to cost inflation.","The same gadget can be used across many queries without per-query adaptation, making the attack cheap and easy to automate.","Black-box transfer means the attack can target proprietary routers without querying them, as long as a surrogate router trained on similar data is available.","Low-perplexity gadget variants make perplexity-based filtering ineffective as a standalone defense, so defenders need workload-level or user-level anomaly detection.","The attack can also be reversed to reroute queries to the weak model, showing the control plane can be pushed in either direction."],"supporting_citations":[{"why":"Provides the four open-source routing algorithms and trained parameters that serve as the white-box targets and surrogates for all of the paper's main experiments.","marker":"[47]"},{"why":"One of the prior prescriptive routing algorithms the paper builds on, contributing the BERT-classifier routing approach and the cost-quality framing.","marker":"[27]"},{"why":"Unify, one of the three commercial routers evaluated in the black-box attack, showing that the attack transfers to a proprietary router with a neural quality scorer.","marker":"[12]"},{"why":"NotDiamond, another commercial router evaluated, providing a cost-aware routing API that the paper tests against.","marker":"[7]"},{"why":"OpenRouter, the third commercial router evaluated, which routes among three models and is partially confused by some gadgets.","marker":"[11]"},{"why":"Martian, the fourth commercial router considered; it is excluded from the main evaluation because it appeared to ignore the supplied model list.","marker":"[5]"},{"why":"Supplies the OpenAI embedding model used by the similarity-weighted ranking router, making it load-bearing for the RSW attack.","marker":"[6]"},{"why":"Establishes perplexity-based filtering as a proposed defense that the paper shows can be evaded by perplexity-constrained gadget optimization.","marker":"[16]"},{"why":"Provides the MT-Bench benchmark and the LLM-as-a-judge methodology used to evaluate response quality after confounding.","marker":"[71]"}],"fun_headline_variants":["One token trick reroutes LLM queries to the pricey model","LLM router attack: a single confounder gadget forces premium model","Fixed token sequence exploits LLM routers for cost inflation","Adversarial gadget reroutes LLM routers to the strong model","LLM routers? One token prefix sends you to the pricey model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The black-box and commercial-router attacks assume that a gadget optimized against a surrogate router trained on similar data will transfer to the actual target router, which is never independently verified for the proprietary routers.","fun_headline_variants_meta":{"raw":{"variants":["One token trick reroutes LLM queries to the pricey model","LLM router attack: a single confounder gadget forces premium model","Fixed token sequence exploits LLM routers for cost inflation","Adversarial gadget reroutes LLM routers to the strong model","LLM routers? One token prefix sends you to the pricey model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000539,"raw_usage":{"total_tokens":2558,"prompt_tokens":892,"completion_tokens":1666,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1577}},"tokens_in":508,"tokens_out":1666,"duration_ms":11945,"temperature":1.0,"reasoning_tokens":1577,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:20:30.050651+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to take a commercial router like Unify or NotDiamond, train a surrogate router on a deliberately different distribution (e.g., only code queries instead of general chat), and check whether gadgets optimized on that surrogate still reroute queries on the target at the reported upgrade rates; if the upgrade rate drops to near zero, the transfer assumption fails.","supporting_citations":[{"cited_title":"Hybrid LLM: Cost-efficient and quality-aware query routing,","cited_arxiv_id":null,"evidence_quote":"One of the prior prescriptive routing algorithms the paper builds on, contributing the BERT-classifier routing approach and the cost-quality framing."},{"cited_title":"Unify LLM router,","cited_arxiv_id":null,"evidence_quote":"Unify, one of the three commercial routers evaluated in the black-box attack, showing that the attack transfers to a proprietary router with a neural quality scorer."},{"cited_title":"Notdiamond LLM router,","cited_arxiv_id":null,"evidence_quote":"NotDiamond, another commercial router evaluated, providing a cost-aware routing API that the paper tests against."},{"cited_title":"Openrouter LLM router,","cited_arxiv_id":null,"evidence_quote":"OpenRouter, the third commercial router evaluated, which routes among three models and is partially confused by some gadgets."},{"cited_title":"Martian LLM router,","cited_arxiv_id":null,"evidence_quote":"Martian, the fourth commercial router considered; it is excluded from the main evaluation because it appeared to ignore the supplied model list."},{"cited_title":"New embedding models and API updates,","cited_arxiv_id":null,"evidence_quote":"Supplies the OpenAI embedding model used by the similarity-weighted ranking router, making it load-bearing for the RSW attack."},{"cited_title":"Judging LLM- as-a-judge with MT-Bench and chatbot arena,","cited_arxiv_id":null,"evidence_quote":"Provides the MT-Bench benchmark and the LLM-as-a-judge methodology used to evaluate response quality after confounding."}],"review_version":1}