{"id":"4f62369c-3725-47ba-835a-6d1657f4c287","arxiv_id":"2411.09837","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RAR uses shadow inference plus a stored library of large-model reasoning hints to continuously shift requests from an expensive LLM to a cheaper one.","lead":"This paper introduces RAR, a router that learns on the job which requests a small language model can handle, using reasoning guides from a large model to teach the small model. The authors report cutting expensive-model calls by about half on MMLU subtests while keeping 90.5% of their alignment-based quality metric.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 50.2% cost-reduction claim is unverified because the paper never states whether stronger-FM calls made to generate guides are included in the 'used stronger FM' metric; if they are excluded, actual strong-model inference could exceed the oracle baseline.","rationale":"The reader's verdict is REJECT, and I agree that rejection is appropriate. The reader identifies semantic similarity as the weakest assumption on the quality side: the paper defines 'aligned response' as semantic similarity to the stronger FM's output (Section III-A) and then uses 'number of aligned responses' as the quality measure, even though MMLU has ground-truth answers that could have been used to validate the proxy. That is a valid concern, and it alone would prevent the paper from fully supporting its '90.5% of response quality' claim. The concern I treat as most load-bearing is on the cost side, because even granting the quality proxy, the headline cost reduction is not established if guide-generation calls are omitted from the strong-model usage count. Section III-D explicitly describes guides as generated by the stronger FM, and Section IV-A3 never says whether those calls are included in 'number of times stronger FM is used.' The separate guide-source tracking in Figure 7 makes exclusion plausible. If guide-generation calls are excluded, RAR's actual strong-model inference load could exceed the oracle static router, directly invalidating the central 50.2% claim. A secondary concern is that the experimental protocol reuses the same 754 samples across five stages, so the reported savings are largely cache hits on repeated requests rather than evidence of adaptation to new requests; the paper acknowledges this in Section V-C as an ideal-case setup. None of this requires questioning the authors' integrity; the fix is to make the cost accounting explicit and re-run the experiment with every strong-model call instrumented. Since the reader already rejected the paper, my analysis does not change the verdict, but it strengthens the reasons for rejection.","tokens_in":11850,"tokens_out":6514,"duration_ms":75165,"concrete_test":"Re-run the RQ1 experiment with explicit logging of every stronger-FM API call, separating (a) full response generation for the user, (b) guide generation, and (c) any additional judge or scoring calls. Recompute the 'used stronger FM' metric as the total count of stronger-FM calls, and also as token cost or dollar cost. If total RAR strong-model calls are not statistically significantly below the oracle static router's 562 calls, the 50.2% reduction claim is an accounting artifact and the headline should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result is that RAR reduces stronger-FM usage by 50.2% while retaining 90.5% of response quality versus an oracle static router (Section IV-B2). The cost side of this claim is not well-defined. In Section III-D, Case 2, a guide is 'acquired either from a guide memory or generated by the stronger FM.' Generating a guide is an inference call to the stronger FM. However, the evaluation procedure (Section IV-A3) only says that the authors 'record the number of aligned responses and the number of times stronger FM is used,' without defining whether guide-generation calls are counted in that number. The separate reporting in Figure 7 of 'guide generation with stronger FM' versus 'guide memory' suggests these events are tracked separately, which makes it plausible that Figure 4's 'Used stronger FM' counts only direct response calls, not guide calls. If so, a request that is answered by the stronger FM and then also needs a guide for the weaker FM is counted once in the cost metric while actually incurring two strong-model calls. The oracle static router, by contrast, makes one strong-model call per routed request. Reported RAR strong usage is about 280 calls versus 562 for the oracle; adding even modest guide-generation calls could erase or reverse that reduction. This is not merely a reporting nit: the central efficiency claim depends on which calls are counted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Real-time Adaptive Routing (RAR), a continual-learning router for software powered by layered foundation models. When a static predictive router sends a request to a stronger, more expensive FM, RAR performs shadow inference with a weaker, cheaper FM: if the weak response is semantically similar to the strong response (an 'aligned response'), the request and any guide are stored in skill/guide memory; future similar requests are then served by the weak FM, optionally with a cached guide. The paper evaluates RAR on three filtered subsets of MMLU (professional law, moral scenarios, high-school psychology) using Mistral-7B as the weak model and GPT-4o or Llama-3-70B as the strong model. It reports a 50.2% reduction in strong-model calls while retaining 90.5% of response quality relative to an oracle static router (p<0.001), and reports evidence of intra-domain and some inter-domain guide generalization.","tokens_in":12139,"tokens_out":6693,"duration_ms":69798,"significance":"RAR addresses a practically relevant cost/quality trade-off, and the core idea - using strong-model step-by-step reasoning as reusable in-context guides to improve routing decisions over time - is plausible and interesting. The paper evaluates across three domains and two different strong models, and the intra-domain guide-reuse trend in Figure 7 is a useful empirical finding. However, the headline quality claim is measured by semantic similarity to the strong model's response rather than by correctness or user-facing quality, and the cost metric does not state whether guide-generation calls to the strong model are counted. These issues mean the headline 50.2% / 90.5% numbers are not yet established. The paper would be substantially strengthened by reporting MMLU accuracy and a complete accounting of all strong-model calls, including guide generation.","major_comments":[{"comment":"The central cost-reduction claim (50.2% fewer stronger-FM calls, Section IV-B2) is computed from 'the number of times stronger FM is used' (Section IV-A3), but the paper never states whether stronger-FM calls made to generate guides under Case 2 (Section III-D) are included. Since Figure 7 separately tracks 'guide generation with stronger FM' versus 'guide memory', it is plausible that Figure 4's 'Used stronger FM' counts only direct response calls and excludes guide-generation calls. If so, a request answered by the strong model that also requires a fresh guide incurs two strong-model calls under RAR, while the oracle static router incurs only one; adding these guide calls could materially reduce or even reverse the reported cost advantage. Please define the cost metric precisely and re-report the efficiency numbers with all stronger-FM calls, including guide generation, included.","section":"III-D, IV-A3, IV-B2"},{"comment":"The paper claims to maintain '90.5% of response quality,' but the evaluation metric is the number of 'aligned responses,' defined in Section III-A as responses that are semantically similar to the stronger FM's output, not responses that are correct or high-quality. The authors themselves note that 'aligned response' differs from 'correct response.' Because RAR's objective (increase aligned responses) and the evaluation metric (number of aligned responses) are the same quantity, the retention figure is partly a restatement of the optimization target rather than independent evidence of quality. Since the evaluation uses MMLU, ground-truth answers are available; please report accuracy (or another user-oriented quality metric) for RAR and all baselines, and use that for the quality-retention claim.","section":"III-A, IV-B2"},{"comment":"The datasets are not the full MMLU subsets named in the abstract. Figure 3 shows that all samples that the weaker FM can answer correctly are discarded, leaving only the 754 law, 359 psychology, and 675 moral-scenario failing samples. Consequently, the reported 50.2% and 90.5% figures apply only to a filtered, deliberately more difficult subset, and the abstract should not describe this as evaluation on 'different subsets of MMLU' without that qualification. Moreover, each stage reuses the same samples, so the cumulative aligned-response counts and the cost reduction over stages partly reflect cached solutions to repeated, identical requests; please also report performance on novel requests or on a single pass to separate memorization/caching from generalization.","section":"IV-A1, Figure 3"},{"comment":"The main results depend on the similarity-score threshold (set to 0.2 in Section IV-A2), the retry period for Case 3 re-checking (Section III-D), and the number of experiment stages (Section IV-A3), but no sensitivity analysis is reported for any of these. The threshold directly controls the exploration/exploitation trade-off and defines which responses count as aligned, so the headline numbers should be shown over a range of thresholds (and, if feasible, over different retry periods and stage counts). Without such analysis, it is hard to know how robust the 50.2% / 90.5% result is to reasonable hyperparameter choices.","section":"IV-A2, III-F"}],"minor_comments":[{"comment":"The abstract and Section IV-B2 report '50.2% reduction,' while the RQ1 summary in Section IV-B2 reports '50.1% reduction'; please make these numbers consistent or explain the difference.","section":"Abstract, IV-B2"},{"comment":"The 'oracle static router' baseline is constructed using privileged knowledge of the dataset (profiling determines which samples the weaker model can answer). This is a useful upper-bound baseline, but the paper should state explicitly that it is an oracle and not a realistic deployable router, so readers do not interpret the comparison as one against a practical static router.","section":"IV-B1"},{"comment":"The caption of Table I says 'difference between the cumulative number of aligned responses vs. stronger FM (lower is better)' and reports values such as '15.0%' and '47,1%'. Please define the formula used, fix the comma in '47,1%', and state whether these are per-stage or cumulative percentages.","section":"Table I"},{"comment":"Reference [26] and [27] are the same paper (Shnitzer et al., “Large Language Model Routing with Benchmark Datasets”) with duplicate entries; please cite it once.","section":"References"},{"comment":"The statistical testing is described only as 'Chi-square test with 95% confidence interval' without stating which contingency tables were compared, what the comparison groups were, or the test statistics and degrees of freedom. Please report the actual test setup so the p<0.001 values are verifiable.","section":"IV-A3"}],"recommendation":"major_revision","confidential_remarks":"The contribution is incremental but potentially useful for the FM-routing community. My main concern is that the headline efficiency and quality claims rest on two underspecified metrics: the cost metric may omit guide-generation calls, and the quality metric equates semantic similarity with response quality. Both are fixable within the scope of the manuscript by re-analysis and re-reporting. I would also encourage the authors to release code and data to support reproducibility, as the paper currently provides neither."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a plausible and honest paper about a useful idea, but the two headline numbers ('50.2% fewer expensive calls,' '90.5% quality retention') are not actually nailed down by the experiments as written. I would send it to review, but only with a clear expectation of major revision.\n\nWhat's new: the loop of static router, shadow inference on the weaker model, storing requests and generated guides in a vector memory, and using that memory to route future requests to the weak model. Each piece exists, but the closed-loop adaptation for routing is a reasonable combination. The strongest evidence is Figure 7: with each stage, more successful guides come from memory rather than fresh strong-model calls. That shows intra-domain generalization is real, not just memorization.\n\nSoft spots, in order of size.\n\nFirst, the cost metric is ambiguous. Section IV-A3 says they record 'the number of times stronger FM is used,' but never defines whether guide-generation calls are included. In Case 2, a guide can be generated by the stronger FM on top of the response call. Figure 7 tracks guide source separately from memory, which strongly suggests the 'Used stronger FM' count in Figure 4 counts only direct response calls. If that's true, a request that needs both a response and a fresh guide costs two strong-model calls, and the 50.2% reduction over the oracle (which makes one call per difficult request) could disappear or reverse. This is the load-bearing issue.\n\nSecond, 'quality' is alignment with the stronger FM's response, not correctness. MMLU has ground truth; they could report accuracy, but don't. Their alignment metric is honest about being a proxy, but then the 90.5% number is '90.5% of alignment,' not '90.5% of quality' as most readers will interpret it.\n\nThird, the oracle static router is privileged information, so it's a conservative baseline—good for a first test, but no real router (e.g., RouteLLM) is compared, so the practical claim 'improves on existing routing' is not directly demonstrated. The threshold on similarity is a free parameter and not sensitivity-analyzed.\n\nThe paper is clearly written and the threats-to-validity section is honest about the repeated-samples concern. No signs of data fabrication. It's just that the headline claims outrun the measurements.\n\nRecommendation: engage with it for peer review, but make the authors define the cost metric exactly, count guide calls, and report accuracy on the benchmark. If the cost claim survives that, the paper is a solid empirical contribution.\n\nBest,\n[Your name]","headline":"Plausible closed-loop routing idea with honest limitations, but the headline cost and quality numbers are not yet supported because guide calls may be excluded and quality is measured by alignment, not correctness.","tokens_in":12654,"tokens_out":4143,"would_cite":false,"duration_ms":44005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAR claims that a router can cut expensive-model calls by 50.2% while retaining 90.5% of response quality by teaching the weak model with strong-model-generated guides.","keywords":["LLM routing","layered foundation models","in-context learning","continual learning","chain-of-thought guides","cost-quality trade-off","semantic similarity","MMLU benchmark"],"falsifier":"On the same MMLU subsets, score RAR's responses against the benchmark's ground-truth answers instead of against the strong model's responses: if the requests RAR routes to the weak model have substantially lower accuracy than the strong model's accuracy, or if aligned responses often disagree with the correct multiple-choice answer, the quality-retention claim is refuted.","tokens_in":11666,"feed_emoji":"🔀","tokens_out":8328,"duration_ms":73709,"temperature":0.7,"pith_summary":"The paper proposes Real-time Adaptive Routing (RAR), a router for software that uses a cheaper, weaker language model by default and a stronger, more expensive one as fallback. The claim is that the router can improve over time by having the strong model supply step-by-step reasoning guides that the weak model uses in its prompt, so that requests originally sent to the strong model can later be handled by the weak model. On subsets of the MMLU multiple-choice benchmark, RAR reports routing 50.2% fewer requests to the strong model while preserving 90.5% of the response quality of an oracle static router, with comparable results on two additional domains and a second strong model. The practical point is that the expensive model can continuously train the cheap model during normal operation, lowering inference cost without retraining weights.","feed_headline":"Router cuts costly AI calls by 50% at 90% quality","feed_subtitle":"A layered LLM system learns guides from its strong model and reuses them so the weak model handles more requests over time.","key_machinery":"The mechanism is a learned loop of shadow inference, semantic comparison, and memory reuse. When the static router would pick the strong model, RAR also generates a weak-model response in the background; a vector-embedding similarity check against the strong model's response decides whether the weak model can serve this type of request alone. If not, RAR asks the strong model for a guide, a step-by-step reasoning hint that does not contain the answer, and runs the weak model again with the guide prepended to the request. Successful request-guide pairs are stored in a vector database indexed by cosine similarity of sentence embeddings, and the matching threshold acts as an exploration-versus-exploitation dial that controls how eagerly the system reuses old guides instead of requesting new ones.","core_discovery":"The central discovery is that routing decisions do not have to stay fixed after deployment. Each time the static router sends a request to the strong model, RAR shadows the request by also asking the weak model to answer it; if the weak model's response is semantically aligned with the strong model's, the request is saved as a skill, and future similar requests are routed directly to the weak model. If the weak model fails alone, RAR obtains a guide from the strong model and retries the weak model with that guide in context, saving successful request-guide pairs for reuse. In the reported experiments this adaptation reduces strong-model use by 50.2% and retains 90.5% of the aligned-response count of an oracle static router, with statistical significance at $p<0.001$; the stored guides show within-domain reuse and a smaller amount of cross-domain transfer. The paper defines an aligned response as one semantically similar to the strong model's response, not necessarily the ground-truth-correct one.","pith_inferences":["Because RAR's quality metric is alignment with the strong model's output, the system inherits the strong model's errors: a confident but wrong strong-model answer would be treated as the reference, so the reported 90.5% shows fidelity to the strong model, not objective accuracy.","A natural next experiment, flagged by the paper's own validity discussion, is to test RAR on open-ended generation with an LLM judge for semantic comparison; the multiple-choice format is a constrained case that may overstate transferability.","The similarity threshold is effectively a cost-quality control: raising it forces more fresh strong-model guides, lowering it favors reuse and accepts less similar requests.","In edge deployments the same loop acts as a personalization mechanism: guides accumulate from the user's own request history, so the on-device weak model should improve for that user's recurring needs while keeping data on the device."],"forward_implications":["A deployed layered LLM service can expect roughly half of the requests that a static router would send to the expensive model to be absorbed by the cheap model after enough guide accumulation, at a small alignment-cost penalty.","The benefit grows with request repetition and topical clustering: similar or recurring user requests become progressively cheaper because they are served from skill and guide memory.","Guide reuse is not per-question memorization; the paper reports intra-domain generalization, with one guide helping multiple related questions, and small cross-domain transfer of 6-7% over an unguided weak model.","Because the guide memory is filled from the stronger FM's own reasoning, the routing loop converts expensive-model calls into reusable assets, so long-running deployments get more efficient rather than staying at the static router's cost level."],"supporting_citations":[{"why":"defines the static predictive routing approach that RAR extends by adding continuous adaptation.","marker":"[23]"},{"why":"supplies the MMLU benchmark and question-answering task used in every experiment.","marker":"[12]"},{"why":"provides the per-model evaluated version of the dataset used to select failing samples.","marker":"[13]"},{"why":"establishes chain-of-thought reasoning prompting, the baseline and guide-generation idea RAR builds on.","marker":"[32]"},{"why":"the chosen weaker FM whose responses and guided performance are measured.","marker":"[14]"},{"why":"one of the two stronger FMs that supplies reference responses and guides.","marker":"[3]"},{"why":"the other stronger FM, included to show the routing loop is not tied to one model.","marker":"[24]"},{"why":"the embedding model used for semantic similarity comparisons and memory indexing.","marker":"[28]"},{"why":"an in-context continual-learning skill-library design that motivates the guide memory component.","marker":"[29]"}],"fun_headline_variants":["Adaptive router slashes expensive AI calls by half","Continuous learning router cuts costs by 50% at 90% quality","Router learns to reuse guides, cutting expensive calls 50%","Self-improving router reduces AI costs by 50%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a response counts as good when it is semantically similar to what the stronger model would have said; if that similarity does not track correctness, the 90.5% quality-retention figure does not show real quality.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive router slashes expensive AI calls by half","Continuous learning router cuts costs by 50% at 90% quality","Router learns to reuse guides, cutting expensive calls 50%","Self-improving router reduces AI costs by 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1209,"prompt_tokens":940,"completion_tokens":269,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":199}},"tokens_in":556,"tokens_out":269,"duration_ms":2990,"temperature":1.0,"reasoning_tokens":199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:15:24.507326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the same MMLU subsets, score RAR's responses against the benchmark's ground-truth answers instead of against the strong model's responses: if the requests RAR routes to the weak model have substantially lower accuracy than the strong model's accuracy, or if aligned responses often disagree with the correct multiple-choice answer, the quality-retention claim is refuted.","supporting_citations":[{"cited_title":"https://ai.meta.com/ blog/llama- 3- 2- connect- 2024- vision- edge- mobile- devices/ [Accessed: (Sept","cited_arxiv_id":null,"evidence_quote":"defines the static predictive routing approach that RAR extends by adding continuous adaptation."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"establishes chain-of-thought reasoning prompting, the baseline and guide-generation idea RAR builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"one of the two stronger FMs that supplies reference responses and guides."},{"cited_title":"https://huggingface.co/sentence-transformers/all- MiniLM-L12-v2","cited_arxiv_id":null,"evidence_quote":"the embedding model used for semantic similarity comparisons and memory indexing."},{"cited_title":"V oyager: An Open-Ended Em- bodied Agent with Large Language Models","cited_arxiv_id":null,"evidence_quote":"an in-context continual-learning skill-library design that motivates the guide memory component."}],"review_version":1}