Pith. sign in

REVIEW 10 cited by

A Unified Approach to Routing and Cascading for LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.10347 v3 pith:64LSJYUE submitted 2024-10-14 cs.CL

Jasper Dekoninck , Maximilian Baader , Martin Vechev This is my paper · ORCID
classification cs.CL
keywords routingcascadingmodelparadigmsselectionstrategiesstrategyapproaches
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The availability of a wide range of large language models (LLMs) embedded in various agentic systems has significantly increased the potential of model selection strategies to improve the cost-performance tradeoff. Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found. However, current approaches face three key limitations: they (1) lack formal proofs of optimality, (2) fail to identify the conditions under which these strategies are most effective to improve the cost-performance tradeoff, and (3) are unable to combine both paradigms for further improvements. To address these issues, we first derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy. Further, we propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy. Through our analysis, we identify good quality estimators as the critical factor for the success of model selection paradigms. Finally, in our experiments, we show that cascade routing consistently outperforms the individual approaches by a large margin and we analyze quality estimators to determine when routing and/or cascading are useful paradigms for model selection.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching

    cs.LG 2026-07 conditional novelty 7.0 of 10

    CAPC combines query-agnostic compression with prompt caching and is reported cheapest in 16/16 LongBench configurations, but the empirical support has unresolved specification and consistency gaps.

  2. How Much of the Routing Gap Is Real? Decomposing the Router-to-Oracle Gap into Reproducible Specialist Advantage and Single-Draw Label Noise

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Under stochastic decoding, 12–36% of the LLM router-to-oracle gap is single-draw noise unreachable by any single-commit router, while the rest is recoverable specialist advantage; best-of-K sampling closes the floor.

  3. Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Hidden-state traces of frozen LLMs/VLMs can be read by lightweight trained heads to predict when to defer, clarify, call tools, or abstain, cutting routed inference cost 27–90%.

  4. Orchestration for Domain-specific Edge-Cloud Language Models

    cs.DB 2025-07 conditional novelty 6.0 of 10

    ECO-LLM jointly selects query processing, retrieval, and model components per query, cutting cost by 60% and latency up to 6x versus model routing in edge-cloud tests.

  5. Influence of Prompt Engineering on Small Language Models for Guarded Query Routing

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Few-shot prompt optimization lifts small language models' guarded-routing scores on GQR-Bench, bringing Qwen3.5 9B to 95.74, within 0.3 points of Gemma 3 27B without weight updates.

  6. Universal Model Routing for Efficient LLM Inference

    cs.CL 2025-02 conditional novelty 5.0 of 10

    UniRoute represents each language model by its error rates on a few prompt clusters, letting a router choose among models it has never seen during training.

  7. Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes

    cs.CL 2026-08 conditional novelty 4.0 of 10

    Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.

  8. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

  9. Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems

    cs.AI 2025-02 conditional novelty 4.0 of 10

    A survey that classifies LLM routing strategies into pre-generation and post-generation approaches and four implementation families, framed as a performance-cost optimization problem.

  10. T-TAMER: Provably Taming Trade-offs in ML Serving

    cs.LG 2025-09 reject novelty 3.0 of 10

    T-TAMER claims recall is necessary and sufficient for provably optimal early-exit and cascade serving policies, but the main extensions are under-derived and partly reduce to known Gittins-index results.

Pith tools