REVIEW 23 cited by
Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.
Forward citations
Cited by 23 Pith papers
-
Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents
Per-task routing between text, image, and hybrid observations of a browser page does not currently beat one fixed choice, because the labels needed to learn routing exist only where the agent already succeeds; only a ...
-
TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning
A heterogeneous-graph router jointly selects the optimal modality (text, vision, or both) and model per time series query, beating prior routing baselines and generalizing to unseen models and tasks.
-
A Workflow-Aware Serving Layer for Agentic Applications
A workflow-aware serving layer compiles per-node model-verifier-backend plans with an ILP and adapts only uncommitted work via pre-solved pressure rungs and residual re-solves.
-
Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First
A 7B searcher's sandbox-verified handoff lets a cheap coding model match a frontier model's solve rate on SWE-bench Pro at roughly one fifth the cost, while the router that picks the fixer contributes nothing on this ...
-
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
PyroDash lets a small model decide mid-generation to hand off once to a frozen LLM, achieving 64.04% accuracy (+6.36 points vs LLM) at lambda=0.05 and 96.4% lower estimated cost at lambda=0.6.
-
HACO: Hedged Agent Computing for Reliable LLM Systems
HACO adaptively selects a reliability-constrained hedge set of candidate LLM agent instances for each role invocation, improving robustness under changing deployment conditions at lower token/latency cost than full-po...
-
Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating
A training-free router for local-cloud LLM collaboration uses prompt-varied self-consistency with Bayesian early stopping to control offload ratios and beat baselines.
-
Optimal-Agent-Selection: State-Aware Routing Framework for Efficient Multi-Agent Collaboration
A state-aware contrastive router that selects the most relevant agent at each step improves multi-agent LLM accuracy by up to 23.8% while using a fraction of the tokens of fixed-pipeline baselines.
-
Adaptive LLM Routing under Budget Constraints
LLM routing is framed as a budget-constrained contextual bandit, solved by a preference-prior initialized LinUCB variant with an online multi-choice knapsack cost policy.
-
Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts
A DRL router using graph attention state abstraction and QoS-aware rewards improves average QoS by up to 35.78% over four baselines in simulated edge LLM routing.
-
DSSD: Efficient Edge-Device LLM Deployment and Collaborative Inference via Distributed Split Speculative Decoding
DSSD moves the resample step of speculative decoding onto the device, replacing the upload of many token distributions with a single download when a draft is rejected.
-
Orchestration for Domain-specific Edge-Cloud Language Models
ECO-LLM jointly selects query processing, retrieval, and model components per query, cutting cost by 60% and latency up to 6x versus model routing in edge-cloud tests.
-
IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory
An IRT-based router that models each LLM's latent ability and each query's difficulty outperforms RouterBench on cost-performance reward across ID and OOD benchmarks.
-
Trading Human Curation for Synthetic Augmentation in RLVR
Gated synthetic augmentations of a 10-task human base substitute for ~87 extra human RLVR tasks on aggregate held-out pass@1, with cost-adjusted trade rate ρ_cost in [1.4×, 11.6×].
-
Towards Generalized Routing: Model and Agent Orchestration for Adaptive and Efficient Inference
MoMA routes each query to a specialized agent or to the cost-optimal LLM, using judge-trained performance scores, a Pareto cost frontier, and TOPSIS selection.
-
Balancing Information Accuracy and Response Timeliness in Networked LLMs
For binary questions, combining m specialized LLMs with a Bayesian majority rule improves accuracy, and the paper derives the optimal m that trades accuracy against system delay.
-
Semantic Scheduling for LLM Inference
A semantic scheduler for LLM inference uses urgency labels and estimated remaining compute to cut waiting times for urgent requests, tested on emergency medical data.
-
Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration
A survey of multi-LLM systems in edge computing, covering architectures, enabling technologies, trust mechanisms, applications, and open datasets for edge general intelligence.
-
Divide, Specialize, and Route: A New Approach to Efficient Ensemble Learning
A difficulty-based ensemble that routes instances to specialized models is proposed, but reported gains are not tested against standard ensemble baselines and are filtered to favorable cases.
-
FAA Framework: A Large Language Model-Based Approach for Credit Card Fraud Investigations
The FAA framework automates credit card fraud investigations with GPT-4o agents and reports 98-99% fraud-detection F1, though the evaluation is weakened by self-referential LLM scoring.
-
Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Sign in to comment.