REVIEW 6 cited by
CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models have achieved remarkable success in various tasks but suffer from high computational costs during inference, limiting their deployment in resource-constrained applications. To address this issue, we propose a novel Collaborative Inference with Token-lEvel Routing (CITER) framework that enables efficient collaboration between small and large language models (SLMs \& LLMs) through a token-level routing strategy. Specifically, CITER routes non-critical tokens to an SLM for efficiency and routes critical tokens to an LLM for generalization quality. We formulate router training as a policy optimization, where the router receives rewards based on both the quality of predictions and the inference costs of generation. This allows the router to learn to predict token-level routing scores and make routing decisions based on both the current token and the future impact of its decisions. To further accelerate the reward evaluation process, we introduce a shortcut which significantly reduces the costs of the reward estimation and improving the practicality of our approach. Extensive experiments on five benchmark datasets demonstrate that CITER reduces the inference costs while preserving high-quality generation, offering a promising solution for real-time and resource-constrained applications. Our data and code are available at https://github.com/aiming-lab/CITER.
Forward citations
Cited by 6 Pith papers
-
Divergence Decoding: Training-Free Capability Fusion
Divergence Decoding routes each token to either a domain specialist or a general reasoning LLM based on Jensen-Shannon divergence, outperforming either model alone on most tested scientific tasks.
-
TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI
Task-consistent sticky routing with a contextual bandit improves accuracy-latency trade-offs on agentic benchmarks by learning from delayed task-level reward.
-
PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference
PyroDash lets a small model decide mid-generation to hand off once to a frozen LLM, achieving 64.04% accuracy (+6.36 points vs LLM) at lambda=0.05 and 96.4% lower estimated cost at lambda=0.6.
-
Accelerating Heterogeneous Agent Collaboration in Dynamic Edge Networks
PRADA distills a process reward model into an edge screening policy and uses a threshold-based server scheduler to retain most LLM reasoning accuracy while sharply cutting multi-user latency.
-
RLAE: Reinforcement Learning-Assisted Ensemble for LLMs
RLAE uses PPO and MAPPO policies to assign per-span ensemble weights across 7B-8B LLMs; it claims up to 3.3% accuracy improvement over prior ensemble baselines but underperforms on several tested tasks.
-
Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.
Discussion (0). Continue with ORCID to comment.