REVIEW 8 cited by
Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.
Forward citations
Cited by 8 Pith papers
-
General Non-Clairvoyant KV-Cache Scheduling via Regime-Aware Routing
A regime-aware routing meta-scheduler is O(1)-competitive for total completion time on general non-clairvoyant KV-cache batch scheduling with arbitrary prompts and responses.
-
Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Apps
Commercial AI video chat apps differ by 4× in video bitrate, 10× in framerate, and from zero to 10+ minutes of visual memory, with none replying in under 1.5 seconds.
-
Auto-Scaling Heterogeneous Neural Processing Units for Energy and Cost-Efficient LLM Serving
NeuScale routes LLM inference requests to the most energy/cost-efficient configuration of heterogeneous NPU chips using roofline allocation and runtime auto-scaling.
-
Online Linear Programming for Multi-Objective Routing in LLM Serving
Online LP bid-price routing with warm-started dual gradient updates substantially improves multi-SLO LLM decode routing over heuristics in simulation.
-
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.
-
Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts
A DRL router using graph attention state abstraction and QoS-aware rewards improves average QoS by up to 35.78% over four baselines in simulated edge LLM routing.
-
Clairvoyant: Predictive Shortest-Job-First Admission for Serial LLM Inference
Predictive shortest-job-first admission using an XGBoost classifier on 19 lexical prompt features cuts short-request P50 latency by 70-76% in serial LLM backends under burst load, per the paper's RTX 4090 benchmarks.
-
Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models
A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.
Discussion (0). Continue with ORCID to comment.