REVIEW 12 cited by
Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in TetriInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computationsaturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that TetriInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in turns of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.
Forward citations
Cited by 12 Pith papers
-
Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework
Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.
-
Sangam: Efficiently Serving Diffusion LLMs with the AR Stack
Deficit token-budget scheduling plus hybrid prefill overflow makes AR-style colocated and disaggregated serving work for cached diffusion LLMs with block-sized, recurring prefills.
-
CoCoScale: Leveraging Layer-wise Scaling to Unlock the Potential of Online LLM Serving
Layer-wise data-parallel replication of hot Transformer layers onto reclaimed idle GPUs reduces LLM serving cold-start latency 97.9–99.3% and average latency 20.7–28.1% while attaining 100% SLO on production traces.
-
TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure
TensorCast packages tensor lifecycle management into a programmable service layer and reports it can match specialized LLM systems while enabling new cross-component optimization policies.
-
Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling
AFlex combines operator-level disaggregation with per-operator DVFS to cut LLM serving energy per token by up to 49% without violating P90 TTFT/TPOT SLOs.
-
SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfer
Selective three-path KV cache transfer (profile-proactive, parallel on-demand, speculative) cuts time-to-second-token up to 4.3× vs full transfer on bandwidth-limited cloud GPUs with near-baseline accuracy and decode speed.
-
AutoSLO: Practical Latency SLOs on Cloud Data Warehouses -- Extended Version
AutoSLO meets query latency SLOs on multi-cluster cloud warehouses while cutting cost ~26% versus the next-best baseline by combining forecast-driven planning, reactive scaling, and concurrency-aware routing.
-
Efficient Multi-round LLM Inference over Disaggregated Serving
AMPD adaptively routes incremental prefill tasks between prefill and decode workers and reorders queued prefill jobs to improve SLO attainment for multi-round LLM inference under prefill-decode disaggregation.
-
Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelism
Hetis dynamically splits LLM attention across heterogeneous GPUs at the granularity of individual heads while keeping dense computation on a selected subset, and reports up to 2.25x throughput gains over Splitwise and Hexgen.
-
Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.
-
Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving
Nexus performs proactive intra-GPU disaggregation of prefill and decode, using an analytical cost model and greedy search to dynamically partition SMs, achieving up to 2.2x throughput gains over vLLM.
-
Beyond the Buzz: A Pragmatic Take on Inference Disaggregation
Disaggregated serving (separate prefill and decode GPU pools) expands the throughput-interactivity Pareto frontier mainly for prefill-heavy workloads and models larger than about 10B parameters, provided the prefill-t...
Discussion (0). Sign in to comment.