REVIEW 6 cited by
KVDirect: Distributed Disaggregated LLM Inference
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode stages, is a promising approach to improving hardware utilization and service quality. However, due to inefficient inter-node communication, existing systems restrict disaggregated inference to a single node, limiting resource allocation flexibility and reducing service capacity. This paper introduces KVDirect, which optimizes KV cache transfer to enable a distributed disaggregated LLM inference. KVDirect achieves this through the following contributions. First, we propose a novel tensor-centric communication mechanism that reduces the synchronization overhead in traditional distributed GPU systems. Second, we design a custom communication library to support dynamic GPU resource scheduling and efficient KV cache transfer. Third, we introduce a pull-based KV cache transfer strategy that reduces GPU resource idling and improves latency. Finally, we implement KVDirect as an open-source LLM inference framework. Our evaluation demonstrates that KVDirect reduces per-request latency by 55% compared to the baseline across diverse workloads under the same resource constraints.
Forward citations
Cited by 6 Pith papers
-
Spatial Prefix Caching for Wireless Edge LLM Inference: A Stochastic-Geometry and Queueing Framework
A stochastic-geometry and queueing framework derives prefix-cache association and TTFT coverage for wireless edge LLM inference, revealing load-dependent caching effects.
-
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.
-
An Internet for the KV Cache: Rethinking Classical Infrastructure Boundaries in the LLM Inference Age
Proposes an 'Internet for the KV Cache' where compute, storage, and network are jointly managed across cloud boundaries to decide when to store, transfer, or recompute cached LLM state.
-
Online Linear Programming for Multi-Objective Routing in LLM Serving
Online LP bid-price routing with warm-started dual gradient updates substantially improves multi-SLO LLM decode routing over heuristics in simulation.
-
Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference
HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.
-
Beyond the Buzz: A Pragmatic Take on Inference Disaggregation
Disaggregated serving (separate prefill and decode GPU pools) expands the throughput-interactivity Pareto frontier mainly for prefill-heavy workloads and models larger than about 10B parameters, provided the prefill-t...
Discussion (0). Continue with ORCID to comment.