Pith. sign in

REVIEW 15 cited by

MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.17565 v3 pith:KVV2YESP submitted 2024-06-25 cs.DC

Cunchen Hu , Heyang Huang , Junhao Hu , Jiang Xu , Xusheng Chen , Tao Xie , Chenxi Wang , Sa Wang
show 3 more authors
Yungang Bao Ninghui Sun Yizhou Shan
This is my paper · ORCID
classification cs.DC
keywords memservecachingcontextdisaggregatedmemoryservingcacheelastic
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language model (LLM) serving has transformed from stateless to stateful systems, utilizing techniques like context caching and disaggregated inference. These optimizations extend the lifespan and domain of the KV cache, necessitating a new architectural approach. We present MemServe, a unified system that integrates both inter-request and intra-request optimizations. MemServe introduces MemPool, an elastic memory pool managing distributed memory and KV caches across serving instances. Using MemPool APIs, MemServe combines context caching with disaggregated inference for the first time, supported by a global scheduler that enhances cache reuse through a global prompt tree-based locality-aware policy. Tests show that MemServe significantly improves job completion time and time-to-first-time.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Spatial Prefix Caching for Wireless Edge LLM Inference: A Stochastic-Geometry and Queueing Framework

    cs.IT 2026-08 accept novelty 7.0 of 10

    A stochastic-geometry and queueing framework derives prefix-cache association and TTFT coverage for wireless edge LLM inference, revealing load-dependent caching effects.

  2. Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework

    cs.DC 2026-07 conditional novelty 7.0 of 10

    Agent serving faces non-inference bottlenecks (up to 48% of latency), a 4.4x serving-capacity loss from long context, and 4.9x cost amplification from snapshot-based sandbox suspension.

  3. From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving

    cs.DC 2026-06 accept novelty 6.5 of 10

    KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.

  4. Energy-Efficient LLM Serving via Disaggregated Attention--FFN and Flexible Frequency Scaling

    cs.DC 2026-08 conditional novelty 6.0 of 10

    AFlex combines operator-level disaggregation with per-operator DVFS to cut LLM serving energy per token by up to 49% without violating P90 TTFT/TPOT SLOs.

  5. [AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows

    cs.DC 2026-07 conditional novelty 6.0 of 10

    AAFLOW+ elevates KV cache to a first-class distributed state object with materialize/transfer/fork operators, replacing text replay and cutting multi-agent TTFT up to 50× under an analytical cost model.

  6. Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Execution

    cs.AI 2025-09 conditional novelty 6.0 of 10

    Auras, a perception-generation disaggregation framework with a public context buffer and asynchronous pipeline executor, raises embodied-agent throughput by 2.54x on average without losing accuracy (102.7%).

  7. Taming the Chaos: Coordinated Autoscaling for Heterogeneous and Disaggregated LLM Inference

    cs.DC 2025-08 conditional novelty 6.0 of 10

    HeteroScale coordinates scaling of prefill and decode pools using decode TPS as a single robust signal, reporting a 26.6 percentage point GPU utilization gain in production.

  8. Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics

    cs.DB 2025-08 conditional novelty 6.0 of 10

    SISO caches representative centroid embeddings of similar queries and adjusts its similarity threshold under load, improving semantic cache hit ratios for LLM serving by up to 1.71x with a small accuracy cost.

  9. DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch

    cs.DC 2026-07 conditional novelty 5.0 of 10

    DualDecoder prefetches sparse KV-cache entries using a co-executed speculative token, removing GPU-resident auxiliary state and improving long-context LLM decoding throughput by up to 2.62x.

  10. Beyond the Buzz: A Pragmatic Take on Inference Disaggregation

    cs.DC 2025-06 conditional novelty 5.0 of 10

    Disaggregated serving (separate prefill and decode GPU pools) expands the throughput-interactivity Pareto frontier mainly for prefill-heavy workloads and models larger than about 10B parameters, provided the prefill-t...

  11. CoDec: Prefix-Shared Decoding Kernel for LLMs

    cs.LG 2025-05 conditional novelty 5.0 of 10

    CoDec combines KV-cache reads across requests that share a prefix, yielding average 1.9x decode-attention speedup and 120.9x less global memory traffic versus FlashDecoding.

  12. HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference

    cs.DC 2025-02 conditional novelty 5.0 of 10

    HACK performs attention matrix multiplications directly on 2-bit quantized key-value cache entries, eliminating per-iteration dequantization and cutting end-to-end latency in disaggregated LLM serving.

  13. DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing

    cs.LG 2025-11 conditional novelty 4.0 of 10

    DuetServe dynamically splits a GPU's compute cores between prefill and decode only when a latency model predicts trouble, improving serving throughput by up to 1.3x at similar or better token latency.

  14. Secure Multi-LLM Agentic AI and Agentification for Edge General Intelligence by Zero-Trust: A Survey

    cs.NI 2025-08 conditional novelty 4.0 of 10

    A survey proposing zero-trust architecture for multi-LLM systems in edge computing, with a taxonomy of model- and system-level defenses and a conceptual framework.

  15. DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Acceleration

    cs.CL 2025-06 conditional novelty 4.0 of 10

    DAM derives per-layer and per-head attention masks from a calibration dataset and extrapolates them to long inputs, matching full-attention retrieval accuracy while reducing memory and compute.

Pith tools