REVIEW 4 cited by
Stateful Large Language Model Serving with Pensieve
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Stateful Large Language Model Serving with Pensieve
read the original abstract
Large Language Models (LLMs) are wildly popular today and it is important to serve them efficiently. Existing LLM serving systems are stateless across requests. Consequently, when LLMs are used in the common setting of multi-turn conversations, a growing log of the conversation history must be processed alongside any request by the serving system at each turn, resulting in repeated processing. In this paper, we design $Pensieve$, a system optimized for multi-turn conversation LLM serving. $Pensieve$ maintains the conversation state across requests by caching previously processed history to avoid duplicate processing. $Pensieve$'s multi-tier caching strategy can utilize both GPU and CPU memory to efficiently store and retrieve cached data. $Pensieve$ also generalizes the recent PagedAttention kernel to support attention between multiple input tokens with a GPU cache spread over non-contiguous memory. Our evaluation shows that $Pensieve$ can achieve $1.14$-$3.0\times$ the throughput of vLLM and TensorRT-LLM and significantly reduce latency.
Forward citations
Cited by 4 Pith papers
-
Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving
Execution-state capsules enable graph-bound full-state checkpointing and sub-millisecond restore for LLMs including KV and recurrent states, yielding 3.9x-27x TTFT speedups in on-device physical-AI serving.
-
SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling
Balancing only first-turn agent requests while keeping follow-ups cache-aware raises LLM cluster TPS 10-16% (colocation) and prefill TPS 2-34% (disaggregation) over SOTA schedulers on production agent traces.
-
HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory
HyMCache makes SSD-backed CXL-hybrid memory behave like a DRAM-tier KV cache for multi-turn LLM serving by prefetching known prefix blocks and isolating writes, achieving ~3x over local caching and ~70% of distributed...
-
ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories
ITME uses CXL-hybrid memories for byte-addressable remote memory expansion in LLM inference, achieving up to 35.7% throughput improvement over conventional CPU-offloading.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.