REVIEW 8 cited by
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
PagedAttention is a popular approach for dynamic memory allocation in LLM serving systems. It enables on-demand allocation of GPU memory to mitigate KV cache fragmentation -- a phenomenon that crippled the batch size (and consequently throughput) in prior systems. However, in trying to allocate physical memory at runtime, PagedAttention ends up changing the virtual memory layout of the KV cache from contiguous to non-contiguous. Such a design leads to non-trivial programming and performance overheads. We present vAttention -- an approach that mitigates fragmentation in physical memory while retaining the contiguity of KV cache in virtual memory. We achieve this by decoupling the allocation of virtual and physical memory using CUDA virtual memory management APIs. We also introduce various LLM-specific optimizations to address the limitations of CUDA virtual memory support. Overall, vAttention is a simpler, portable, and performant alternative to PagedAttention: it supports various attention kernels out-of-the-box and improves LLM serving throughput by up to 1.23x compared to the use of PagedAttention-based kernels of FlashAttention and FlashInfer.
Forward citations
Cited by 8 Pith papers
-
What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents
KV-cache eviction, prompt compression, recurrent state bounding, and agent memory consolidation are unified as one rate-distortion problem with a shared lower bound, shared failure mode, and transferable mechanisms.
-
Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Keyless Attention eliminates the key projection in transformers via value-space routing, cutting KV cache size by 50% with competitive or superior performance on language modeling and zero-shot benchmarks.
-
Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving
Pattern-aware speculative tool execution cuts agent end-to-end latency by roughly half and observed tool latency by about 1.8× by overlapping predicted tools with LLM generation.
-
HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
HGCA splits attention between GPU (dense, recent KV) and CPU (sparse, salient KV) and merges partial results with exact log-sum-exp fusion, scaling long-context decoding on commodity GPUs.
-
cuRAMSES: Scalable AMR Optimizations for Large-Scale Cosmological Simulations
Recursive k-section domain decomposition, Morton-key hashing, and GPU dispatch cut communication and memory bottlenecks in RAMSES while preserving conservation to ~0.5%.
-
AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up
The framework trains small models on synthetic AI-generated data to produce PFD/PID text, then validates two examples by manual DWSIM setup, leaving the industrial-viability claim unproven.
-
Position: AI Scaling: From Up to Down and Out
AI scaling is reframed as three paradigms: Scaling Up, Scaling Down, and Scaling Out, with future gains predicted to come from down and out.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
Discussion (0). Continue with ORCID to comment.