REVIEW 7 cited by
KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The development of large language models (LLMs) has significantly expanded model sizes, resulting in substantial GPU memory requirements during inference. The key and value storage of the attention map in the KV (key-value) cache accounts for more than 80\% of this memory consumption. Nowadays, most existing KV cache compression methods focus on intra-layer compression within a single Transformer layer but few works consider layer-wise compression. In this paper, we propose a plug-and-play method called \textit{KVSharer}, which shares the KV cache between layers to achieve layer-wise compression. Rather than intuitively sharing based on higher similarity, we discover a counterintuitive phenomenon: sharing dissimilar KV caches better preserves the model performance. Experiments show that \textit{KVSharer} can reduce KV cache computation by 30\%, thereby lowering memory consumption without significantly impacting model performance and it can also achieve at least 1.3 times generation acceleration. Additionally, we verify that \textit{KVSharer} is compatible with existing intra-layer KV cache compression methods, and combining both can further save memory.
Forward citations
Cited by 7 Pith papers
-
From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
KV-cache serving systems concentrate into five archetypes under a four-axis taxonomy, with ownership explaining residual distributed design variance and seven measurement gaps blocking next steps.
-
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
DepthWeave-KV achieves 8.3x KV cache memory reduction with near-full-cache task quality by factorizing key-value states across transformer layers using shared bases and token-adaptive residuals.
-
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Frequency-guided inter-layer KV sharing with logit-aware head routing nearly matches full-cache long-context accuracy at about 3.9× lower peak KV memory.
-
TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference
TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.
-
DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression
A dynamic attention-aware metric for task-agnostic prompt compression that combines entropy and attention scores, with iterative entropy recalculation, outperforms prior entropy-based compressors on multiple benchmarks.
-
Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing
Krul dynamically selects per-conversation cross-layer KV cache compression from attention similarity patterns, cutting TTFT by 1.28x to 2.68x and KV storage by 1.33x to 2.35x with less than 1% average accuracy loss on...
-
CE-LSLM: Efficient Large-Small Language Model Inference and Communication via Cloud-Edge Collaboration
CE-LSLM reports lower edge inference latency by reusing cloud LLM KV caches in edge SLMs, but the cross-model reuse mechanism is asserted, not demonstrated.
Discussion (0). Continue with ORCID to comment.