Pith. sign in

REVIEW 4 cited by

DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.14838 v4 pith:3SCIASRY submitted 2024-12-19 cs.CL

Xiabin Zhou , Wenbin Wang , Minyan Zeng , Jiaxian Guo , Xuebo Liu , Li Shen , Min Zhang , Liang Ding This is my paper · ORCID
classification cs.CL
keywords cachedynamickvcompressionadaptivelayerlayersllmsmaximum
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization. Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics and reducing the retention of essential information. However, we observe distinct activation patterns across layers in various tasks, highlighting the need for adaptive strategies tailored to each task's unique demands. Based on this insight, we propose DynamicKV, a method that dynamically optimizes token retention by adjusting the number of tokens retained at each layer to adapt to the specific task. DynamicKV establishes global and per-layer maximum KV cache budgets, temporarily retaining the maximum budget for the current layer, and periodically updating the KV cache sizes of all preceding layers during inference. Our method retains only 1.7% of the KV cache size while achieving ~85% of the Full KV cache performance on LongBench. Notably, even under extreme compression (0.9%), DynamicKV surpasses state-of-the-art (SOTA) methods by 11% in the Needle-in-a-Haystack test using Mistral-7B-Instruct-v0.2. The code will be released.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Region-labeled tokens in LLM agent traces have order-of-magnitude different attention lifetimes, and a decay-plus-pinning eviction policy preserves system facts under fixed KV budgets while recency collapses as context grows.

  2. CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Offline-learned head-reliability and risk-threshold tables make prefill-only KV compression recover about 97.7% of uncompressed LongBench accuracy at a 512-token-per-layer memory budget.

  3. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  4. TPLA: Tensor Parallel Latent Attention for Efficient Disaggregated Prefill and Decode Inference

    cs.LG 2025-08 conditional novelty 5.0 of 10

    TPLA splits the latent KV cache across tensor-parallel GPUs while keeping every head's full view, yielding 1.79x and 1.93x decode speedups on DeepSeek-V3 and Kimi-K2 at 32K context with modest accuracy loss.

Pith tools