Pith. sign in

REVIEW 7 cited by

Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matters

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.12335 v2 pith:BNSYW7MR submitted 2024-06-18 cs.CL cs.LG

Zhiyu Guo , Hidetaka Kamigaito , Taro Watanabe This is my paper · ORCID
classification cs.CLcs.LG
keywords tokenattentionvalueimportancellmscachescorestasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Scaling the context size of large language models (LLMs) enables them to perform various new tasks, e.g., book summarization. However, the memory cost of the Key and Value (KV) cache in attention significantly limits the practical applications of LLMs. Recent works have explored token pruning for KV cache reduction in LLMs, relying solely on attention scores as a token importance indicator. However, our investigation into value vector norms revealed a notably non-uniform pattern questioning their reliance only on attention scores. Inspired by this, we propose a new method: Value-Aware Token Pruning (VATP) which uses both attention scores and the $ \ell_{1} $ norm of value vectors to evaluate token importance. Extensive experiments on LLaMA2-7B-chat and Vicuna-v1.5-7B across 16 LongBench tasks demonstrate that VATP outperforms attention-score-only baselines in over 12 tasks, confirming the effectiveness of incorporating value vector norms into token importance evaluation of LLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  2. Stability Implies Redundancy: Delta Attention Selective Halting for Efficient Long-Context Prefilling

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    DASH selectively halts stabilized tokens by monitoring layer-wise self-attention deltas, delivering prefill speedups on language and vision tasks without accuracy loss or hardware incompatibility.

  3. Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Models

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A training-free two-stage token pruning method for video-language models, using eigenvalue decay of token correlations to set a content-adaptive retention ratio.

  4. ART: Attention Run-time Termination for Efficient Large Language Model Decoding

    cs.CL 2026-04 conditional novelty 6.0 of 10

    ART early-terminates FlashAttention-style KV block traversal when intermediate attention outputs stabilize in scale and direction, speeding decoding up to ~20% atop existing KV methods.

  5. CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    Offline-learned head-reliability and risk-threshold tables make prefill-only KV compression recover about 97.7% of uncompressed LongBench accuracy at a 512-token-per-layer memory budget.

  6. GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction

    cs.CL 2025-08 conditional novelty 5.0 of 10

    GraphKV adds a similarity-based decay penalty to existing top-k KV cache eviction scores, improving retention diversity and accuracy on LongBench and needle-in-a-haystack tasks.

  7. Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decoding

    cs.LG 2025-08 reject novelty 4.0 of 10

    This paper proposes filtering cloud-verification requests by combining token-level uncertainty with attention-based importance, claiming energy savings up to 40.7% in wireless hybrid LLM inference.

Pith tools