Pith. sign in

REVIEW 12 cited by

No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.18096 v1 pith:LSPXIHOA submitted 2024-02-28 cs.LG cs.AI

June Yong Yang , Byeongwook Kim , Jeongin Bae , Beomseok Kwon , Gunho Park , Eunho Yang , Se Jung Kwon , Dongsoo Lee This is my paper · ORCID
classification cs.LGcs.AI
keywords cachepairscompressionprecisionsizecontainedcontexteven
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models~(LLMs). However, the memory footprint of the KV cache poses a critical bottleneck in LLM deployment as the cache size grows with batch size and sequence length, often surpassing even the size of the model itself. Although recent methods were proposed to select and evict unimportant KV pairs from the cache to reduce memory consumption, the potential ramifications of eviction on the generative process are yet to be thoroughly examined. In this paper, we examine the detrimental impact of cache eviction and observe that unforeseen risks arise as the information contained in the KV pairs is exhaustively discarded, resulting in safety breaches, hallucinations, and context loss. Surprisingly, we find that preserving even a small amount of information contained in the evicted KV pairs via reduced precision quantization substantially recovers the incurred degradation. On the other hand, we observe that the important KV pairs must be kept at a relatively higher precision to safeguard the generation quality. Motivated by these observations, we propose \textit{Mixed-precision KV cache}~(MiKV), a reliable cache compression method that simultaneously preserves the context details by retaining the evicted KV pairs in low-precision and ensure generation quality by keeping the important KV pairs in high-precision. Experiments on diverse benchmarks and LLM backbones show that our proposed method offers a state-of-the-art trade-off between compression ratio and performance, compared to other baselines.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Seen, Said, or Forgotten? A Causal Audit of Visual KV Memory Across Dialog Turns

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Current attention is not a safe-forgetting signal for visual KV memory; damage from eviction concentrates in visually dependent turns, and only explicitly verbalized facts are reliably rescued by assistant text.

  2. LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

    cs.LG 2026-07 accept novelty 7.0 of 10

    Page-local rank-8 spectral key summaries let sparse decode selection track the exact mass oracle and match FullKV quality at ~2% attended tokens with 2× latency cut at 1M context.

  3. XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization

    cs.LG 2025-08 conditional novelty 7.0 of 10

    XQuant caches quantized layer inputs and rematerializes Keys and Values on the fly, achieving up to 12.5x KV cache memory savings with about 0.1 perplexity degradation.

  4. Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Training checkpoint-specific orthogonal KV-cache rotations with a DCT spectral-concentration objective cuts zfp-compression KL divergence to the full-cache model by 44% at matched rates.

  5. Token Sparse Attention: Efficient Long-Context Inference with Interleaved Token Selection

    cs.CL 2026-02 unverdicted novelty 6.0 of 10

    Token Sparse Attention uses dynamic per-head token compression and decompression during attention to achieve up to 3.23x speedup at 128K context with under 1% accuracy loss.

  6. CaliDrop: KV Cache Compression with Calibration

    cs.CL 2025-07 conditional novelty 6.0 of 10

    CaliDrop adds a stale-query calibration term on top of token eviction, improving accuracy at high KV compression ratios with modest throughput overhead.

  7. MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference

    cs.LG 2025-06 conditional novelty 6.0 of 10

    MadaKV adaptively splits the KV cache budget by attention-head modality preference and compensates across layers, cutting cache memory by 80-95% and speeding decoding by 1.3-1.5x with small accuracy loss.

  8. TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization

    cs.CL 2025-05 conditional novelty 6.0 of 10

    TailorKV combines 1-bit quantization in shallow attention layers with dynamic Top-K token retrieval in deeper layers to serve 128k-context Llama-3.1-8B on a single 24GB GPU with a small accuracy loss.

  9. PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    PM-KVQ introduces progressive bit-width shrinking, per-block memory allocation, and positional-interpolation calibration to make 2-bit KV cache quantization nearly lossless on long-CoT LLMs.

  10. NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache

    cs.LG 2025-05 conditional novelty 6.0 of 10

    NSNQuant applies a Normalize-Shift-Normalize transform plus a Hadamard rotation to make KV cache channels match a standard normal distribution, so one codebook trained on random noise can quantize them without calibration.

  11. SpindleKV: A Novel KV Cache Reduction Method Balancing Both Shallow and Deep Layers

    cs.CL 2025-07 conditional novelty 5.0 of 10

    SpindleKV compresses LLM KV cache by evicting low-attention tokens in deep layers and replacing near-duplicate key and value vectors in shallow layers with a shared codebook, while preserving benchmark accuracy at 15 ...

  12. MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MoQAE learns chunk-level KV cache bit-width selection with a mixture-of-experts router, improving accuracy and efficiency over prior quantization methods.

Pith tools