Pith. sign in

REVIEW 5 cited by

SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.06219 v3 pith:RTVUFAF5 submitted 2024-05-10 cs.LG cs.CL

Haojie Duanmu , Zhihang Yuan , Xiuhong Li , Jiangfei Duan , Xingcheng Zhang , Dahua Lin This is my paper · ORCID
classification cs.LGcs.CL
keywords cachequantizationskvqaccuracyllmschannelscontexthigh
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) can now handle longer sequences of tokens, enabling complex tasks like book understanding and generating lengthy novels. However, the key-value (KV) cache required for LLMs consumes substantial memory as context length increasing, becoming the bottleneck for deployment. In this paper, we present a strategy called SKVQ, which stands for sliding-window KV cache quantization, to address the issue of extremely low bitwidth KV cache quantization. To achieve this, SKVQ rearranges the channels of the KV cache in order to improve the similarity of channels in quantization groups, and applies clipped dynamic quantization at the group level. Additionally, SKVQ ensures that the most recent window tokens in the KV cache are preserved with high precision. This helps maintain the accuracy of a small but important portion of the KV cache.SKVQ achieves high compression ratios while maintaining accuracy. Our evaluation on LLMs demonstrates that SKVQ surpasses previous quantization approaches, allowing for quantization of the KV cache to 2-bit keys and 1.5-bit values with minimal loss of accuracy. With SKVQ, it is possible to process context lengths of up to 1M on an 80GB memory GPU for a 7b model and up to 7 times faster decoding.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

    cs.AR 2026-02 conditional novelty 6.0 of 10

    Harmonia runs LLM inference with all activations in block floating point (BFP) and a 4-bit KV cache, reporting 3.08x average speedup, 2.03x energy savings, and under 1% accuracy loss on LongBench.

  2. Which Heads Matter for Reasoning? RL-Guided KV Cache Compression

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A small set of "reasoning heads" found by RL can keep full KV cache while other heads are compressed to a constant size, giving 20–50% cache savings with near-lossless accuracy.

  3. PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs

    cs.CL 2025-05 conditional novelty 6.0 of 10

    PM-KVQ introduces progressive bit-width shrinking, per-block memory allocation, and positional-interpolation calibration to make 2-bit KV cache quantization nearly lossless on long-CoT LLMs.

  4. Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache

    cs.CL 2025-06 conditional novelty 5.0 of 10

    Compressing the less context-sensitive dimensions of the LLM KV cache with a truncated Fourier transform preserves long-context accuracy on LongBench and needle-in-a-haystack tests, while the claimed memory savings ar...

  5. Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents

    cs.AI 2025-02 conditional novelty 5.0 of 10

    The authors propose episodic memory, with five defining properties, as the unifying framework needed for LLM agents to learn and remember over long time horizons.

Pith tools