Pith. sign in

REVIEW 3 major objections 4 minor 47 references

A runtime meter certifies how much KV-cache compression damages each request, per step.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:37 UTC pith:I4GXE53G

load-bearing objection WitCert delivers a genuinely new runtime-certified bound for KV-cache quantization, but the fp16 witness store may void the deployed guarantee. the 3 major comments →

arxiv 2607.28699 v1 pith:I4GXE53G submitted 2026-07-30 cs.AR cs.AI

WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization

Fanzhe Wei , Li Liu This is my paper
classification cs.AR cs.AI
keywords KV-cache quantizationtotal variation boundruntime certificationsoftmax perturbationband-norm witnesssubtractive ditherattention repairLLM serving
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that the harm done by KV-cache compression can be bounded soundly at runtime, per layer, per attention head, per decoding step, instead of being judged only by offline benchmark averages. The bound is a total-variation upper bound on the difference between exact and compressed attention, computed from per-frequency-band norms of the quantization residual stored at cache-write time plus the current query. That makes compression a closed loop: a serving system can see in real time whether the request it is serving is being degraded, and page exact values back in to repair it. The paper reports that meter-driven gating restores a failing fp8 cache from 22.8 to 79.7 on hard needle-in-a-haystack retrieval tasks, with the difference from uncompressed bounded at [+0.0, +0.8], and that a certified int8 cache holds 1.88 times more KV tokens at the same memory.

Core claim

WitCert's load-bearing result is an identity: if the logit error of every cached token is bounded by c_t, then the total variation between exact and compressed attention satisfies TV(p, p̃) ≤ ½(A² − 1), where A = Σ_t p̃_t e^{c_t} is the attention-weighted mean of e^c. The paper proves the per-token bound c_t = d^(−1/2) Σ_b ‖q_b‖ w_{t,b} can be computed at decode time from the current query and a witness — the per-band Euclidean norms of the quantization residual, stored once at write time — and that this witness is position-invariant because RoPE rotations are unitary within each frequency band. The authors therefore claim a runtime-computable, data-dependent, worst-case valid TV bound that

What carries the argument

The band-norm residual witness. At write time, for each cached token, the residual between the exact and compressed key is split into B frequency bands of the rotary embedding, and the Euclidean norm of each band is stored (32 bytes per token per head). At read time, Cauchy–Schwarz within each band plus the current query gives the per-token error bound c_t = d^(−1/2) Σ_b ‖q_b‖ w_{t,b}; because RoPE is orthogonal on each band, the witness is valid at every query position. This c_t is then fed to the inequality TV(p,p̃) ≤ ½(A²−1), whose exponential term A = Σ_t p̃_t e^{c_t} is itself attention-weighted and merges cleanly into the softmax's log-sum-exp reduction, making the whole meter fuse int

Load-bearing premise

The probabilistic certificate assumes the query is statistically independent of the dither noise used at quantization time; in free-running generation the query depends on earlier attention outputs and hence on those very residuals, so the formal guarantee does not apply — only empirical validation (0/1000 adaptive requests, 100M+ observed cells) supports that mode.

What would settle it

Run free-running decoding on an adversarially chosen long prompt with a fixed dither seed, computing exact and compressed attention at every step; a single (layer, head, step) cell whose true total variation exceeds the certificate would falsify the system's deployment soundness. The paper's own validation used 50 ordinary prompts and found zero violations, so a directed search that concentrates attention on high-residual tokens would test the independence premise directly.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any cache-preserving quantization scheme becomes observable in live serving: the meter's per-step coverage predicts end-task safety at benchmark scale.
  • Meter-driven gating restores the quality floor of broken compression: a raw fp8 cache jumps from 22.8 to 79.7 on hard retrieval tasks, with paired difference from uncompressed bounded at [+0.0, +0.8].
  • A certified int8 cache holds 1.88× more KV tokens at the same memory, with the certificate adding roughly 12% kernel-level overhead on an fp16 store and essentially zero once storage is already quantized.
  • Aggressive 2-bit schemes survive through cross-layer error cancellation, not per-step fidelity; in a 28-layer sweep no single layer's pollution alone loses anything (0/28).
  • At a 1% request-level failure budget, the probabilistic certificate halves the page-in rate and authorizes compression 3.43 bits/dim deeper than the previous deterministic bound.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The layer-pollution sweep suggests a composed multi-layer certificate could be far tighter than any per-layer bound; the paper leaves that as the natural next target, and nothing in the framework forbids it.
  • Because the Tier-B probabilistic guarantee is formally valid only for non-adaptive queries, live free-running decoding rests on empirical validation; until a martingale-style adaptive bound exists, the deterministic Tier-A witness is the only formally covered mode in deployment.
  • The same witness machinery might extend to token-eviction compression, where the exact distortion equals dropped attention mass; the paper's negative evidence on cheap online witnesses for eviction turns this into a concrete open problem rather than a settled dead end.
  • The finding that subtractive dither is quality-neutral, with all quality gain coming from the outlier bypass, implies the probabilistic certificate costs no accuracy — a useful calibration for practitioners adopting dithered quantization.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes WitCert, a runtime meter for KV-cache quantization that computes a per-(layer, head, decode-step) upper bound on the total variation between exact and compressed attention. Tier A is a deterministic bound: for any cache-preserving black-box quantizer, per-band norms of the write-time residual are stored as a witness, and Theorem 2 combines these with the current query via Cauchy–Schwarz and RoPE band unitarity to obtain per-token logit-error bounds, which Theorem 1 propagates to a TV bound. Tier B is a probabilistic alternative for a controlled subtractively dithered INT8 quantizer, using a sub-Gaussian proxy and a request-level failure budget; the probabilistic guarantee is explicitly stated for non-adaptive queries only. The authors report SGLang integration, meter-driven gating that restores benchmark quality (e.g., raw FP8 from 22.8 to 79.7 on hard RULER tasks), and an analysis showing that aggressive schemes fail through cross-layer accumulation rather than per-step infidelity. Core inequalities and several experiments are accompanied by a Lean 4 development, adversarial validation, and extensive reproduction artifacts.

Significance. If the identified gaps are closed, this is a significant contribution: it moves KV-cache compression from offline-average validation to a runtime-observable quantity with a per-request, data-dependent guarantee, which is stronger than prior worst-case certificates such as the tanh bound. The mathematical core is simple and largely correct, and the paper is unusually careful in reporting retractions, negative results, and explicit scope limitations. The reproducibility apparatus—four independent implementations, a Lean development, adversarial per-query validation, and published raw JSON—is a genuine strength. However, the deployed soundness claim currently rests on an unstated precision assumption in the fp16 witness store, and the machine-checking claim as stated exceeds what Appendix A actually formalizes. These are local but load-bearing issues for the central claim of a sound runtime meter.

major comments (3)
  1. [§4.1, Eq. (2); Theorem 2] The witness w_{t,b} is stored in fp16 (32 B/tok/head) with no stated rounding mode. Theorem 2 requires the exact band norm to form a sound upper bound c_t = d^{-1/2} Σ_b ||q_b|| w_{t,b}; standard round-to-nearest-even can round downward, in which case the deployed c_t is not an upper bound on the true logit error. The paper must either require upward rounding to the next representable fp16 value, store an explicitly proven upper bound, add a rounding error term, or demonstrate that the kernel's arithmetic cannot round down. Without this, the Tier A meter is not provably sound on the deployed path, and the Lean theorems listed in Appendix A do not cover this step.
  2. [Appendix A / Abstract] The abstract and §7 state that the core theorems are machine-checked in Lean 4, but the compiled theorem list (L1–L4) contains only the e-form TV bound (Theorem 1), the uniform-dither sub-Gaussian lemma, the request-level union bound, and the blockwise/per-token refinement. Theorem 2, the black-box band-norm logit bound that drives Tier A, and Lemma 1 (RoPE band unitarity) are not among the formalized statements. The claim should be narrowed to the theorems actually checked, or Theorem 2 and Lemma 1 should be added to the formal development. This matters because Theorem 2 is the load-bearing link between the stored witness and the TV certificate.
  3. [§4.2 vs. §6.3.8, Table 17] Tier B's probabilistic guarantee is explicitly limited to non-adaptive queries (§4.2), with free-running decoding supported only by empirical validation (0/1000 adaptive requests). However, Table 17 reports 'Coverage (τ=0.2)' measured in live SGLang serving, which is free-running autoregressive decoding, without distinguishing formal certificate coverage from empirical validation. The 'certified INT8 cache' language in the abstract and §6.3.8 can therefore be read as extending a guarantee the theorem does not provide. The serving coverage numbers should either be labeled as empirical validation under adaptivity, be recomputed with the adaptive-safe Tier A, or be accompanied by a clear statement that the formal Tier B guarantee does not apply to those live coverage values.
minor comments (4)
  1. [§6.3.9, Table 18] The kivi+gate row is labeled 'certified' at τ=0.2, but the raw kivi-2bit Tier A witness is saturated (~5×10⁻⁵ coverage), so the gate decision before repair is made in the risk-ranked regime. Clarify whether 'certified' refers to the post-repair state (zeroed witnesses) or to the gating decision itself.
  2. [§5, Table 1 / §5.1] The n=4 long-CoT coverage table is explicitly a no-collapse check, but the constant 0.800/0.857 values across four samples with different lengths look suspiciously identical; state whether the rounded values hide meaningful variation or are exact to the displayed precision.
  3. [§7] The code repository is listed as private ('to be opened at publication'). For a paper whose claims rest on reproducible measurements and a proof–kernel contract, providing the code and Lean sources to reviewers is important; consider an anonymous downloadable artifact.
  4. [Appendix A] The phrase 'compiled, not transcribed' is helpful, but the list contains many non-core supporting lemmas; a table mapping each body theorem (Theorem 1, Theorem 2, Lemma 1, Lemma 2, Proposition 1) to its Lean file and statement name would make the coverage transparent.

Circularity Check

0 steps flagged

No significant circularity: the meter bounds are derived from definitions and classical inequalities, and no fitted quantity is renamed as a prediction.

full rationale

The derivation chain is self-contained rather than circular. Theorem 1 is an inequality relating TV(p,p̃) to a softmax-weighted exponential of per-token bounds c_t, conditional only on |ε_t| ≤ c_t. Theorem 2 constructs c_t by Cauchy–Schwarz from the stored band norms w_{t,b} = ‖r_{t,b}‖, which are measured residuals, not free parameters fitted to the predicted TV. Lemma 1 (RoPE band unitarity) is a direct algebraic fact. The Tier-B certificate rests on classical subtractive-dither theory and machine-checked lemmas; its request-level budget is an explicit design parameter, not a fitted value. The paper also reports an unsound candidate bound, a counterexample, a sound replacement, negative results, and multiple retractions, which is inconsistent with a hidden reduction of the conclusion to the inputs. Non-circular caveats worth noting: (1) Sec. 4.1 stores the witness in fp16 without stating a rounding mode, so the deployed Tier-A c_t could in principle cease to be an upper bound; this is a soundness implementation gap, not circularity. (2) Sec. 4.2 explicitly limits the probabilistic guarantee to non-adaptive queries and labels adaptive validation as validation, not proof. (3) Appendix A lists Lean-checked theorems L1–L4 but does not list the band-norm Cauchy–Schwarz bound; this is a coverage caveat, not a circular dependence. None of these issues makes the central derivation equivalent to its own inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

No new physical entities are postulated; the witness and certificate are algorithmic artifacts with direct in-paper measurement. The central claim rests on classical inequalities, RoPE structure, a controlled quantizer, and two load-bearing domain assumptions: non-adaptive queries for the probabilistic tier, and exact/guarded witness storage.

free parameters (5)
  • outlier RoPE pair count m = default 4; sweep 0/2/8/16; zero-fallback at 16
    Chosen by ablation over coverage vs memory (Tables 6, 14, 15); affects certificate tightness and byte cost, not theorem soundness.
  • band count B = 16 (system integration), 8 (offline study)
    Design choice; larger B tightens the Cauchy-Schwarz bound but costs 2B B/tok/head storage.
  • gate thresholds τ_K, τ_V = 0.2, 0.05; risk-ranked τ=5
    Operator-set output-error budget / traffic knob; selected from profiling in risk-ranked mode, not fit to the theorem.
  • request-level failure budget δ_req = 1e-2
    User-specified risk budget allocated by union bound over L·H·T; the coverage-δ trade is measured, not used to fit constants.
  • scale granularity G_c = 32 channels per group (token-level)
    Paging-safe choice determined by SGLang slot allocation; selected by empirical quality comparison (Table 4).
axioms (6)
  • standard math Band-wise Cauchy-Schwarz with RoPE band unitarity
    Used to derive Theorem 2 and Lemma 1; assumes each RoPE frequency pair lies in a single band and rotation is orthogonal on that band.
  • domain assumption Classical subtractive-dither residual independence and uniformity
    Basis of Lemma 2 and Tier B; imported from Gray & Stockham [14] / Lipshitz [26] and required to hold in the shipped kernel.
  • domain assumption No clipping on non-outlier channels: s = amax/(Q − 1/2)
    The kernel must guarantee the uniform residual distribution; paper reports contract tests for no clipping, but the theorem depends on this engineering premise.
  • domain assumption Non-adaptive queries for Tier B: query independent of stored dither
    Explicit in Sec. 4.2; fails in free-running decoding, so adaptive use is only empirically validated, not theorem-backed.
  • domain assumption Exact (or one-sided safe) storage of witness norms
    Theorem 2 needs exact ||r_t,b||; the paper stores fp16 witness without describing an upward-rounding guard (Sec. 4.1).
  • domain assumption Philox dither reconstruction exact via five-tuple
    The read side must reproduce the write dither; the five-tuple counter is invariant to scheduling and CUDA-graph replay.

pith-pipeline@v1.3.0-alltime-deepseek · 35308 in / 14944 out tokens · 142730 ms · 2026-08-03T00:37:39.490775+00:00 · methodology

0 comments
read the original abstract

KV-cache quantization is validated today by offline benchmark averages; a deployed system cannot tell whether compression is damaging the request it is serving right now. We give it a provably sound runtime meter, a "DTrace for KV quantization": a per-(layer, head, step) upper bound on the total variation between exact and compressed attention. The meter has two tiers: a deterministic band-norm-witness bound, sound for any cache-preserving black-box quantizer and for any query (adaptive-safe, worst-case Cauchy-Schwarz plus RoPE band-unitarity), and a tighter probabilistic certificate for a controlled subtractively-dithered INT8 quantizer under an explicit request-level failure budget (stated for non-adaptive queries; core theorems machine-checked in Lean 4). Three results. Observability: the meter enters SGLang through an environment-guarded patch, and any scheme registered as one tensor function is measured in live serving. Repair: meter-driven gating, risk-ranked where the witness is saturated and certified where it is informative, empirically restores the quality floor at benchmark scale. For example, raw-cast FP8 improves from 22.8 back to 79.7 on hard RULER tasks, with the difference from uncompressed bounded at [+0.0, +0.8] by a paired test. Analysis: aggressive schemes survive on cross-layer error cancellation, not per-step fidelity. In a 28-layer sweep, no single layer's pollution alone loses anything (0/28), and the certified INT8 cache serves 1.88 times more KV tokens at the same memory in SGLang.

Figures

Figures reproduced from arXiv: 2607.28699 by Fanzhe Wei, Li Liu.

Figure 1
Figure 1. Figure 1: The universal meter on six black-box quantizers, per-cell diagnostics ( [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The WitCert observatory. Any scheme’s residual leaves a witness at write time; the [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Meter-guided repair of a broken fp8 KV cache (24-prompt needle task). Four configura [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sound, not clairvoyant: meter coverage at [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Meter gating at benchmark scale (labels per Sec. 4.3: risk-ranked at [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Aggressive schemes survive on structure, not per-step fidelity: per-step danger rate (frac [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The gate is selective and tunable: one-time page-in traffic (24 requests) as the gate [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 31 linked inside Pith

  1. [1]

    Yi: Open foundation models by 01.AI.arXiv:2403.04652, 2024

    01.AI. Yi: Open foundation models by 01.AI.arXiv:2403.04652, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.arXiv:2107.07511, 2021

    Pith/arXiv arXiv 2021

  3. [3]

    LongBench: Abilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, XiaoLiu, AohanZeng, LeiHou, YuxiaoDong, JieTang, andJuanziLi. LongBench: Abilingual, multitask benchmark for long context understanding. InACL, 2024. arXiv:2308.14508

    Pith/arXiv arXiv 2024

  4. [4]

    STAR-KV: Low-rank KV cache compression via soft thresholding for adaptive rank control.arXiv:2606.08382, 2026

    Priyansh Bhatnagar, Ashkan Moradifirouzabadi, Se-Hyun Yang, SeungJae Lee, Jungwook Choi, and Mingu Kang. STAR-KV: Low-rank KV cache compression via soft thresholding for adaptive rank control.arXiv:2606.08382, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    Runtime-certified bounded-error quantized attention.arXiv:2605.20868, 2026

    Dean Calver. Runtime-certified bounded-error quantized attention.arXiv:2605.20868, 2026

    Pith/arXiv arXiv 2026

  6. [6]

    Abdelfattah, and Kai-Chiang Wu

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, Chong-Yan Chen, Yu-Fang Hu, Pei-Shuo Wang, Ning-Chi Huang, Luis Ceze, Mohamed S. Abdelfattah, and Kai-Chiang Wu. Palu: Compressing KV-cache with low-rank projection. InICLR, 2025. arXiv:2407.21118

    Pith/arXiv arXiv 2025

  7. [7]

    Compactor: Calibrated query-agnostic KV cache compression with approximate leverage scores.arXiv:2507.08143, 2025

    Vivek Chari and Benjamin Van Durme. Compactor: Calibrated query-agnostic KV cache compression with approximate leverage scores.arXiv:2507.08143, 2025

    arXiv 2025

  8. [8]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InNeurIPS, 2022. arXiv:2205.14135

    Pith/arXiv arXiv 2022

  9. [9]

    Flash-decoding for long-context inference.https://pytorch.org/blog/flash-decoding/, 2023

    Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference.https://pytorch.org/blog/flash-decoding/, 2023

    2023

  10. [10]

    The Lean 4 theorem prover and programming language

    Leonardo de Moura and Sebastian Ullrich. The Lean 4 theorem prover and programming language. InCADE-28, 2021

    2021

  11. [11]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv:2501.12948, 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Expected attention: KV cache com- pression by estimating attention from future queries distribution.arXiv:2510.00636, 2025

    Alessio Devoto, Maximilian Jeblick, and Simon Jégou. Expected attention: KV cache com- pression by estimating attention from future queries distribution.arXiv:2510.00636, 2025. 36

    arXiv 2025

  13. [13]

    A simple and effective L2 norm-based strategy for KV cache compression.arXiv:2406.11430, 2024

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective L2 norm-based strategy for KV cache compression.arXiv:2406.11430, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Gray and Thomas G

    Robert M. Gray and Thomas G. Stockham. Dithered quantizers.IEEE Transactions on Information Theory, 39(3):805–812, 1993

    1993

  15. [15]

    Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization.arXiv:2401.18079, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    RULER: What’s the real context size of your long-context language models? InCOLM, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? InCOLM, 2024. arXiv:2404.06654

    Pith/arXiv arXiv 2024

  17. [17]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al. Mistral 7B.arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Lee, Sangdoo Yun, and Hyun Oh Song

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, and Hyun Oh Song. KVzip: Query-agnostic KV cache compression with context reconstruction. InNeurIPS, 2025. arXiv:2505.23416

    arXiv 2025

  19. [19]

    Streaming attention approximation via discrepancy theory.arXiv:2502.07861, 2025

    Ekaterina Kochetkova, Kshiteej Sheth, Insu Han, Amir Zandieh, and Michael Kapralov. Streaming attention approximation via discrepancy theory.arXiv:2502.07861, 2025

    arXiv 2025

  20. [20]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InSOSP, 2023. arXiv:2309.06180

    Pith/arXiv arXiv 2023

  21. [21]

    Rakhshan, and Guillaume Rabusseau

    Damien Lesens, Beheshteh T. Rakhshan, and Guillaume Rabusseau. KQ-SVD: Compressing the KV cache with provable guarantees on attention fidelity.arXiv:2512.05916, 2025

    arXiv 2025

  22. [22]

    CommVQ: Commutative vector quantization for KV cache compression

    Junyan Li, Yang Zhang, Muhammad Yusuf Hassan, Talha Chafekar, Tianle Cai, Zhile Ren, Pengsheng Guo, Foroozan Karimzadeh, Colorado Reed, Chong Wang, and Chuang Gan. CommVQ: Commutative vector quantization for KV cache compression. InICML, 2025. arXiv:2506.18879

    Pith/arXiv arXiv 2025

  23. [23]

    RaBitQ- Cache: Rotated binary quantization for KVCache in long context LLM inference

    Wenhao Li, Jinhao Dong, Hailin Zhang, Wenhang Shi, Wei Lu, and Xiaoyong Du. RaBitQ- Cache: Rotated binary quantization for KVCache in long context LLM inference. InICML,

  24. [24]

    SnapKV: LLM knows what you are looking for before generation

    Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. SnapKV: LLM knows what you are looking for before generation. InNeurIPS, 2024. arXiv:2404.14469

    Pith/arXiv arXiv 2024

  25. [25]

    RoPE-aware bit allocation for KV-cache quan- tization.arXiv:2606.24033, 2026

    Fengfeng Liang, Yuechen Zhang, and Jiaya Jia. RoPE-aware bit allocation for KV-cache quan- tization.arXiv:2606.24033, 2026

    Pith/arXiv arXiv 2026

  26. [26]

    Lipshitz, Robert A

    Stanley P. Lipshitz, Robert A. Wannamaker, and John Vanderkooy. Quantization and dither: A theoretical survey.Journal of the Audio Engineering Society, 40(5):355–375, 1992

    1992

  27. [27]

    KIVI: A tuning-free asymmetric 2bit quantization for KV cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. InICML,

  28. [28]

    Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, and Lukas Cavigelli

    Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, and Lukas Cavigelli. KVarN: Variance-normalized KV-cache quantization mitigates error accumulation in reasoning tasks.arXiv:2606.03458, 2026

    Pith/arXiv arXiv 2026

  29. [29]

    kvpress: LLM KV cache compression made easy.https://github.com/NVIDIA/ kvpress, 2024

    NVIDIA. kvpress: LLM KV cache compression made easy.https://github.com/NVIDIA/ kvpress, 2024

    2024

  30. [30]

    YaRN: Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. InICLR, 2024. arXiv:2309.00071

    Pith/arXiv arXiv 2024

  31. [31]

    Qwen2.5 technical report.arXiv:2412.15115, 2024

    Qwen Team. Qwen2.5 technical report.arXiv:2412.15115, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    The risk of KV cache compression.arXiv:2607.01520, 2026.https://arxiv.org/abs/2607. 01520

    Pith/arXiv arXiv 2026

  33. [33]

    Salmon, Mark A

    John K. Salmon, Mark A. Moraes, Ron O. Dror, and David E. Shaw. Parallel random numbers: As easy as 1, 2, 3. InSC, 2011

    2011

  34. [34]

    KV-Latent: Dimensional-level KV cache reduction with frequency-aware rotary positional embedding

    Luohe Shi, Zuchao Li, Lefei Zhang, Guoming Liu, Baoyuan Qi, and Hai Zhao. KV-Latent: Dimensional-level KV cache reduction with frequency-aware rotary positional embedding. In ACL, 2025. arXiv:2507.11273

    Pith/arXiv arXiv 2025

  35. [35]

    RoFormer: Enhanced transformer with rotary position embedding.arXiv:2104.09864, 2021

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.arXiv:2104.09864, 2021

    Pith/arXiv arXiv 2021

  36. [36]

    Zico Kolter, and Zhuang Liu

    Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models.arXiv:2402.17762, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Quest: Query-aware sparsity for efficient long-context LLM inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context LLM inference. InICML, 2024. arXiv:2406.10774

    Pith/arXiv arXiv 2024

  38. [38]

    The Lean mathematical library

    The mathlib Community. The Lean mathematical library. InCPP, 2020

    2020

  39. [39]

    Springer, 2005

    Vladimir Vovk, Alexander Gammerman, and Glenn Shafer.Algorithmic Learning in a Random World. Springer, 2005

    2005

  40. [40]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InICLR, 2024. arXiv:2309.17453

    Pith/arXiv arXiv 2024

  41. [41]

    RAP: KV-cache com- pression via RoPE-aligned pruning.arXiv:2602.02599, 2026

    Jihao Xin, Tian Lyu, David Keyes, Hatem Ltaief, and Marco Canini. RAP: KV-cache com- pression via RoPE-aligned pruning.arXiv:2602.02599, 2026

    arXiv 2026

  42. [42]

    ThinK: Thinner key cache by query-driven pruning

    Yuhui Xu, Zhanming Jie, Hanze Dong, Lei Wang, Xudong Lu, Aojun Zhou, Amrita Saha, Caiming Xiong, and Doyen Sahoo. ThinK: Thinner key cache by query-driven pruning. arXiv:2407.21018, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    FlashInfer: Efficient and customizable attention engine for LLM inference serving.arXiv:2501.01005, 2025

    Zihao Ye, Lequn Chen, Ruihang Lai, et al. FlashInfer: Efficient and customizable attention engine for LLM inference serving.arXiv:2501.01005, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    H2O: Heavy-hitter oracle for efficient generative inference of LLMs

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen. H2O: Heavy-hitter oracle for efficient generative inference of LLMs. InNeurIPS, 2023. arXiv:2306.14048. 38

    Pith/arXiv arXiv 2023

  45. [45]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. InNeurIPS, 2024. arXiv:2312.07104

    Pith/arXiv arXiv 2024

  46. [46]

    EliteKV: Scalable KV cache compression via RoPE frequency selection and joint low-rank projection.arXiv:2503.01586, 2025

    Yuhao Zhou, Sirui Song, Boyang Liu, Zhiheng Xi, Senjie Jin, Xiaoran Fan, Zhihao Zhang, Wei Li, and Xuanjing Huang. EliteKV: Scalable KV cache compression via RoPE frequency selection and joint low-rank projection.arXiv:2503.01586, 2025. 39

    Pith/arXiv arXiv 2025

  47. [2024]

    arXiv:2402.02750. 37

    Pith/arXiv arXiv