Pith. sign in

REVIEW 2 major objections 8 minor 77 references

Page-local spectral summaries let long-context decode attend only ~2% of tokens while matching full-cache quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 11:42 UTC pith:I4OJDWYC

load-bearing objection Solid systems paper: page-local spectral summaries actually track the exact-LSE oracle, ship in vLLM, and win where envelope/shared-basis selectors collapse—soft spot is that the fidelity microscope is thinner than the 100K+ quality headline. the 2 major comments →

arxiv 2607.24555 v1 pith:I4OJDWYC submitted 2026-07-27 cs.LG cs.AI

LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

Junsung Hwang This is my paper
classification cs.LG cs.AI
keywords KV cachesparse attentionlong-context decodingpage-local low-ranklog-sum-exp selectionGQAvLLM plugin
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long-context LLM serving is limited by having to read the entire key-value cache on every decode step. This paper argues that attention keys are low-rank inside small pages but high-rank across the whole sequence, so a shared low-rank basis throws away the page-specific directions that matter for selection. LOCKS builds a compact spectral summary for each page (about a tenth the size of that page’s cache), reconstructs the within-page logits from it, scores each page by log-sum-exp attention mass, and attends only the top pages—without reading candidate keys or values during selection. At a fixed 2048-token budget it tracks a read-every-key oracle on retrieval tasks, stays near full-cache quality on long-document QA, and holds up on long reasoning where other selectors collapse, while cutting per-token decode latency by about 2× at million-token context. The practical claim is that locality of the key geometry is enough to make aggressive sparse decode both accurate and deployable as a drop-in serving plugin.

Core claim

Attention keys concentrate their energy in a page’s own low-rank eigenbasis, so a per-page rank-8 spectral summary is a high-fidelity surrogate for exact page attention mass. Ranking pages on reconstructed log-sum-exp from that resident summary alone recovers the carrier pages that decide tasks, tracks exact-mass selection down to tight budgets, and matches full-cache aggregate quality at 100K+ context while attending roughly 2% of tokens.

What carries the argument

Per-page spectral summary: center the page’s keys, take a rank-r eigenbasis of the page Gram, store basis/coefficients/centroid in compact integers, then reconstruct within-page logits and reduce by log-sum-exp to score mass without reading keys. Theorem 1 composes reconstruction error into a retention guarantee for the group-shared top pages.

Load-bearing premise

That a fixed small page and a quantized rank-8 local basis keep ranking of the few task-critical pages faithful enough that selecting on the summary alone, with no exact key rescore, stays near the exact-mass oracle.

What would settle it

On the same long-context retrieval and reasoning suites, measure whether reconstructed-LSE page ranking loses carrier pages or task score relative to exact page log-sum-exp as budget shrinks, or whether matched-byte shared (global/sequence) bases ever match page-local mass and recall.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fixed small attended budgets can replace full-cache reads at 100K–1M context without large quality loss when selection uses page-local mass estimates.
  • Shared low-rank or envelope summaries are structurally weaker for page selection than page-local factors at the same state size.
  • Decode latency and bytes-read per step can drop by large factors while the full cache stays resident, with selection itself reading only summaries.
  • A training-free page summary can be integrated into an unmodified production serving stack with full CUDA-graph batched decode.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If page-local summaries are sufficient for selection, keeping only summaries on-accelerator and tiering full KV off-device becomes a natural next serving path.
  • The same locality argument may transfer to other block-sparse attention designs that currently use shared sketches or second-order moments.
  • Reasoning workloads may be especially sensitive to carrier retention under irreversible eviction, which favors re-selecting each step from a resident summary.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. The paper presents LOCKS, a training-free page-level KV-cache selector for long-context decoding. Each 16-token page gets its own rank-8 spectral summary (centroid + per-page eigenbasis + coefficients, quantized to int4/int8, ~10% of the page's KV bytes), from which within-page logits are reconstructed and reduced by log-sum-exp to a page-mass estimate; selection ranks these estimates without reading any candidate keys or values. The paper supports the design with (i) an impossibility result for fixed shared projections (Prop. 1), (ii) a composed retention bound for the unquantized summary (Thm. 1), (iii) a microscope study separating mass capture from carrier-page recall (App. B), and (iv) same-engine task and systems evaluations against FullKV, an exact-LSE oracle, and ported baselines (Quest, ShadowKV, RocketKV, KVzip, R-KV, TriAttention, LazyEviction), with an explicit per-method budget-accounting audit (App. C.1). Headline claims: at the shipped 2048-token per-head budget LOCKS matches FullKV aggregate quality at 100K+ context on InfiniteBench while attending ~2% of tokens, tracks the exact-LSE oracle on RULER down to small budgets, and achieves 2.0× lower per-token decode latency than dense attention at 1M tokens on an H200, shipping as a plugin for unmodified vLLM with full CUDA-graph decode.

Significance. If the results hold, this is a useful and unusually well-documented contribution to query-aware sparse decoding. The specific strengths worth naming: the evaluation is same-engine against FullKV and a read-every-key exact-LSE oracle on identical records; the budget-accounting audit of App. C.1 (Quest's GQA-union traffic, ShadowKV's fixed block, RocketKV's traffic-vs-residency split, KVzip's question/generation exclusion) is the kind of comparison hygiene this literature frequently skips; the efficiency claims come with a full measurement protocol including dispersion, lockstep/throttle guards, and matched-pair batched measurements (App. C.6, Tables 2, C.9); and the theoretical component is appropriately scoped — Lemma 1 is explicitly attributed to concurrent/prior work, Prop. 1's class coverage is stated precisely, and the Remark after Thm. 1 plainly discloses that the deployed quantized summary is outside the proven bound. Code, kernels, and harnesses are released. The page-local-vs-shared-scope finding with matched-bytes controls (Tables D.11–D.16) is independently informative even for readers not interested in the system. The main caveat on impact is that the mechanistic story (

major comments (2)
  1. [§5.1, App. B (Tables B.1–B.2) vs. Table C.8] The causal chain (page-local concentration → ranking fidelity → carrier survival) is instrumented only at 32K on Llama-3.1-8B and Qwen3-4B (App. B: five RULER-32K records per model), while the 100K+ headline rests on GLM-4-9B-Chat-1M task scores alone. This matters because η in Thm. 1 is a max over pages, and the page count grows ~8× from 32K to 256K, so tail-error pages (Table B.2: group-share log error max 18.01) have more chances to coincide with a carrier. There is direct evidence this is not hypothetical at 100K+: Table C.8, kv-retr rows, LOCKS scores 0/2/6/14 at b=64–512 versus FullKV 28, recovering only at b≥1024 — i.e., on the headline model at 100K+, the tight-budget carrier failures the 32K microscope rates as rare do occur. The shipped-budget claim itself (43.6 vs 43.0 at b=2048) is directly measured and unaffected. What is not currently supported is (a) the abstract's unquali
  2. [§4, Thm. 1 and Remark; Table B.2] Thm. 1's retention constant e^{-4η} is vacuous at the measured deployed-precision error magnitudes: with Table B.2's group-share log-error max of 18.01 (Qwen r8i4), e^{-4η} ≈ 10^{-31}, so the bound certifies nothing about the deployed summary — and even the unquantized variant (max 17.32) is vacuous. The paper discloses this honestly in the Remark ('the retention constant is qualitative at measured magnitudes'), but the Introduction's chain claim ('every link holds, theoretically through attention-output error (Thm. 1)') reads stronger than what the theorem delivers at deployment. Since Table B.2 already provides percentile data (p50/p95), a quantile- or high-probability-form retention statement (e.g., retention at η = p95 error, with the residual failure mass bounded separately) would restore a quantitative link at essentially no new measurement cost, and would also make the theory hone
minor comments (8)
  1. [App. C.4] Tables C.6–C.7 (MATH-500, AIME26): AIME26 is 30 problems × 4 seeds, so differences of a few points (e.g., LOCKS 65.0 vs Oracle 62.5 at b=1024; 63.3 vs FullKV 59.2 at b=2048) are well within binomial noise; the text's 'nominally matches or exceeds' hedging is appreciated, but adding Wilson or bootstrap CIs to these two tables would prevent over-reading.
  2. [Table 1 / §5.1] The InfiniteBench evaluation uses a seeded 50-record subset per task with bootstrap CIs (good), but at b=512 the claim 'highest measured mean of any deployable baseline' (LOCKS 41.1±2.4 vs RocketKV 38.5±2.7) has overlapping intervals; 'measured mean' phrasing is accurate, yet a sentence noting the CI overlap would be fairer to the reader.
  3. [§5.1, Fig. 2a] LongBench-v1 average context is ~11K tokens, so its role as evidence for 'long-document QA' is weak relative to the 100K+ framing; consider noting the context-length distribution when citing the within-a-point result.
  4. [Fig. 2] Fig. 2's common nominal-budget x-axis is not cost-matched across methods (Quest's GQA-union reads up to G·b; App. C.1 documents this). The audit is excellent, but a one-line pointer in the figure caption would help readers who do not reach App. C.1.
  5. [Fig. 1c] Fig. 1 panel (c) is dense: the per-row metric values (logit MAE, LSE MAE, rank, recall, carrier-kept) are hard to attribute to rows at print size; consider a small table instead.
  6. [App. D] Tables D.11 and D.15 leave the Llama-3.1-8B codebook cells unmeasured ('–'); since the codebook outranks the per-sequence basis on Qwen, either filling these cells or stating why the vintage sweep was not rerun would strengthen the frontier claim.
  7. [App. C.6] Table C.9's build column at 128K (2904±2282 ms) is dominated by one boot's stall; the explanation is given and commendable, but reporting the median alongside the mean for this heavy-tailed quantity would be more informative.
  8. [§3, Prop. 1] Prop. 1's hypothesis covers shared linear sketches with a centroid; codebooks and moment cores are explicitly outside it and handled empirically. A half-sentence in §3 reminding the reader of this scoping when the dose-response table is discussed would avoid the impossibility result being read more broadly than stated.

Circularity Check

0 steps flagged

No significant circularity: theory is class-relative with external attributions, and quality claims are measured against FullKV/exact-LSE oracles on third-party benchmarks.

full rationale

The paper's derivation chain does not reduce predictions to their inputs by construction. Lemma 1's mass-ranking optimality is explicitly attributed to Tzachristas et al. and Tian et al. and scoped as minimax among deterministic value-blind selectors, not presented as a fitted or self-derived uniqueness fact. Proposition 1 and Theorem 1 are proved in App. A from standard Lipschitz/ranking-robustness arguments (Eckart–Young for residual optimality via Cor. 1); the Remark honestly states that Thm. 1 covers only the unquantized summary and that deployed int4/int8 retention is measured, not certified. Rank r=8, int4, and B=16 are hyperparameters swept and ablated (Fig. 5, App. D), not hidden fits that redefine the target. Task quality is evaluated against external FullKV, an exact-LSE oracle, and third-party suites (LongBench-v1, RULER, InfiniteBench, MATH-500, AIME26) with same-engine identical records. No self-citation load-bearing chain exists (single-author first appearance of LOCKS). Skeptical concerns about microscope coverage (five RULER-32K records) versus 100K+ GLM claims are generalization/correctness risks, not circular reductions. Score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central serving claim rests on standard transformer attention math, the established value-blind page-mass selection target, the empirical local low-rank geometry of keys, and several engineering hyperparameters (page size, rank, quantization, fixed token budget). No new physical entities are postulated; the ‘spectral summary’ is a constructed data structure, not an ontological invention.

free parameters (5)
  • page size B = 16
    Fixed at 16 tokens throughout quality and efficiency results; larger pages need more rank at matched attended tokens (Fig. 5).
  • summary rank r = 8
    Shipped rank-8 chosen via ablation of recall vs exact ranking and state cost; not derived from a uniqueness theorem.
  • basis/coeff quantization = r8i4 (basis int4, coeff/centroid int8)
    int4 basis + int8 coefficients/centroid shipped after precision sweep; retention under quantization is measured, not bounded.
  • per-head token budget b = 2048 (shipped); sweep 64–2048
    Operating point for the headline quality/latency claim; 2048 is the shipped budget where FullKV parity is reported.
  • always-kept sink and recent pages = 2 pages counted inside b
    Two of k page slots reserved by design inside b; standard sparse-decode choice that affects coverage.
axioms (5)
  • domain assumption Decoder self-attention with softmax(q⊤k/√d) and paged KV cache is the serving model.
    §2 setup; standard GQA decoder assumption for vLLM-class engines.
  • domain assumption Exact page log-sum-exp mass is the right value-blind selection target; top-k by mass is minimax-optimal among deterministic value-blind rules.
    Lemma 1; identity attributed to Tzachristas et al. / Tian et al.; LOCKS adopts rather than re-derives the target.
  • domain assumption Attention keys are locally low-rank at page scope though globally high-rank.
    Core empirical premise of §3 and Table D.11–D.12; supported by measurements vs isotropic null, not proved for all models.
  • standard math Eckart–Young / truncated SVD gives the residual-optimal rank-r per-page linear reconstruction among orthonormal bases.
    Cor. 1 cites Eckart & Young 1936; used to justify the spectral basis.
  • ad hoc to paper Fixed per-head token budget b (not a fixed fraction) is the relevant serving knob.
    §2 provisioning argument; shapes all quality curves and the 2% attended-tokens headline.
invented entities (2)
  • Per-page spectral summary (centroid + rank-r basis Vj + coefficients ci, quantized) independent evidence
    purpose: Resident surrogate enabling reconstructed within-page logits and LSE page-mass scores without reading keys.
    Constructed representation, not a new physical object; independent evidence is empirical ranking/task fidelity and the unquantized error bound in Thm. 1.
  • Group share-average combine rule for GQA shared page selection independent evidence
    purpose: One shared top-k page set across query heads in a KV group from averaged normalized page-mass shares.
    Cor. 3 proves it maximizes equal-weight group-average coverage among shared sets; related to TokenSelect’s soft vote, with optimality claim as contribution.

pith-pipeline@v1.2.0-grok45-kimik3 · 54171 in / 3766 out tokens · 74814 ms · 2026-07-31T11:42:49.718452+00:00 · methodology

0 comments
read the original abstract

Serving large language models at long context is bottlenecked by the key-value (KV) cache, which is read in full at every decode step. Attention keys are locally low-rank though globally high-rank: shared low-rank bases discard page-specific directions that a page's own compact basis retains. LOCKS gives every page its own spectral summary (resident, about a tenth the cache's size), reconstructs within-page logits, estimates each page's attention mass by log-sum-exp, and attends only the top pages; selection itself reads no candidate keys or values. Selecting on this summary alone stays within about a point of the full cache on long-document QA (LongBench-v1), tracks the read-every-key oracle on retrieval-dense RULER down to the smallest budgets, and shows its largest margins on long-form reasoning (AIME26, MATH-500), where baseline selectors collapse. At its shipped $2048$-token budget LOCKS matches FullKV aggregate quality at $100$K$+$ context while attending about $2\%$ of the tokens, and halves per-token decode latency ($2.0\times$ at $1$M tokens) against dense attention. LOCKS ships as a drop-in plugin for unmodified vLLM, with batched decode running in full CUDA graphs.

Figures

Figures reproduced from arXiv: 2607.24555 by Junsung Hwang.

Figure 1
Figure 1. Figure 1: The locality thesis in one figure (Llama-3.1-8B, Qwen3-4B twin and full protocol in App. B). (a) Exact vs. reconstructed page-mass share at matched bytes. (b) Carrier retention vs. per-head budget; the ceiling is 100 by construction. (c) The error ladder at matched bytes, one scope per row. (d) RULER-16K capability families at a 256-token budget. Page scope hugs the exact diagonal, keeps carriers down to a… view at source ↗
Figure 2
Figure 2. Figure 2: Retrieval and QA vs. budget (Llama-3.1-8B [18], greedy, same-engine FullKV, identical records across methods): (a) LongBench-v1 (14-subset mean, average context ∼11K tokens), (b) RULER-16K and (c) RULER-32K (13-task means). The x-axis is a common nominal budget; native budget semantics and measured traffic and residency differ across methods, audited in App. C.1 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reasoning vs. budget (Qwen3-4B [62], thinking on; per-arm sampling settings in App. C): (a) AIME26, (b) MATH-500 [21, 34] (16384-token cap) 16K 64K 256K 1M context length 0 5 10 15 20 25 TPOT (ms) (a) FA3 FI LOCKS 2.0x 16K 64K 256K 1M context length 10 0 10 1 10 2 10 3 TTFT (s) (b) 16K 64K 256K 1M context length 0 20 40 60 80 bytes read / step (GB) (c) 9.8x [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Decode efficiency at long and extreme context (GLM-4-9B-Chat-1M [53], H200 NVL node; full measure￾ment protocol in App. C.6, software and hardware in App. F). Measured at budget b=2048, where LOCKS matches the full cache on the quality suites of §5.1 and §5.2. (a) Per-token latency versus context, 2.0× below the dense baselines (FA3, FlashInfer) at 1M. (b) Steady prefill TTFT remains at parity; the one-tim… view at source ↗
Figure 5
Figure 5. Figure 5: Rank and basis precision at matched attended tokens (1024 per head; Llama-3.1-8B). Recall vs. the exact ranking up to each page size’s cap (r=B−1, where the bf16 basis reaches 100 by construction). int8 tracks bf16; int4 costs a few points and flattens past r∼16; int2 stays near centroid level at every rank, so int4 is the lowest tested precision that preserves the benefit of added rank. Full grid released… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 41 linked inside Pith

  1. [1]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    doi:10.18653/v1/2024.acl-long.172 2024

  2. [2]

    RocketKV: Accelerating long-context LLM inference via two-stage KV cache compression

    Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po-An Tsai, Zhiding Yu, and Alexey Tumanov. RocketKV: Accelerating long-context LLM inference via two-stage KV cache compression. InProceedings of the 42nd International Conference on Machine Learning (ICML), pages 3358–3392. PMLR, 2025. arXiv:2502.14051

    Pith/arXiv arXiv 2025

  3. [3]

    Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Pith/arXiv arXiv 2004

  4. [4]

    R-KV: Redundancy-aware KV cache compression for reasoning models

    Zefan Cai, Wen Xiao, Hanshi Sun, Cheng Luo, Yikai Zhang, Ke Wan, Yucheng Li, Yeyang Zhou, Li-Wen Chang, Jiuxiang Gu, Zhen Dong, Anima Anandkumar, Abedelkadir Asi, and Junjie Hu. R-KV: Redundancy-aware KV cache compression for reasoning models. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2505.24133

    arXiv 2025

  5. [5]

    Runtime-certified bounded-error quantized attention, 2026

    Dean Calver. Runtime-certified bounded-error quantized attention, 2026. arXiv:2605.20868

    Pith/arXiv arXiv 2026

  6. [6]

    Palu: Compressing KV-cache with low-rank projection

    Chi-Chih Chang, Wei-Cheng Lin, Chien-Yu Lin, et al. Palu: Compressing KV-cache with low-rank projection. In International Conference on Learning Representations (ICLR), 2025. arXiv:2407.21118

    Pith/arXiv arXiv 2025

  7. [7]

    Value-aware stochastic KV cache eviction for reasoning models, 2026

    Ting-Yun Chang, Harvey Yiyun Fu, Deqing Fu, Chenghao Yang, Jesse Thomason, and Robin Jia. Value-aware stochastic KV cache eviction for reasoning models, 2026. arXiv:2606.03928

    Pith/arXiv arXiv 2026

  8. [8]

    MagicPIG: LSH sampling for efficient LLM generation

    Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, et al. MagicPIG: LSH sampling for efficient LLM generation. In International Conference on Learning Representations (ICLR), 2025. Spotlight; OpenReview ALzTQUgW8a

    2025

  9. [9]

    Guess-verify-refine: Data-aware top-K for sparse-attention decoding on Blackwell via temporal correlation, 2026

    Long Cheng, Ritchie Zhao, Timmy Liu, Mindy Li, Xianjie Qiao, Kefeng Duan, Yu-Jung Chen, Xiaoming Chen, Bita Darvish Rouhani, and June Yang. Guess-verify-refine: Data-aware top-K for sparse-attention decoding on Blackwell via temporal correlation, 2026. arXiv:2604.22312

    Pith/arXiv arXiv 2026

  10. [10]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    Pith/arXiv arXiv 1904

  11. [11]

    DeepSeek-V4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

    DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence.arXiv preprint arXiv:2606.19348, 2026

    arXiv 2026

  12. [12]

    Gonzalez, and Ion Stoica

    Aditya Desai, Kumar Krishna Agrawal, Shuo Yang, Alejandro Cuadron, Luis Gaspar Schroeder, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. vAttention: Verified sparse attention. InInternational Conference on Learning Representations (ICLR), 2026. arXiv:2510.05688

    Pith/arXiv arXiv 2026

  13. [13]

    Kascade: A practical sparse attention method for long-context LLM inference, 2025

    Dhruv Deshmukh, Saurabh Goyal, Nipun Kwatra, and Ramachandran Ramjee. Kascade: A practical sparse attention method for long-context LLM inference, 2025. arXiv:2512.16391

    arXiv 2025

  14. [14]

    Expected attention: KV cache compression by estimating attention from future queries distribution, 2025

    Alessio Devoto, Maximilian Jeblick, and Simon Jégou. Expected attention: KV cache compression by estimating attention from future queries distribution, 2025. arXiv:2510.00636

    arXiv 2025

  15. [15]

    The approximation of one matrix by another of lower rank.Psychometrika, 1(3): 211–218, 1936

    Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank.Psychometrika, 1(3): 211–218, 1936

    1936

  16. [16]

    Kevin Zhou

    Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S. Kevin Zhou. Ada-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. OpenReview tcisuhGsQZ; arXiv:2407.11550

    Pith/arXiv arXiv 2025

  17. [17]

    SeerAttention: Learning intrinsic sparse attention in your LLMs

    Yizhao Gao, Zhichen Zeng, Dayou Du, et al. SeerAttention: Learning intrinsic sparse attention in your LLMs. arXiv preprint arXiv:2410.13276, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters, 2024

    Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in KV cache reduction: Value also matters, 2024. arXiv:2406.12335

    Pith/arXiv arXiv 2024

  20. [20]

    ZipCache: Accurate and efficient KV cache quantization with salient token identification

    Yefei He, Luoming Zhang, Weijia Wu, Jing Liu, Hong Zhou, and Bohan Zhuang. ZipCache: Accurate and efficient KV cache quantization with salient token identification. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2405.14256

    Pith/arXiv arXiv 2024

  21. [21]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021

    2021

  22. [22]

    SVDq: 1.25-bit and 410x key cache compression for LLM attention.arXiv preprint arXiv:2502.15304, 2025

    Yankun Hong, Xing Li, Hui-Ling Zhen, Xianzhi Yu, Wulong Liu, and Mingxuan Yuan. SVDq: 1.25-bit and 410x key cache compression for LLM attention.arXiv preprint arXiv:2502.15304, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    KVQuant: Towards 10 million context length LLM inference with KV cache quantization

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In NeurIPS, 2024. 10 APREPRINT- JULY28, 2026

    2024

  24. [24]

    Multipole attention for efficient long context reasoning

    Coleman Hooper, Sebastian Zhao, Luca Manolache, et al. Multipole attention for efficient long context reasoning. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2506.13059

    arXiv 2025

  25. [25]

    Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Hierarchical sparse attention done right: Toward infinite context modeling

    Xiang Hu, Xinyu Wei, Hao Gu, et al. Hierarchical sparse attention done right: Toward infinite context modeling. arXiv preprint arXiv:2607.02980, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention

    Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, et al. MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. InNeurIPS, 2024

    2024

  28. [28]

    Lee, Sangdoo Yun, and Hyun Oh Song

    Jang-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W. Lee, Sangdoo Yun, and Hyun Oh Song. KVzip: Query- agnostic KV cache compression with context reconstruction. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. Oral; OpenReview JFygzwx8SJ; arXiv:2505.23416

    arXiv 2025

  29. [29]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon et al. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023

    2023

  30. [30]

    InfiniGen: Efficient generative inference of large language models with dynamic KV cache management

    Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim. InfiniGen: Efficient generative inference of large language models with dynamic KV cache management. InOSDI, 2024

    2024

  31. [31]

    Rakhshan, and Guillaume Rabusseau

    Damien Lesens, Beheshteh T. Rakhshan, and Guillaume Rabusseau. KQ-SVD: Compressing the KV cache with provable guarantees on attention fidelity, 2025. arXiv:2512.05916

    arXiv 2025

  32. [32]

    Efficient low rank attention for long-context inference in large language models

    Tenghui Li, Guoxu Zhou, Xuyang Zhao, Yuning Qiu, and Qibin Zhao. Efficient low rank attention for long-context inference in large language models. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2510.23649

    arXiv 2025

  33. [33]

    Snapkv: LLM knows what you are looking for before generation

    Yuhong Li et al. Snapkv: LLM knows what you are looking for before generation. InNeurIPS, 2024

    2024

  34. [34]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  35. [35]

    Twilight: Adaptive attention sparsity with hierarchical top-p pruning

    Chaofan Lin, Jiaming Tang, Shuo Yang, Hanshuo Wang, Tian Tang, Boyu Tian, Ion Stoica, Song Han, and Mingyu Gao. Twilight: Adaptive attention sparsity with hierarchical top-p pruning. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. Spotlight; OpenReview Ve693NkzcU; arXiv:2502.02770

    arXiv 2025

  36. [36]

    RetrievalAttention: Accelerating long-context LLM inference via vector retrieval

    Di Liu, Meng Chen, Baotong Lu, et al. RetrievalAttention: Accelerating long-context LLM inference via vector retrieval. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  37. [37]

    ClusterKV: Manipulating LLM KV cache in semantic space for recallable compression, 2024

    Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. ClusterKV: Manipulating LLM KV cache in semantic space for recallable compression, 2024. arXiv:2412.03213

    Pith/arXiv arXiv 2024

  38. [38]

    ReasonAlloc: Hierarchical decoding-time KV cache budget allocation for reasoning models, 2026

    Wenhao Liu, Hao Shi, Yunhe Li, Weizhi Fei, Xiangyuan Wang, Mengzhe Ruan, Hanxu Hou, Peisong Wang, Linqi Song, and Shuang Qiu. ReasonAlloc: Hierarchical decoding-time KV cache budget allocation for reasoning models, 2026. arXiv:2606.11164

    Pith/arXiv arXiv 2026

  39. [39]

    KIVI: A tuning-free asymmetric 2bit quantization for KV cache

    Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. KIVI: A tuning-free asymmetric 2bit quantization for KV cache. InICML, 2024

    2024

  40. [40]

    Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, Zhiqi Huang, Huan Yuan, Suting Xu, Xinran Xu, Guokun Lai, Yanru Chen, Huabin Zheng, Junjie Yan, Jianlin Su, Yuxin Wu, Neo Y . Zhang, Zhilin Yang, Xinyu Zhou, Mingxing Zhang, and Jiezhong Qiu. MoBA: Mixture of block attention for long-cont...

    Pith/arXiv arXiv 2025

  41. [41]

    TriAttention: Efficient long reasoning with trigonometric KV compression.arXiv preprint arXiv:2604.04921, 2026

    Weian Mao, Xi Lin, Wei Huang, Yuxin Xie, Tianfu Fu, Bohan Zhuang, Song Han, and Yukang Chen. TriAttention: Efficient long reasoning with trigonometric KV compression.arXiv preprint arXiv:2604.04921, 2026

    Pith/arXiv arXiv 2026

  42. [42]

    Random-access infinite context length for transformers

    Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. Landmark Attention; arXiv:2305.16300

    Pith/arXiv arXiv 2023

  43. [43]

    SALS: Sparse attention in latent space for KV cache compression

    Junlin Mu, Hantao Huang, Jihang Zhang, Minghui Yu, Tao Wang, and Yidong Li. SALS: Sparse attention in latent space for KV cache compression. InAdvances in Neural Information Processing Systems (NeurIPS), 2025. arXiv:2510.24273

    arXiv 2025

  44. [44]

    The sparse frontier: Sparse attention trade-offs in transformer LLMs.arXiv preprint arXiv:2504.17768, 2025

    Piotr Nawrot et al. The sparse frontier: Sparse attention trade-offs in transformer LLMs.arXiv preprint arXiv:2504.17768, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, 2026. 11 APREPRINT- JULY28, 2026

    2026

  46. [46]

    SparQ attention: Bandwidth-efficient LLM inference

    Luka Ribar, Ivan Chelombiev, Luke Hudlass-Galley, Charlie Blake, Carlo Luschi, and Douglas Orr. SparQ attention: Bandwidth-efficient LLM inference. InICML, 2024

    2024

  47. [47]

    Eigen attention: Attention in low-rank space for KV cache compression

    Utkarsh Saxena, Gobinda Saha, Sakshi Choudhary, and Kaushik Roy. Eigen attention: Attention in low-rank space for KV cache compression. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 15332–15344, 2024

    2024

  48. [48]

    Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Noam Shazeer. Fast transformer decoding: One write-head is all you need.arXiv preprint arXiv:1911.02150, 2019

    Pith/arXiv arXiv 1911

  49. [49]

    Loki: Low-rank keys for effi- cient sparse attention

    Prajwal Singhania, Siddharth Singh, Shwai He, Soheil Feizi, and Abhinav Bhatele. Loki: Low-rank keys for effi- cient sparse attention. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2406.02542

    Pith/arXiv arXiv 2024

  50. [50]

    LongFlow: Efficient KV cache compression for reasoning models, 2026

    Yi Su, Zhenxu Tian, Dan Qiao, Yuechi Zhou, Juntao Li, and Min Zhang. LongFlow: Efficient KV cache compression for reasoning models, 2026. arXiv:2603.11504

    Pith/arXiv arXiv 2026

  51. [51]

    ShadowKV: KV cache in shadows for high-throughput long-context LLM inference

    Hanshi Sun, Li-Wen Chang, Wenlei Bao, et al. ShadowKV: KV cache in shadows for high-throughput long-context LLM inference. InProceedings of the 42nd International Conference on Machine Learning (ICML). PMLR, 2025. Spotlight

    2025

  52. [52]

    Quest: Query-aware sparsity for efficient long-context LLM inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context LLM inference. InICML, 2024

    2024

  53. [53]

    ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    Team GLM. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    COBS: Cumulant order block sparse attention.arXiv preprint arXiv:2607.09052, 2026

    Alexander Tian, Aditya Ghai, Sanjit Neelam, Zaal Vasania, and Akshay Mishra. COBS: Cumulant order block sparse attention.arXiv preprint arXiv:2607.09052, 2026

    Pith/arXiv arXiv 2026

  55. [55]

    A mathematical theory of top-ksparse attention via total variation distance, 2025

    Georgios Tzachristas, Lei Deng, Ioannis Tzachristas, Gong Zhang, and Renhai Chen. A mathematical theory of top-ksparse attention via total variation distance, 2025. arXiv:2512.07647

    arXiv 2025

  56. [56]

    SPLA: Block sparse plus linear attention for long context modeling.arXiv preprint arXiv:2601.22379, 2026

    Bailin Wang, Dan Friedman, Tao Lei, and Chong Wang. SPLA: Block sparse plus linear attention for long context modeling.arXiv preprint arXiv:2601.22379, 2026

    arXiv 2026

  57. [57]

    Prism: Spectral- aware block-sparse attention, 2026

    Xinghao Wang, Pengyu Wang, Xiaoran Liu, Fangxu Liu, Jason Chu, Kai Song, and Xipeng Qiu. Prism: Spectral- aware block-sparse attention, 2026. arXiv:2602.08426

    Pith/arXiv arXiv 2026

  58. [58]

    TokenSelect: Efficient long-context inference and length extrapolation for LLMs via dynamic token-level KV cache selection

    Wei Wu, Zhuoshi Pan, Chao Wang, Liyi Chen, Yunchu Bai, Tianfu Wang, Kun Fu, Zheng Wang, and Hui Xiong. TokenSelect: Efficient long-context inference and length extrapolation for LLMs via dynamic token-level KV cache selection. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025. arXiv:2411.02886

    arXiv 2025

  59. [59]

    InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. InfLLM: Training-free long-context extrapolation for LLMs with an efficient context memory. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2402.04617

    Pith/arXiv arXiv 2024

  60. [60]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. InICLR, 2024

    2024

  61. [61]

    DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads

    Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han. DuoAttention: Efficient long-context LLM inference with retrieval and streaming heads. InInternational Conference on Learning Representations (ICLR), 2025. arXiv:2410.10819

    Pith/arXiv arXiv 2025

  62. [62]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    No token left behind: Reliable KV cache compression via importance-aware mixed precision quantization.arXiv preprint arXiv:2402.18096, 2024

    June Yong Yang, Byeongwook Kim, Jeongin Bae, et al. No token left behind: Reliable KV cache compression via importance-aware mixed precision quantization.arXiv preprint arXiv:2402.18096, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Adaptive mass-segmented KV compression for long-context reasoning.arXiv preprint arXiv:2605.23200, 2026

    Junzhe Yang and Xiaoyu Shen. Adaptive mass-segmented KV compression for long-context reasoning.arXiv preprint arXiv:2605.23200, 2026

    Pith/arXiv arXiv 2026

  65. [65]

    TidalDecode: Fast and accurate LLM decoding with position persistent sparse attention

    Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, and Zhihao Jia. TidalDecode: Fast and accurate LLM decoding with position persistent sparse attention. InInternational Conference on Learning Representations (ICLR), 2025

    2025

  66. [66]

    Self-indexing KVCache: Predicting sparse attention from compressed keys, 2026

    Xu Yang, Jiapeng Zhang, Dongyang Zhao, Guo Chen, and Zhuo Tang. Self-indexing KVCache: Predicting sparse attention from compressed keys, 2026. arXiv:2603.14224

    arXiv 2026

  67. [67]

    HARD-KV: Head-adaptive regularization for decoding-time KV compression.arXiv preprint arXiv:2606.28831, 2026

    Yuxuan Yang, Feiyang Ren, Bowen Zeng, Dalin Zhang, Jinpeng Chen, Gang Chen, and Huan Li. HARD-KV: Head-adaptive regularization for decoding-time KV compression.arXiv preprint arXiv:2606.28831, 2026

    Pith/arXiv arXiv 2026

  68. [68]

    BLASST: Dynamic BLocked attention sparsity via softmax thresholding

    Jiayi Yuan, Cameron Shinn, Kai Xu, Jingze Cui, George Klimiashvili, Guangxuan Xiao, Perkz Zheng, Bo Li, et al. BLASST: Dynamic BLocked attention sparsity via softmax thresholding. InProceedings of Machine Learning and Systems (MLSys), 2026. MLSys 2026 (Oral); arXiv:2512.12087. 12 APREPRINT- JULY28, 2026

    Pith/arXiv arXiv 2026

  69. [69]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention

    Jingyang Yuan, Huazuo Gao, Damai Dai, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23078–23097, 2025. Best Paper Award; arXiv:2502.11089

    Pith/arXiv arXiv 2025

  70. [70]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020

    2020

  71. [71]

    GLM-5.2: Built for long-horizon tasks

    Z.ai. GLM-5.2: Built for long-horizon tasks. https://z.ai/blog/glm-5.2, 2026. Open-weight model release, June 2026; arXiv:2602.15763 is the GLM-5 series report, not GLM-5.2

    Pith/arXiv arXiv 2026

  72. [72]

    PQCache: Product quantization-based KVCache for long context LLM inference.Proceedings of the ACM on Management of Data, 3(3):201:1–201:30, 2025

    Hailin Zhang, Xiaodong Ji, Yilin Chen, et al. PQCache: Product quantization-based KVCache for long context LLM inference.Proceedings of the ACM on Management of Data, 3(3):201:1–201:30, 2025. arXiv:2407.12820

    Pith/arXiv arXiv 2025

  73. [73]

    LazyEviction: Lagged KV eviction with attention pattern observation for efficient long reasoning

    Haoyue Zhang, Hualei Zhang, Xiaosong Ma, Jie Zhang, and Song Guo. LazyEviction: Lagged KV eviction with attention pattern observation for efficient long reasoning. InProceedings of the 64th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pages 36335–36352, 2026. arXiv:2506.15969

    arXiv 2026

  74. [74]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhenyu Zhang et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. In NeurIPS, 2023

    2023

  75. [75]

    Tactic: Adaptive sparse attention with clustering and distribution fitting for long-context LLMs.arXiv preprint arXiv:2502.12216, 2025

    Kan Zhu, Tian Tang, Qinyu Xu, et al. Tactic: Adaptive sparse attention with clustering and distribution fitting for long-context LLMs.arXiv preprint arXiv:2502.12216, 2025

    Pith/arXiv arXiv 2025

  76. [76]

    Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, and Pin-Yu Chen

    Yuxuan Zhu, David H. Yang, Mohammad Mohammadi Amiri, Keerthiram Murugesan, Tejaswini Pedapati, and Pin-Yu Chen. OjaKV: Context-aware online low-rank KV cache compression. InFindings of the Association for Computational Linguistics: ACL 2026, 2026. arXiv:2509.21623

    Pith/arXiv arXiv 2026

  77. [77]

    Fast KV compaction via attention matching.arXiv preprint arXiv:2602.16284, 2026

    Adam Zweiger, Xinghong Fu, Han Guo, and Yoon Kim. Fast KV compaction via attention matching.arXiv preprint arXiv:2602.16284, 2026. Accepted as a poster at ICML 2026. 13 Appendix Table of Contents AProofs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....

    Pith/arXiv arXiv 2026