Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The critical KV footprint makes eviction methods comparable, and PruLong beats DuoAttention by about 12 points on recall while patched chunked eviction wins ICL and RAG.

desk verdict A genuinely useful new metric for KV eviction comparison, with a solid but incremental training method; the headline numbers should be read as estimates pending variance reporting and a wider hardware sweep. read the letter →

arxiv 2506.17121 v1 pith:VBGYU2E3 submitted 2025-06-20 cs.CL

Adithya Bhaskar , Alexander Wettig , Tianyu Gao , Yihe Dong , Danqi Chen This is my paper · ORCID
classification cs.CL
keywords KVcacheevictionlong-contextlanguagemodelsfootprintcriticalattentionheadspecializationstreamingheadschunkedpre-fillinghard-concretemask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that fair comparison of KV-eviction methods needs a time-aggregated memory measure, not a final cache size: the KV footprint, the count of un-evicted KV entries summed over all timesteps and normalized to full causal attention. On top of it, the critical KV footprint is the smallest footprint at which a method still keeps 90% of full-attention performance. Under that metric, post-fill eviction methods such as PyramidKV and SnapKV carry a large hidden cost because they hold all KVs through pre-fill, and patched chunked eviction recovers much of it. The learned head-specialization method PruLong reduces the critical footprint on recall by roughly 12 points relative to DuoAttention, and no single method is best across all eight task categories.

What carries the argument

The central object is the KV footprint: the number of KV entries that have not been evicted, aggregated across all pre-fill and decoding timesteps, divided by the count for full causal attention. The critical KV footprint is the smallest such value at which a method retains 90% of full-attention performance. The argument is carried by comparing methods on this single axis; the method that carries PruLong is a hard-concrete reparametrization of Bernoulli masks over attention heads, trained with a next-token prediction loss and a Lagrange penalty that drives total sparsity toward a target, so discrete streaming-versus-retrieval head assignments are optimized end-to-end and discretized only after training. Streaming heads attend only to a local window plus sink tokens, while retrieval heads keep full context.

What would settle it

Run PruLong, DuoAttention, and patched PyramidKV at their critical footprints on a production-style inference stack that uses grouped-query attention, batched chunked pre-fill, and a real allocator, and compare end-to-end throughput and peak GPU memory; if a method with a larger theoretical footprint matches or beats PruLong on those metrics at the same task scores, the central ranking claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the critical KV footprint makes different long-context inference methods commensurable, and that measuring with it changes the apparent leaderboard. Post-fill eviction methods scored poorly not because their final caches are large but because they keep every KV alive during pre-fill; patched chunked eviction lets PyramidKV reach the lowest critical footprint on ICL, RAG, LongQA, and summarization, while PruLong, trained with a next-token-prediction objective, hard-concrete masks, and natural long-context data, delivers the lowest critical footprint on recall, re-ranking, and HTML-to-TSV. The paper further shows that both learned head-specialization methods lose up to 20 points when pre-fill chunks shrink from 32K to 8K, and that no method compresses the reasoning-heavy travel-planning task meaningfully.

Load-bearing premise

The load-bearing premise is that memory cost is proportional to the time-aggregated count of un-evicted KV entries, independent of kernel implementation, grouped-query attention replication, and memory allocator behavior; if that proportionality fails, critical-footprint rankings may not reproduce on real hardware.

Editorial extensions

If this is right

  • Critical KV footprint should replace raw compression ratio as the headline comparison for eviction methods, since it penalizes methods that evict late even if their final cache is small.
  • Post-fill eviction methods can be made pre-fill friendly: patched chunked eviction cuts PyramidKV's critical footprint by roughly 30% on recall relative to naive chunking and makes it the best method on ICL, RAG, LongQA, and summarization.
  • PruLong's learned head masks give recall, re-ranking, and HTML-to-TSV a 10-15% smaller critical footprint than DuoAttention, confirming that optimizing next-token loss on natural long-context data beats reconstruction on synthetic data.
  • Pre-filling in 8K chunks dominates the Pareto frontier of footprint versus performance, but head-specialization methods degrade by up to 20 points at that chunk size, so chunk size is a first-class hyperparameter.
  • No KV-eviction method compresses travel-planning reasoning below the 90% threshold, so memory savings on such tasks require something beyond head specialization or token-drop heuristics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the footprint metric is a proxy for achievable memory cost, not realized memory cost; a serving stack with grouped-query replication, batching, and allocator effects could change which of these methods is cheapest, so the paper's rankings should be read as kernel-independent upper bounds.
  • Editorial inference: the 90% retention threshold is a convention; deployments with stricter score requirements would likely see the ranking shift because methods differ in how sharply their performance curves drop below the threshold.
  • Editorial inference: the observed chunk-size sensitivity of PruLong suggests a testable fix—training the head masks with multiple chunk sizes or with a distribution over chunk sizes—which could remove the need to match training and inference chunking.
  • Editorial inference: the complementary strengths of patched PyramidKV (ICL, RAG) and PruLong (recall) point to a hybrid that keeps retrieval heads learned while routing non-retrieval heads through chunked eviction; nothing in the paper rules such a combination out.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces the KV footprint, defined as the time-aggregated number of un-evicted KV entries normalized to full causal attention, and proposes the critical KV footprint as the smallest footprint at which a method retains 90% of full-attention performance. It uses this metric to compare existing KV eviction methods, adapts post-fill eviction methods such as SnapKV and PyramidKV to chunked pre-filling via "patched" chunked eviction, and proposes PruLong, an end-to-end method that learns global versus streaming attention heads from natural long-context data using hard-concrete masks and a next-token prediction objective. The evaluation spans HELMET and LongProc tasks with up to 128K-token contexts and reports that PruLong improves over DuoAttention by roughly 12 points on recall tasks, while patched PyramidKV attains the lowest critical footprint on several ICL/RAG tasks but is unreliable on recall.

Significance. The paper addresses a real gap: KV eviction methods are difficult to compare because they are evaluated at fixed sparsity levels and target different inference stages. The proposed KV footprint is a conceptually clean, implementation-agnostic normalization, and the paper's catalog of method classes is likely to be useful to the community. The code release, the hardware measurements in Appendix F, and the explicit discussion of limitations are strengths. If the empirical rankings are robust, the claim that PruLong reduces the critical KV footprint by about 12 points on recall over DuoAttention is a meaningful contribution. However, the central role of the critical-footprint ranking makes the interpolation procedure and the strength of the hardware validation load-bearing issues, not presentation details.

major comments (3)
  1. [Section 2.1 and Table 2] The critical KV footprint values in Table 2 are obtained by linear interpolation from a coarse hyperparameter grid (ten sparsity levels) with no error bars, no multiple runs, and no uncertainty analysis. If the recall curves are flat or noisy near the 90% threshold, the headline gap between PruLong (46%) and DuoAttention (58%) could change substantially. Additionally, entries such as "<34" and ">98" depend on the endpoints of the grid and need explanation of what happens outside the evaluated range. The authors should report each grid point used for interpolation, provide confidence intervals or bootstrap estimates over task instances, and justify that linear interpolation is accurate enough for the claimed rankings.
  2. [Appendix F and Limitations] The paper's central claim is that the KV footprint is a fair, unified basis for ranking eviction methods, but the hardware validation in Appendix F is performed at a single operating point for each method (70% head sparsity for recency methods and 30% cache for chunked eviction). The appendix itself states that "precise rankings depend on implementation details beyond KV evictions" and that practical efficiency varies across frameworks, and the limitations section concedes that the KV footprint "may not correlate perfectly with throughput or other hardware metrics." Because the 12-point recall advantage and the Table 2 rankings are the main empirical contributions, a single-point check is not sufficient; the authors should provide hardware measurements across the sparsity grid used to define the critical footprint, or at least at the critical footprint of each method, and discuss conditions under which the idealized ranking could diverge from real memory and throughput rankings.
  3. [Section 5.2 and Figure 3] The claim that patched PyramidKV is the "most effective method for retaining performance in ICL and RAG settings" is based on critical footprints at or below the lowest grid point, such as "<34" for RAG and "<36" for ICL. Since these values are outside the measured range, the ranking between patched PyramidKV and PruLong on these tasks is not actually resolved; the paper should evaluate at smaller footprints or report the performance at the lowest measured grid point explicitly rather than using unbounded inequalities in the headline table.
minor comments (6)
  1. [Appendix B, Eq. (5)] The variable s is used both for the logistic-sigmoid output in the hard-concrete reparameterization and for the sparsity function s(π), which is confusing. Please use a different symbol, such as g for the sigmoid output, to avoid collision.
  2. [Section 5.2] The word "suprisingly" should be "surprisingly".
  3. [Appendix C] The text says results with other metrics of KV usage are reported in Appendix C, but the peak-KV discussion appears in Appendix A. Please fix the cross-reference.
  4. [Table 2] The use of strict inequalities like "<34" and ">98" without a clarifying note is unclear; a short caption explaining what these symbols mean would improve readability.
  5. [Section 3] The claim that selecting a single set of KV entries per query group reduces memory by a factor of 8x for Llama-3.1-8B-Instruct needs a brief explanation of why the memory reduction is a factor of 8, since the KV cache in GQA is already shared across query heads in a group.
  6. [Section 6] The conclusion says PruLong achieved a "10–15% reduction in the critical KV footprint over the next best method in 3 out of 6 tasks," but Table 2 shows task-dependent differences that are sometimes larger and sometimes not statistically grounded; consider aligning this summary with the actual reported values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; PruLong's footprint reductions are measured on held-out tasks, and the paper's own limitation statement concerns hardware validity, not construction.

full rationale

The central claim—that PruLong reaches a lower critical KV footprint than DuoAttention on recall and that patched PyramidKV leads on ICL/RAG—is an empirical comparison. Section 2.1 defines the KV footprint as the time-aggregated number of un-evicted KV entries normalized to full causal attention, and the critical KV footprint as the smallest footprint at which a method retains F=90% of full-attention performance. That is a measurement protocol, not a quantity whose definition presupposes any method's ranking. Each method is evaluated on a grid of sparsity levels (Section 5.1), and the held-out task scores determine the crossing point. PruLong's masks are trained with the next-token prediction loss and a sparsity regularizer (Eq. 2) on pre-training data from Gao et al. (2025); evaluation on HELMET and LongProc tasks is outside that training distribution. The only self-referential-looking result, Figure 5, shows that a model regularized toward the evaluation sparsity performs best at that sparsity; this is an expected property of the regularizer and is presented as an ablation, not as a prediction from a fitted parameter. Self-citations to Gao et al., Yen et al., and Ye et al. supply training data and benchmarks; they do not supply the claimed footprint reductions. The acknowledged limitation in 'Limitations and future work'—'Since the KV footprint assumes an idealized model, it may not correlate perfectly with throughput or other hardware metrics'—is a validity caveat, not a circularity; Appendix F provides a partial hardware check at a single operating point and explicitly notes that precise rankings depend on implementation details. No load-bearing step reduces to its own inputs by construction, so no circular step is identified.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper's central claims rest on a small set of hyperparameter choices and on the assumption that an idealized time-aggregated KV count tracks real inference cost. No new physical or architectural entities are introduced; the comparison depends on two modeling choices, F=90% and chunk size, and on the learned head-type dichotomy.

free parameters (6)
  • F (performance retention threshold) = 0.9
    Critical KV footprint is defined as retaining F=90% of full-attention performance; the cutoff is a modeling choice and different F would change all critical-footprint numbers.
  • Target sparsity t for PruLong = swept from 0.1 to 1.0; 0.7 used in ablations
    The sparsity regularizer in Eq. (2) drives masks toward a chosen t, and the reported critical footprint depends on the evaluation sparsity grid.
  • Local window size = 1024 tokens
    Streaming heads retain only the last 1024 tokens; inherited from prior work and held fixed across experiments.
  • Attention sink size = 128 tokens
    Streaming heads always retain 128 initial tokens; held fixed across experiments.
  • Prefill chunk size = 32768 tokens, with 8192 in ablations
    Chunk size determines how early KVs can be evicted in chunked eviction and strongly affects the footprint; the paper reports sensitivity to it.
  • PyramidKV importance window k = 64 tokens
    Post-fill eviction scores KV importance using the last 64 input tokens; used both in the original method and in the patched chunked version.
assumptions (5)
  • domain assumption Transformer autoregressive decoding requires storing all previous KV states, and KV memory dominates long-context inference.
    Section 1 introduction; if serving used recurrent or linear-attention layers, the KV-eviction problem and footprint metric would not apply.
  • domain assumption The KV footprint, counting un-evicted entries aggregated over time, is a faithful proxy for real memory and throughput.
    Section 2.1 and the limitations paragraph; Appendix F shows only rough correlation and says exact rankings depend on implementation details.
  • domain assumption Attention heads can be cleanly split into retrieval and streaming heads, and a mask learned on next-token prediction transfers to downstream tasks.
    Section 4, Eq. (1); the PruLong method assumes this dichotomy and that masks trained on natural long-context data generalize to evaluation tasks.
  • standard math The hard-concrete reparametrization gives usable gradients for discrete mask optimization.
    Appendix B, Eqs. (4)-(7); relies on Louizos et al. (2018) and the Gumbel-concrete literature.
  • domain assumption Evaluation datasets from HELMET and LongProc are representative of long-context applications, and full-attention Llama-3.1-8B-Instruct is a valid reference.
    Section 5.1; task selection is limited to categories where the 8B model has non-trivial performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?." pith.science (2026). https://pith.science/paper/VBGYU2E3

@misc{pith2026250617121,
  author       = {Pith},
  title        = {Pith review of: Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBGYU2E3}},
  note         = {Machine review of arXiv:2506.17121}
}
read the original abstract

Language models handle increasingly long contexts for tasks such as book summarization, but this leads to growing memory costs for the key-value (KV) cache. Many prior works have proposed ways of discarding KVs from memory, but their approaches are tailored to favorable settings, obscuring caveats like high peak memory and performance degradation, and a fair comparison between methods is difficult. In this paper, we propose the *KV footprint* as a unified metric, which accounts for both the amount of KV entries stored and their lifespan in memory. We evaluate methods based on the smallest footprint they attain while preserving performance in both long-context understanding and generation, with context lengths of up to 128K tokens. This metric reveals the high peak memory of prior KV eviction methods. One class of methods -- *post-fill eviction* -- has a high footprint due to being incompatible with eviction during pre-filling. We adapt these methods to be able to evict KVs during pre-filling, achieving substantially lower KV footprints. We then turn to *recency eviction* methods, wherein we propose PruLong, an end-to-end optimization method for learning which attention heads need to retain the full KV cache and which do not. PruLong saves memory while preserving long-context performance, achieving 12% smaller KV footprint than prior methods while retaining performance in challenging recall tasks. Our paper clarifies the complex tangle of long-context inference methods and paves the way for future development to minimize the KV footprint.

Figures

Figures reproduced from arXiv: 2506.17121 by the authors.

Figure 1
Figure 1. The KV footprint [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of relevant methods for efficient inference discussed in Section [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance vs. KV footprint for the baselines and PruLong (green). The gray dashed line [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Left: Pre-filling with a smaller chunk size (8K tokens vs. 32K tokens) tends to deteriorate [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: We investigate how masks trained with different sparsities perform when evaluated at [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: , we observe an interesting trend where updating the weights of the model leads to the best Recall at a chunk pre-filling size of 128 tokens—corresponding to the sliding window block size during training—but deteriorates at greater chunk sizes. We hypothesize that upda…
Figure 7
Figure 7. Figure 7: Performance vs. Peak KV for the baselines and PruLong. The gray dashed line denotes [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Performance vs. KV Footprint for the baselines and PruLong. The gray dashed line denotes [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference

    cs.AR 2026-07 conditional novelty 7.0 of 10

    Six HBF stacks plus FlashAccel co-design deliver 2.54× decode throughput and 1.93× energy efficiency per GPU versus HBM-only under a 100 ms latency constraint.

  2. Which Heads Matter for Reasoning? RL-Guided KV Cache Compression

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A small set of "reasoning heads" found by RL can keep full KV cache while other heads are compressed to a constant size, giving 20–50% cache savings with near-lossless accuracy.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages · cited by 2 Pith papers

  1. [1]

    DuoAttention and PruLong usually achieve the highest throughput and the lowest peak memory

    work page

  2. [2]

    # Input” and “# Output

    Line 6 then stretches this distribution to the interval [−0.1,1]≡[l,r] , and the excess probability on either side is accumulated into a delta function at 0 and 1 (line 7). This places a non-zero probability weight on the support 0,1 to better represent the discrete nature of the modeled variables. The hard concrete reparametrization allows us to re-expre...

    work page 2024

  3. [3]

    The combination of P + C is usually the best-performing variant in both the PyramidKV and SnapKV groups

    Patching (P) leads to a higher score in most settings, and does not substantially affect the throughput or peak memory. The combination of P + C is usually the best-performing variant in both the PyramidKV and SnapKV groups

    work page

  4. [4]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    Association for Computational Linguistics. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-traini...

    arXiv 2024

  5. [5]

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han

    URLhttps://openreview.net/forum?id=poE54GOq2l. Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for LLM compression and acceleration. InMLSys, 2024. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng...

    arXiv 2024

  6. [6]

    URLhttps://arxiv.org/abs/2502.13189. Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. InInternational Conference on Learning Representations,

    arXiv

  7. [8]

    Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen

    URLhttps://arxiv.org/abs/2501.05414. Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effectively and thoroughly. InInternational Conference on Learning Representations (ICLR), 2025. Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao,...

    arXiv 2025

  8. [11]

    It does not usually affect performance

    Mean-pooling (C) usually reduces peak memory utilization by around 25%, but reduces throughput slightly for PyramidKV and SnapKV . It does not usually affect performance

    work page

Show all 12 references
  1. [13]

    On the other hand, the precise values of the real metrics are noisy and show some variation across different runs

    The ranking of methods generally tracks with those in Section 5, demonstrating that our idealized metrics lead to reliable takeaways. On the other hand, the precise values of the real metrics are noisy and show some variation across different runs. Why not just use hardware me...

  2. [2017]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi

    URLhttps://openreview.net/forum?id=S1jE5L5gl. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Gr...

    2023 arXiv

  3. [2023]

    doi: 10.18653/v1/2023.emnlp-main.298

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298/. Yash Akhauri, Ahmed F AbouElhamayed, Yifei Gao, Chi-Chih Chang, Nilesh Jain, and Mohamed S. Abdelfattah. TokenButler: Token importance is predict...

    2023 arXiv

  4. [2025]

    Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini

    URLhttps://arxiv.org/abs/2501.12948. Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. A simple and effective l_2 norm-based strategy for KV cache compression. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.