Pith. sign in

REVIEW 3 major objections 6 minor 54 references

QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Attention drift breaks irreversible KV eviction; QEvict's quantized tier recovers what was dropped.

desk verdict QEvict's recoverable three-tier KV cache is a genuine contribution with strong eviction-baseline results, but the quantization overclaim is contradicted by its own GSM8K tables and the routing diagnostics are thinner than the abstract suggests. read the letter →

arxiv 2608.05326 v1 pith:Q2YBIOSC submitted 2026-08-05 cs.LG cs.CL

Ayushman Garg , Akshita Gupta , Shaswata Bhattacharya , Abhishek Gupta , Sandeep Kumar , Manoj Kumar This is my paper · ORCID
classification cs.LGcs.CL
keywords KVcachecompressionrecoverableevictionquantizedtiersattentiondriftlong-contextdecodingcumulativescoringmemory-efficientinferencepromotion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the standard premise of KV-cache eviction—once a token is dropped, it can safely be forgotten—breaks down during decoding, because token and window importance drift as generated queries evolve. It introduces QEvict, a three-tier cache manager that keeps high-confidence windows in full precision, stores an intermediate tier in 2-bit quantized form, and permanently deletes only the lowest-confidence windows; a quantized window can be promoted back to full precision when its cumulative attention score rises. Under a fixed byte budget, this design holds more historical context than permanent eviction, and the paper reports consistent quality–memory improvements over representative eviction and quantization baselines on LongBench, RULER, and GSM8K. The practical stake is that a fixed cache budget can preserve context that would otherwise be lost to an irreversible decision.

What carries the argument

The central mechanism is a window-level cumulative attention score used to rank and route contiguous cache windows. At each routing event, the score $S^\ell_t(w)$ accumulates attention probabilities assigned to the tokens of window $w$ by all query heads over the most recent $\Omega$ decoding steps; the top $K_f$ windows go to the full-precision tier, the next $K_q$ to the quantized tier, and the rest are evicted. Quantized windows keep accumulating scores and can be promoted back to full precision, while a write-once quantization ledger reuses the original low-bit representation on every subsequent migration so approximation error does not compound. Two diagnostics carry the empirical argument: Future Missed Mass measures the fraction of future full-cache attention that lands on already-discarded states, and Global LIR measures how often an inactive window later re-enters the active set.

What would settle it

Run QEvict against its own one-way policy (Full→Quantized→Evicted, with promotion disabled) on the full LongBench, RULER, and GSM8K suites at the same 5% and 20% byte budgets, keeping every other component identical. If disabling the recoverable tier does not lower scores, then recoverability is not what drives the reported gains.

Watch

Extended reading notes

Core claim

The paper's core claim is that binary retain-or-delete eviction systematically discards states that will later receive substantial attention, and that this loss is partly avoidable. Using full-cache attention traces, it defines Future Missed Mass—the share of future attention assigned to states the policy has already made inaccessible—and Global LIR—the rate at which historically inactive windows re-enter the full-precision set—and shows that a non-negligible middle region of windows takes attention during decoding but falls outside the small permanently retained set. QEvict operationalizes this by routing contiguous windows among three tiers: a full-precision tier, a persistent INT2 recoverable tier that remains available to attention, and permanent eviction. The paper reports that at matched memory budgets this recoverable tier reduces the attention mass assigned to permanent eviction and translates into higher scores across long-context understanding, retrieval, and reasoning benchmarks.

Load-bearing premise

QEvict assumes that cumulative attention scores measured on its own compressed cache are a reliable guide to which windows will be needed later, even though the cache has already been reshaped by earlier routing decisions.

Editorial extensions

If this is right

  • Under a fixed byte budget, QEvict preserves a broader historical region than permanent eviction: in the paper's byte-matched diagnostic, the evicted share of FullKV attention mass drops from 64.8% (two-tier binary eviction) to 35.7% (three-tier with an INT2 recoverable tier).
  • Tighter budgets make recovery more valuable: on LongBench at a 5% KV-cache budget, QEvict reports +9.7 points (Llama) and +4.7 points (Mistral) over the strongest matched eviction baseline, with smaller gains at 20%.
  • Promotion is exercised in practice, not just permitted: in the pilot R3 configuration, enabling Q→F promotion produced 639 recorded transitions, raised Global LIR from 0% to 6.6%, and reduced Future Missed Mass from 16.8% to 16.1%.
  • The hierarchy is scoring-agnostic: substituting exponentially decayed attention, attention p-norms, or key-vector norms for cumulative attention leaves the method functional, so the recoverable-tier design is separable from the particular importance estimator.
  • The systems cost is backend-dependent: with eager SDPA, QEvict improves decoding throughput and time-per-output-token relative to Full-KV, while with FlashAttention-2 it cuts peak GPU memory by 29.7% at the price of lower throughput; fused low-bit kernels are stated as future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An open question the paper does not settle is whether restoring 2-bit approximations rather than exact vectors will degrade on tasks that are highly sensitive to value noise; the paper's own diagnostic shows an attention-mass ratio of 0.866 for the quantized tier, so the effect is measurable.
  • The diagnostics Future Missed Mass and Global LIR could be adopted as standard reporting metrics for any eviction policy, since they quantify the information a policy permanently forfeits without requiring a full rerun on every benchmark.
  • The same three-tier principle could generalize to other non-stationary streaming state in long-context systems, such as activation caches or retrieved document summaries, wherever exact precision is only needed for the currently active region.
  • Adaptive, discourse-aware window boundaries are the most direct next step; the diagnostics suggest that very large fixed windows sacrifice allocation granularity, and segmenting by topical coherence could reduce the churn between tiers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes QEvict, a three-tier KV-cache management scheme that routes historical windows among full-precision, recoverable low-bit (INT2/INT4), and permanently evicted tiers under a fixed byte budget. It introduces two diagnostics, Future Missed Mass (FMM) and Global LIR, to characterize decoding-time importance drift and the revival of inactive windows, and it reports a systems implementation with a write-once quantization ledger intended to prevent error accumulation across tier transitions. The evaluation covers LongBench, RULER, and GSM8K on three instruction-tuned LLMs at multiple KV-memory budgets, reporting the best LongBench macro-average in all six model-budget settings, the best RULER macro-average among matched-memory eviction baselines at 20% budget on Llama-3.1-8B-Instruct, and a mixed set of GSM8K results against quantization baselines.

Significance. If the routing scores are reliable, the central claim is significant: QEvict offers a practical mechanism for converting irreversible eviction into recoverable, byte-constrained retention while preserving exact full precision for the most important regions. The paper is unusually thorough in its matched-memory benchmark tables, byte-accounting methodology, Pareto analyses, and systems measurements, and the write-once quantization ledger is a thoughtful design point that explicitly addresses error compounding. The reported LongBench and RULER gains are conditional on validating the endogenous cumulative-attention scores used for routing; the abstract's 'consistently improves' claim is broader than the GSM8K evidence supports.

major comments (3)
  1. [§4.2–4.3, Appendix I.3–I.4] Eq. (1) accumulates attention probabilities produced by the compressed cache itself, and those probabilities are attenuated in the quantized tier (Table 15 reports R_Q = 0.866). The same scores drive the top-K assignments in Eq. (5). The only direct validation, Appendix I.3, computes QSA as cosine agreement over two articles and 256 generated tokens; cosine agreement (mean 0.9824, per-head minimum 0.7874) does not measure ordering agreement among windows near the full/quantized boundary, and R_Q = 0.866 implies a systematic under-ranking of low-bit windows that could suppress promotion and accelerate eviction in a self-reinforcing loop. Appendix I.4 shows the deployed policy already revives 6.18% of inactive full-precision episodes versus 0.98% for the oracle. I request a long-sequence study (e.g., RULER 32K or a LongBench multi-document task) comparing deployed compressed-cache rankings with FullKV reference rankings using rank-based agreement (Spearman/Kendall or boundary overlap), together with promotion/demotion accuracy. Without this, the benchmark gains cannot be confidently attributed to correct routing rather than to a favorable interaction between scoring bias and benchmark distribution.
  2. [§5 and Appendix F (Tables 8–10, Fig. 3)] The abstract's claim that 'across LongBench, RULER, and GSM8K, QEvict consistently improves over representative eviction and quantization baselines' is contradicted by the GSM8K quantization comparisons. At the 20% budget, QEvict scores 79.09 on Llama, 40.12 on Mistral, and 79.42 on Qwen, while at least one quantization baseline is higher in every model: ZipCache achieves 84.46/40.64/89.99 and KVQuant-3b achieves 83.18/41.79/89.76. The same pattern appears in the main-text Figure 3. Please qualify the claim to LongBench/RULER, or state explicitly the performance-memory criterion under which these GSM8K operating points count as improvements.
  3. [§3, Appendix I.2] Observation II and Figure 1b attribute lower Future Missed Mass to the three-tier design, but FMM is defined over the set of inaccessible positions E_r in Eq. (13); any policy that leaves more positions accessible will, by construction, tend to have lower FMM on the same FullKV trace. The R2 versus R3 comparison is byte-matched, so the comparison is not vacuous, but the conclusion that routing is preserving future-relevant windows requires rank-level validation rather than mass-share accounting alone. This definitional dependence should be stated explicitly, and the rank-based validation requested above should be supplied to support the causal interpretation.
minor comments (6)
  1. [Table 2] The quantization comparison group repeats the QEvict row twice; remove the duplicate or replace it with an explicit marker that QEvict is the same operating point.
  2. [Appendix F.1] The protocol text attributes GSM8K to Mirzadeh et al. (2024), but GSM8K is Cobbe et al. (2021); the Mirzadeh citation is GSM-Symbolic and should be moved or removed.
  3. [Table 10] The table header misspells 'Qwen2.5-7B-Instruct' as 'Qwen2.5-7B-Instruc'.
  4. [§5 and Tables 2, 3, 6] The paper states that QEvict achieves the best macro-average in all six model-budget settings, but no macro-average row is shown in the tables; adding explicit macro-average rows (or an appendix table of them) would make the claim directly verifiable.
  5. [Appendix K] The promotion ablation uses different source articles across the two compared variants and is described as 'directional evidence'; this limitation should be stated in the main text if the promotion result is used to motivate the method.
  6. [Appendix J] The phrase 'used by method such SnapKV' contains a typo and should read 'used by methods such as SnapKV'.

Circularity Check

1 steps flagged · score 4.0 of 10

FMM diagnostic is self-definitional; benchmark claims remain externally grounded.

  1. self definitional [Section 3, Observation II; Appendix I.2, Eq. (13); Table 14]
    "Moreover, disabling access to the R3 quantized tier produces the FMM curve in Figure 1a, showing that these windows remain relevant to subsequent decoding. ... Future Missed Mass (FMM). At routing event r, let E^{ℓ,h}_r denote the historical positions made inaccessible by the evaluated policy."

    FMM (Eq. 13) is defined as the fraction of future FullKV attention assigned to positions in E_r, the set the policy makes inaccessible. In the 'R3: FP tier only' ablation, the quantized tier is made inaccessible, so E_r is enlarged exactly by the quantized windows. By Eq. (13), FMM must increase by the future attention mass of those windows; the gap between the dashed and solid FMM curves is therefore, by construction, the very quantity used to assert that the windows 'remain relevant.' The diagnostic cannot independently establish the value of the recoverable tier; it restates the definition of FMM. The LongBench, RULER, and GSM8K comparisons are external benchmarks and do not inherit this tautology.

full rationale

The central empirical claim of QEvict is not circular: the benchmark comparisons on LongBench, RULER, and GSM8K are evaluated against external baselines and an uncompressed FullKV reference, and the reported quality-memory improvements are not derived from the paper's own diagnostics. The main load-bearing circular element is the Future Missed Mass argument used to motivate the recoverable tier: because FMM (Eq. 13) counts only states made inaccessible by the policy, comparing a policy with a quantized accessible tier to an ablation in which that tier is inaccessible guarantees an FMM increase equal to the future attention mass of the disabled windows. The paper uses this guaranteed gap as evidence that the quantized windows 'remain relevant,' which is a definitional restatement rather than an independent observation. The endogenous routing scores (Eqs. 1 and 5) are computed from the compressed cache's own attention and so form a feedback loop, but this is a validation gap and correctness risk, not a derivation that reduces to its inputs; the paper reports the oracle-vs-policy deviation (6.18% versus 0.98% revival rate) and the attention-mass ratio R_Q = 0.866, acknowledging the divergence. The choice of routing interval and quantized-tier fraction is tuned on the evaluation benchmarks, which is a test-set selection concern rather than a circularity. Overall, one motivational diagnostic is self-definitional, but the principal empirical contribution stands on external benchmarks.

Assumptions & free parameters 7 free parameters · 5 assumptions · 2 invented entities

No new physical entities are introduced; the new items are diagnostic metrics and system components. The main assumptions are domain assumptions about attention-based scoring, window coherence, and quantization fidelity. Free parameters are the hyperparameters chosen by ablation on LongBench, plus constants used in the diagnostics.

free parameters (7)
  • q (quantized-tier fraction) = 0.70
    Fraction of the historical budget assigned to the quantized tier; chosen by ablation in Appendix G.2 on LongBench tasks, then fixed for all benchmarks.
  • Omega (routing window size) = 8
    Routing interval and window size; chosen by ablation in Appendix G.1 on LongBench subsets, then fixed for all benchmarks.
  • Sink tokens = 5
    Protected attention-sink prefix size; fixed across experiments following the StreamingLLM convention, not swept.
  • Recent tokens = 32 default; 128 on LongBench; 64 on RULER; 25% of budget on GSM8K
    Size of the protected local region; varies by benchmark and is set by protocol rather than independently justified.
  • Recoverable-tier precision = INT2
    INT2 chosen over INT4 in ablation G.3 because it provides broader historical coverage under a fixed byte budget.
  • FMM future horizon H = 32
    Future horizon for the Future Missed Mass diagnostic in Appendix I.2; a chosen constant that affects reported diagnostic values.
  • Global LIR inactivity threshold m = 3
    Default inactivity threshold for Global LIR episodes in Appendix I.4; the paper verifies the same trend for m in {1,2,4,8}.
assumptions (5)
  • domain assumption Cumulative attention over contiguous windows is a stable and sufficient signal for future window importance.
    Section 4.2 defines cumulative scores and Section 4.3 routes windows by these scores; this is the central empirical premise of the hierarchy, despite the paper's own Observation III showing drift.
  • domain assumption FullKV attention traces can serve as an oracle for what a compressed policy should have retained.
    Appendix I.1 uses uncompressed FullKV attention to compute Future Missed Mass and Global LIR, assuming the full-cache model's future attention is the correct reference for compressed policies.
  • domain assumption INT2 quantization preserves enough information to keep rankings and attention useful.
    Section 4.3 and Appendix I.3 rely on measured cosine agreement 0.9824, but the attention-mass ratio 0.866 shows magnitude attenuation, so the assumption is empirical rather than guaranteed.
  • domain assumption Natural language relevance is structured in contiguous spans, so windows are the correct routing unit.
    Section 2 argues from local-coherence literature; the system uses fixed-size contiguous windows and does not test adaptive boundaries.
  • domain assumption Benchmark prompts and greedy decoding are representative of long-context inference.
    Evaluation uses LongBench, RULER, and GSM8K with greedy decoding; no distribution-shift analysis is given for other decoding schemes or model families.
invented entities (2)
  • Future Missed Mass (FMM)
    purpose: Diagnostic measuring the fraction of future FullKV attention assigned to states a policy has permanently discarded.
    A definitional metric whose values depend on the chosen future horizon H and the FullKV oracle assumption; it has no falsifiable handle outside the paper's protocol.
  • Global LIR
    purpose: Diagnostic measuring how often historically inactive windows re-enter the full-precision set.
    A metric over selected-set trajectories; not independently falsifiable outside the paper's definitions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding." pith.science (2026). https://pith.science/paper/Q2YBIOSC

@misc{pith2026260805326,
  author       = {Pith},
  title        = {Pith review of: QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2YBIOSC}},
  note         = {Machine review of arXiv:2608.05326}
}
read the original abstract

Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention-derived scores. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again. We show that this assumption is brittle during decoding. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full-cache model. To characterize this behaviour, we introduce Future Missed Mass and Global LIR, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions. We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction. QEvict maintains high-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest-confidence windows. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full-precision. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions. Across long-context understanding, retrieval, and reasoning benchmarks, QEvict consistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention

Figures

Figures reproduced from arXiv: 2608.05326 by the authors.

Figure 1
Figure 1. Diagnostics motivating window-level routing and recoverable low-bit retention. (a) Contiguous-window routing reduces Future Missed Mass, while the gap between the solid and dashed R3 curves shows the future utility of the quantized tier. (b) At the same KV-cache budget, three-tier allocation preserves substantially more ground-truth attention mass than binary full-precision retention and eviction. uncertainty in fut… view at source ↗
Figure 2
Figure 2. Overall workflow of the QEvict framework. Stage 1 partitions the input sequence into contiguous KV windows while preserving sink and recent tokens in full precision. Stage 2 accumulates attention scores over each window to produce a stable importance ranking. Stage 3 routes windows under a fixed byte budget into a full-precision tier, a recoverable INT2 tier that remains available for attention and later promotion, … view at source ↗
Figure 3
Figure 3. GSM8K accuracy under KV-cache compression. Accuracy–memory trade-offs of QEvict and representative eviction and quantization baselines across three instruction-tuned language models. The resulting scores induce a layer-wise ranking over his￾torical windows. Although we use cumulative attention as the default importance estimator, the hierarchy requires only an ordering and is compatible with alternative ranking func… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: LongBench performance–memory Pareto analysis for Llama-3.1-8B-Instruct. Rows correspond to Single￾Document QA, Multi-Document QA, Summarization, and Few-Shot Learning. The left and right columns compare QEvict with eviction and quantization baselines, respectively. Das…
Figure 5
Figure 5. Figure 5: LongBench performance–memory Pareto analysis for Mistral-7B-Instruct-v0.2. Rows correspond to Single￾Document QA, Multi-Document QA, Summarization, and Few-Shot Learning. The left and right columns compare QEvict with eviction and quantization baselines, respectively. …
Figure 6
Figure 6. Figure 6: LongBench performance–memory Pareto analysis for Qwen2.5-7B-Instruct.Rows correspond to Single-Document QA, Multi-Document QA, Summarization, and Few-Shot Learning. The left and right columns compare QEvict with eviction and quantization baselines, respectively. Dashed…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 28 canonical work pages

  1. [1]

    Advances in Neural Information Processing Systems , volume=

    Attention Is All You Need , author=. Advances in Neural Information Processing Systems , volume=

  2. [2]

    2025 , eprint=

    SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference , author=. 2025 , eprint=

    work page 2025

  3. [3]

    arXiv preprint arXiv:2310.06825 , year=

    Mistral 7B , author=. arXiv preprint arXiv:2310.06825 , year=

    arXiv

  4. [4]

    The Fourteenth International Conference on Learning Representations, 2026a

    DefensiveKV: Taming the fragility of KV cache eviction in LLM inference , author=. The Fourteenth International Conference on Learning Representations, 2026a. URL https://openreview. net/forum , year=

    work page

  5. [5]

    2025 , eprint=

    WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference , author=. 2025 , eprint=

    work page 2025

  6. [6]

    2025 , eprint=

    ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models , author=. 2025 , eprint=

    work page 2025

  7. [7]

    ICLR , year=

    Efficient Streaming Language Models with Attention Sinks , author=. ICLR , year=

  8. [8]

    NeurIPS , year=

    Zhenyu Zhang and Ying Sheng and Tianyi Zhou and Tianlong Chen and Lianmin Zheng and Ruisi Cai and Zhao Song and Yuandong Tian and Christopher R\'. NeurIPS , year=

    work page

Show all 54 references
  1. [9]

    NeurIPS , year=

    Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time , author=. NeurIPS , year=

  2. [10]

    NeurIPS , year=

    SnapKV: LLM Knows What You are Looking for Before Generation , author=. NeurIPS , year=

  3. [11]

    PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling , author=

  4. [12]

    PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference , author=

  5. [13]

    Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference , author=

  6. [14]

    ICLR , year=

    CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences , author=. ICLR , year=

  7. [15]

    KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache , author=

  8. [16]

    NeurIPS , year=

    KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization , author=. NeurIPS , year=

  9. [17]

    NeurIPS , year=

    MiniCache: KV Cache Compression in Depth Dimension for Large Language Models , author=. NeurIPS , year=

  10. [18]

    KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing , author=

  11. [19]

    GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM , author=

  12. [20]

    LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy , author=

  13. [21]

    SOSP , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. SOSP , year=

  14. [22]

    Advances in Neural Information Processing Systems , volume=

    Language models are few-shot learners , author=. Advances in Neural Information Processing Systems , volume=

  15. [23]

    arXiv preprint arXiv:2302.13971 , year=

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    arXiv

  16. [24]

    Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Yang, Amy and Fan, Angela and others , journal=. The

  17. [25]

    Yang, An and Yang, Baosong and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Zhou, Chang and Li, Chengpeng and Li, Chengyuan and Liu, Dayiheng and Huang, Fei and others , journal=

  18. [26]

    Proceedings of Machine Learning and Systems , volume=

    Efficiently scaling transformer inference , author=. Proceedings of Machine Learning and Systems , volume=

  19. [27]

    Achiam, Josh and Adler, Steven and Agarwal, Sandhini and Ahmad, Lama and Akkaya, Ilge and Aleman, Florencia Leoni and Almeida, Diogo and Altenschmidt, Janko and Altman, Sam and Anadkat, Shyamal and others , journal=

  20. [28]

    arXiv preprint arXiv:2408.05646 , year=

    Eigen Attention: Attention in Low-Rank Space for KV Cache Compression , author=. arXiv preprint arXiv:2408.05646 , year=

    arXiv

  21. [29]

    International Conference on Learning Representations , year=

    Efficient Streaming Language Models with Attention Sinks , author=. International Conference on Learning Representations , year=

  22. [30]

    arXiv preprint arXiv:2406.02069 , year=

    PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling , author=. arXiv preprint arXiv:2406.02069 , year=

    arXiv

  23. [31]

    OpenReview preprint , year=

    Identify Critical KV Cache in LLM Inference from an Output Perturbation Perspective , author=. OpenReview preprint , year=

  24. [32]

    arXiv preprint arXiv:2503.12491 , year=

    CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences , author=. arXiv preprint arXiv:2503.12491 , year=

  25. [33]

    arXiv preprint arXiv:2510.01290 , year=

    ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models , author=. arXiv preprint arXiv:2510.01290 , year=

    arXiv

  26. [34]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , year=

    LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , year=

  27. [35]

    Conference on Language Modeling , year=

    RULER: What's the Real Context Size of Your Long-Context Language Models? , author=. Conference on Language Modeling , year=

  28. [36]

    arXiv preprint arXiv:2407.10671 , year=

    Qwen2 Technical Report , author=. arXiv preprint arXiv:2407.10671 , year=

    arXiv

  29. [37]

    arXiv preprint arXiv:2410.10819 , year=

    DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads , author=. arXiv preprint arXiv:2410.10819 , year=

    arXiv

  30. [38]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  31. [39]

    5-coder technical report , author=

    Qwen2. 5-coder technical report , author=. arXiv preprint arXiv:2409.12186 , year=

    arXiv

  32. [40]

    Advances in Neural Information Processing Systems , volume=

    Zipcache: Accurate and efficient kv cache quantization with salient token identification , author=. Advances in Neural Information Processing Systems , volume=

  33. [41]

    arXiv preprint arXiv:2410.05229 , year=

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models , author=. arXiv preprint arXiv:2410.05229 , year=

    arXiv

  34. [42]

    Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages

    Hearst, Marti A. Text Tiling: Segmenting Text into Multi-paragraph Subtopic Passages. Computational Linguistics. 1997

    1997

  35. [43]

    Modeling Local Coherence: An Entity-Based Approach

    Barzilay. Modeling Local Coherence: An Entity-Based Approach. Computational Linguistics. 2008. doi:10.1162/coli.2008.34.1.1

    2008 doi

  36. [44]

    How Contextual are Contextualized Word Representations? C omparing the Geometry of BERT , ELM o, and GPT -2 Embeddings

    Ethayarajh, Kawin. How Contextual are Contextualized Word Representations? C omparing the Geometry of BERT , ELM o, and GPT -2 Embeddings. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natur...

    2019 doi

  37. [45]

    and Zettlemoyer, Luke and Levy, Omer

    Joshi, Mandar and Chen, Danqi and Liu, Yinhan and Weld, Daniel S. and Zettlemoyer, Luke and Levy, Omer. S pan BERT : Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics. 2020. doi:10.1162/tacl_a_00300

    2020 doi

  38. [46]

    2024 , editor =

    Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , booktitle =. 2024 , editor =

    2024

  39. [47]

    C entering: A Framework for Modeling the Local Coherence of Discourse

    Grosz. C entering: A Framework for Modeling the Local Coherence of Discourse. Computational Linguistics. 1995

    1995

  40. [48]

    Discourse Segmentation by Human and Automated Means

    Passonneau. Discourse Segmentation by Human and Automated Means. Computational Linguistics. 1997

    1997

  41. [49]

    Text Segmentation as a Supervised Learning Task

    Koshorek. Text Segmentation as a Supervised Learning Task. Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018. doi:10.18653/v1/N18-2075

    2018 doi

  42. [50]

    What Does BERT Look at? An Analysis of BERT ' s Attention

    Clark, Kevin and Khandelwal, Urvashi and Levy, Omer and Manning, Christopher D. What Does BERT Look at? An Analysis of BERT ' s Attention. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. 2019. doi:10.18653/v1/W19-4828

    2019 doi

  43. [51]

    2024 , editor =

    Ribar, Luka and Chelombiev, Ivan and Hudlass-Galley, Luke and Blake, Charlie and Luschi, Carlo and Orr, Douglas , booktitle =. 2024 , editor =

    2024

  44. [52]

    Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    A simple and effective l\_2 norm-based strategy for kv cache compression , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages=

    2024

  45. [53]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2110.14168 , year=

    Training verifiers to solve math word problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.