Pith. sign in

REVIEW 3 major objections 6 minor 38 references

Lag-Relative Sparse Attention In Long Context Training

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning an LLM with a static sparse attention mask derived from LagKV scores makes it substantially more robust to key-value cache compression, and on a long-context question-answer benchmark it outperforms fine-tuning with full…

desk verdict A promising post-training trick for making models tolerate KV cache compression, but the paper overclaims the average result and skips the control that would show LagKV's mask actually matters. read the letter →

arxiv 2506.11498 v1 pith:3FNF7K5W submitted 2025-06-13 cs.CL

Manlai Liang , Wanyi Huang , Mandi Liu , Huaijun Li , Jinlong Li This is my paper
classification cs.CL
keywords sparseattentionKVcachecompressionlong-contextfine-tuningLagRULERbenchmarktrain-inferencemismatchquestion-answertuningstaticmask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models lose accuracy when inference-time KV-cache compression discards context they were never trained to ignore. This paper claims that fine-tuning with a sparse attention pattern fixed in advance, Lag-Relative Sparse Attention (LRSA), built from the LagKV scoring heuristic, trains the model to work well with compressed memory. On the RULER synthetic long-context benchmark, the LRSA-fine-tuned model scores higher on average than a vanilla-fine-tuned model, especially on question-answer tasks and at 2x and 4x compression ratios. The point matters because LRSA adds no parameters and only modest overhead, so if the claim holds it is a cheap way to close the train-inference gap for compressed long-context serving.

What carries the argument

The central object is the LRSA static attention mask. LagKV assigns each historical key or value a score by min-max normalizing the current KV chunk against the next chunk and taking the softmax of the standard deviation, retaining the top rL tokens per chunk. LRSA turns that selection into a differentiable, query-independent mask used chunk-by-chunk during prefill, so gradients flow through a context that is already compressed the way inference will compress it. Because the mask is independent of the query, the KV cache can be physically condensed rather than merely masked, which is what keeps the method compatible with efficient attention kernels.

What would settle it

An ablation that fine-tunes the same model with random static masks of the same sparsity and evaluates on RULER QA tasks at 2x and 4x compression; if random masks match LRSA's average scores, the benefit is not coming from the LagKV selection criterion.

Watch

Extended reading notes

Core claim

The paper's central claim is that the mismatch between full-attention training and compressed-context inference can be largely fixed by making training attention sparse in exactly the way inference will be sparse. LRSA uses LagKV's query-independent importance score, the min-max normalized standard deviation of K and V in the next lag window, to choose, for each chunk of incoming tokens, the top-K historical key-value pairs to attend to, and applies that same static mask during fine-tuning. The authors report that the 1.5B base model fine-tuned this way outperforms the same model fine-tuned with vanilla attention on the RULER synthetic evaluation set, with the clearest margins in QA tasks and at higher compression ratios; it also degrades more slowly as compression goes from 1x to 4x.

Load-bearing premise

The load-bearing premise is that the LagKV score, the normalized standard deviation of Keys and Values in the following chunk, really identifies the history the model needs; if that heuristic selects the wrong tokens, LRSA would train the model to ignore useful context and its advantage over vanilla fine-tuning would shrink or vanish.

Editorial extensions

If this is right

  • After LRSA fine-tuning, the model scores higher than the vanilla-fine-tuned model on RULER's question-answer tasks both with and without compression.
  • The LRSA-fine-tuned model degrades more slowly as the KV-cache retention ratio drops from 1x to 4x, meaning compression robustness is learned rather than patched at inference.
  • Because the mask is query-independent and static, the KV cache can be physically condensed during training, keeping LRSA compatible with efficient attention implementations and gradient-based fine-tuning.
  • LRSA introduces no additional parameters and has a small, stable effect on training loss, so it can be dropped into an existing post-training run without a separate compression step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LRSA advantage is real, the strongest form of the claim is that training-time attention sparsity should mirror the inference-time compression pattern, not just the sparsity ratio; matching the retained-token set during training may matter more than the training budget.
  • A direct extension would be to co-tune the lag size and retention ratio so they match the deployment memory budget; the paper fixes L=1024 and r=0.5 during training while testing other ratios at inference.
  • The paper does not compare LagKV-based selection against random masks or attention-based selection, so the contribution of the specific importance score is untested; such an ablation would clarify whether any static sparse training mask is enough or LagKV's heuristic is essential.
  • If LRSA generalizes beyond QA fine-tuning, retrieval and summarization long-context tasks should show the same robustness under compression, since they also depend on precise recall of earlier tokens.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Lag-Relative Sparse Attention (LRSA), a sparse-attention mask for post-training LLMs, derived from the LagKV compression importance score. LRSA processes long sequences in chunks, retaining the sink, a fixed-size lagging window, and a top-K subset of historical KV pairs selected by LagKV, and uses this static mask during fine-tuning. The authors fine-tune Qwen2.5-1.5B-Base on a QA-style long-context dataset and evaluate on RULER under LagKV compression at sequence lengths 4k–32k and retention ratios 1x–4x, comparing against vanilla fine-tuning and the base model. They claim LRSA achieves better QA performance and higher average RULER scores, and makes models more robust to KV-cache compression.

Significance. If the claims were fully supported, LRSA would be an attractive, parameter-free post-training technique that closes the train-inference gap for KV-cache compression, with low compute overhead and compatibility with chunked FlashAttention-style implementations. The paper reports a working implementation in Megatron and demonstrates on a 1.5B model that sparse fine-tuning can yield some gains under compression. However, the evidence is partially inconsistent with the paper's own statements, and the most distinctive component—the LagKV selection mechanism—is not tested against cheaper or simpler sparse-mask baselines, so the source of the observed gains is unidentified. With additional controls and corrected claims, this could be a useful contribution.

major comments (3)
  1. [Section 4.3 (Table 1), Section 5] The paper's claim that LRSA 'consistently outperforms' vanilla attention on QA tasks and that its average RULER score is higher is contradicted by the reported data. For example, at LS=4k,r=1x, the average of the 13 RULER tasks is 81.9 for vanilla fine-tuning versus 79.1 for LRSA, and on QA1 vanilla scores 76.4 vs 74.0 for LRSA. Similar reversals occur at LS=8k,r=1x (QA1: 57.6 vs 54.6) and LS=16k,r=1x (QA1: 57.6 vs 56.4). The consistent-improvement claim should be restricted to the compressed conditions (r=2x, 4x), where LRSA does generally show gains, or the table should be re-analyzed.
  2. [Section 3.2, Section 4.3] The central attribution of LRSA's benefit to LagKV-based token selection is untested because the only baselines are vanilla fine-tuning and the base model. No ablation is provided with a random mask, a fixed/stride mask, or an attention-score-based mask at the same sink/lag/retention budget. As a result, the robustness gains at 2x and 4x compression could be a generic effect of training with any sparse mask that teaches the model to cope with missing KV pairs, rather than evidence that the LagKV importance score identifies the 'salient' tokens. This control is load-bearing for the method's novelty, since LRSA's only difference from generic sparse training is the LagKV-derived mask.
  3. [Section 4.3, Abstract, Section 5] The paper claims improved 'robustness of the LLM with key-value compression' (Abstract) and concludes robustness to key-value compression generally, but all inference-time compression in Table 1 uses LagKV—the same heuristic that generates the LRSA training mask. This makes the evaluation conditions match the training conditions by construction and does not test whether LRSA confers robustness to other KV-compression methods (e.g., H2O, SnapKV, StreamingLLM). The authors should either evaluate with additional compression algorithms or restrict the claim to robustness under LagKV compression.
minor comments (6)
  1. [Section 1] The abbreviation 'LSRA' appears in the phrase 'LSRA introduces a structured sparsity pattern' and should be 'LRSA'.
  2. [Section 5] The sentence 'the average score of the fine-tuned model with LRSA the RULER synthetic evaluation set is higher than that with vanilla attention' is missing a word (likely 'on' or 'in').
  3. [Table 1] The CWE column contains apparent typos: '70.24' appears in several 16k rows and should likely be '70.2'; the table caption also reads 'T able 1' with a space.
  4. [Figures 1–4] The figures are not visible in the submitted text (they appear as '/uni...' sequences); please ensure the compiled PDF renders them.
  5. [Section 3.1] The citation for the notation system is left as '[?]' in the text; please fill in the missing reference.
  6. [Section 4.3] The paper does not report standard deviations or multiple random seeds; given the mixed results in Table 1, a measure of variance would aid interpretation.

Circularity Check

1 steps flagged · score 4.0 of 10

Matched LagKV training/evaluation masks make the robustness result a train–test alignment, with LagKV-specific selection untested against random-mask controls.

  1. other [Section 3.2 (LRSA) and Section 4.3 (RULER Benchmark)]
    "In the work of LagKV [16], it proposes to use the lag-relative information to determine the importance of historical tokens. ... Based onscore i and the retention ratior, select the toprLtokens to remain in the KV cache to complete the compression process. ... Following their approach, the LRSA mask matrix takes the form like Figure 1. ... The fine-tuned model with LRSA shows a considerably slower performance degradation as the compression ratio times increase from1×to4×in RULER and specific QA tasks."

    The LRSA training mask is generated by the exact LagKV scoring method (Eqs. 4-8) that is also used for the inference-time KV compression at evaluation. Section 3.2 says the mask follows LagKV; the same score_i and retention ratio r select the top rL tokens in both training and test. Therefore, the model is trained to attend only to tokens LagKV will later retain, making the reported robustness gain a matched train/test alignment. No random-mask, stride-mask, or alternative-compressor ablation is provided, so the causal contribution of LagKV's specific token selection is untested. The vanilla baseline provides independent empirical content, but the central anchoring premise is imported from a same-first-author citation, making the circularity partial.

full rationale

This is an empirical fine-tuning paper rather than a formal derivation. The main comparison between LRSA fine-tuning and vanilla fine-tuning uses an external benchmark (RULER), fixed training data, and a fixed base model, so the central robustness claim has independent empirical content. However, the specific version of robustness being claimed is robustness to LagKV compression: the LRSA training mask is exactly the LagKV eviction mask described in Section 3.2, and the test-time compression that defines the reported 2x/4x gains is also LagKV. This means the training and testing conditions are aligned by construction. The paper does not test whether the particular LagKV-selected positions matter beyond any sparse-mask training; a random or stride mask at the same budget, or an evaluation under H2O/SnapKV, would be needed to establish that. The LagKV salience premise is also sourced from a same-first-author preprint (reference [16]), which is load-bearing because it justifies the exact positions trained against. This is not complete circularity because the model could have failed to learn the masked distribution and because the vanilla baseline is an independent control. It is also not merely a self-citation; it is a missing-control and matched-condition issue. Therefore the score is 4 rather than higher.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or symbolic entities. It relies on LagKV's scoring and chunking as given, plus dataset and benchmark assumptions. The two key hand-set hyperparameters, L=1024 and r=0.5, are not ablated, and the sink size is not reported.

free parameters (4)
  • lag_size L = 1024 in training; 128/256/512/1024 in evaluation
    Chosen without ablation; defines the lagging window for LagKV scoring in Section 4.1.
  • retention_ratio r = 0.5 in training; 1x/2x/4x in evaluation
    Chosen without ablation; determines the top-K count in the sparse mask.
  • sink_size S = not specified
    Inherited from LagKV and required by the compression, but never reported in this paper.
  • number_of_filling_chunks = 2
    Section 3.2 says the number of chunks processed simultaneously can be optimized but gives no ablation.
assumptions (5)
  • domain assumption LagKV scoring identifies tokens worth attending to
    Section 3.2 assumes that min-max normalized standard deviation of K and V is a reliable importance signal; no ablation against random or attention-based masks is provided.
  • domain assumption Static query-independent masks during training transfer to inference-time LagKV compression
    The experiments use the same LagKV mask at train and test time; the paper does not test transfer to other compressors.
  • domain assumption Concatenated synthetic documents preserve QA coherence
    Section 4.1 augments short samples by concatenating unrelated paragraphs; the paper assumes this does not corrupt question-answer pairs.
  • domain assumption RULER synthetic benchmark approximates useful long-context QA performance
    Evaluation uses RULER only, with no real long-context QA benchmark or human evaluation.
  • standard math Standard softmax attention with RoPE
    Section 3.1 relies on the standard attention equations and rotary position embeddings without modification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lag-Relative Sparse Attention In Long Context Training." pith.science (2026). https://pith.science/paper/3FNF7K5W

@misc{pith2026250611498,
  author       = {Pith},
  title        = {Pith review of: Lag-Relative Sparse Attention In Long Context Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3FNF7K5W}},
  note         = {Machine review of arXiv:2506.11498}
}
read the original abstract

Large Language Models (LLMs) have made significant strides in natural language processing and generation, yet their ability to handle long-context input remains constrained by the quadratic complexity of attention computation and linear-increasing key-value memory footprint. To reduce computational costs and memory, key-value cache compression techniques are commonly applied at inference time, but this often leads to severe performance degradation, as models are not trained to handle compressed context. Although there are more sophisticated compression methods, they are typically unsuitable for post-training because of their incompatibility with gradient-based optimization or high computation overhead. To fill this gap with no additional parameter and little computation overhead, we propose Lag-Relative Sparse Attention(LRSA) anchored by the LagKV compression method for long context post-training. Our method performs chunk-by-chunk prefilling, which selects the top K most relevant key-value pairs in a fixed-size lagging window, allowing the model to focus on salient historical context while maintaining efficiency. Experimental results show that our approach significantly enhances the robustness of the LLM with key-value compression and achieves better fine-tuned results in the question-answer tuning task.

Figures

Figures reproduced from arXiv: 2506.11498 by the authors.

Figure 1
Figure 1. For efficiency, the prefilling process processes multiple chunks simultaneously. The number of filling chunks can be optimized up to different hardware and context lengths. For instance, with two chunks at a time: – Attention Filling 1 (Attn-Fill 1): The sink portion and two lag-sized KV segments are fed into the attention calculation. The attention mask is a full￾size matrix with the upper triangle masked out. Mean… view at source ↗
Figure 1
Figure 1. Lag-Relative Sparse Attention: Chunked-Mask is generated by the LagKV al￾gorithm. – Attention Filling 2 (Attn-Fill 2): The next two chunks are concatenated to the compressed KV cache. The third chunk’s queries require no additional masking, while the fourth chunk’s queries are masked for the second chunk’s portion. The masked indices are provided by the new compression process. – Repeat step 2 until all chunks proce… view at source ↗
Figure 2
Figure 2. Training Loss of Fine-tuned Qwen2.5-1.5B-Base (w/o LRSA) 4.2 Training Loss [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: Question-Answer-focused Statistics from RULER benchmark – LRSA improves the robustness of compression. The fine-tuned model with LRSA shows a considerably slower performance degradation as the compres￾sion ratio times increase from 1× to 4× in RULER and specific QA tas…
Figure 4
Figure 4. Figure 4: Average Statistics of RULER benchmark QA-related evaluation sets. Moreover, the average score of the fine-tuned model with LRSA the RULER synthetic evaluation set is higher than that with vanilla attention. In addition, we show that integrating LRSA into the Megatron f…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 9 canonical work pages

  1. [1]

    arXiv preprint arXiv:2004.05150 (2020)

    Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020)

    arXiv 2020

  2. [2]

    ACM transactions on intelligent systems and technology15(3), 1–45 (2024)

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al.: A survey on evaluation of large language models. ACM transactions on intelligent systems and technology15(3), 1–45 (2024)

    2024

  3. [3]

    arXiv preprint arXiv:1904.10509 (2019)

    Child, R., Gray, S., Radford, A., Sutskever, I.: Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019)

    arXiv 2019

  4. [4]

    Advances in neural information pro- cessing systems35, 16344–16359 (2022)

    Dao, T., Fu, D., Ermon, S., Rudra, A., Ré, C.: Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in neural information pro- cessing systems35, 16344–16359 (2022)

    2022

  5. [5]

    arXiv preprint arXiv:2402.13753 (2024) Lag-Relative Sparse Attention In Long Context Training 11

    Ding, Y., Zhang, L.L., Zhang, C., Xu, Y., Shang, N., Xu, J., Yang, F., Yang, M.: Longrope: Extending llm context window beyond 2 million tokens. arXiv preprint arXiv:2402.13753 (2024) Lag-Relative Sparse Attention In Long Context Training 11

    arXiv 2024

  6. [6]

    arXiv preprint arXiv:2407.11550 (2024)

    Feng, Y., Lv, J., Cao, Y., Xie, X., Zhou, S.K.: Ada-kv: Optimizing kv cache eviction by adaptive budget allocation for efficient llm inference. arXiv preprint arXiv:2407.11550 (2024)

    arXiv 2024

  7. [7]

    arXiv preprint arXiv:2310.01801 (2023)

    Ge, S., Zhang, Y., Liu, L., Zhang, M., Han, J., Gao, J.: Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801 (2023)

    arXiv 2023

  8. [8]

    Hsieh, C.P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y., Ginsburg, B.: Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 (2024)

    arXiv 2024

Show all 38 references
  1. [9]

    HUAWEI: Ascend/mindspeed-llm (2024), https://gitee.com/ascend/MindSpeed- LLM

    2024

  2. [10]

    arXiv preprint arXiv:2502.20766 (2025)

    Lai, X., Lu, J., Luo, Y., Ma, Y., Zhou, X.: Flexprefill: A context-aware sparse attention mechanism for efficient long-sequence inference. arXiv preprint arXiv:2502.20766 (2025)

    2025 arXiv

  3. [11]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Li, D., Sun, Z., Hu, X., Hu, B., Zhang, M.: Cmt: A memory compression method for continual knowledge learning of large language models. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 39, pp. 24413–24421 (2025)

    2025

  4. [12]

    arXiv preprint arXiv:2412.19442 (2024)

    Li, H., Li, Y., Tian, A., Tang, T., Xu, Z., Chen, X., Hu, N., Dong, W., Li, Q., Chen, L.: A survey on large language model acceleration based on kv cache management. arXiv preprint arXiv:2412.19442 (2024)

    2024 arXiv

  5. [13]

    In: International Conference on Machine Learning

    Li, Y., Yu, Y., Zhang, Q., Liang, C., He, P., Chen, W., Zhao, T.: Losparse: Struc- tured compression of large language models based on low-rank and sparse ap- proximation. In: International Conference on Machine Learning. pp. 20336–20350. PMLR (2023)

    2023

  6. [14]

    arXiv preprint arXiv:2412.10319 (2024)

    Li, Y., Jiang, H., Wu, Q., Luo, X., Ahn, S., Zhang, C., Abdi, A.H., Li, D., Gao, J., Yang, Y., et al.: Scbench: A kv cache-centric analysis of long-context methods. arXiv preprint arXiv:2412.10319 (2024)

    2024 arXiv

  7. [15]

    Advances in Neural Information Processing Systems37, 22947–22970 (2024)

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., Chen, D.: Snapkv: Llm knows what you are looking for before generation. Advances in Neural Information Processing Systems37, 22947–22970 (2024)

    2024

  8. [16]

    Liang, M., Zhang, J., Li, X., Li, J.: Lagkv: Lag-relative information of the kv cache tells which tokens are important (2025), https://arxiv.org/abs/2504.04704

    2025 arXiv

  9. [17]

    Advances in Neural Information Processing Systems36, 52342–52364 (2023)

    Liu, Z., Desai, A., Liao, F., Wang, W., Xie, V., Xu, Z., Kyrillidis, A., Shrivas- tava, A.: Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time. Advances in Neural Information Processing Systems36, 52342–52364 (2023)

    2023

  10. [18]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  11. [19]

    arXiv preprint arXiv:2406.16747 (2024)

    Lou, C., Jia, Z., Zheng, Z., Tu, K.: Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747 (2024)

    2024 arXiv

  12. [20]

    arXiv preprint arXiv:2307.06435 (2023)

    Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., Mian, A.: A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435 (2023)

    2023 arXiv

  13. [21]

    NVIDIA: Llm kv cache compression made easy (2024), https://github.com/NVIDIA/kvpress

    2024

  14. [22]

    NVIDIA: Nvidia/megatron-lm (2024), https://github.com/NVIDIA/Megatron- LM

    2024

  15. [23]

    arXiv preprint arXiv:2309.00071 (2023) 12 Manlal Liang, Wanyi Huang, Mandi Liu, Huaijun Li, and Jinlong Li

    Peng, B., Quesnelle, J., Fan, H., Shippole, E.: Yarn: Efficient context window ex- tension of large language models. arXiv preprint arXiv:2309.00071 (2023) 12 Manlal Liang, Wanyi Huang, Mandi Liu, Huaijun Li, and Jinlong Li

    2023 arXiv

  16. [24]

    Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin,...

    2025 arXiv

  17. [25]

    arXiv preprint arXiv:1909.08053 (2019)

    Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., Catanzaro, B.: Megatron-lm: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv:1909.08053 (2019)

    2019 arXiv

  18. [26]

    Neurocomputing568, 127063 (2024)

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced trans- former with rotary position embedding. Neurocomputing568, 127063 (2024)

    2024

  19. [27]

    arXiv preprint arXiv:2503.10501 (2025)

    Tan, X., Ye, P., Tu, C., Cao, J., Yang, Y., Zhang, L., Zhou, D., Chen, T.: Token- carve: Information-preserving visual token compression in multimodal large lan- guage models. arXiv preprint arXiv:2503.10501 (2025)

    2025 arXiv

  20. [28]

    arXiv preprint arXiv:2407.15891 (2024)

    Tang, H., Lin, Y., Lin, J., Han, Q., Hong, S., Yao, Y., Wang, G.: Razorat- tention: Efficient kv cache compression through retrieval heads. arXiv preprint arXiv:2407.15891 (2024)

    2024 arXiv

  21. [29]

    In: Proceedings of the 41st Inter- national Conference on Machine Learning

    Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., Han, S.: Quest: query-aware sparsity for efficient long-context llm inference. In: Proceedings of the 41st Inter- national Conference on Machine Learning. ICML’24, JMLR.org (2024)

    2024

  22. [30]

    Advances in neural information pro- cessing systems30(2017)

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez,A.N.,Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems30(2017)

    2017

  23. [31]

    Neurocomputing 639, 130193 (2025)

    Wang, Q., Fu, Y., Cao, Y., Wang, S., Tian, Z., Ding, L.: Recursively summarizing enables long-term dialogue memory in large language models. Neurocomputing 639, 130193 (2025)

    2025

  24. [32]

    arXiv preprint arXiv:2403.07378 (2024)

    Wang, X., Zheng, Y., Wan, Z., Zhang, M.: Svd-llm: Truncation-aware singu- lar value decomposition for large language model compression. arXiv preprint arXiv:2403.07378 (2024)

    2024 arXiv

  25. [33]

    arXiv preprint arXiv:2402.02244 (2024)

    Wang, X., Salmani, M., Omidi, P., Ren, X., Rezagholizadeh, M., Eshaghi, A.: Beyond the limits: A survey of techniques to extend the context length in large language models. arXiv preprint arXiv:2402.02244 (2024)

    2024 arXiv

  26. [34]

    arXiv preprint arXiv:2309.17453 (2023)

    Xiao, G., Tian, Y., Chen, B., Han, S., Lewis, M.: Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453 (2023)

    2023 arXiv

  27. [35]

    arXiv preprint arXiv:2502.11089 (2025)

    Yuan, J., Gao, H., Dai, D., Luo, J., Zhao, L., Zhang, Z., Xie, Z., Wei, Y., Wang, L., Xiao, Z., et al.: Native sparse attention: Hardware-aligned and natively trainable sparse attention. arXiv preprint arXiv:2502.11089 (2025)

    2025 arXiv

  28. [36]

    Advances in neural information processing systems33, 17283–17297 (2020)

    Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al.: Big bird: Transformers for longer sequences. Advances in neural information processing systems33, 17283–17297 (2020)

    2020

  29. [37]

    arXiv preprint arXiv:2505.23520 (2025)

    Zhang, Y., Guo, D., Wu, F., Zhu, G., Ding, D., Zhang, Y.: Anchoratten- tion: Difference-aware sparse attention with stripe granularity. arXiv preprint arXiv:2505.23520 (2025)

    2025 arXiv

  30. [38]

    Advances in Neural Information Processing Systems36, 34661–34710 (2023)

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., Ré, C., Barrett, C., et al.: H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems36, 34661–34710 (2023)

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.