Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Rectified Sparse Attention

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Periodic dense refreshes keep sparse attention near-lossless for long-context decoding, the paper argues.

desk verdict ReSA's periodic dense rectification is a sound, useful idea and the quality results mostly support it, but the paper's central speedup claim is unverifiable until the p notation is fixed. read the letter →

arxiv 2506.04108 v2 pith:OW7XPX7K submitted 2025-06-04 cs.CL

Yutao Sun , Tianzhu Ye , Li Dong , Yuqing Xia , Jian Chen , Yizhao Gao , Shijie Cao , Jianyong Wang
show 1 more author
Furu Wei
This is my paper
classification cs.CL
keywords long-contextinferencesparseattentionKVcacherectificationtest-timescalingdecoderaccelerationgroupquery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long sequences make transformer decoding slow because every new token must reread the full key-value cache. Sparse attention reads only a subset, but its approximations accumulate errors in the cache, so quality degrades as generation continues. This paper proposes Rectified Sparse Attention (ReSA), which alternates block-sparse decoding with a dense forward pass every f tokens to rebuild the cache exactly. The claim is that this periodic rectification keeps errors bounded within a window, making generation quality nearly identical to dense attention while cutting inference cost. On math reasoning, language modeling, and retrieval, ReSA reports near-lossless accuracy with up to 2.4x end-to-end speedup at 256K token contexts.

What carries the argument

Group block sparse attention (GBSA) with shared head-grouping: each GQA group pools its queries and selects the same top blocks via Quest-style min/max block descriptors, so all heads in a group read the same small set of contiguous key blocks. Dense rectification is the companion mechanism: every f decoding steps, the last f generated tokens are re-encoded with dense attention in one parallel forward pass, replacing the sparsely-computed KV entries and updating block statistics. The rectification frequency f controls the bound on accumulated error and the amortized cost; n = max(nmin, ceil(M p)) controls how many blocks each query attends to.

What would settle it

Profile the ReSA decoding kernel at the stated defaults (block size 16, p=0.9, f=32) and count the KV bytes fetched per query. If the count is close to the full cache, the 2.4x speedup cannot be reproduced; if the count matches an active ratio near 0.1, then the paper's formula and defaults misstate the sparsity that produces the results.

Watch

Extended reading notes

Core claim

ReSA's central claim is that error accumulation in sparse decoding is curable by periodically refreshing the KV cache with dense attention. Instead of trying to make every sparse step lossless, the method lets a cheap block-sparse pass generate up to f tokens, then re-encodes those f tokens with exact attention to reset the cache. Because the cache is rebuilt exactly, any approximation error is confined to the last f tokens; the paper argues this suffices to keep the model aligned with its pretraining distribution, recovering most of the dense baseline's accuracy across math reasoning, language modeling, and RULER retrieval while reducing memory access per step to roughly 1/b + p + 1/f of the cache.

Load-bearing premise

The reported speedups depend on the sparse pass reading only a small fraction of the KV cache, but the paper's default parameter p=0.9 and its memory-access formula imply nearly full-cache reads, so the efficiency claim rests on an unstated mismatch between the defined and actual sparsity level.

Editorial extensions

If this is right

  • Sparse decoding alone degrades with length; ReSA closes most of the gap on math reasoning, matching dense average accuracy on the 7B model (60.52 vs 60.72).
  • At 256K context, ReSA reports 2.28x (FP16) and 2.44x (INT4) end-to-end speedups over dense attention.
  • RULER retrieval accuracy with sparsity ratio 0.9 (0.559) slightly exceeds the dense baseline (0.549), so short-generation retrieval is not hurt.
  • Rectification amortizes well: at f=32, its overhead is about 28-33% of attention latency at long contexts, and even f=128 retains most of the quality gain.
  • The method is compatible with serving optimizations like continuous batching and chunked prefill, and with more advanced block-selection mechanisms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own ablation mentions an 'attention computation ratio' of 0.1 while the default p is stated as 0.9, suggesting the experiments that yield speedups actually use an active block ratio near 0.1 that the text does not clearly reconcile with the formula.
  • If rectification truly bounds error to a window of f tokens, then much larger rectification intervals, or combining with learned retrieval, could extend the speed-quality frontier beyond the reported settings.
  • ReSA's speedup over self-speculation suggests a broader design principle: periodic exact recomputation of a small suffix can replace per-token verification for approximate inference schemes, not just sparse attention.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Rectified Sparse Attention (ReSA), a training-free inference method that alternates block-sparse decoding with periodic dense rectification of the KV cache. The motivation is that sparse decoding alone accumulates KV-cache errors and degrades generation quality over long horizons; a dense forward pass every f steps is intended to bound this error accumulation. The authors evaluate ReSA on math reasoning, language modeling, and RULER retrieval tasks using Qwen2.5 models, and report near-dense quality with up to 2.42x end-to-end speedup at 256K sequence length. The core qualitative claim, that periodic rectification substantially closes the sparse-decoding quality gap, is supported by the tables and ablations. However, the quantitative efficiency claim is currently not verifiable because the parameter p is used inconsistently as both an active block ratio and a sparsity ratio, and because the speedup experiments are not reported with enough detail.

Significance. If the efficiency claim can be made precise, ReSA would be a practically useful contribution: it is training-free, compatible with existing serving optimizations such as continuous batching and chunked prefill, and the quality results suggest that rectification is an effective fix for error accumulation in sparse decoding. The paper also ships code, which supports reproducibility. The central qualitative finding is credible and not circular: ReSA is an algorithm with hyperparameters (f, p, b, nmin, nlocal) selected by standard ablation, and the error-bounding statement is a design assertion rather than a quantity derived from fitted values. Nevertheless, the load-bearing speedup claim depends on resolving the meaning of p and on reporting the exact configuration used in the kernel and end-to-end measurements. Until that is fixed, the practical significance of the efficiency results cannot be assessed.

major comments (4)
  1. [§2.1, Eq. (5); §2.3, Eq. (6); §3.1] The parameter p is defined inconsistently. Eq. (5) sets n = max(nmin, ceil(M×p)), so p is the active block ratio, whereas §2.3 and §3.1 call p the sparsity ratio and insert it directly into the memory-access formula. With the stated default p=0.9, Eq. (6) gives 1/16 + 0.9 + 1/32 ≈ 0.99, i.e., essentially no memory savings, which contradicts the 2.44x INT4 speedup reported in §3.5.2. Conversely, §3.6 states that the attention computation ratio is reduced to 0.1, which only holds if p is a sparsity ratio and the active fraction is 0.1. The paper must state unambiguously which quantity p denotes, correct Eq. (5) to n = ceil(M×(1−p)) if p is the sparsity ratio, and report the actual active block fraction used in the speedup experiments.
  2. [§3.3, Figure 5] The sparsity-ratio ablation is internally inconsistent. The text reports a 'noticeable performance gap between p=0.98 and p=0.95' and states that 'p=0.8 sparsity achieves perplexity comparable to the dense setting,' yet the legend of Figure 5 shows ReSA with p=0.2, p=0.1, and p=0.05. These are different quantities (active ratios versus sparsity ratios), and neither set matches the default p=0.9 claimed in §3.1. Please align the figure labels, the text, and the actual experimental values so that the default selection is reproducible.
  3. [§3.4, Table 2] The RULER experiments use the same ambiguous p. The table heading says 'sparsity level x' and the text says 'as the sparsity ratio increases from p=0.95 to p=0.9,' even though 0.9 is a smaller sparsity ratio; the text also says 'a lower sparsity ratio generally leads to faster inference,' which is only true if p is an active ratio, not a sparsity ratio. Please re-express these results in terms of either active blocks or sparsity consistently and report the actual values used in the experiments.
  4. [§3.5.2, §5, Abstract] The end-to-end speedup claim is not presented with enough detail to be checked. The abstract and conclusion report 2.42x, while §3.5.2 reports 2.28x in FP16 and 2.44x in INT4; the figures plot latency but do not give numerical values. Please report the raw latencies, state whether prefill time is included, clarify the shared-KV-cache evaluation protocol introduced in §3.5, and provide the exact (b, f, active-block fraction) configuration used for the 256K measurements.
minor comments (4)
  1. [§1, §3.1, §3.3] There are several wording and typographical errors, including 'evalutaion architectures,' 'dataets inclue,' 'Se evaluate,' and 'nlocal = 1to avoid performance degradation.' These should be corrected.
  2. [Table 1] The row labeled 'Sparsedense2' is not defined in the text or the caption; please explain what configuration it represents.
  3. [§2.1, Eq. (2)] The mask notation M^{ijk} = M_{ij⌊k/b⌋} is confusing given that M is defined as an element of {0,1}^{h×n×n/b}; please clarify the index conventions.
  4. [Appendix B, Table 3] The speedup comparison with self-speculation would benefit from a short description of the batching, kernel settings, and acceptance-rate assumptions, since the 1.92x average speedup is otherwise difficult to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ReSA's quality and efficiency claims are empirical, and hyperparameters (f, p, b) are ablation-tuned inputs rather than fitted outputs.

full rationale

The manuscript does not derive its central results from quantities that are defined in terms of those results. ReSA combines an external block-sparse attention design (Quest) with shared grouping (NSA) and a proposed dense rectification step; each component is described by explicit algorithms and equations. The active block count is set by Eq. (5) using an active ratio p, and the memory-access estimate in Eq. (6) is an asymptotic accounting formula, not a fitted predictor. The claimed 2.42x speedup is reported as a measured end-to-end latency result (Figures 7 and 8), not as an output of Eq. (6). The hyperparameters f, p, and b are selected by ablation experiments (Figures 5 and 9, Table 2), which is standard empirical tuning rather than circular fitting. No self-citation is load-bearing: SeerAttention is mentioned only as an optional compatible extension and is not used to justify ReSA's central mechanism. The notation p is used inconsistently (active ratio in Eq. 5 vs. sparsity ratio in Eq. 6 and Section 3.5.1), and this ambiguity undermines the verifiability of the efficiency analysis, but it does not make any derivation equivalent to its inputs by construction. That is a correctness or reproducibility concern, not circularity. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central mechanism rests on the assumption that periodic dense refresh bounds KV-cache error, which is a design assertion rather than a derived theorem. The hyperparameters f, p, b, nmin, and nlocal are tuned for performance, and the paper does not provide a principled selection rule. No new entities are introduced.

free parameters (5)
  • rectification frequency f = 32 (default)
    Chosen via ablation in Figure 9; affects the trade-off between quality and overhead.
  • sparsity/active ratio p = 0.9 (default)
    Ambiguous in the paper: Eq 5 uses it as the active block ratio, while the text and memory analysis treat it as a sparsity ratio. Tuned via ablations.
  • block size b = 16
    Fixed in experiments; not swept in the paper.
  • minimum block number nmin = 16
    Set to avoid performance degradation on short sequences.
  • local block count nlocal = 1
    Number of recent blocks always preserved for local coherence.
assumptions (3)
  • domain assumption Periodic dense rectification bounds KV cache error within a constant window
    Stated in Section 2.2 without formal proof; empirically supported by Figure 9.
  • domain assumption Block min/max descriptors preserve retrieval quality
    Inherited from Quest [23]; ReSA relies on this for block selection.
  • domain assumption Dense attention matches the pretraining distribution, so refreshing with dense attention restores alignment
    Implicit in the motivation of Section 1 and 2.2; not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rectified Sparse Attention." pith.science (2026). https://pith.science/paper/OW7XPX7K

@misc{pith2026250604108,
  author       = {Pith},
  title        = {Pith review of: Rectified Sparse Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OW7XPX7K}},
  note         = {Machine review of arXiv:2506.04108}
}
abstract

Efficient long-sequence generation is a critical challenge for Large Language Models. While recent sparse decoding methods improve efficiency, they suffer from KV cache misalignment, where approximation errors accumulate and degrade generation quality. In this work, we propose Rectified Sparse Attention (ReSA), a simple yet effective method that combines block-sparse attention with periodic dense rectification. By refreshing the KV cache at fixed intervals using a dense forward pass, ReSA bounds error accumulation and preserves alignment with the pretraining distribution. Experiments across math reasoning, language modeling, and retrieval tasks demonstrate that ReSA achieves near-lossless generation quality with significantly improved efficiency. Notably, ReSA delivers up to 2.42$\times$ end-to-end speedup under decoding at 256K sequence length, making it a practical solution for scalable long-context inference. Code is available at https://aka.ms/ReSA-LM.

Figures

Figures reproduced from arXiv: 2506.04108 by the authors.

Figure 1
Figure 1. Sparse decoding perfor￾mance becomes worse with increas￾ing decoding length due to error ac￾cumulation of KV cache. Recent works [18, 23] used sparse decoding to alleviate this issue. These methods selectively attend to a subset of the con￾text, achieving accuracy comparable to dense attention on long inputs while reducing computational cost. However, as shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of ReSA. After completing the prefill stage, the model enters sparse decoding. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of Group Block Sparse Attention. For each group of query heads, we perform [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Top-3 next-token prediction accu￾racy with different sparsity ratio. long-context reasoning tasks, whereas Sparse Decoding leads to performance degradation as decoding progresses. Additionally, manually enforcing dense layers for the first two layers does not result in…
Figure 6
Figure 6. Figure 6: Kernel-level latency breakdown across different sequence lengths. While Sparse Decoding [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: End-to-end latency with INT4. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Ablation studies on different rectification frequencies [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SeerAttention-R: Sparse Attention Adaptation for Long Reasoning

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A learned gate selects the important KV blocks during long decoding, preserving math reasoning accuracy while skipping up to 90% of attention work.

Reference graph

Works this paper leans on

28 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills.arXiv preprint arXiv:2308.16369, 2023

    Amey Agrawal, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S Gulavani, and Ramachandran Ramjee. Sarathi: Efficient llm inference by piggybacking decodes with chunked prefills.arXiv preprint arXiv:2308.16369, 2023

    arXiv 2023

  2. [2]

    Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Training generalized multi-query transformer models from multi-head checkpoints.arXiv preprint arXiv:2305.13245, 2023

    arXiv 2023

  3. [3]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024

    arXiv 2024

  4. [4]

    Magicpig: Lsh sampling for efficient llm generation.arXiv preprint arXiv:2410.16179, 2024

    Zhuoming Chen, Ranajoy Sadhukhan, Zihao Ye, Yang Zhou, Jianyu Zhang, Niklas Nolte, Yuandong Tian, Matthijs Douze, Leon Bottou, Zhihao Jia, et al. Magicpig: Lsh sampling for efficient llm generation.arXiv preprint arXiv:2410.16179, 2024

    arXiv

  5. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [6]

    Flash-Decoding for long-context inference.https://crfm.stanford.edu/2023/10/12/flashdecoding.html, 2023

    Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-Decoding for long-context inference.https://crfm.stanford.edu/2023/10/12/flashdecoding.html, 2023

    work page 2023

  7. [7]

    Marlin: Mixed-precision auto-regressive parallel inference on large language models.arXiv preprint arXiv:2408.11743, 2024

    Elias Frantar, Roberto L Castro, Jiale Chen, Torsten Hoefler, and Dan Alistarh. Marlin: Mixed-precision auto-regressive parallel inference on large language models.arXiv preprint arXiv:2408.11743, 2024

    arXiv 2024

  8. [8]

    Seerattention: Learning intrinsic sparse attention in your llms.arXiv preprint arXiv:2410.13276, 2024

    Yizhao Gao, Zhichen Zeng, Dayou Du, Shijie Cao, Peiyuan Zhou, Jiaxing Qi, Junjie Lai, Hayden Kwok-Hay So, Ting Cao, Fan Yang, et al. Seerattention: Learning intrinsic sparse attention in your llms.arXiv preprint arXiv:2410.13276, 2024

    arXiv

Show all 28 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  2. [10]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.14008, 2024

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems.arXiv preprint arXiv:2402.1...

    2024 arXiv

  3. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

    2021

  4. [12]

    Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference

    Connor Holmes, Masahiro Tanaka, Michael Wyatt, Ammar Ahmad Awan, Jeff Rasley, Samyam Rajbhandari, Reza Yazdani Aminabadi, Heyang Qin, Arash Bakhtiari, Lev Kurilenko, et al. Deepspeed-fastgen: High-throughput text generation for llms via mii and deepspeed-inference. arXiv prepr...

    2024 arXiv

  5. [13]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 10

    2024 arXiv

  6. [14]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. InInternational Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  7. [15]

    Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:3843–3857, 2022

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models.Advances in Neural Information Processing Systems, 35:...

    2022

  8. [16]

    Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty.arXiv preprint arXiv:2401.15077, 2024

    2024 arXiv

  9. [17]

    Mario: Math reasoning with code interpreter output–a reproducible pipeline.arXiv preprint arXiv:2401.08190, 2024

    Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output–a reproducible pipeline.arXiv preprint arXiv:2401.08190, 2024

    2024 arXiv

  10. [18]

    Clusterkv: Manipulating llm kv cache in semantic space for recallable compression.arXiv preprint arXiv:2412.03213, 2024

    Guangda Liu, Chengwei Li, Jieru Zhao, Chenqi Zhang, and Minyi Guo. Clusterkv: Manipulating llm kv cache in semantic space for recallable compression.arXiv preprint arXiv:2412.03213, 2024

    2024 arXiv

  11. [19]

    Moba: Mixture of block attention for long-context llms

    Enzhe Lu, Zhejun Jiang, Jingyuan Liu, Yulun Du, Tao Jiang, Chao Hong, Shaowei Liu, Weiran He, Enming Yuan, Yuzhi Wang, et al. Moba: Mixture of block attention for long-context llms. arXiv preprint arXiv:2502.13189, 2025

    2025 arXiv

  12. [20]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv pre...

    2024 arXiv

  13. [21]

    Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding.arXiv preprint arXiv:2408.11049, 2024

    Ranajoy Sadhukhan, Jian Chen, Zhuoming Chen, Vashisth Tiwari, Ruihang Lai, Jinyuan Shi, Ian En-Hsu Yen, Avner May, Tianqi Chen, and Beidi Chen. Magicdec: Breaking the latency- throughput tradeoff for long context generation with speculative decoding.arXiv preprint arXiv:2408.1...

    2024 arXiv

  14. [22]

    Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.arXiv preprint arXiv:2404.11912, 2024

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding.arXiv preprint arXiv:2404.11912, 2024

    2024 arXiv

  15. [23]

    Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774, 2024

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest: Query-aware sparsity for efficient long-context llm inference.arXiv preprint arXiv:2406.10774, 2024

    2024 arXiv

  16. [24]

    Infllm: Training-free long-context extrapolation for llms with an efficient context memory.arXiv preprint arXiv:2402.04617, 2024

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrapolation for llms with an efficient context memory.arXiv preprint arXiv:2402.04617, 2024

    2024 arXiv

  17. [25]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  18. [26]

    An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, et al. Qwen2. 5-1m technical report.arXiv preprint arXiv:2501.15383, 2025

    2025 arXiv

  19. [27]

    Orca: A distributed serving system for Transformer-based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-based generative models. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, 2022

    2022

  20. [28]

    Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025

    Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, YX Wei, Lean Wang, Zhiping Xiao, et al. Native sparse attention: Hardware-aligned and natively trainable sparse attention.arXiv preprint arXiv:2502.11089, 2025. 11 A Pseudo Code of Flash D...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.