Pith. sign in

REVIEW 3 major objections 4 minor 48 references

Windowing the draft's attention cuts long-context speculation cost up to 44%

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:08 UTC pith:SDO3IU5Z

load-bearing objection A solid, well-measured systems paper on windowing the built-in MTP draft; the central result holds, but the cost decomposition has a small leak. the 3 major comments →

arxiv 2607.21535 v1 pith:SDO3IU5Z submitted 2026-07-23 cs.LG cs.CLcs.PF

Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

Alagappan Valliappan This is my paper
classification cs.LG cs.CLcs.PF
keywords speculative decodingmulti-token predictiondraft attentionattention sinksliding windowKV cachelong contextlossless acceleration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

At million-token context, the built-in draft head that makes speculative decoding cheap is itself the bottleneck: it runs full attention over the entire KV cache at every draft step, so its cost grows linearly with context and can make speculation slower than no speculation. The paper's claim is that this 'draft-KV tax' can be removed by windowing the draft's attention only—a fixed attention sink plus a sliding window of recent tokens—while leaving the full-attention target verification untouched. This is lossless by construction: the target decides every accepted token, so windowing changes which tokens are proposed, never which are accepted. Across three architecture families at 1M context the per-decode-step cost falls by 28–44% with acceptance preserved, and the unread draft KV is reclaimed as a compact ring buffer.

Core claim

The central discovery is that a built-in Multi-Token-Prediction (MTP) draft head—a small extra head that predicts several future tokens in one pass—pays an O(context-length) attention read per draft step, so at million-token context the draft phase adds roughly 92–138% on top of the bare verification cost, nearly doubling the decode step and, on hard low-acceptance tasks, pushing deep native drafts below the no-speculation baseline. The paper shows that restricting the draft's attention to a fixed sink plus the most recent W tokens (4K in practice) eliminates this tax while preserving the target's verified output distribution exactly in exact arithmetic. Windowing changes only the proposal d

What carries the argument

The central mechanism is the *draft-attention window*: a StreamingLLM-style attention sink plus sliding window applied only to the draft head's KV read, not to the target. It is implemented by truncating the draft's per-request KV index to the first `n_sink` tokens plus the last `W` tokens; because RoPE positions are baked into cached keys, this does not disturb scores. The argument is carried by a per-decode-step cost identity, t_step = t_verify + t_ctx_draft + γ·t_fwd_draft, where the two O(S) draft-phase terms shrink to O(W) under windowing while the verify term is untouched. The losslessness argument is the speculative-decoding acceptance rule: the full-context target distribution decide

Load-bearing premise

The load-bearing premise is that the windowed draft's proposals are accepted by the full-context target often enough that the per-step cost saving is never outweighed by a change in acceptance length.

What would settle it

Take a needle-in-haystack task at 1M context with the needle placed just outside the 4K recent window (and not in the sink), and compare acceptance length and end-to-end latency of Windowed-MTP versus the native full-context draft. If the acceptance drop exceeds the cost-side saving, the central claim is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • At 1M context the draft's KV working set is bounded by a constant window (~4K), so per-step speculation cost no longer scales with context length—the saving widens as the prompt grows.
  • The target's verified output distribution is unchanged (greedy-exact in exact arithmetic; bf16 verifier noise affects native and windowed drafts identically).
  • End-to-end latency improves by the same cost-side margin at matched acceptance, and by more where windowing also lifts acceptance; hard tasks where native MTP dropped below dense are restored to net-positive speedup.
  • The draft KV pool, measured at 7.7–11% of total KV, can be compacted into a ring buffer, freeing memory for extra concurrent requests or deeper drafts at the same budget.
  • Windowing composes with tree-shaped speculation and with KV quantization (fp8), and is training-free and drop-in, so it can be gated on a context-length threshold.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference beyond the paper: the same windowing should transfer to trained draft heads (e.g., EAGLE-style) and diffusion drafters, which pay the same O(S) draft-attention tax; the paper itself flags this as natural future work.
  • Inference beyond the paper: because the windowed draft never reads intermediate draft KV, the draft's prefill pass can be pruned to O(W+sink), which would lower time-to-first-token on long prompts—an extension the paper leaves to future work.
  • Inference beyond the paper: an adaptive draft-depth controller could tune depth per input cheaply, since the windowed draft's cost no longer rises steeply with depth; the paper's best-depth sweep already hints that shallower depths often suffice.
  • Inference beyond the paper: the measured margin is framework-relative; on a leaner serving runtime that lacks a per-step O(S) index build, the headline 28–44% would shrink toward the durable per-forward attention-slope reduction (22–40% in the paper's own fit).

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper identifies a long-context 'draft-attention tax' in built-in Multi-Token-Prediction (MTP/NEXTN) heads: at million-token context the draft's full-context attention read dominates the decode step, eroding or inverting speculative-decoding speedups. It proposes Windowed-MTP, which restricts only the draft's attention to a StreamingLLM-style sink plus a sliding window while leaving the target's full-attention verification intact. The paper argues this is training-free, drop-in, and lossless by construction, and reports per-decode-step cost reductions of +28% to +44% over the native MTP draft at 1M context across three architecture families (Qwen3.6-35B GDN-MoE, Qwen3.5-122B GDN-MoE, and a Mamba2-hybrid NoPE 120B), with acceptance preserved within confidence intervals, end-to-end speedups over native MTP of +4% to +53%, and reclamation of 7.7-11% of total KV as dead draft KV. The evidence includes an analytic latency model (Eq. 1-3), phase-decomposed kernel timings (App. B), a backend ablation, a best-depth comparison, batch/TP Pareto sweeps, and a reproducibility package.

Significance. If the central claims hold, this is a practically important systems contribution. It directly attacks a real bottleneck for long-context serving with built-in speculative heads, and the intervention is simple enough to be adopted in production. The losslessness logic is a standard speculative-decoding theorem and is correctly stated. The paper's strengths are the direct phase-decomposed measurements (App. B), the backend ablation showing Triton is a handicap rather than the source of the win (Table 5), the explicit separation of durable per-forward slope reduction from implementation-dependent index-build costs (Table 6 and §7), and the release of a reproduction package. The claimed 'input-invariant' cost-side margin, if confirmed, would be a durable hardware-level effect. The main weakness is that the exact decomposition used to attribute the entire saving to the draft phase is undermined by a violation of the 't_verify identical' assumption in one model, and the headline cells rest on single runs.

major comments (3)
  1. [App. B, Table 7; Eq. (1)] The claim that t_verify is identical across native and windowed is contradicted by the Nemotron row of Table 7: tovh_verify ('fixed speculative bookkeeping') is 5.8 ms native vs 4.6 ms windowed, a 1.2 ms drop inside t_verify, in the same cell where acceptance length drops from 3.75 to 3.61. The paper asserts these terms are acceptance-independent, but the observed drop tracks the acceptance change, not the window. Because the intercept drop Δc in Table 6 is attributed entirely to removal of the draft's O(S) index build (Δc = Δt_ctx_draft), and the headline saving Δt_draft in Table 1 includes that intercept drop, part of the reported +28.3% Nemotron margin may be an acceptance/bookkeeping artifact rather than a reduction in the draft's full-context KV read. Please trace the specific kernels inside tovh_verify that change, report them as a function of accepted length, and recompute the cos
  2. [§6, Tables 1/2/13; App. A 'Statistical protocol'] The headline percentages — +28% to +44% per-step cost reduction, +4% to +53% end-to-end win, and the 'never regresses' claim — are supported by a single seeded run per cell. The paper does report within-run aggregation over many decode steps, but the central cross-cell claims (e.g., q35 NIAH-mv win/nat of 1.04×, q122 at 2K in Table 10 net 0.97×) are small enough that single runs cannot distinguish a real effect from run-to-run noise. Given the strength of the 'input-invariant' and 'never regresses' language, please provide repeat runs (at least 3) or bootstrap confidence intervals for Tables 1, 2, and 13, and for the best-depth Table 3 where margins are as small as 1.22×. This is a load-bearing point for the quantitative claims, not just a presentation issue.
  3. [§4, §1 contribution 2] The paper claims an 'all-cell greedy output diff showing no divergence beyond the verifier's pre-existing bf16 non-determinism' as an empirical confirmation of losslessness, but no diff results are reported anywhere in the paper or appendices: there is no table of token-level agreement counts, no false-positive rate, and no comparison of native-vs-dense divergence against windowed-vs-native divergence. The distributional losslessness is a theorem and does not need this support, but the empirical claim of 'no added divergence' is unsupported as written. Please add a table with per-cell divergence counts (number of differing tokens, number of cells) and, if possible, a direct comparison of native-vs-dense versus windowed-vs-native divergence to substantiate the 'same verify-noise envelope' statement.
minor comments (4)
  1. [Eq. (3) and Table 1] Eq. (3) defines win/native = (step_native/step_window)·(AL_window/AL_native), but Table 1 labels its last column 'win/nat' as the matched-acceptance ratio. Please clarify in the caption that Table 1 reports the acceptance-cancelled cost-side ratio and Table 2 reports the full end-to-end ratio including acceptance changes.
  2. [App. B, Table 6] The fitted slopes and intercepts in Table 6 are reported with R² but without standard errors. Given the attribution hinges on Δc ≈ Δt_ctx_draft, reporting standard errors or confidence intervals would strengthen the decomposition and would also make the 'input-invariant' claim more falsifiable.
  3. [§6, Table 13] Table 13 uses abbreviations (NIAH-s/mv/mq, VT, CWE, FWE, Code-QA) without expansion in the caption. The reader can infer from the text, but expanding them in the caption would improve readability.
  4. [App. A, pseudocode] The pseudocode for the windowing change is helpful, but the ring-buffer slot mapping is only described in a comment ('slot = ring_base(r) + logical_pos'). A one-sentence explanation of how the ring wraps and how the +d slack is used would make the implementation claim easier to verify.

Circularity Check

0 steps flagged

No significant circularity: the central results are direct measurements, and the losslessness claim is a standard property of speculative decoding rather than an output fed back into the method.

full rationale

The paper's central claims rest on direct measurements of per-decode-step wall time, acceptance length, and end-to-end latency, with native and windowed draft arms compared under the same engine, kernels, and inputs. The 'lossless by construction' argument is the textbook rejection-sampling property of speculative decoding: the target verifies every accepted token, so changing the draft distribution changes only acceptance length, not the target distribution. This is not circular; it is the definition of the method. The per-step decomposition T_iter = gamma*t_fwd_draft + c is a descriptive least-squares fit to measured iteration times, used to attribute the observed speedup into a once-per-step index-build term and a per-forward attention term. It is not a prediction derived from the claimed effect, and the headline win/native ratios are directly measured step times, not values recovered from the fit. The one concern that might be raised—the nsys phase table showing a 1.2 ms drop in Nemotron's tovh_verify term despite the paper asserting t_verify is identical across arms—is a possible attribution or measurement error, not circularity: it could overstate the share of the saving credited to the draft phase, but it does not make the result equivalent to its inputs. The paper also discloses implementation-dependent components and conservative lower bounds, further indicating the derivation is not self-referential. No load-bearing self-citation or imported uniqueness theorem appears; cited prior work supplies standard baselines and techniques, and the SGLang contributor reference is not used to justify the paper's central claims. Therefore no circular step meets the required standard of a quoted reduction to the paper's own inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new physical or architectural entities are postulated. The method is a system configuration (window + sink + ring buffer) built on existing mechanisms; the main free choices are the window size and the per-model descriptive latency-fit parameters.

free parameters (2)
  • draft window size W and sink size nsink = W=4032, nsink=64
    Hand-chosen operating point from a one-input dose-response sweep (§5, Table 10); not auto-tuned, and a few off-point windows dip net-negative for Qwen-122B/Nemotron. The method depends on this configuration for the claimed gains.
  • latency-fit slope/intercept per model = slope 1.19→0.71 / 1.16→0.85 / 1.45→1.13; intercepts 19.3→14.2 / 27.6→21.5 / 24.6→19.1 ms
    Least-squares fit of T_iter = γ·t_fwd + c (App. B, Table 6) used to attribute the per-step saving to per-forward attention vs. the index build. Descriptive fit to the same measured data, not a prediction.
axioms (5)
  • standard math Speculative decoding's target-verification rule exactly preserves the target output distribution when the draft proposal distribution changes.
    Used in §4 'Losslessness' to assert that windowing the draft only changes which tokens are proposed, never which are accepted.
  • domain assumption Forward kernel time is content-independent at fixed context (acceptance-independent), so the per-step cost margin is input-invariant.
    Invoked in §3 and Table 1 to label the cost-side margin input-invariant; the paper measures content spread ≤7% but does not prove universality.
  • domain assumption Per-decode-step time decomposes additively into t_verify and t_draft, with t_verify identical across native and windowed modes.
    Eq. 1 and App. B, Tables 6-8; the entire attribution of the measured saving to the draft phase depends on this decomposition.
  • domain assumption The reduced paged-KV block table is exactly the draft's attention key set, with no hidden full-context reads or score changes from RoPE.
    Section 4 mechanism; supported by allocation logs and the implementation, but not formally verified.
  • domain assumption The tested built-in MTP/NEXTN draft heads run full softmax/GQA attention over the entire KV cache at every draft step.
    Central to the existence of the long-context draft tax; the paper states this for the three models but acknowledges it is not universal across all architectures.

pith-pipeline@v1.3.0-alltime-deepseek · 23245 in / 17735 out tokens · 164550 ms · 2026-08-01T07:08:15.721930+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context." pith.science (2026). https://pith.science/paper/SDO3IU5Z

@misc{pith2026260721535,
  author       = {Pith},
  title        = {Pith review of: Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SDO3IU5Z}},
  note         = {Machine review of arXiv:2607.21535}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million-token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost -- precisely where speculation is most valuable. The effect compounds with draft length (a deep native draft can turn net-negative, slower than no speculation) and sharpens under hybrid/linear-attention targets, where cheaper verification leaves the draft's full-attention read exposed. We apply a StreamingLLM-style sliding window plus attention sink to the draft's attention only (Windowed-MTP), leaving full-attention verification intact. It is training-free, drop-in, and lossless by construction: the full-attention target still decides every accepted token, so windowing changes only which tokens are proposed, never which are accepted. It bounds the draft's KV working set to a constant, dropping ~99% of KV entries at 1M. Across three architecture families (Qwen GDN-MoE 35B/122B and a Mamba2-hybrid NoPE 120B) at 1M context on a single GPU in SGLang, windowing cuts the per-decode-step cost over the shipping native MTP draft by +28% to +44%, an input-invariant margin that widens with context. Since per-token latency is this cost divided by acceptance length, at matched acceptance end-to-end decode latency improves by the same amount, and more where windowing also lifts acceptance, while preserving the target's verified output distribution. Finally, the unread draft KV -- 7.7-11% of total KV at 1M -- is reclaimed via a compact ring buffer at no acceptance or quality cost.

Figures

Figures reproduced from arXiv: 2607.21535 by Alagappan Valliappan.

Figure 1
Figure 1. Figure 1: Draft attention before/after windowing. The [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-position conditional acceptance αj (d=7, γ=6, 1M) on the retrieval hero input (niah_multiquery_enum), native (full-1M) vs. windowed (4032+64), for the three models (columns). The windowed profile tracks native within the 95% Wilson intervals at essentially every position: α1 is unchanged and the deep positions decay together, so windowing preserves the acceptance shape rather than collapsing it. Net AL… view at source ↗
Figure 3
Figure 3. Figure 3: The windowing wedge widens with context. Input-invariant, matched-acceptance cost-side ratio stepnat/stepwin−1 (Titer=AL · TPOT, so acceptance cancels) vs. context length S, at d=7, B=1, bf16, on niah_multiquery_enum. All three savings grow toward 1M; the 1M endpoints match [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Throughput–latency Pareto at 1M (B sweep over d ∈ {3, 5, 7}; single B200, TP1). Rows: Qwen3.6-35B, Qwen3.5-122B, Nemotron-3-120B; columns: niah_multiquery_enum and fwe (up￾and-right is better). Windowed-MTP+ring (blue) holds the frontier over native MTP (red) and Dense (grey) in five of six panels; the exception is Nemotron+FWE (see text). The ring also fits more resident 1M requests (native MTP OOMs a bat… view at source ↗
Figure 5
Figure 5. Figure 5: TP2 batch-concurrency Pareto frontier (2×B200, 1M, d=7, niah_multiquery_enum): per-user decode speed (x) vs. per-physical-GPU throughput (y), sweeping batch B (non-dominated points only). Windowed-MTP (blue) vs. native MTP (red) vs. Dense (grey). Cross-hardware generality (H100). The win is not specific to Blackwell memory bandwidth. We reproduce the single-GPU result on one NVIDIA H100-80GB using the publ… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 13 linked inside Pith

  1. [1]

    LongBench v2 : Towards deeper understanding and reasoning on realistic long-context multitasks

    Yushi Bai et al. LongBench v2 : Towards deeper understanding and reasoning on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Medusa: Simple LLM inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning (ICML), 2024

    2024

  3. [3]

    SpecExtend : A drop-in enhancement for speculative decoding of long sequences

    Jungyoub Cha, Hyunjong Kim, and Sungzoon Cho. SpecExtend : A drop-in enhancement for speculative decoding of long sequences. In Findings of the Association for Computational Linguistics (ACL Findings), 2026. arXiv:2505.20776

    arXiv 2026

  4. [5]

    MagicDec : Breaking the latency-throughput tradeoff for long context generation with speculative decoding

    Jian Chen, Vashisth Tiwari, Ranajoy Sadhukhan, Zhuoming Chen, Jinyuan Shi, Ian En-Hsu Yen, and Beidi Chen. MagicDec : Breaking the latency-throughput tradeoff for long context generation with speculative decoding. arXiv preprint arXiv:2408.11049, 2024

    Pith/arXiv arXiv 2024

  5. [6]

    DFlash : Block diffusion for flash speculative decoding, 2026

    Jian Chen, Yesheng Liang, and Zhijian Liu. DFlash : Block diffusion for flash speculative decoding, 2026. arXiv:2602.06036

    Pith/arXiv arXiv 2026

  6. [8]

    Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024

    2024

  7. [10]

    DeepSpec / DSpark : Speculative decoding for DeepSeek models, 2025

    DeepSeek-AI . DeepSpec / DSpark : Speculative decoding for DeepSeek models, 2025. https://github.com/deepseek-ai/DeepSpec

    2025

  8. [13]

    RULER : What's the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER : What's the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024

    2024

  9. [14]

    BABILong : Testing the limits of LLM s with long context reasoning-in-a-haystack

    Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong : Testing the limits of LLM s with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  10. [15]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention . Symposium on Operating Systems Principles (SOSP), 2023

    2023

  11. [16]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML), 2023

    2023

  12. [17]

    EAGLE : Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE : Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning (ICML), 2024 a

    2024

  13. [18]

    EAGLE-2 : Faster inference of language models with dynamic draft trees

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2 : Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858, 2024 b

    Pith/arXiv arXiv 2024

  14. [19]

    EAGLE-3 : Scaling up inference acceleration of large language models via training-time test

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-3 : Scaling up inference acceleration of large language models via training-time test. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2025

    2025

  15. [20]

    YaRN : Efficient context window extension of large language models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN : Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023

    Pith/arXiv arXiv 2023

  16. [21]

    Qwen3 technical report, 2025

    Qwen Team . Qwen3 technical report, 2025. arXiv preprint arXiv:2505.09388

    Pith/arXiv arXiv 2025

  17. [22]

    SGLang : DFLASH speculative decoding support ( --speculative-draft-window-size )

    SGLang contributors . SGLang : DFLASH speculative decoding support ( --speculative-draft-window-size ). https://github.com/sgl-project/sglang/pull/22077, 2026. Merged 2026-04-07; draft-KV sliding window for DFLASH and EAGLE-3

    2026

  18. [23]

    RoFormer : Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer : Enhanced transformer with rotary position embedding. Neurocomputing, 568, 2024

    2024

  19. [24]

    Quest : Query-aware sparsity for efficient long-context LLM inference

    Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han. Quest : Query-aware sparsity for efficient long-context LLM inference. In International Conference on Machine Learning (ICML), 2024

    2024

  20. [25]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations (ICLR), 2024

    2024

  21. [26]

    LongSpec : Long-context lossless speculative decoding with efficient drafting and verification

    Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, and Bo An. LongSpec : Long-context lossless speculative decoding with efficient drafting and verification. In Annual Meeting of the Association for Computational Linguistics (ACL), 2026. arXiv:2502.17421

    Pith/arXiv arXiv 2026

  22. [27]

    SGLang : Efficient execution of structured language model programs

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, et al. SGLang : Efficient execution of structured language model programs. Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  23. [28]

    International Conference on Machine Learning (ICML) , year=

    Fast inference from transformers via speculative decoding , author=. International Conference on Machine Learning (ICML) , year=

  24. [29]

    arXiv preprint arXiv:2302.01318 , year=

    Accelerating large language model decoding with speculative sampling , author=. arXiv preprint arXiv:2302.01318 , year=

    Pith/arXiv arXiv

  25. [30]

    Medusa: Simple

    Cai, Tianle and Li, Yuhong and Geng, Zhengyang and Peng, Hongwu and Lee, Jason D and Chen, Deming and Dao, Tri , booktitle=. Medusa: Simple

  26. [31]

    Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang , booktitle=

  27. [32]

    Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang , journal=

  28. [33]

    Chen, Jian and Liang, Yesheng and Liu, Zhijian , year=

  29. [34]

    arXiv preprint arXiv:2412.19437 , year=

    Pith/arXiv arXiv

  30. [35]

    International Conference on Learning Representations (ICLR) , year=

    Efficient streaming language models with attention sinks , author=. International Conference on Learning Representations (ICLR) , year=

  31. [36]

    2026 , note=

    Yang, Penghui and Du, Cunxiao and Zhang, Fengzhuo and Wang, Haonan and Pang, Tianyu and Du, Chao and An, Bo , booktitle=. 2026 , note=

    2026

  32. [37]

    2026 , note=

    Cha, Jungyoub and Kim, Hyunjong and Cho, Sungzoon , booktitle=. 2026 , note=

    2026

  33. [38]

    arXiv preprint arXiv:2605.09992 , year=

    Attention Drift: What Autoregressive Speculative Decoding Models Learn , author=. arXiv preprint arXiv:2605.09992 , year=

    Pith/arXiv arXiv

  34. [39]

    Peng, Bowen and Quesnelle, Jeffrey and Fan, Honglu and Shippole, Enrico , journal=

  35. [40]

    Su, Jianlin and Ahmed, Murtadha and Lu, Yu and Pan, Shengfeng and Bo, Wen and Liu, Yunfeng , journal=

  36. [41]

    arXiv preprint arXiv:2306.15595 , year=

    Extending context window of large language models via positional interpolation , author=. arXiv preprint arXiv:2306.15595 , year=

    Pith/arXiv arXiv

  37. [42]

    arXiv preprint arXiv:2312.00752 , year=

    Mamba: Linear-time sequence modeling with selective state spaces , author=. arXiv preprint arXiv:2312.00752 , year=

    Pith/arXiv arXiv

  38. [43]

    Transformers are

    Dao, Tri and Gu, Albert , booktitle=. Transformers are

  39. [44]

    Hsieh, Cheng-Ping and Sun, Simeng and Kriman, Samuel and Acharya, Shantanu and Rekesh, Dima and Jia, Fei and Ginsburg, Boris , booktitle=

  40. [45]

    arXiv preprint arXiv:1911.05507 , year=

    Compressive transformers for long-range sequence modelling , author=. arXiv preprint arXiv:1911.05507 , year=

    Pith/arXiv arXiv 1911

  41. [46]

    Kuratov, Yuri and Bulatov, Aydar and Anokhin, Petr and Rodkin, Ivan and Sorokin, Dmitry and Sorokin, Artyom and Burtsev, Mikhail , journal=

  42. [47]

    Bai, Yushi and others , journal=

  43. [48]

    Chen, Jian and Tiwari, Vashisth and Sadhukhan, Ranajoy and Chen, Zhuoming and Shi, Jinyuan and Yen, Ian En-Hsu and Chen, Beidi , journal=

  44. [49]

    2025 , note=

    Qwen3 technical report , author=. 2025 , note=

    2025

  45. [50]

    Efficient memory management for large language model serving with

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E and Zhang, Hao and Stoica, Ion , journal=. Efficient memory management for large language model serving with

  46. [51]

    Zheng, Lianmin and Yin, Liangsheng and Xie, Zhiqiang and Sun, Chuyue and others , journal=

  47. [52]

    2026 , howpublished=

    2026

  48. [53]

    Tang, Jiaming and Zhao, Yilong and Zhu, Kan and Xiao, Guangxuan and Kasikci, Baris and Han, Song , booktitle=