Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

SpecMemo: Speculative Decoding is in Your Pocket

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A memory-aware inference engine keeps 96% of speculative decoding speed while cutting generation-time buffer memory by 65%.

desk verdict A useful systems idea with thin evidence: memory budgeting for speculative decoding is worth exploring, but the 65%/96% claims are not yet demonstrated. read the letter →

arxiv 2506.01986 v1 pith:WY4QHQCP submitted 2025-05-16 cs.LG cs.AIcs.DC

Selin Yildirim , Deming Chen This is my paper · ORCID
classification cs.LGcs.AIcs.DC
keywords speculativedecodingmemory-constrainedinferenceKVcacheallocationtree-basedattentionmasksmulti-turnchatbotsdistributedbatcheddevice-awarememorybudgeting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding speeds up text generation by drafting several candidate tokens per step and verifying them together, but the extra buffers it needs exceed what many consumer GPUs have. This paper claims much of that memory is wasted, since candidate branches in the draft tree are highly similar before verification and deeper branches can be pruned in advance without losing accepted tokens. The proposed engine, SpecMemo, models total memory as the sum of base model, parallel decoding heads, the stored key-value (KV) attention cache, and runtime buffers, then picks a pruned tree mask, head count, and quantization level to fit a target device before generation starts. On a 24 GB GPU it reports a 65% reduction in generation-time buffer memory over a 20-query multi-turn chat while keeping 96% of speculative throughput. On eight smaller server GPUs it distributes a 70B-parameter model and runs batched speculative decoding, achieving roughly double the throughput of distributed vanilla decoding.

What carries the argument

The load-bearing object is the tree-based attention mask, which fixes how many candidate tokens are speculated in one step and how they branch; its cost is $N = \sum_{i=0}^{l} k^i$ nodes and $S$ leaf sequences. SpecMemo's memory model prices the runtime buffers as $bNw + bSlw + bSl^2w$, where $b$ is batch size, $l$ is the number of decoding heads, and $w$ is vocabulary size, and combines that with the base model, the heads, and the KV cache. Two mechanisms carry the argument: a device-aware optimizer that pre-computes the minimum KV cache for a target number of chatbot queries and then prunes the mask with the scaled logistic function $y = \mathrm{clip}(1 - e^{-ax}(1-bx), 0, 1)$, which removes most nodes from deeper levels; and a batched verification scheme that pads accepted tokens to uniform tensor shapes, remaps position IDs so padding is ignored, and sets attention to $-\infty$ for cached padding positions.

What would settle it

Run the same pruning policy on several diverse multi-turn dialogue datasets and measure per-step acceptance length and throughput against the full speculative tree; if throughput retention drops well below 96% or the accepted-branch distribution shifts, the redundancy premise fails. Independently, recompute the KV-cache lower bound with the per-head dimension that the paper's Eq. 1 drops; if the modeled budget underestimates observed memory on a target GPU, the memory guarantee is not reliable.

Watch

Extended reading notes

Core claim

SpecMemo's central claim is that the memory footprint of parallel-head speculative decoding has a simple four-part structure—base model, decoding heads, KV cache, and runtime buffers—and that the buffer term, which is often the bottleneck on small GPUs, can be shrunk without sacrificing acceptance. The evidence for shrinkability is empirical: pre-verification embeddings of the candidate branches in a speculative tree have high pairwise cosine similarity, so a static tree that keeps first-level diversity but prunes deeper levels retains most of the accepted sequence length. Using a 44-node pruned mask instead of the default 64-node mask cuts per-query buffer allocation from 55 MB to 19.5 MB, which accumulates to 390 MB versus 1.1 GB over the same 20-query benchmark conversation, while 96% of the original throughput remains. For the distributed setting, SpecMemo splits a 70B-parameter model across eight GPUs, shares the decoding heads and KV cache across batches, and reports a 2x speedup over batched vanilla decoding, with throughput rising from 5.7 to 46.0 tokens per second as batch size grows from 1 to 10.

Load-bearing premise

The argument collapses if the candidate branches of a speculative tree are not redundant enough in real multi-turn conversations: the static pruned mask that preserves 96% throughput on the evaluation benchmark could prune away branches that matter elsewhere.

Editorial extensions

If this is right

  • Multi-turn chatbots with speculative decoding can run on GPUs with 8–16 GB of memory, where the default configuration would otherwise run out of memory.
  • A fixed memory budget serves more user turns: the measured 390 MB versus 1.1 GB buffer gap corresponds to nearly three times the conversational length from the same device.
  • Large models that do not fit on a single GPU can be served from a cluster of small GPUs with speculation, roughly doubling throughput over vanilla distributed decoding.
  • Automatic pre-generation selection of head count, KV-cache size, and mask shape removes manual tuning and prevents runtime out-of-memory failures.
  • The budgeting procedure is plug-and-play for other parallel-head speculative decoding methods, extending the memory savings beyond the demonstrated system.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 65% figure describes generation-time buffer memory only; total device memory savings will be smaller when the base-model weights dominate, so the headline reduction should not be extrapolated to total footprint.
  • If branch redundancy is context-dependent, static pruning could be combined with per-step branch scoring to adapt the mask to actual acceptance statistics, potentially retaining more speed at even smaller memory cost.
  • The batching scheme's padding introduces wasted computation that grows with acceptance-length variance, so continuous batching or variable-length batches could push the throughput curve further; the paper lists continuous batching as future work.
  • A direct check of cosine-similarity redundancy on longer, multi-domain conversations would show whether the static mask is a general property of speculative trees or an artifact of the particular benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents SpecMemo, a device-aware inference engine that reduces the memory footprint of speculative decoding on memory-constrained GPUs. It formulates KV-cache and buffer memory models, prunes tree-based attention masks on the basis of candidate-branch redundancy, and automatically adjusts decoding heads, model precision, and mask size to fit a given device budget. The authors report that on a single Nvidia Titan RTX, SpecMemo reduces buffer memory by 65% (from about 1.1 GB to 390 MB across 20 MT-Bench queries) while retaining 96% of speculative decoding throughput. They also propose a distributed and batched speculative decoding scheme for Llama-2-70B-Chat on eight AMD MI250 GPUs, claiming a 2x speedup over distributed vanilla decoding and an 8x throughput improvement at batch size 10.

Significance. The problem addressed—deploying speculative decoding on consumer-grade GPUs with scarce memory—is practically important, and the paper's core idea of pruning candidate trees based on empirically observed branch redundancy is plausible and complementary to existing KV-cache compression work. The distributed batched speculative decoding extension for small-GPU clusters is a useful novelty that goes beyond draft-model-based batching studies. If the measurements are reproducible, the system could make speculative decoding more accessible on edge devices. However, the theoretical memory model contains clear errors, the headline 65% reduction applies only to a small buffer component rather than total generation memory, and the throughput-retention claim rests on a single post-hoc selected mask without variance estimates or hold-out validation. These issues currently limit the strength of the paper's central claims.

major comments (5)
  1. [Section 3.1, Eq. (1)] Equation (1) defines Memory_KV = 2*h*b*k*x*p, but the sentence below it defines d as the hidden dimension per attention head, and the standard KV cache size must include a factor of d (each KV entry is a vector of size d). As written, the predicted KV cache size is too small by a factor of d (e.g., 128 for Vicuna-7B), which would invalidate the memory-budget decisions made by Algorithm 1 and the 'theoretical guide' in Figure 3. Please correct the formula or explicitly explain how d is absorbed into another variable.
  2. [Section 3.1, Eq. (4)] Equation (4) asserts Memory_heads = 0.6GB * l, claiming that the size of Medusa heads is constant. This is not defended; the size of a decoding head depends on the base model's hidden dimension and the head architecture, and 0.6 GB per head appears far too large for a 7B-class model. Because this term enters the total-memory computation in Equation (6) and influences the quantization and pruning choices in Algorithm 1, please provide measured head sizes for the models actually used and remove the model-independence claim.
  3. [Abstract and Section 4 (Figure 11)] The abstract and contributions claim 'reduced generation-memory by 65%', but the evidence in Figure 11 is a comparison of buffer allocation only (55 MB vs 19.5 MB per query, and 1.1 GB vs 390 MB across a 20-query conversation). Buffer memory is a small fraction of total generation memory, which also includes model weights, Medusa heads, and KV cache. Please report the total memory footprint before and after SpecMemo and phrase the claim as buffer-memory reduction unless the total generation memory is indeed reduced by 65%.
  4. [Section 4 (Figure 12)] The 96% throughput-retention figure is reported for the mask 1-10-16-17, which was selected after exploring 15 masks in Figures 11 and 12. No error bars, random seeds, or train/test separation are described, so this number is a post-hoc best-case result rather than a predictive validation. Please provide repeated runs with variance estimates, and either evaluate on a hold-out benchmark or specify an a priori mask-selection criterion that does not use the same benchmark on which the claim is reported.
  5. [Section 3.2 and Appendix A.1.1] The pruning strategy is motivated by branch-redundancy evidence from a single story-generation task with Vicuna-7B (Figures 2 and 14). The paper's target workload is multi-turn chatbots (MT-Bench), and it is plausible that open-domain dialogue has less branch redundancy than story continuation; if so, pruning could degrade acceptance length and throughput on MT-Bench. Please include branch-redundancy or acceptance-length measurements on MT-Bench, and also report a sensitivity analysis for the hand-picked pruning-curve coefficients a=0.02 and b=0.1 in Figure 5.
minor comments (6)
  1. [Section 5] There is a typo: 'capacilities' should be 'capabilities'.
  2. [Equation (2)] The expression for the attention mask is typeset ambiguously; the union over i and j needs explicit index sets and a clear definition of Node_{i,j}.
  3. [Figure 5] The formula in the caption is missing a minus sign in the exponent; please write it unambiguously, e.g., y = clip(1 - exp(-a x)(1 - b x), 0, 1) if that is the intended form.
  4. [Figure 12] The x-axis lists mask names, but the correspondence to the buffer-size values in Figure 11 is not fully legible; please add a legend or clearer annotation so each mask can be identified.
  5. [Algorithm 1] The loop that reduces the number of heads (lines 21-27) updates new_heads, but the later quantization step uses 'new_precision' without specifying how the precision level is chosen; please clarify the relationship between head count reduction and precision selection.
  6. [References] Reference [18] appears to be MT-Bench-101, while the main text cites MT-Bench as [14]; please ensure the correct benchmark is referenced throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the memory reduction follows arithmetically from the buffer model, and the throughput retention is an empirical measurement, not a quantity derived from the pruning rule.

full rationale

I walked the paper's derivation chain. The memory-budget formulas (Eqs. 1-6) are capacity models: Eq. 3 defines buffer size as a function of node count N, sequence count S, tree depth l, and vocabulary size w, so the reported 65% buffer-memory reduction from using a 44-node mask instead of a 64-node mask follows arithmetically from the formula. The paper does not claim Eq. 3 predicts acceptance rate or throughput. The 96% throughput retention is presented as a measured result on MT-Bench for the specific 1-10-16-17 mask, not as a consequence of the memory model. Although Algorithm 2 selects a configuration after measuring speedup on the same benchmark, that is an in-sample selection-bias or generalization concern, not a definitional equivalence, and it does not make the measured throughput equal to the selection criterion by construction. Medusa [1] is cited as the base speculative-decoding system and its architecture is used, but no load-bearing premise is justified solely by a self-citation; the pruning rationale is supported by the paper's own cosine-similarity and branch-acceptance analyses, however limited. The Limitations section notes context-length and batching restrictions, which are not circularity admissions. Overall, the central claims are self-contained empirical and arithmetic results, so no significant circularity is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's memory budget rests on asserted formulas (Eqs 1-6) rather than a derivation, and the pruning heuristic rests on an empirical cosine-similarity observation with no quantified threshold. The system introduces no new physical entities, but it depends on assumptions about buffer reuse, a model-independent Medusa head size, and layer-wise distribution.

free parameters (3)
  • Pruning curve coefficients a and b = a=0.02, b=0.1
    Hand-picked in the scaled logistic function y = clip(1 - e^{a x (1 - b x)}, 0, 1) that sets per-level pruning rates (Figures 5 and 6); these values determine mask size, memory use, and throughput.
  • Default number of decoding heads = 5, reduced to as low as 2
    Algorithm 1 starts with default_heads=5 and searches downward until memory fits; the chosen head count is a per-device hyperparameter that trades memory against speed.
  • Default tree shape = 64 nodes, 42 leaf sequences
    The Medusa default tree is the initial configuration before pruning; the reported savings (e.g., 44-node mask) are relative to this starting point, so the headline numbers depend on the choice of baseline.
assumptions (5)
  • domain assumption KV cache memory is exactly 2*h*b*k*x*p (Eq 1), with no separate per-head dimension factor
    Section 3.1, Eq 1 omits the per-head dimension d mentioned in the text; SpecMemo's budget calculation depends on this formula being correct.
  • domain assumption Medusa decoding heads consume a constant 0.6 GB per head regardless of base model (Eq 4)
    This constant is near Llama-70B-class head size but is used in experiments with Vicuna-7B, so it may misstate the head memory component of the budget.
  • domain assumption Intermediate buffers are reused across generation steps, so only the largest buffer allocation counts toward peak memory
    Section 3.1 states that buffer space is reused across steps; if reuse is not achievable in the implementation, the buffer reduction numbers overstate the savings.
  • domain assumption Pre-verification cosine similarity among candidate branches is high enough that static pruning preserves acceptance rate
    Appendix A.1.1 and Figure 2 show heatmaps, but the paper gives no quantitative relationship between similarity and acceptance-rate retention.
  • domain assumption Layer-wise distribution of the model across GPUs with per-GPU KV cache slices is an efficient unit of parallelism
    Section 3.3 assumes the communication overhead of layer distribution is outweighed by speculative decoding speedups; no communication model or sensitivity study is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecMemo: Speculative Decoding is in Your Pocket." pith.science (2026). https://pith.science/paper/WY4QHQCP

@misc{pith2026250601986,
  author       = {Pith},
  title        = {Pith review of: SpecMemo: Speculative Decoding is in Your Pocket},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WY4QHQCP}},
  note         = {Machine review of arXiv:2506.01986}
}
read the original abstract

Recent advancements in speculative decoding have demonstrated considerable speedup across a wide array of large language model (LLM) tasks. Speculative decoding inherently relies on sacrificing extra memory allocations to generate several candidate tokens, of which acceptance rate drives the speedup. However, deploying speculative decoding on memory-constrained devices, such as mobile GPUs, remains as a significant challenge in real-world scenarios. In this work, we present a device-aware inference engine named SpecMemo that can smartly control memory allocations at finer levels to enable multi-turn chatbots with speculative decoding on such limited memory devices. Our methodology stems from theoretically modeling memory footprint of speculative decoding to determine a lower bound on the required memory budget while retaining speedup. SpecMemo empirically acquires a careful balance between minimizing redundant memory allocations for rejected candidate tokens and maintaining competitive performance gains from speculation. Notably, with SpecMemo's memory management, we maintain 96% of overall throughput from speculative decoding on MT-Bench, with reduced generation-memory by 65% on single Nvidia Titan RTX. Given multiple constrained GPUs, we build on top of previous speculative decoding architectures to facilitate big-model inference by distributing Llama-2-70B-Chat model, on which we provide novel batched speculative decoding to increase usability of multiple small server GPUs. This novel framework demonstrates 2x speedup over distributed and batched vanilla decoding with the base model on eight AMD MI250 GPUs. Moreover, inference throughput increases remarkably 8x with batch size 10. Our work contributes to democratized LLM applications in resource-constrained environments, providing a pathway for faster and cheaper deployment of real-world LLM applications with robust performance.

Figures

Figures reproduced from arXiv: 2506.01986 by the authors.

Figure 1
Figure 1. SpecMemo Inference Engine Architecture usage, thus mostly targeting data-center GPUs. Deployment of such solution on consumer-grade hardware, such as GPUs integrated into laptops and edge devices, remains a significant challenge when not addressed carefully, due to the substantial memory demand at inference-time. This aspect make existing speculative decoding methods impractical for the use in real-world application… view at source ↗
Figure 2
Figure 2. Pre-verification cosine similarity of 42 candidate sequences present in original Medusa[ [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Memory breakdown by the theoretical guide provided in Equation 6. Attention mask choice of size 64 causes OOM in FP16, whereas the smaller mask can succeed. 0 200 400 600 800 1,000 1,200 1,400 1,600 1,800 16 18 20 22 24 1 GB 5 GB 7 GB # of Tokens Memory Allocation (GB) Pre-Allocating KV Cache 1 Branch Quarter Tree Half Tree Original Tree [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Tree pruning function for a growing full attention tree with arity k = 10. 1 2 3 4 Decoding Heads 0.0 0.2 0.4 0.6 0.8 1.0 Pruning Rate 0.0000 0.9274 0.9809 0.9951 y = clip(1 e ax (1 bx), 0, 1), a=0.02 ,b=0.1 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Distributed LLama-2-70B[8, 19] on 8 AMD MI250 GPUs[11], where base model and decoding heads are shared by multiple batches. KV cache is also distributed for model layer splits. Custom tree build from tree features: SpecMemo derives budget-friendly and computationally f…
Figure 9
Figure 9. Figure 9: A toy example of batched Medusa[1], where attention masks filter pad tokens in the cache. Iteration verification determines whether to discard or keep tokens. Batches have varying length of acceptance; 2,4,1, and 2, respectively. Uniform tensor shape is obtained via pa…
Figure 10
Figure 10. Figure 10: Token latency exploration of Vicuna￾7B[15] while varying inference hyper-parameters and mask size on story telling benchmark. Us￾ing 3 heads with mask of size 44 gives the best latency. 1-10-10-10-10 1_10_23_9 1_10_19_14 1_10_16_17 1_10_20_13 1_10_23_10 1_10_21_12 1_1…
Figure 11
Figure 11. Figure 11: Explored tree-based attention masks with their per-level node counts given in mask labels, i.e, 1-10-19-14 nodes. Annotated numbers on query-level allocation show the average accep￾tance length (τ ) of corresponding masks. (41, 34) (43, 27) (44, 26) (44, 23) (44, 26) …
Figure 13
Figure 13. Figure 13: Comparison of SpecMemo against memory scaling ratios ranging [PITH_FULL_IMAGE:figures/full_fig_p009_13.png]
Figure 14
Figure 14. Figure 14: Distribution of selected branches and their contribution to total generation length over [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding

    cs.AR 2026-08 conditional novelty 6.0 of 10

    A co-designed edge accelerator that avoids loading redundant expert weights when speculative decoding meets mixture-of-experts, cutting latency by up to 56.3% and energy by up to 44.1% in simulation.

Reference graph

Works this paper leans on

36 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Lee, Deming Chen, Tri Dao,Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, 2024,https://arxiv

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, Tri Dao,Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads, 2024,https://arxiv. org/abs/2401.10774

    arXiv 2024

  2. [2]

    Haifeng Qian, Sujan Kumar Gonugondla, Sungsoo Ha, Mingyue Shang, Sanjay Krishna Gouda, Ramesh Nallapati, Sudipta Sengupta, Xiaofei Ma, Anoop Deoras,BASS: Batched Attention-optimized Speculative Sampling, 2024,https://arxiv.org/abs/2404.15778

    arXiv 2024

  3. [3]

    Qidong Su, Christina Giannoula, Gennady Pekhimenko,The Synergy of Speculative Decoding and Batching in Serving Large Language Models, 2023,https://arxiv.org/abs/2310.18813

    arXiv 2023

  4. [4]

    Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang,EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, 2024,https://arxiv.org/abs/2406.16858

    arXiv 2024

  5. [5]

    Yuhui Li, Fangyun Wei, Chao Zhang, Hongyang Zhang,EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty, 2025,https://arxiv.org/abs/2401.15077

    arXiv 2025

  6. [6]

    NVIDIA Corporation,NVIDIA A100 Tensor Core GPU Architecture, https://www.nvidia.com/ en-us/data-center/a100/

    work page

  7. [7]

    NVIDIA Corporation,NVIDIA L40 GPU, https://www.nvidia.com/content/dam/en-zz/ Solutions/Data-Center/datasheets/L-40/product-brief-L40.pdf

    work page

  8. [8]

    Huggingface,Meta Llama-2-70B-Chat-Hf, https://huggingface.co/meta-llama/ Llama-2-70b-chat-hf

    work page

Show all 36 references
  1. [9]

    NVIDIA Corporation,NVIDIA GeForce RTX 4090 GPU Architecture, https://www.nvidia.com/ en-us/geforce/graphics-cards/40-series/rtx-4090/

  2. [10]

    NVIDIA Corporation,NVIDIA Titan RTX GPU Architecture, https://www.nvidia.com/en-us/ design-visualization/rtx-6000/

  3. [11]

    Advanced Micro Devices,AMD MI250 GPU Architecture, https://www.amd.com/en/products/ accelerators/instinct/mi200/mi250.html

  4. [12]

    Advanced Micro Devices,AMD AI X1 Pro,https://www.minisforum.com/pages/ai-x1-pro

  5. [13]

    Advanced Micro Devices,AMD Radeon GPU, https://www.amd.com/en/products/graphics/ desktops/radeon.html

  6. [14]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica,Judging LLM-as-a- Judge with MT-Bench and Chatbot Arena, 2023,https://arxiv.org/abs/2306.05685

    2023 arXiv

  7. [15]

    LMSys ORG,Vicuna 7B v1.3,https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  8. [16]

    Microsoft,Windows laptops,https://www.microsoft.com/en-us/windows?r=1

  9. [17]

    Rush, Thomas Wolf,Zephyr: Direct Distillation of LM Alignment, 2023, https://arxiv.org/abs/2310.16944

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf,Zephyr: Direct Distillation of LM Alignment, 202...

    2023 arXiv

  10. [18]

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, Wanli Ouyang,MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues, Proceedings of the 62nd Annual Meeting of the A...

    2024 doi

  11. [19]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  12. [20]

    Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, Beidi Chen, Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding, 2024, https://arxiv.org/ abs/2402.12374. 10

    2024 arXiv

  13. [21]

    Hang Wu, Jianian Zhu, Yinghui Li, Haojie Wang, Biao Hou, and Jidong Zhai,SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models, 2025, arXiv:2505.07680, https://arxiv.org/abs/2505.07680

    2025 arXiv

  14. [22]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia,SpecInfer: Accelerating Large Language Model Serving with Tre...

    2024

  15. [23]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper, Accelerating Large Language Model Decoding with Speculative Sampling, 2023, arXiv:2302.01318, https://arxiv.org/abs/2302.01318

    2023 arXiv

  16. [24]

    Ming Yin, Minshuo Chen, Kaixuan Huang, and Mengdi Wang,A Theoretical Perspective for Speculative Decoding Algorithm, 2024, arXiv:2411.00841,https://arxiv.org/abs/2411.00841

    2024 arXiv

  17. [25]

    Ziteng Sun, Uri Mendlovic, Yaniv Leviathan, Asaf Aharoni, Jae Hun Ro, Ahmad Beirami, and Ananda Theertha Suresh,Block Verification Accelerates Speculative Decoding, 2025, arXiv:2403.10444, https: //arxiv.org/abs/2403.10444

    2025 arXiv

  18. [26]

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, Beidi Chen, H 2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models,arXiv preprint arXiv:2306....

    2023 arXiv

  19. [27]

    Zheng Wang, Boxiao Jin, Zhongzhi Yu, Minjia Zhang, Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks,arXiv preprint arXiv:2407.08454, 2024, https: //arxiv.org/abs/2407.08454

    2024 arXiv

  20. [28]

    Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need,arXiv preprint arXiv:1706.03762, 2023, https: //arxiv.org/abs/1706.03762

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need,arXiv preprint arXiv:1706.03762, 2023, https: //arxiv.org/abs/1706.03762

    2023 arXiv

  21. [29]

    Zili Wang, Robert Zhang, Kun Ding, Qi Yang, Fei Li, Shiming Xiang, Continuous Speculative Decoding for Autoregressive Image Generation,arXiv preprint arXiv:2411.11925, 2024, https://arxiv.org/ abs/2411.11925

    2024 arXiv

  22. [30]

    Doohyuk Jang, Sihwan Park, June Yong Yang, Yeonsung Jung, Jihun Yun, Souvik Kundu, Sung-Yub Kim, Eunho Yang, LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding, InThe Thirteenth International Conference on Learning Representations, 2025, https...

    2025

  23. [31]

    Fu, Christopher Ré, Azalia Mirhoseini, Hydragen: High-Throughput LLM Inference with Shared Prefixes,arXiv preprint arXiv:2402.05099, 2024, https: //arxiv.org/abs/2402.05099

    Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y . Fu, Christopher Ré, Azalia Mirhoseini, Hydragen: High-Throughput LLM Inference with Shared Prefixes,arXiv preprint arXiv:2402.05099, 2024, https: //arxiv.org/abs/2402.05099

    2024 arXiv

  24. [32]

    Penghui Yang, Cunxiao Du, Fengzhuo Zhang, Haonan Wang, Tianyu Pang, Chao Du, Bo An, LongSpec: Long-Context Speculative Decoding with Efficient Drafting and Verification,arXiv preprint arXiv:2502.17421, 2025,https://arxiv.org/abs/2502.17421

    2025 arXiv

  25. [33]

    org/abs/2501.19309

    Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Schönfeld, Ali Thabet, Jonas Kohler, Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment,arXiv preprint arXiv:2501.19309, 2025, https:...

    2025 arXiv

  26. [34]

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang, LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models,arXiv preprint arXiv:2308.16137, 2024, https://arxiv.org/abs/2308.16137

    2024 arXiv

  27. [35]

    Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, Wen Xiao, PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling,arXiv preprint arXiv:2406.02069, 2025, https://arxiv.org/abs/2406. 02069

    2025 arXiv

  28. [36]

    Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, Jianfeng Gao, Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs,arXiv preprint arXiv:2310.01801, 2024, https: //arxiv.org/abs/2310.01801. A Appendix / supplemental material Table 2 presents the f...

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.