Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SwiftSpec claims that asynchronous speculative decoding—draft and target models on separate GPU groups—cuts LLM decoding latency by an average of 1.75x over state-of-the-art systems, reaching 348 tokens/s for Llama-3-70B on 8 Hopper GPUs.

desk verdict Solid systems contribution with real ablations, but the 1.75x headline is measured against an internal baseline, not external SOTA—worth reviewing, needs honest reporting and artifacts. read the letter →

arxiv 2506.11309 v1 pith:BMZWCNLM submitted 2025-06-12 cs.DC cs.LG

Ziyi Zhang , Ziheng Jiang , Chengquan Jiang , Menghan Yu , Size Zheng , Haibin Lin , Henry Hoffmann , Xin Liu This is my paper
classification cs.DCcs.LG
keywords speculativedecodinglowlatencytensorparallelismasynchronousKVcachemanagementkernelfusionLLMservingtree-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SwiftSpec argues that the next big win in low-latency LLM decoding comes from removing the serial dependency between the draft and verify phases of speculative decoding. It does this by putting the draft model and the target model on separate GPU groups that work concurrently, so the small model proposes tokens while the large model checks them, and only the verified tokens are exchanged. On five model families and six datasets it reports an average 1.75x speedup over the best existing systems, and 348 tokens/s for Llama-3-70B on eight Hopper GPUs. If this holds, interactive applications would feel dramatically faster without changing model weights.

What carries the argument

The load-bearing mechanism is the parallel tree generation loop plus the KV-cache invariant. Throughout decoding, the draft model's KV cache stores verified tokens contiguously as a prefix and tree tokens right after it; after each verification the draft tree re-roots at the last verified token and the cache is reorganized to keep only the still-useful subtree. The three latency-optimized kernels (GEMM fused with all-reduce, a mask attention kernel with no explicit barriers, and fused SwiGLU) use NCCL's low-latency store/load protocol to hide communication, and the kernel set is what makes small batch sizes (bs=8) efficient.

What would settle it

Run the same five model families on a held-out set of single-request workloads (for example long code completion and multi-turn chat) with independent hyperparameter sweeps for vLLM, SGLang, and TensorRT-LLM, and compare tokens/s; the central claim would be falsified if the average speedup over the best baseline drops below about 1.1x.

Watch

Extended reading notes

Core claim

On the paper's own terms, SwiftSpec establishes that asynchronous, disaggregated speculative decoding can combine the compression-ratio benefits of tree-based drafting with the latency benefits of tensor parallelism, a combination that prior systems could not achieve. The central discovery is that the draft model does not need to know the verification outcome before proposing the next batch: a maximum-likelihood tree expansion keeps the proposals good enough (compression ratio only about 9% lower than serial in the reported Qwen2-72B case) while the target model's GPU group keeps verifying. The KV cache is the glue: verified tokens are moved into a prefix cache, and the surviving draft-tree states are reorganized contiguously after it so that no useful computation is thrown away.

Load-bearing premise

The reported 1.75x speedup depends on the claim that all baselines were run at their best tuned configurations on the six chosen datasets, while SwiftSpec's own design parameters (batch size, tree depth, GPU split) were selected on the same benchmark data.

Editorial extensions

If this is right

  • For any model family with an independent smaller draft model, decoding latency can be cut roughly in half relative to today's serving frameworks, without retraining or changing target weights.
  • Draft and target models can each be assigned different tensor-parallelism degrees, so small draft models stop wasting GPU resources that they cannot use.
  • Parallel tree generation preserves most of the compression ratio of tree-based speculative decoding (about 91% of serial in the reported Qwen2-72B case) while keeping the draft model off the critical path.
  • The latency-optimized kernels alone account for at least a 16% end-to-end improvement, and the fused attention kernel is 30-56% faster than FlashAttention at the tested context lengths.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An open question the paper leaves implicit is whether the same disaggregation can be adapted to EAGLE-style drafts, whose input depends on the target model's hidden states; a testable extension would relax that dependency by feeding the draft model a predicted or stale hidden state.
  • The method's gains are demonstrated at batch size 8 on a single 8-GPU node; the same techniques could be stress-tested at larger batch sizes or across nodes with slower interconnects, where the paper suggests the communication savings from disaggregation would matter even more.
  • The claim of being the fastest known system at this scale rests on a point-in-time comparison; an easy check is to re-run the same workloads against newer releases of the baseline frameworks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. SwiftSpec proposes a disaggregated, asynchronous speculative decoding system for single-request LLM decoding. It partitions GPUs into draft and target groups, allows the draft model to expand a tree of candidate tokens while the target model verifies the previous tree, and reorganizes the draft KV cache so that partially rejected branches can be reused. The paper also introduces latency-optimized fused kernels (GEMM plus all-reduce, masked attention, SwiGLU) for low-batch int4 AWQ models. The evaluation reports decoding speeds on five model families and six datasets, with an average 1.75x speedup claimed over the internal serial baseline SwiftSpec-base, and 348 tokens/s for Llama3-70B on 8 H800 GPUs.

Significance. The disaggregated design is a plausible and useful systems contribution: Table 1 and Figure 9 provide evidence that draft and target models have different optimal tensor-parallelism degrees, and the ablations in Section 5.3 indicate that parallel tree generation contributes 1.43-1.50x and kernel fusion contributes 1.16-1.21x end-to-end. The kernel microbenchmarks in Table 7 are informative, and Section 6 is candid about the main limitations: EAGLE-style draft models are not supported and the system targets light-load, low-batch settings. However, the headline claim of a 1.75x speedup over state-of-the-art speculative decoding systems is not supported by the evaluation as written, and the absence of tuning details, error bars, and artifacts makes the quantitative claims difficult to verify. The core idea is worth pursuing, but the claims and evidence must be brought into line.

major comments (5)
  1. [Abstract and Section 5.2 (Figure 7)] The abstract states an average 1.75x speedup over state-of-the-art speculative decoding systems, but Section 5.2 reports that 1.75x is the speedup over SwiftSpec-base, an internal baseline, while the external tree-based system SGLang with EAGLE is available for only two of the five model families. For the other three families, SGLang is benchmarked in autoregressive mode, and vLLM and TensorRT-LLM support only sequence-based speculation, which the paper itself says is not comparable. The stated average against state-of-the-art speculative decoding systems is therefore not established. The claim should be rephrased, and per-family comparisons against the best available external tree-based configuration should be reported.
  2. [Section 5.1] The baseline tuning procedure is not documented. The text says each baseline was run in an extensive set of configurations with the best average chosen, but no search space, per-configuration results, or final configurations are given. Since SwiftSpec's own parameters (batch size bs=8, width w=8, tree expansions d, and GPU split x) are selected by profiling on the same benchmarks (Sections 3.1 and 5.5), the reported speedups could be inflated if the baselines were not tuned with equal effort. Please provide the tuning ranges and the final chosen configurations, or show sensitivity to these choices.
  3. [Section 3.2] The consistency-management contribution is described only for the draft model. The introduction says SwiftSpec reorganizes the KV cache of both the draft and target models, and Table 2 claims fine-grained reorganization with zero waste, but Section 3.2 explains only the draft cache's prefix/tree organization. The target model also consumes tree-structured inputs and needs a consistent cache for the accepted path; without a description of target-side KV management, or an explicit statement that the target cache is recomputed, the zero-waste and consistency claims are not substantiated.
  4. [Sections 5.2-5.4] All end-to-end and kernel timings are presented as single point estimates with no error bars, repetition counts, or confidence intervals. For a low-latency system where kernel-level differences such as 1.16x/1.21x (Figure 8) and the 8.2% attention improvement (Section 5.3) are close to run-to-run noise, this is insufficient. Please report multiple runs and variance, and state the number of repetitions used for each measurement.
  5. [Abstract and Section 7] The claim that SwiftSpec is the fastest known system for low-latency LLM serving at this scale is not backed by a survey of existing systems. The evaluation compares only with vLLM, SGLang, TensorRT-LLM, and SwiftSpec-base; other speculative-decoding systems listed in Table 2 (PEARL, AMUSD, PipeInfer) are not benchmarked. Please restrict the claim to the fastest among the systems evaluated, or provide evidence of a broader search.
minor comments (6)
  1. [Section 5.2] The six datasets are listed in Section 5.1, but Figure 7 and the text report only per-model-family averages. A per-dataset table or decomposition would help verify the claim that results cover all six datasets.
  2. [Section 4] The description of padding for arbitrary tensor parallelism says padded attention heads do not contribute to the final result, but it is not explained how the padded heads are masked or zeroed after the attention computation; please specify the mechanism.
  3. [Table 5] The support matrix uses check and cross symbols without a legend; please define the symbols in the caption so that the support claims are unambiguous.
  4. [Section 5.5 and Figure 9] Figure 9 is described as target TP=2/4/6, but the corresponding draft TP (6/4/2) is not shown in the figure; please state the full GPU split for each configuration.
  5. [Section 2.3.2 and Figure 2] The right-hand panel's y-axis is labeled compression ratio, but the text describes average number of correct guesses; please clarify whether these are the same quantity.
  6. [Reproducibility] The paper does not provide code, configuration scripts, or raw benchmark data. For a systems paper whose claims rest on end-to-end numbers, releasing artifacts, or at least a detailed appendix with per-dataset and per-configuration results, would substantially improve verifiability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's claims are empirical systems measurements with internal and external baselines, and no derivation reduces to its own inputs.

full rationale

SwiftSpec's central claims are engineering results, not derived predictions, so there is no derivation chain that reduces to its inputs. The design parameters (bs=8, w=8, d, GPU split) are selected by profiling and validated through ablations, which is standard systems practice rather than fitting a parameter to a target result. The end-to-end speedups are measured against vLLM, SGLang, TensorRT-LLM, and the internal SwiftSpec-base, and the kernel-level gains are measured against FlashAttention, FlashInfer, and baseline serving frameworks. No claim in the paper is equivalent by construction to a fitted value or to a self-citation: the 1.75x number is an actual measurement against SwiftSpec-base, and the abstract's phrasing 'state-of-the-art speculative decoding systems' overstates that comparison because SGLang runs autoregressively for three of the five model families, but this is a benchmark-validity and reporting-accuracy concern, not circularity. Self-citations in the references (e.g., CacheGen [23]) are not load-bearing for any central claim, and no uniqueness theorem or ansatz smuggled in via citation appears. The paper is self-contained against external baselines and microbenchmarks, so the correct circularity verdict is a score of 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several tuned system parameters and unproven but plausible domain assumptions. No new physical entities are introduced. The paper is transparent about parameter choices, but the completeness of the tuning and the representativeness of the benchmark are major determinants of whether the speedups generalize.

free parameters (4)
  • target batch size (bs) = 8
    Set in Section 5.5 using Figure 2 and runtime measurements: 'We set bs = 8 and w = 8 empirically to balance the acceptance ratio and running time.' The end-to-end speedup claim depends on this value.
  • draft expansion width (w) = 8
    Set empirically alongside bs in Section 5.5; larger widths give diminishing compression-ratio gains (Figure 2) and increase draft inference time.
  • tree expansions per round (d) = r or r+1, chosen per model pair
    In Sections 3.1 and 5.5, d is derived from the profiled ratio of target to draft inference time and then the authors 'choose the configuration with the higher decoding speed' among the two candidates. This is a tuned per-model parameter.
  • GPU allocation for target model (x) = 6 for Deepseek-Coder 33B and Qwen2-72B; 4 for others
    Chosen via a profile phase (Section 3.1 and Figure 9) that 'try[ies] out different x to find which configuration yields the fastest average decoding speed.' This split is model-dependent and tuned on the benchmark workloads.
assumptions (5)
  • domain assumption The NCCL LL protocol, using PTX volatile atomic loads and stores without explicit barriers, is race-free and correct for the fused kernels.
    The fused GEMM-all-reduce and attention kernels in Section 3.3 rely on this protocol for implicit synchronization. The paper does not provide a formal memory-model argument or verification.
  • domain assumption Maximum-likelihood tree expansion (sum of log-softmax probabilities along each path) yields draft trees with high enough acceptance to make speculation profitable.
    Used to select which leaves to expand in Algorithm 1 and Section 3.1. Empirical evidence is given, but there is no theoretical guarantee that this heuristic is near-optimal.
  • domain assumption The KV-cache reorganization invariant (verified tokens contiguously in prefix, tree tokens after, with re-rooting and pruning) preserves the model's output distribution.
    Section 3.2 describes the invariant and an example, but no formal equality proof or randomized testing is provided. A violation would change generated text.
  • domain assumption 4-bit AWQ quantization of the transformer weights does not materially change model quality, so decoding speedups are not offset by quality loss.
    The paper applies int4 AWQ to all target and draft models and reports no accuracy or quality metrics on the six datasets. The speed claims are thus for quantized models only.
  • domain assumption An 8-GPU H800 NVLink node is representative of the 'Nvidia Hopper GPUs' referenced in the headline claims.
    The abstract and conclusion generalize to Hopper GPUs, but H800 differs from H100 in memory bandwidth and NVLink speed for some variants, which could affect the portability of the 348 tokens/s figure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding." pith.science (2026). https://pith.science/paper/BMZWCNLM

@misc{pith2026250611309,
  author       = {Pith},
  title        = {Pith review of: SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BMZWCNLM}},
  note         = {Machine review of arXiv:2506.11309}
}
read the original abstract

Low-latency decoding for large language models (LLMs) is crucial for applications like chatbots and code assistants, yet generating long outputs remains slow in single-query settings. Prior work on speculative decoding (which combines a small draft model with a larger target model) and tensor parallelism has each accelerated decoding. However, conventional approaches fail to apply both simultaneously due to imbalanced compute requirements (between draft and target models), KV-cache inconsistencies, and communication overheads under small-batch tensor-parallelism. This paper introduces SwiftSpec, a system that targets ultra-low latency for LLM decoding. SwiftSpec redesigns the speculative decoding pipeline in an asynchronous and disaggregated manner, so that each component can be scaled flexibly and remove draft overhead from the critical path. To realize this design, SwiftSpec proposes parallel tree generation, tree-aware KV cache management, and fused, latency-optimized kernels to overcome the challenges listed above. Across 5 model families and 6 datasets, SwiftSpec achieves an average of 1.75x speedup over state-of-the-art speculative decoding systems and, as a highlight, serves Llama3-70B at 348 tokens/s on 8 Nvidia Hopper GPUs, making it the fastest known system for low-latency LLM serving at this scale.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting

    cs.CL 2026-07 accept novelty 6.0 of 10

    PTD accelerates autoregressive LLM decoding up to ~2× by guiding the target model to explore multiple coherent draft paths via a progressive, pruned tree in a single forward pass.

  2. ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling

    cs.LG 2025-07 conditional novelty 3.0 of 10

    ODIA routes 60% of function-calling traffic in a music app to a small 1.3B model, reducing expected latency by 45% and median latency by 78%.

Reference graph

Works this paper leans on

51 extracted references · 18 canonical work pages · cited by 2 Pith papers

  1. [1]

    Tam- ing Throughput-Latency tradeoff in LLM inference with Sarathi-Serve

    Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gulavani, Alexey Tumanov, and Ramachandran Ramjee. Tam- ing Throughput-Latency tradeoff in LLM inference with Sarathi-Serve. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 117–134, Santa Clara, CA, July

    work page

  2. [2]

    Pipeinfer: Accelerating llm inference usingasynchronouspipelinedspeculation, 2024

    Branden Butler, Sixing Yu, Arya Mazaheri, and Ali Jannesari. Pipeinfer: Accelerating llm inference usingasynchronouspipelinedspeculation, 2024. URL https://arxiv.org/abs/2407.11798

    arXiv 2024

  3. [3]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simplellminferenceaccelerationframework with multiple decoding heads, 2024. URL https: //arxiv.org/abs/2401.10774

    arXiv 2024

  4. [4]

    Large language models vs

    Kevin Matthe Caramancion. Large language models vs. search engines: Evaluating user preferences across varied information retrieval scenarios, 2024. URL https://arxiv.org/abs/2401.05761

    arXiv 2024

  5. [5]

    Accelerating large language model decod- ing with speculative sampling, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irv- ing, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decod- ing with speculative sampling, 2023. URL https: //arxiv.org/abs/2302.01318

    arXiv 2023

  6. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

    arXiv 2021

  7. [8]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URLhttps://arxiv.org/abs/2110.14168

    arXiv 2021

  8. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xi- aokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai...

    arXiv 2025

Show all 51 references
  1. [10]

    Dapple: A pipelined data parallel approach for training large models, 2020

    Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, and Wei Lin. Dapple: A pipelined data parallel approach for training large models, 2020. URL https://arxiv.org/abs/2007.01045

    2020 arXiv

  2. [11]

    Longcoder: A long-range pre-trained language model for code completion, 2023

    Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Ju- lian McAuley. Longcoder: A long-range pre-trained language model for code completion, 2023. URL https://arxiv.org/abs/2306.14893

    2023 arXiv

  3. [12]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. URLhttps://arxiv.or...

    2024 arXiv

  4. [13]

    Unlocking the potential of chat- gpt: A comprehensive exploration of its applica- tions, advantages, limitations, and future direc- tions in natural language processing, 2025

    Walid Hariri. Unlocking the potential of chat- gpt: A comprehensive exploration of its applica- tions, advantages, limitations, and future direc- tions in natural language processing, 2025. URL https://arxiv.org/abs/2304.02017

    2025 arXiv

  5. [14]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, Hy- oukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism,

  6. [15]

    Language models for code completion: A practical evaluation, 2024

    Maliheh Izadi, Jonathan Katzy, Tim van Dam, Marc Otten, Razvan Mihai Popescu, and Arie van Deursen. Language models for code completion: A practical evaluation, 2024. URL https://arxiv.org/abs/ 2402.16197

    2024 arXiv

  7. [16]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Jacob De- vlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and ...

    2019

  8. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient mem- ory management for large language model serving with pagedattention, 2023. URL https://arxiv. org/abs/2309.06180

    2023 arXiv

  9. [18]

    Fast inference from transformers via speculative de- coding, 2023

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative de- coding, 2023. URL https://arxiv.org/abs/2211. 17192

    2023

  10. [19]

    Eagle-2: Faster inference of language models withdynamicdrafttrees, 2024

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-2: Faster inference of language models withdynamicdrafttrees, 2024. URL https://arxiv. org/abs/2406.16858

    2024 arXiv

  11. [20]

    Awq: Activation-aware weight quantization for llm com- pression and acceleration, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm com- pression and acceleration, 2024. URL https:// arxiv.org/abs/2306.00978

    2024 arXiv

  12. [21]

    Parallel speculative decoding with adaptive draft length, 2024

    Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. Parallel speculative decoding with adaptive draft length, 2024. URL https:// arxiv.org/abs/2408.11850

    2024 arXiv

  13. [22]

    Opti- mizing speculative decoding for serving large lan- guage models using goodput, 2024

    Xiaoxuan Liu, Cade Daniel, Langxiang Hu, Woosuk Kwon, Zhuohan Li, Xiangxi Mo, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Opti- mizing speculative decoding for serving large lan- guage models using goodput, 2024. URL https: //arxiv.org/abs/2406.14066

    2024 arXiv

  14. [23]

    Cachegen: Kv cache compression and stream- ing for fast large language model serving, 2024

    Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang. Cachegen: Kv cache compression and stream- ing for fast large language model s...

    2024 arXiv

  15. [24]

    Amusd: Asynchronous multi- devicespeculativedecodingforllmacceleration, 2024

    Bradley McDanel. Amusd: Asynchronous multi- devicespeculativedecodingforllmacceleration, 2024. URLhttps://arxiv.org/abs/2410.17375

    2024 arXiv

  16. [25]

    Specinfer: Accelerating large language model serving with tree-based speculative infer- ence and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, 17 Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhi- hao Jia. Specinfer: Accelerating large language model serving wi...

    2024

  17. [26]

    Abstrac- tive text summarization using sequence-to-sequence rnns and beyond, 2016

    Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos santos, Caglar Gulcehre, and Bing Xiang. Abstrac- tive text summarization using sequence-to-sequence rnns and beyond, 2016. URLhttps://arxiv.org/ abs/1602.06023

    2016 arXiv

  18. [27]

    Nvidia/tensorrt-llm: A tensorrt toolbox for optimized large language model inference

    Nvidia. Nvidia/tensorrt-llm: A tensorrt toolbox for optimized large language model inference. https: //github.com/NVIDIA/TensorRT-LLM

  19. [28]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URLhttps: //arxiv.org/abs/2303.08774

    2024 arXiv

  20. [29]

    Splitwise: Efficient generative llm inference using phase splitting, 2024

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ri- cardo Bianchini. Splitwise: Efficient generative llm inference using phase splitting, 2024. URL https://arxiv.org/abs/2311.18677

    2024 arXiv

  21. [30]

    Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053

    2020 arXiv

  22. [31]

    Tatsu-lab/stanford-alpaca: Code and documentation to train stanford’s alpaca mod- els, and generate the data

    Tatsu-Lab. Tatsu-lab/stanford-alpaca: Code and documentation to train stanford’s alpaca mod- els, and generate the data. https://github.com/ tatsu-lab/stanford_alpaca

  23. [32]

    CUTLASS, January

    Vijay Thakkar, Pradeep Ramani, Cris Cecka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Fengqi Qiao, Piotr Majcher, Paul Springer, Markus Hohnerbach, Jin Wang, and Manish Gupta. CUTLASS, January

  24. [33]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023

  25. [34]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706. 03762

    2023

  26. [35]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https://arxiv.org/abs/2201.11903

    2023 arXiv

  27. [36]

    Smoothquant: Ac- curate and efficient post-training quantization for large language models, 2024

    Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Ac- curate and efficient post-training quantization for large language models, 2024. URLhttps://arxiv. org/abs/2211.10438

    2024 arXiv

  28. [37]

    When search engine services meet large language models: Visions and challenges, 2024

    Haoyi Xiong, Jiang Bian, Yuchen Li, Xuhong Li, Mengnan Du, Shuaiqiang Wang, Dawei Yin, and Sumi Helal. When search engine services meet large language models: Visions and challenges, 2024. URL https://arxiv.org/abs/2407.00128

    2024 arXiv

  29. [38]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Hao- ran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jin...

    2024 arXiv

  30. [39]

    Palr: Personal- ization aware llms for recommendation, 2023

    Fan Yang, Zheng Chen, Ziyan Jiang, Eunah Cho, Xiaojiang Huang, and Yanbin Lu. Palr: Personal- ization aware llms for recommendation, 2023. URL https://arxiv.org/abs/2305.07622

    2023 arXiv

  31. [40]

    Orca: A distributed serving system for Transformer-Based generative models

    Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. Orca: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521–538, Carlsbad, CA, July

  32. [41]

    Fltrnn: Faithful long-horizon taskplanningforroboticswithlargelanguagemodels

    Jiatao Zhang, Lanling Tang, Yufan Song, Qiwei Meng, Haofu Qian, Jun Shao, Wei Song, Shiqiang Zhu, and Jason Gu. Fltrnn: Faithful long-horizon taskplanningforroboticswithlargelanguagemodels. In 2024 IEEE International Conference on Robotics 18 and Automation (ICRA), pages 6680–...

    2024

  33. [42]

    Prepacking: A simple method for fast prefilling and increased throughput in large language models, 2024

    Siyan Zhao, Daniel Israel, Guy Van den Broeck, and Aditya Grover. Prepacking: A simple method for fast prefilling and increased throughput in large language models, 2024. URL https://arxiv.org/abs/2404. 09529

    2024

  34. [43]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm- as-a-judge with mt-bench and chatbot arena, 2023. URLhttps://arxiv.org/abs/2306.05685

    2023 arXiv

  35. [44]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs,

  36. [45]

    DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving

    Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24)...

  37. [46]

    Code-as-monitor: Constraint-aware vi- sual programming for reactive and proactive robotic failure detection, 2025

    Enshen Zhou, Qi Su, Cheng Chi, Zhizheng Zhang, Zhongyuan Wang, Tiejun Huang, Lu Sheng, and He Wang. Code-as-monitor: Constraint-aware vi- sual programming for reactive and proactive robotic failure detection, 2025. URL https://arxiv.org/ abs/2412.04455. 19

    2025 arXiv

  38. [49]

    URLhttps://arxiv.org/abs/2312.07104

    arXiv

  39. [51]

    ISBN 978-1-939133- 40-3

    USENIX Association. ISBN 978-1-939133- 40-3. URL https://www.usenix.org/conference/ osdi24/presentation/zhong-yinmin

  40. [2019]

    URLhttps://arxiv.org/abs/1811.06965

    arXiv

  41. [2022]

    ISBN 978-1-939133- 28-1

    USENIX Association. ISBN 978-1-939133- 28-1. URL https://www.usenix.org/conference/ osdi22/presentation/yu

  42. [2023]

    URLhttps://github.com/NVIDIA/cutlass

  43. [2024]

    ISBN 978-1-939133- 40-3

    USENIX Association. ISBN 978-1-939133- 40-3. URL https://www.usenix.org/conference/ osdi24/presentation/agrawal

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.