Pith. sign in

REVIEW 12 cited by

SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.09781 v4 pith:FIOEL4S7 submitted 2023-05-16 cs.CL cs.DCcs.LG

Xupeng Miao , Gabriele Oliaro , Zhihao Zhang , Xinhao Cheng , Zeyu Wang , Zhengxin Zhang , Rae Ying Yee Wong , Alan Zhu
show 7 more authors
Lijie Yang Xiaoxiang Shi Chunan Shi Zhuoming Chen Daiyaan Arfeen Reyna Abhyankar Zhihao Jia
This is my paper
classification cs.CLcs.DCcs.LG
keywords specinfertokengenerativeinferenceservingmodelspeculativetree
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8x for distributed LLM inference and by 2.6-3.5x for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/

Discussion (0). Sign in to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Beyond KV Reconstruction: Functional Reconstruction for MLA Draft Models in Speculative Decoding

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A post-conversion training objective that reconstructs the original attention function of MLA-converted draft models improves speculative-decoding acceptance in most of 64 matched task cells.

  2. Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Strictly pre-answer hidden states of a looped transformer add significant AUROC over surface shortcuts for predicting correctness, and the readout yields decision-level gains but no generative control.

  3. Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models

    cs.CL 2025-11 conditional novelty 6.0 of 10

    ODB-dLLM accelerates diffusion LLM inference by adaptively shortening responses and using a jump-share speculative decoding strategy, achieving 2.6-6.3x over Fast-dLLM.

  4. TetriServe: Efficiently Serving Mixed DiT Workloads

    cs.LG 2025-10 conditional novelty 6.0 of 10

    TetriServe's step-level, deadline-aware sequence parallelism improves SLO attainment for mixed-resolution diffusion transformer serving by up to 32% over fixed-SP systems.

  5. HiSpec: Hierarchical Speculative Decoding for LLMs

    cs.CL 2025-10 conditional novelty 6.0 of 10

    HiSpec uses early-exit layers to verify draft tokens midway through the model, reporting 1.28×-2.01× faster decoding over baseline speculative decoding, but the accuracy claim is not empirically tested.

  6. Kinetics: Rethinking Test-Time Scaling Laws

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A memory-aware test-time scaling law shows small models are overestimated and sparse attention is needed for efficient scaling.

  7. AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism

    cs.CL 2025-06 conditional novelty 6.0 of 10

    AdaDecode speeds up LLM generation by predicting tokens at early layers when confidence is high, running the skipped layers in parallel, and verifying the output exactly matches standard decoding.

  8. Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Reflective Verification fuses a target LLM's normal and reflection-prompted logits to accept semantically correct draft tokens, increasing accepted draft length and decoding speed by 5-15%.

  9. Towards the Holographic Characteristic of LLMs for Efficient Short-text Generation

    cs.CL 2026-01 conditional novelty 5.0 of 10

    LLMs tend to assign high probability to target keywords in the first two generation steps; a plugin using those keywords with parallel constrained generation gives comparable short-text quality and large speedups on s...

  10. Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions

    cs.CL 2025-08 conditional novelty 5.0 of 10

    Meta reports EAGLE-based speculative decoding optimizations for Llama models, achieving state-of-the-art latency (about 4 ms/token for Llama4 Maverick) and 1.4-2.0x speedups for large batches.

  11. SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding

    eess.AS 2025-07 reject novelty 4.0 of 10

    SpecASR accelerates LLM-based ASR by 3.04x-3.79x over autoregressive decoding using adaptive draft lengths, draft token recycling, and sparse token trees, but the speedups are simulated from Whisper proxy models rathe...

  12. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

Pith tools