Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

SPEED-Bench is a benchmark for speculative decoding that claims speedups depend heavily on the prompt distribution and serving regime, and that standardized diverse workloads are needed to measure them fairly.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:35 UTC pith:EEWGKQH2

load-bearing objection A genuinely useful benchmark for speculative decoding, with a diversity-driven split that is broader but not proven representative — worth reviewing. the 3 major comments →

arxiv 2604.09557 v2 pith:EEWGKQH2 submitted 2026-02-10 cs.DC cs.AI

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

Talor Abramovich , Maor Ashkenazi , Izzy Putterman , Benjamin Chislett , Tiyasa Mitra , Bita Darvish Rouhani , Ran Zilberstein , Yonatan Geifman This is my paper
classification cs.DC cs.AI
keywords speculative decodingLLM inferencebenchmarksemantic diversitythroughput evaluationacceptance ratedraft lengthdata selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SPEED-Bench is a benchmark for speculative decoding (SD), a technique that uses a small draft model to guess several tokens ahead and then verifies them in one pass, speeding up LLM inference. The paper argues that SD speedups depend heavily on what text the model is given and on the serving conditions, so existing evaluations—often small, repetitive, and run one request at a time—mislead. To fix that, SPEED-Bench provides a qualitative split whose prompts are chosen to maximize semantic diversity, and a throughput split with fixed input-length buckets from 1k to 32k tokens to support batch-size scaling. A unified measurement framework runs the same tokenized sequences in production-grade serving engines, and the paper uses it to show that synthetic token inputs overestimate throughput, that optimal draft length shifts with batch size, and that vocabulary pruning degrades some domains more than others. A sympathetic reader would take away that SD comparisons need standardized, diverse, and realistic workloads to be trustworthy.

Core claim

The paper argues that speculative decoding speed cannot be judged on narrow or synthetic workloads; evaluation needs semantically diverse, realistic data and production-grade measurement. Its Qualitative Split selects prompts by minimizing pairwise cosine similarity among embeddings via greedy selection with local swap refinement, cutting average semantic similarity by about 40% versus an earlier benchmark. A Throughput Split adds real prompts in fixed input-length buckets (1k–32k tokens) across three entropy tiers, enabling throughput-latency Pareto curves. A measurement framework normalizes tokenization and chat templates so all engines process identical sequences. Empirically, random-toke

What carries the argument

The load-bearing object is the greedy subset-selection algorithm with local swap refinement: it minimizes the sum of pairwise cosine similarities between row-normalized prompt embeddings, so the chosen prompts span the semantic space rather than clustering. For the throughput regime, the key identity is Speedup = (t_ar * AL) / t_sd, which separates domain-dependent acceptance length (AL) from system-dependent per-step latencies; the Throughput Split provides realistic workloads for measuring those latencies, allowing domain-specific speedups to be estimated without exhaustive testing.

Load-bearing premise

The whole benchmark rests on the assumption that maximizing semantic diversity in embedding space (cosine similarity) produces a prompt set whose diversity matches the real-world data distribution that matters for speculative decoding; nothing in the paper independently verifies that connection.

What would settle it

Sample, say, 1,000 prompts uniformly from logged production traffic for a coding assistant and a chat service, run a fixed drafter-target pair through the same measurement framework, and compare category-level acceptance lengths and the ranking across domains with SPEED-Bench's results; if the rankings diverge materially, the diversity-selection premise fails. Similarly, if random-token prompts on a mixture-of-experts model activate experts as evenly as real prompts do, the claimed synthetic-input distortion collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark is adopted, published SD speedups will become comparable across methods, because all measurements run on the same diverse prompts and the same underlying engine implementations.
  • Synthetic token inputs should be retired for SD benchmarking: the paper measures a 23% average overestimate in throughput, and shows random inputs also distort expert routing in mixture-of-experts models.
  • Optimal draft length is not a fixed property of a drafter; practitioners need to re-tune it for their batch size, since shorter drafts win in compute-bound high-batch regimes.
  • Vocabulary-pruned drafters should be expected to lose accuracy on multilingual, retrieval-augmented, and summarization workloads even when math and coding look fine.
  • Drafter training context length matters: models trained on short sequences degrade sharply beyond their training input length, and long-context scaling like YaRN recovers much of the accuracy.
  • The authors identify a boundary on their own framework: above about 256 concurrent requests, the Python client-side loop can become a bottleneck, so the highest-concurrency regime is not yet fully covered.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the selection algorithm optimizes embedding-space cosine distance, but embedding diversity is only a proxy for the task-level properties that determine acceptance rates; a validation against randomly sampled production traffic would strengthen the claim of representativeness.
  • Editorial inference: the speedup decomposition suggests a calibration protocol: measuring per-step latencies once per hardware/engine/batch configuration and reusing them with domain-specific ALs could let the community extrapolate speedups to new domains cheaply.
  • Editorial inference: the paper's finding that low-diversity benchmarks hide drafter differences implies that previously published comparisons on small, repetitive datasets may need re-checking, especially for multilingual and long-context claims.
  • Boundary noted by the authors: their measurement client can become a bottleneck above roughly 256 concurrent requests, so the very-high-concurrency regime is not fully covered.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SPEED-Bench proposes a benchmark suite for speculative decoding (SD) with two data splits and a measurement framework. The Qualitative Split selects 880 prompts from 18 public sources by greedily minimizing pairwise cosine similarity among prompt embeddings (Eq. 2, Algorithm 1), yielding 11 categories with 80 samples each. The Throughput Split provides fixed-ISL buckets (1k–32k) across three entropy classes for throughput-latency evaluation at batch sizes up to 512. The framework integrates with vLLM, TensorRT-LLM, SGLang, and SpecBench, and the paper reports experiments on Llama 3.3 70B, GPT-OSS 120B, DeepSeek R1, Qwen3 235B, and Qwen3-Next, covering acceptance rates, speedups, vocabulary pruning, synthetic-input artifacts, draft-length selection, and training-ISL effects. The paper claims SPEED-Bench will establish a unified evaluation standard for practical SD comparisons.

Significance. If the dataset and measurement pipeline are sound, SPEED-Bench is a useful step toward standardized SD evaluation. Its strengths include: the use of production-grade engines rather than high-level stubs; the Throughput Split, which addresses a real gap in batch-size/ISL coverage; the validation of a proxy formula for domain-specific speedups (Appendix H); and the demonstrations that synthetic random inputs distort both SD and baseline MoE throughput (Appendix G) and that vocabulary pruning degrades multilingual and RAG performance. The paper is also transparent about its conflict of interest and about acknowledged limitations (e.g., the GIL at BS>256, the unconfirmed causes of long-context EAGLE3 degradation). The main risk is whether the Qualitative Split's diversity-maximized subset is representative enough of real workloads to support the paper's strong claim of a 'unified evaluation standard.'

major comments (3)
  1. [§5, Eq. (2), Algorithm 1; Appendix C] The Qualitative Split is constructed by minimizing pairwise cosine similarity in an embedding space. This is a coverage/diversity objective, not a representativeness objective: it selects mutually dissimilar prompts that tend to lie in low-density regions and down-weights common, clustered prompt types. Appendix C shows that the optimized subset is stable across random seeds, but stability under resampling is not the same as unbiasedness relative to the source distribution or to real deployment workloads. Since the aggregate AL/speedup numbers in Table 1 and the comparisons with SpecBench in §8.3 rest entirely on this subset, the paper should either (a) validate that aggregate conclusions (draft-method rankings, category orderings) agree with those obtained from a uniform/stratified sample of the same source pools, or (b) compare the selected prompts against a held-out realistic workload
  2. [Table 1; §8.3 and Figure 5] The central quantitative results are reported without confidence intervals, standard errors, or repeated-run variation. With 80 samples per category and, apparently, a single measurement per cell, small differences between methods (e.g., the mean AL values of 2.44 for Vanilla and EAGLE3 on Llama 3.3 70B at Temperature=0; the mean AL differences across domains) cannot be assessed statistically. The claims that SPEED-Bench 'corrects' the SpecBench impression, or that one drafter outperforms another, need error bars or significance tests. This is especially important because the stated purpose of the benchmark is to enable reliable cross-method comparisons.
  3. [§5, 'Selection Algorithm'] The entire Qualitative Split depends on a single pre-trained embedder, OpenAI's text-embedding-3-large, used to compute cosine similarities. There is no evidence that the selected subset is stable across embedding models or that this embedding geometry is aligned with the task properties that actually determine SD acceptance rates and acceptance lengths. I request an ablation: recompute the selection with at least one other embedder (or an ensemble) and report subset overlap and the resulting differences in per-category AL/speedups. Without such evidence, the benchmark's diversity claims and the subsequent empirical conclusions are tied to a single, proprietary embedding choice.
minor comments (5)
  1. [Abstract and Conclusion] The abstract says 'unified evaluation standard' and the conclusion says 'establishes a unified evaluation ecosystem.' This is stronger than what the evidence currently supports, given the representativeness concern above; consider using 'a step toward' or 'a benchmark for.'
  2. [§5 vs Appendix A] The text says the Qualitative Split aggregates '18 publicly available datasets,' while Table 2 reports '24 distinct data sources' for SPEED-Bench. Please clarify whether this counts only the Qualitative Split or the union of both splits.
  3. [Table 2] Minor typo: 'Lanuages' should be 'Languages.' Also, the table lists 'Programming Lanuages' and 'Lanuages in Multilingual' with inconsistent capitalization.
  4. [References] The reference list contains duplicate entries for See et al. 2017 (2017a and 2017b with the same title and pages). Please deduplicate and use a single citation.
  5. [§7] The note that the asyncio/GIL introduces overhead at BS>256 is honest, but it would help to state explicitly which experimental points in Figures 6 and 7 are within the validated regime.

Circularity Check

0 steps flagged

No significant circularity: the benchmark is constructed by explicit, non-fitted selection rules and its claims are supported by direct measurements against external baselines.

full rationale

SPEED-Bench is a benchmark and measurement paper, not a derivation. The Qualitative Split is deliberately constructed to maximize embedding-space diversity (Eq. 2, Algorithm 1); this is a transparent design choice, not a fitted parameter later relabeled as a prediction. Acceptance lengths, speedups, and throughputs are measured directly from engine outputs, not derived from the selection objective. The comparisons with SpecBench in §8.3 are independent empirical measurements on differently sourced data, and the finding that low-diversity benchmarks hide differences is a consequence of the construction but is presented as a demonstration, not as a deduced law. The proxy Equation 3 in Appendix H is an algebraic identity, but it is used explicitly and checked against directly measured end-to-end speedups (Table 10), so it is not a hidden redefinition. The paper contains self-referential elements—NVIDIA employment, TensorRT-LLM, NVIDIA draft checkpoints—but these are disclosed conflicts of interest and are not used as evidence for the central claims. The stated limitations (GIL overhead at BS>256, unconfirmed causes of long-context degradation) are candid and do not hide circularity. The representativeness question raised about the diversity-maximizing selection is a validity concern, not a circularity concern; it does not reduce any claimed result to its own inputs. Therefore no circular step meets the evidentiary bar.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The central claim of benchmark representativeness rests on design choices (embedding-based diversity, entropy labels, and the measurement framework) that are assumptions rather than measured facts. The paper provides some validation but does not independently establish these assumptions.

axioms (3)
  • domain assumption Maximizing semantic diversity in the embedding space (OpenAI text-embedding-3-large) yields a dataset representative of real-world SD workloads.
    The selection algorithm in Section 5 minimizes pairwise cosine similarity, but there is no independent evidence that this diversity metric aligns with the dimensions that affect speculative decoding accuracy.
  • domain assumption The entropy categorization (Low/Mixed/High) correctly reflects the difficulty of speculative decoding across domains.
    The Throughput Split groups data by domain entropy based on a taxonomy from Li et al., but AL measurements only partially validate this ordering (Appendix J shows GPT-OSS EAGLE3 deviates from expected ordering).
  • domain assumption The measurement framework accurately captures streaming response timings without engine-specific overheads.
    Section 7 notes a GIL limitation at BS>256, so timing accuracy is limited at very high concurrency.

pith-pipeline@v1.3.0-alltime-deepseek · 24766 in / 7232 out tokens · 72891 ms · 2026-08-03T02:35:47.052296+00:00 · methodology

0 comments
read the original abstract

Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness. Existing benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments. To address this, we introduce SPEED-Bench, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes. SPEED-Bench offers a carefully curated Qualitative data split, selected by prioritizing semantic diversity across the data samples. Additionally, it includes a Throughput data split, allowing speedup evaluation across a range of concurrencies, from latency-sensitive low-batch settings to throughput-oriented high-load scenarios. By integrating with production engines like vLLM and TensorRT-LLM, SPEED-Bench allows practitioners to analyze system behaviors often masked by other benchmarks. We highlight this by quantifying how synthetic inputs overestimate real-world throughput, identifying batch-size dependent optimal draft lengths and biases in low-diversity data, and analyzing the caveats of vocabulary pruning in state-of-the-art drafters. We release SPEED-Bench to establish a unified evaluation standard for practical comparisons of SD algorithms.

Figures

Figures reproduced from arXiv: 2604.09557 by Benjamin Chislett, Bita Darvish Rouhani, Izzy Putterman, Maor Ashkenazi, Ran Zilberstein, Talor Abramovich, Tiyasa Mitra, Yonatan Geifman.

Figure 1
Figure 1. Figure 1: Overview of the SPEED-Bench ecosystem. (Left) Curation of the Qualitative split, utilizing a custom selection algorithm on prompt embeddings to maximize semantic diversity across categories. (Middle) Construction of the Throughput Split, where data is aggregated and processed into fixed Input Sequence Length (ISL) buckets (1k-32k) across three domain difficulties, supporting large batch sizes (up to 512 pe… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of average semantic similarity between samples (lower is better). SPEED-Bench achieves lower similarity than both random selection and SpecBench across all categories. with Local Swap Refinement (see Algorithm 1). We initialize S with a random index and iteratively append i ∗ = argmini /∈S P j∈S x ⊤ i xj . To escape local minima, we then iteratively swap iout ∈ S with iin ∈/ S if the swap strict… view at source ↗
Figure 3
Figure 3. Figure 3: Average AL on the Qualitative Split. External drafting scales better across DLs. the SpecBench framework excels at evaluating methods us￾ing native PyTorch/HuggingFace, SPEED-Bench focuses on the viability of these methods in deployment. To support a holistic pipeline, we demonstrate how SpecBench models can be evaluated within our framework. The supplementary material includes an example for SpecBench’s M… view at source ↗
Figure 5
Figure 5. Figure 5: Average AL across selected categories in SpecBench vs SPEED-Bench. Target model is Llama 3.3 70B. DL = 7. Full results are in Appendix K. narios. Unlike methods that focus on latency at BS = 1, SPEED-Bench enables the construction of throughput￾latency Pareto curves, providing insights into the interplay between BS, DL, and inference engines. Random data VS SPEED-Bench In Section 6, we iden￾tified the risk… view at source ↗
Figure 4
Figure 4. Figure 4: Average AL across selected categories using GPT-OSS 120B and EAGLE3 drafters (full vs. pruned vocabulary), DL = 3. GLE3 applies vocabulary pruning (usually to 32k tokens), mitigating the computational bottlenecks of the final projec￾tion layer. While effective for standard inputs, this heuristic degrades performance on the ”long tail” of user inputs. We identified the Multilingual category as a particular … view at source ↗
Figure 6
Figure 6. Figure 6: Throughput as a function of user TPS, comparing random input tokens to the Throughput Split (8k). Target is GPT-OSS 120B with EAGLE3 drafter, measured on TensorRT-LLM. DL = 3. Points represent BS from 1 to 128. 0 100 200 300 400 User TPS 0 5000 10000 15000 Output TPS per GPU Draft Length=1 Draft Length=3 w/o SD [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Throughput as a function of user TPS, comparing DL = 1, 3 on the Throughput Split (2k). Target is GPT-OSS 120B with EAGLE3, measured on vLLM. Points represent BS from 2 to 512. in Appendix F: random inputs fail to trigger realistic expert routing in the MoE target model. This leads to inaccurate step latency measurements even without speculation. Optimal DL selection [PITH_FULL_IMAGE:figures/full_fig_p008… view at source ↗
Figure 9
Figure 9. Figure 9: Pairwise similarity matrices for the ’Translation/Multilingual’ category. SpecBench (left) shows dense blocks of high similarity, indicating redundant data. SPEED-Bench (right) shows a dispersed, low-similarity distribution, demonstrating better semantic diversity [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 8
Figure 8. Figure 8: Average AL as a function of ISLs, comparing training ISLs. Target is GPT-OSS 120B with EAGLE3 drafters, measured on vLLM. Dotted lines denote YaRN scaling. Legend labels (1k, 2k, 4k) indicate the maximum ISL used during training. DL = 3. Throughput Split focused on large batches and fixed ISLs, the framework enables the analysis of critical system proper￾ties. Specifically, these splits allow practitioners… view at source ↗
Figure 10
Figure 10. Figure 10: display the pairwise cosine similarity matrices for two categories: Translation/Multilingual and Math, respectively. In these heatmaps, darker green values indicate high semantic similarity (redundancy), while lighter yellow values indicate low similarity (diversity). • SpecBench (Left Column): This figure reveals clusters of highly repetitive prompts (e.g., the same math problem with minor changes, or id… view at source ↗
Figure 9
Figure 9. Figure 9: Pairwise similarity matrices for the ’Translation/Multilingual’ category. SpecBench (left) shows dense blocks of high similarity, indicating redundant data. SPEED-Bench (right) shows a dispersed, low-similarity distribution, demonstrating better semantic diversity [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: illustrates the activation frequency of the top-k experts for a middle layer (Layer 17) in GPT-OSS 120B during the prefill of 8k ISL inputs at a batch size of 32. While SPEED-Bench inputs result in a relatively uniform activation profile, random tokens lead to significant imbalance, where the router disproportionately favors a subset of experts [PITH_FULL_IMAGE:figures/full_fig_p019_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: display the pairwise cosine similarity matrices for two categories: Translation/Multilingual and Math, respectively. In these heatmaps, darker green values indicate high semantic similarity (redundancy), while lighter yellow values indicate low similarity (diversity). • SpecBench (Left Column): This figure reveals clusters of highly repetitive prompts (e.g., the same math problem with minor changes, or id… view at source ↗
Figure 12
Figure 12. Figure 12: tracks the total number of unique experts activated across layers of the model. Notably, processing random tokens fails to activate 20-30% of available experts in certain layers. This lack of coverage is interesting given the high volume of tokens (32 × 8000), confirming that synthetic noise fails to trigger the routing logic that occurs on real semantic workloads. 0 32 64 96 128 10 1 10 3 10 5 10 7 Frequ… view at source ↗
Figure 11
Figure 11. Figure 11: illustrates the activation frequency of the top-k experts for a middle layer (Layer 17) in GPT-OSS 120B during the prefill of 8k ISL inputs at a batch size of 32. While SPEED-Bench inputs result in a relatively uniform activation profile, random tokens lead to significant imbalance, where the router disproportionately favors a subset of experts [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: presents the average AL as a function of ISL for three setups. For Vanilla SD (Llama 3.3 70B) and Native MTP (Qwen3-Next), we observe the expected behavior: Low Entropy prompts (e.g., coding, sorting) yield the highest ALs. High Entropy prompts (e.g., creative writing, roleplay) yield the lowest ALs. Mixed Entropy prompts (e.g., STEM and general knowledge) fall in between. Furthermore, these methods demon… view at source ↗
Figure 12
Figure 12. Figure 12: tracks the total number of unique experts activated across layers of the model. Notably, processing random tokens fails to activate 20-30% of available experts in certain layers. This lack of coverage is interesting given the high volume of tokens (32 × 8000), confirming that synthetic noise fails to trigger the routing logic that occurs on real semantic workloads. 0 32 64 96 128 10 1 10 3 10 5 10 7 Frequ… view at source ↗
Figure 14
Figure 14. Figure 14: Average AL across all categories in SpecBench vs. SPEED-Bench. Target model is Llama 3.3 70B. DL=7, BS=32. L. Inference Engine Comparison In Section 8.4, we briefly discussed the performance differences between inference backends. Here we provide the full comparison between TensorRT-LLM and vLLM [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
Figure 13
Figure 13. Figure 13: presents the average AL as a function of ISL for three setups. For Vanilla SD (Llama 3.3 70B) and Native MTP (Qwen3-Next), we observe the expected behavior: Low Entropy prompts (e.g., coding, sorting) yield the highest ALs. High Entropy prompts (e.g., creative writing, roleplay) yield the lowest ALs. Mixed Entropy prompts (e.g., STEM and general knowledge) fall in between. Furthermore, these methods demon… view at source ↗
Figure 15
Figure 15. Figure 15: compares the throughput of TensorRT-LLM and vLLM. Both frameworks are orchestrated in Python, which can introduce host synchronization overhead and kernel launch latency compared to C++ implementations. To mitigate this, both engines leverage CUDA Graphs to capture and replay device operations with a single launch. We observe that TensorRT-LLM achieves higher throughput in this configuration, largely due … view at source ↗
Figure 14
Figure 14. Figure 14: Average AL across all categories in SpecBench vs. SPEED-Bench. Target model is Llama 3.3 70B. DL=7, BS=32. M. Inference Engine Comparison In Section 8.4, we briefly discussed the performance differences between inference backends. Here we provide the full comparison between TensorRT-LLM and vLLM [PITH_FULL_IMAGE:figures/full_fig_p027_14.png] view at source ↗
Figure 16
Figure 16. Figure 16: AL Stability across various models. Average AL measured on the Throughput Split buckets (1k–32k). Target is GPT-OSS 120B, with three EAGLE3 drafters. Carefully configured RoPE scaling can ensure stability over all context lengths. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_16.png] view at source ↗
Figure 15
Figure 15. Figure 15: compares the throughput of TensorRT-LLM and vLLM. Both frameworks are orchestrated in Python, which can introduce host synchronization overhead and kernel launch latency compared to C++ implementations. To mitigate this, both engines leverage CUDA Graphs to capture and replay device operations with a single launch. We observe that TensorRT-LLM achieves higher throughput in this configuration, largely due … view at source ↗
Figure 16
Figure 16. Figure 16: AL Stability across various models. Average AL measured on the Throughput Split buckets (1k–32k). Target is GPT-OSS 120B, with three EAGLE3 drafters. Carefully configured RoPE scaling can ensure stability over all context lengths. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding

    cs.CL 2026-05 unverdicted novelty 7.0

    PSD is a training-free framework that jointly optimizes spatial unmasking and temporal speculative decoding in diffusion LLMs to reach up to 5.5x tokens per forward pass while preserving accuracy comparable to greedy ...

  2. Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding

    cs.CL 2026-07 accept novelty 6.0

    Joint AR–diffusion training yields one tri-mode LM that switches AR, diffusion, and self-speculation, beating open AR/diffusion models on accuracy and tokens-per-forward.

  3. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

    cs.AI 2026-07 accept novelty 6.0

    DSpark's semi-autoregressive drafter plus load-aware confidence scheduling raises accepted draft length and shifts the production serving Pareto frontier by 60-85% higher per-user speed at matched throughput versus MTP-1.

  4. Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs

    cs.AI 2026-07 conditional novelty 5.5

    Iterative Puzzle plus KD, RL, quantization, and MTP compresses Nemotron-3-Super to 75B total / 9B active parameters with ~2× interactive throughput and 8× 1M-context concurrency while retaining most parent accuracy.

  5. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

  6. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 conditional novelty 3.0

    A survey of large-model inference optimization, organized as a four-layer 'token-operations' taxonomy: multi-model fusion, model optimization, compute-model fusion, and compute-network-model fusion.

Reference graph

Works this paper leans on

57 extracted references · 23 linked inside Pith · cited by 5 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Pard: Accelerating llm inference with low-cost parallel draft model adaptation

    An, Z., Bai, H., Liu, Z., Li, D., and Barsoum, E. Pard: Accelerating llm inference with low-cost parallel draft model adaptation. arXiv preprint arXiv:2504.18583, 2025

    arXiv 2025

  4. [4]

    Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues

    Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Nvidia nemotron 3: Efficient and open intelligence

    Blakeman, A., Grattafiori, A., Basant, A., Gupta, A., Khattar, A., Renduchintala, A., Vavre, A., Shukla, A., Bercovich, A., Ficek, A., et al. Nvidia nemotron 3: Efficient and open intelligence. arXiv preprint arXiv:2512.20856, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Long code arena: a set of benchmarks for long-context code models

    Bogomolov, E., Eliseeva, A., Galimzyanov, T., Glukhov, E., Shapkin, A., Tigina, M., Golubev, Y., Kovrigin, A., van Deursen, A., Izadi, M., and Bryksin, T. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Bojar, O., Buck, C., Federmann, C., Haddow, B., Koehn, P., Leveling, J., Monz, C., Pecina, P., Post, M., Saint-Amand, H., Soricut, R., Specia, L., and Tamchyna, A. s. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp.\ 12--58, Baltimore, Maryland, USA, June 2014. A...

    2014

  8. [8]

    D., Chen, D., and Dao, T

    Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In International Conference on Machine Learning, pp.\ 5209--5235. PMLR, 2024

    2024

  9. [9]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Ch...

    2021

  11. [11]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding

    Chen, Z., May, A., Svirschevski, R., Huang, Y., Ryabinin, M., Jia, Z., and Chen, B. Sequoia: Scalable, robust, and hardware-aware speculative decoding. CoRR, 2024

    2024

  12. [12]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  13. [13]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 3029--3051, 2023

    2023

  14. [14]

    X., and Wen, J.-R

    Dong, Z., Tang, T., Li, J., Zhao, W. X., and Wen, J.-R. Bamboo: A comprehensive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint arXiv:2309.13345, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Break the sequential dependency of llm inference using lookahead decoding

    Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Break the sequential dependency of llm inference using lookahead decoding. In Forty-first International Conference on Machine Learning

  16. [16]

    The llama 3 herd of models, 2024

    Grattafiori, A., Dubey, A., Jauhri, A., and et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  17. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    REST : Retrieval-based speculative decoding

    He, Z., Zhong, Z., Cai, T., Lee, J., and He, D. REST : Retrieval-based speculative decoding. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 1582--1595, Mexico City, Mexico, June 2024. ...

    doi:10.18653/v1/2024.naacl-long.88 2024

  19. [19]

    Moesd: Unveil speculative decoding's potential for accelerating sparse moe

    Huang, Z., Zhu, L., Zhan, Z., Hu, T., Mao, W., Yu, X., Liu, Y., and Zhang, T. Moesd: Unveil speculative decoding's potential for accelerating sparse moe. arXiv preprint arXiv:2505.19645, 2025

    arXiv 2025

  20. [20]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  22. [22]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  23. [23]

    M., Ghaddar, A., Sun, Q., Ma, L., Luo, Y., Li, D., Coates, M., Hao, J., and Zhang, Y

    Li, D., Zhou, J., Brunswic, L. M., Ghaddar, A., Sun, Q., Ma, L., Luo, Y., Li, D., Coates, M., Hao, J., and Zhang, Y. Omni-thinker: Scaling multi-task rl in llms with hybrid reward and task scheduling, 2025 a . URL https://arxiv.org/abs/2507.14783

    arXiv 2025

  24. [24]

    Competition-level code generation with alphacode

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., Hubert, T., Choy, P., de Masson d'Autume, C., Babuschkin, I., Chen, X., Huang, P.-S., Welbl, J., Gowal, S., Cherepanov, A., Molloy, J., Mankowitz, D., Sutherland Robson, E., Kohli, P., de Freitas, N., Kavukcuoglu, K., and Vinyals, O...

    Pith/arXiv arXiv 2022

  25. [25]

    EAGLE : Speculative sampling requires rethinking feature uncertainty

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE : Speculative sampling requires rethinking feature uncertainty. In International Conference on Machine Learning, 2024 a

    2024

  26. [26]

    EAGLE-2 : Faster inference of language models with dynamic draft trees

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE-2 : Faster inference of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing, 2024 b

    2024

  27. [27]

    EAGLE-3 : Scaling up inference acceleration of large language models via training-time test

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE-3 : Scaling up inference acceleration of large language models via training-time test. In Annual Conference on Neural Information Processing Systems, 2025 b

    2025

  28. [28]

    Repobench: Benchmarking repository-level code auto-completion systems, 2024 a

    Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems, 2024 a . URL https://arxiv.org/abs/2306.03091

    Pith/arXiv arXiv 2024

  29. [29]

    Chatqa: Surpassing gpt-4 on conversational qa and rag

    Liu, Z., Ping, W., Roy, R., Xu, P., Lee, C., Shoeybi, M., and Catanzaro, B. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint arXiv:2401.10225, 2024 b

    Pith/arXiv arXiv 2024

  30. [30]

    X., Sha, J., Wang, S., and Wen, J.-R

    Luo, W., Zhao, W. X., Sha, J., Wang, S., and Wen, J.-R. Mmath: A multilingual benchmark for mathematical reasoning. arXiv preprint arXiv:2505.19126, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Vo...

    2024

  32. [32]

    Y., Singh, S., Tang, X., von Werra, L., and Longpre, S

    Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., and Longpre, S. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124, 2023

    Pith/arXiv arXiv 2023

  33. [33]

    Tensorrt‑llm: High‑performance inference for large language models

    NVIDIA . Tensorrt‑llm: High‑performance inference for large language models. https://github.com/NVIDIA/TensorRT-LLM, 2023. Accessed: 2026‑01‑06

    2023

  34. [34]

    gpt-oss-120b & gpt-oss-20b model card, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508.10925

    Pith/arXiv arXiv 2025

  35. [35]

    Paech, S. J. Eq-bench creative writing benchmark v3. https://github.com/EQ-bench/creative-writing-bench, 2025

    2025

  36. [36]

    Mcif: Multimodal crosslingual instruction-following benchmark from scientific talks, 2025

    Papi, S., Züfle, M., Gaido, M., Savoldi, B., Liu, D., Douros, I., Bentivogli, L., and Niehues, J. Mcif: Multimodal crosslingual instruction-following benchmark from scientific talks, 2025. URL https://arxiv.org/abs/2507.19634

    arXiv 2025

  37. [37]

    Ya RN : Efficient context window extension of large language models

    Peng, B., Quesnelle, J., Fan, H., and Shippole, E. Ya RN : Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=wHBfxhZu1u

    2024

  38. [38]

    Humanity's last exam, 2025

    Phan, L., Gatti, A., Han, Z., and et al. Humanity's last exam, 2025. URL https://arxiv.org/abs/2501.14249

    Pith/arXiv arXiv 2025

  39. [39]

    Project gutenberg

    Project Gutenberg . Project gutenberg. https://www.gutenberg.org

  40. [40]

    E.-H., May, A., Chen, T., and Chen, B

    Sadhukhan, R., Chen, J., Chen, Z., Tiwari, V., Lai, R., Shi, J., Yen, I. E.-H., May, A., Chen, T., and Chen, B. Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding. In The Thirteenth International Conference on Learning Representations

  41. [42]

    J., and Manning, C

    See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1073--1083, Vancouver, Canada, July 2017 b . Association for Computational Linguistics. doi:10.18653/v1/P17-1099

    doi:10.18653/v1/p17-1099 2017

  42. [43]

    Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

    2023

  43. [44]

    Ada-leval: Evaluating long-context llms with length-adaptable benchmarks

    Wang, C., Duan, H., Zhang, S., Lin, D., and Chen, K. Ada-leval: Evaluating long-context llms with length-adaptable benchmarks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 3712--3724, 2024 a

    2024

  44. [45]

    Coser: Coordinating llm-based persona simulation of established roles, 2025

    Wang, X., Wang, H., Zhang, Y., Yuan, X., Xu, R., tse Huang, J., Yuan, S., Guo, H., Chen, J., Wang, W., Xiao, Y., and Zhou, S. Coser: Coordinating llm-based persona simulation of established roles, 2025. URL https://arxiv.org/abs/2502.09082

    arXiv 2025

  45. [46]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 0 95266--95290, 2024 b

    2024

  46. [47]

    M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Chen, W., Fu, J., and Peng, J

    Wang, Z. M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Chen, W., Fu, J., and Peng, J. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv: 2310.00746, 2023

    Pith/arXiv arXiv 2023

  47. [48]

    L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conferenc...

    2020

  48. [49]

    Writingbench: A comprehensive benchmark for generative writing, 2025

    Wu, Y., Mei, J., Yan, M., Li, C., Lai, S., Ren, Y., Wang, Z., Zhang, J., Wu, M., Jin, Q., and Huang, F. Writingbench: A comprehensive benchmark for generative writing, 2025. URL https://arxiv.org/abs/2503.05244

    arXiv 2025

  49. [50]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics ACL 2024, pp.\ 7655--7671, Bangkok, Thailand and virtual me...

    doi:10.18653/v1/2024.findings-acl.456 2024

  50. [51]

    Parallelspec: Parallel drafter for efficient speculative decoding

    Xiao, Z., Zhang, H., Ge, T., Ouyang, S., Ordonez, V., and Yu, D. Parallelspec: Parallel drafter for efficient speculative decoding. arXiv preprint arXiv:2410.05589, 2024

    Pith/arXiv arXiv 2024

  51. [52]

    Mimo-v2-flash technical report, 2026

    Xiaomi, L.-C. Mimo-v2-flash technical report, 2026. URL https://arxiv.org/abs/2601.02780

    Pith/arXiv arXiv 2026

  52. [53]

    Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024

    Pith/arXiv arXiv 2024

  53. [54]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025 a

    Pith/arXiv arXiv 2025

  54. [55]

    Longspec: Long-context lossless speculative decoding with efficient drafting and verification

    Yang, P., Du, C., Zhang, F., Wang, H., Pang, T., Du, C., and An, B. Longspec: Long-context lossless speculative decoding with efficient drafting and verification. In ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025 b

    2025

  55. [56]

    Improving massively multilingual neural machine translation and zero-shot translation

    Zhang, B., Williams, P., Titov, I., and Sennrich, R. Improving massively multilingual neural machine translation and zero-shot translation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.\ 1628--1639, Online, July 2020. Association for Computati...

    doi:10.18653/v1/2020.acl-main.148 2020

  56. [57]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

  57. [58]

    L., Huang, J., Yu, C

    Zheng, L., Yin, L., Xie, Z., Sun, C. L., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. Sglang: Efficient execution of structured language model programs. In Conference on Neural Information Processing Systems (NeurIPS), 2024. doi:10.48550/arXiv.2312.07104

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.07104 2024