Pith. sign in

REVIEW 4 major objections 4 minor 41 references

SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that a judge verifier for speculative decoding can be trained from the target model's own likelihoods, removing the need for human labels or verifiable ground truth.

desk verdict SelfJudge has a genuinely new labeling idea and mostly good results, but the 'no ground truth' claim does not survive contact with Section 4.1, where the threshold tau is calibrated using AutoJudge answer labels. read the letter →

arxiv 2510.02329 v2 pith:AQ3KOVKA submitted 2025-09-26 cs.CL cs.AI

Kanghoon Yoon , Minsub Kim , Sungjae Lee , Joonhyung Lee , Sunghyeon Woo , Yeonjun In , Se Jung Kwon , Chanyoung Park
show 1 more author
Dongsoo Lee
This is my paper · ORCID
classification cs.CLcs.AI
keywords speculativedecodingjudgeverificationself-supervisedlabelingsemanticpreservationlikelihoodratioLLMinferenceaccelerationlogisticregressionverifier
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SelfJudge proposes that the verifier used in speculative decoding can be trained automatically from the target model's own judgments rather than from human annotations or task-specific answers. The paper defines a semantic preservation score that compares the target model's likelihood of its original response with the likelihood after replacing one token with the draft token, and labels a replacement as acceptable when the difference is small. These labels, computed offline, train a lightweight logistic-regression verifier on the target model's hidden states; at inference the verifier accepts draft tokens that preserve meaning, and any rejected tokens still go through standard rejection sampling. On math, code, general QA, and summarization tasks, the paper reports more accepted tokens per verification cycle than exact-alignment baselines while losing less task accuracy. If this holds, judge-based speculative decoding stops being limited to domains with checkable answers and becomes a general speed-up technique.

What carries the argument

The central object is the semantic preservation score $s(y, z_i)$, a bidirectional likelihood comparison defined as the log-probability of the draft's alternative token minus the log-probability of the target's own token, both conditioned on the full original response around the position. Because autoregressive models cannot condition on the suffix directly, the score is rewritten as a prefix score plus a suffix likelihood ratio (a log Bayes factor), and computed by feeding the token-substituted response through the target model. This score generates the automatic training labels; the learned logistic regression on hidden states is what makes the score available online, when future tokens are unknown. A two-stage verification scheme, apply the judge first and then apply alignment-based rejection sampling to rejected tokens, keeps the method from discarding valid tokens.

What would settle it

Run SelfJudge on a held-out set of prompts, record every draft token the verifier accepts, and then ask independent human raters or a strong judge model whether replacing that token with the target model's own token changes the meaning of the final response; if a large fraction of accepted substitutions are meaning-changing, the self-supervised labels are not transferring to online contexts.

Watch

Extended reading notes

Core claim

The central claim is that token-level acceptance in speculative decoding can be grounded in semantic preservation measured by the target model itself. For each position where the draft model's top token differs from the target token, SelfJudge computes $s(y,z_i) = \log P(z_i \mid y_{<i}, y_{>i}) - \log P(y_i \mid y_{<i}, y_{>i})$, decomposes it into a prefix log-probability difference plus a suffix log Bayes factor, and labels the replacement acceptable when $s$ exceeds a threshold. The labels become training data for a logistic regression verifier that reads the target model's hidden states during inference. The reported result is that this self-supervised verifier accepts semantically equivalent paraphrases and formatting variants, producing higher accepted length with smaller accuracy degradation than answer-preservation judges across diverse NLP tasks.

Load-bearing premise

The load-bearing premise is that a verifier trained on labels derived from the target model's complete responses will judge draft tokens correctly during inference, when it sees only a partial causal context and no future tokens; the paper does not directly measure this train and inference gap.

Editorial extensions

If this is right

  • Judge verification becomes applicable to open-ended tasks like creative writing and instruction following, where no ground-truth answer exists to label accept or reject decisions.
  • Training verifier data is cheaper than answer-preservation labeling because SelfJudge prefills a short suffix instead of generating an entire alternative completion for every token.
  • A fixed lightweight verifier trained on one mixture of tasks transfers to held-out math, code, QA, and summarization benchmarks within the same target and draft model pair.
  • Because rejected tokens are still checked by rejection sampling, the method bounds quality loss: the only relaxed decisions are the ones the judge accepts, so accuracy degradation is controlled by the judge's precision.
  • End-to-end inference latency improves whenever the draft model produces semantically equivalent but lexically different tokens, which is common for sentence starters, punctuation, synonyms, and formatting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would measure how well the verifier's online accept or reject decisions match the offline semantic preservation labels when the context contains previously accepted draft tokens; large disagreement would indicate that future-token supervision does not transfer to causal decoding contexts.
  • The same labeling procedure could be used to fine-tune or distill the draft model itself toward tokens the target model judges as semantically acceptable, potentially compounding the speedup by making drafts more acceptable before verification.
  • Because the score's suffix term is a Bayes factor, one could weight it by the mutual information between the token and future context, or tune the suffix length per position, rather than using a fixed $N=20$ for all tokens.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SelfJudge, a method for training a judge verifier for speculative decoding without human annotations or verifiable ground truth. The method labels mismatched draft tokens by computing a semantic preservation score, defined as the target-model log-likelihood difference between a token-substituted response and the original response, decomposed into prefix and suffix terms (Eqs. 6-8); tokens above a threshold τ are labeled acceptable. A logistic-regression verifier is trained on target-model hidden states with these labels, and at inference it accepts draft tokens when the verifier score exceeds θ, with a two-stage fallback to standard rejection sampling for rejected tokens. Experiments on Llama-3.1-8B/Llama-3.2-1B and Qwen-2.5-7B/0.5B over GSM8K, MATH-500, LiveCodeBench, CNN/DailyMail, and MMLU report average accepted lengths and task accuracies, claiming superior accuracy-efficiency trade-offs over SD, top-k verification, AutoJudge, and AutoJudge+.

Significance. The contribution is potentially significant. If the claims hold, SelfJudge would remove the task-specific supervision bottleneck of judge decoding: the target model itself generates token-level acceptability labels, the labeling procedure is substantially cheaper than AutoJudge's (8 hours versus 82 hours on the reported workloads), and the verifier is a simple logistic regressor that operates on frozen hidden states. The decomposition of the semantic preservation score into a prefix term and a suffix log Bayes factor is a useful formalization, and the empirical results show SelfJudge achieving higher or comparable accepted lengths than AutoJudge while retaining accuracy across math, code, summarization, and general QA, including an MMLU setting where AutoJudge degrades. The paper also provides a threshold-sweep analysis rather than only a single operating point. These strengths make the work worth pursuing.

major comments (4)
  1. [§4.1, Implementation Details] The central claim that SelfJudge trains verifiers without human annotations or verifiable ground truth is undercut by the threshold calibration: τ is set on 100 GSM8K queries using AutoJudge's answer-preservation labels, specifically τ = quantile(semantic preservation scores of unacceptable tokens determined by AutoJudge, 0.1), with the stated goal of guaranteeing that all tokens rejected by AutoJudge are also rejected by SelfJudge. As a result, the 69,432 training labels inherit the answer-based supervision that the paper claims to avoid, and the evaluation is entangled with the AutoJudge baseline it compares against. Moreover, no self-contained procedure is provided for choosing τ on tasks without verifiable answers, such as CNN/DailyMail or MMLU, so the claimed cross-task generality of 'automatic verifier training' is not demonstrated. The authors should add a sensitivity analysis that selects τ without AutoJudge labels, for example a fixed quantile of the score distribution or a validation set from Dolly15k, and report whether the Table 1 conclusions hold.
  2. [§3.3.2, Eq. (6); §3.4 and §4.1] There is a train/inference distribution shift that the paper does not directly test. Training labels are computed with access to N future tokens y_{>i} of the target-complete response, with N=20 in the experiments, while at inference the verifier must judge acceptability from causal hidden states of a draft-token context (Eq. 4), where previously accepted draft tokens may already deviate from the target distribution. The two-stage rejection sampling can mask verifier errors, so end-to-end accuracy alone does not show that the verifier's ranking is reliable under this shift. I ask for a direct diagnostic, such as verifier precision/recall against oracle labels on actual draft contexts, the score distribution for tokens that are later rejected by the fallback, and an evaluation with stronger draft models or longer generation horizons to test whether the trade-off degrades.
  3. [Table 1 and Table 2; Section 4.2] The headline aggregate Δm/Δtask comparison is not a like-for-like summary. SelfJudge uses 69,432 training labels versus 14,896 for AutoJudge and 39,069 for AutoJudge+ (Table 2), and the averaged row hides AutoJudge-R's collapse to 0.7 Pass@1 on LiveCodeBench, which inflates its average accepted length; SelfJudge's aggregate, by contrast, includes a LiveCodeBench row with 10.0 Pass@1. The paper should report per-task trade-offs, compare against an AutoJudge+ variant trained with the same Dolly15k data, and provide variance or repeated-run statistics for the small accuracy differences before claiming a consistent superior trade-off.
  4. [Throughout, especially §4.2 and Figure 3] The paper reports average accepted length m as the sole efficiency metric and never reports wall-clock latency or the overhead of the logistic verifier. Since the title and abstract claim 'faster speculative decoding,' the method should be evaluated in end-to-end tokens/sec or latency; if accepted length is meant to be the only efficiency metric, that should be explicitly stated and justified.
minor comments (4)
  1. [§4.1, Table 2, Table 3] Model names are inconsistent: Qwen-2.5-8B appears in §4.1 and Table 1, while Qwen-2.5-7B appears in Table 2 and §4.3; Table 3 uses Llama3.2-0.5B where the text reports Llama3.2-1B. Please harmonize all model names.
  2. [Equation (7)] Equation (7) contains notation typos, including the terms 'y_i>i' and the aligned cancellation symbols, which are hard to parse; please rewrite with clean y_{<i} and y_{>i} notation.
  3. [Tables 4-6 and §4.2] The header 'Freuquently' in Tables 4-6 is misspelled and should be 'Frequently'; the column header 'OriginAlter' should be spaced; and 'Observartion1' in §4.2 should be 'Observation1.'
  4. [Appendix B.1, AutoJudge+ description] The AutoJudge+ labeling procedure using LLMs-as-judges is not described with enough detail to be reproducible; please provide the exact judge prompt, the [SAME]/[DIFFERENT] criterion, and the model used for labeling.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the semantic-preservation score is an exact likelihood decomposition, and the verifier is trained on target-model-generated labels; the AutoJudge-calibrated threshold in Section 4.1 is a methodological caveat, not a circular reduction.

full rationale

The paper's derivation chain is self-contained. Equation (7) decomposes the bidirectional log-likelihood difference exactly into prefix and suffix terms; this is algebra, not an assumption. Labels are generated by thresholding the semantic preservation score s(y,z_i) computed from the target model's own likelihoods, and the logistic verifier is trained on those labels and evaluated on external benchmarks (GSM8K, MATH-500, LiveCodeBench, MMLU, CNN/DM). The verifier is not defined in terms of the reported acceptance lengths or task accuracies, so the reported predictions are not equal to the training inputs by construction. The only load-bearing external input is Section 4.1's choice of threshold: 'we set τ = quantile(semantic preservation scores of unacceptable tokens determined by AutoJudge, 0.1)'. This uses AutoJudge's answer-based labels on 100 GSM8K queries to set the labeling threshold, which weakens the claim that the method requires no verifiable ground truth and entangles the AutoJudge comparison. However, this is a hyperparameter calibration rather than a circular derivation in which the predicted quantity equals an input by construction; the speed/accuracy trade-offs remain empirical outcomes contingent on that calibration. This caveat belongs under correctness or novelty risk, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical or model entities are introduced; the judge verifier is a standard logistic regression. The main free parameters are the labeling threshold, suffix length, verifier threshold, and regularization. The core assumptions are about the semantics-likelihood link and train/inference transfer, neither of which is proven independently.

free parameters (4)
  • tau (token labeling threshold) = quantile of AutoJudge-unacceptable token scores on 100 GSM8K queries, 0.1 (numeric value not reported)
    Used to label tokens as acceptable or unacceptable during verifier training; chosen using AutoJudge labels, coupling the method to a baseline that requires ground-truth answers.
  • N (suffix length for semantic preservation score) = 20
    Figure 4 shows the highest accuracy at N=20; this is a hyperparameter fitted to validation performance.
  • theta_R and theta_F (verifier acceptance thresholds) = selected for best recall (-R) and best F1 (-F) on a holdout validation set
    Controls the accuracy-efficiency trade-off; reported results depend on this choice and are presented separately for recall- and F1-optimized variants.
  • L2 regularization C for logistic regression = grid search over 10 log-spaced values in [0.001, 100], best ROC-AUC
    Hyperparameter of the logistic regression verifier, selected on validation data.
assumptions (4)
  • domain assumption The target model's likelihood difference between an original response and a token-substituted response quantifies semantic preservation.
    Core labeling premise in Section 3.3.1; if false, the labels do not reflect meaning preservation and the verifier would not preserve response quality.
  • domain assumption In natural language, the suffix y_>i almost always provides information about the correct token at position i, so bidirectional context reduces uncertainty.
    Used to justify the semantic preservation score and the Appendix A proof; it is asserted as a well-established empirical fact rather than derived.
  • standard math Conditional mutual information identity I(A;B|C) = H(A|C) - H(A|B,C).
    Used in Appendix A for Theorem 1; standard information theory, not in question.
  • domain assumption The logistic regression verifier trained on target-response hidden states will generalize to draft-token contexts at inference.
    Assumed in Sections 3.4 and 4.1; not directly validated, and the train/inference distribution shift is a key risk to the central claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification." pith.science (2026). https://pith.science/paper/AQ3KOVKA

@misc{pith2026251002329,
  author       = {Pith},
  title        = {Pith review of: SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AQ3KOVKA}},
  note         = {Machine review of arXiv:2510.02329}
}
read the original abstract

Speculative decoding accelerates LLM inference by verifying candidate tokens from a draft model against a larger target model. Recent judge decoding boosts this process by relaxing verification criteria by accepting draft tokens that may exhibit minor discrepancies from target model output, but existing methods are restricted by their reliance on human annotations or tasks with verifiable ground truths, limiting generalizability across diverse NLP tasks. We propose SelfJudge, which trains judge verifiers via self-supervision of the target model. Our method measures semantic preservation by assessing whether token-substituted responses preserve the meaning of original responses, enabling automatic verifier training across diverse NLP tasks. Our experiments show SelfJudge achieves superior inference-accuracy trade-offs than judge decoding baselines, offering a broadly applicable solution for faster LLM inference.

Figures

Figures reproduced from arXiv: 2510.02329 by the authors.

Figure 1
Figure 1. Inference efficiency and task performance comparison of SD methods on GSM8K (a,b) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The training data generation process of SelfJudge for the verifier. Our approach com￾pares the likelihood of the replaced response with the original response to measure the semantic preservation score. If semantic preservation score is higher than τ , the replaced token is labeled as acceptable. After the token labeling process, we train the verifier that will be used during the inference phase for draft verificatio… view at source ↗
Figure 3
Figure 3. Speed/Performance comparison across different methods. We report the accuracy, with [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance of SelfJudge over a range of suffix length N. We compute the semantic score by including the likelihood computed on N future tokens. identical to the original. This fundamental difference significantly impacts the robustness when the verifier encounters out…
Figure 5
Figure 5. Figure 5: ROC curve of our judge verifier. We train each verifier for Llama-3.1-8B/Llama-3.2-1B [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 12 canonical work pages

  1. [1]

    Judge decoding: Faster speculative sampling requires going beyond model alignment

    Gregor Bachmann, Sotiris Anagnostidis, Albert Pumarola, Markos Georgopoulos, Artsiom Sanakoyeu, Yuming Du, Edgar Sch \"o nfeld, Ali Thabet, and Jonas K Kohler. Judge decoding: Faster speculative sampling requires going beyond model alignment. In The Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  2. [2]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=PEpbUobfJv

    2024

  3. [3]

    Accelerating large language model decoding with speculative sampling, 2023

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023. URL https://arxiv.org/abs/2302.01318

    arXiv 2023

  4. [4]

    Measuring massive multitask language understanding, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, , and John Schulman. Measuring massive multitask language understanding, 2021. URL arXiv:2110.14168, 2021

    arXiv 2021

  5. [5]

    Gemini2.5: Our newest gemini model with thinking., 2025

    Google DeepMind. Gemini2.5: Our newest gemini model with thinking., 2025. URL https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/

    work page 2025

  6. [6]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

    doi:10.18653/v1/ 2019

  7. [7]

    Break the sequential dependency of llm inference using lookahead decoding, 2024

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding, 2024. URL https://arxiv.org/abs/2402.02057

    arXiv 2024

  8. [8]

    Autojudge: Judge decoding without manual annotation, 2025

    Roman Garipov, Fedor Velikonivtsev, Ruslan Svirschevski, Vage Egiazarian, and Max Ryabinin. Autojudge: Judge decoding without manual annotation, 2025. URL https://arxiv.org/abs/2504.20039

    arXiv 2025

Show all 41 references
  1. [9]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  2. [10]

    REST : Retrieval-based speculative decoding

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. REST : Retrieval-based speculative decoding. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hum...

    2024 doi

  3. [11]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  4. [12]

    Rossi, Yihan Wu, Dinesh Manocha, and Heng Huang

    Zhengmian Hu, Tong Zheng, Vignesh Viswanathan, Ziyi Chen, Ryan A. Rossi, Yihan Wu, Dinesh Manocha, and Heng Huang. Towards optimal multi-draft speculative decoding. In ICLR, 2025

    2025

  5. [13]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learn...

    2025

  6. [14]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  7. [15]

    Mahoney, Amir Gholami, and Kurt Keutzer

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W. Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY,...

    2023

  8. [16]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. S umma C : Re-visiting NLI -based models for inconsistency detection in summarization. Transactions of the Association for Computational Linguistics, 10: 0 163--177, 2022. doi:10.1162/tacl_a_00453. URL http...

    2022 doi

  9. [17]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  10. [18]

    Eagle: speculative sampling requires rethinking feature uncertainty

    Li, Yuhui, Wei, Fangyun, Zhang, Chao, Zhang, and Hongyang. Eagle: speculative sampling requires rethinking feature uncertainty. In International Conference on Macihne Learning, ICML'24. JMLR.org, 2024 a

    2024

  11. [19]

    Llms-as-judges: A comprehensive survey on llm-based evaluation methods, 2024 b

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Llms-as-judges: A comprehensive survey on llm-based evaluation methods, 2024 b . URL https://arxiv.org/abs/2412.05579

    2024 arXiv

  12. [20]

    Eagle-3: Scaling up inference acceleration of large language models via training-time test, 2025

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle-3: Scaling up inference acceleration of large language models via training-time test, 2025. URL https://arxiv.org/abs/2503.01840

    2025 arXiv

  13. [21]

    Let's verify step by step, 2023

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050

    2023 arXiv

  14. [22]

    Online speculative decoding

    Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  15. [23]

    Turning trash into treasure: Accelerating inference of large language models with token recycling, 2025

    Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, and Dongliang Xu. Turning trash into treasure: Accelerating inference of large language models with token recycling, 2025. URL https://arxiv.org/abs/2408.08696

    2025 arXiv

  16. [24]

    Specinfer: Accelerating large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tr...

    2024

  17. [25]

    Abstractive text summarization using sequence-to-sequence RNN s and beyond

    Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, C a g lar Gu l c ehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequence RNN s and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning . Association for Computa...

    2016 doi

  18. [26]

    Faster cascades via speculative decoding

    Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net...

    2025

  19. [27]

    Suffixdecoding: Extreme speculative decoding for emerging ai applications, 2025

    Gabriele Oliaro, Zhihao Jia, Daniel Campos, and Aurick Qiao. Suffixdecoding: Extreme speculative decoding for emerging ai applications, 2025. URL https://arxiv.org/abs/2411.04975

    2025

  20. [28]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024

  21. [29]

    Gonzalez

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive API s. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=tBRNC6YemY

    2024

  22. [30]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI, 2019

    2019

  23. [31]

    Blockwise parallel decoding for deep autoregressive models

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso...

    2018

  24. [32]

    Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=HVK6nl3i97

    2024

  25. [33]

    Spectr: fast speculative decoding via optimal transport

    Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: fast speculative decoding via optimal transport. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023....

    2023

  26. [34]

    Mmlu-pro: a more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: a more robust and challenging multi-task language unders...

    2025

  27. [35]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Sys...

    2022

  28. [36]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  29. [37]

    Distillspec: Improving speculative decoding via knowledge distillation

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Fran c ois Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Repres...

    2024

  30. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  31. [39]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  32. [40]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  33. [41]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.