Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes AdaDecode, which claims to speed up LLM generation by up to 1.73x while producing exactly the same tokens as standard autoregressive decoding.

desk verdict A plausible and useful vertical-acceleration method, but the output-parity guarantee needs an explicit rollback path before the 1.73x speedup can be trusted. read the letter →

arxiv 2506.03700 v1 pith:KIYECSHS submitted 2025-06-04 cs.CL

Zhepei Wei , Wei-Lin Chen , Xinyu Zhu , Yu Meng This is my paper
classification cs.CL
keywords LLMdecodingaccelerationearlyexitinglayerparallelismkey-valuecacheoutputparityverificationthroughputspeeduplongchain-of-thoughtgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AdaDecode is trying to establish that the sequential dependency of autoregressive decoding can be relaxed without changing what the model generates. Its lever is the observation that many tokens—common words, predictable continuations—can be predicted correctly before the transformer has run through all its layers. The method adds small trainable LM heads at intermediate layers, keeps the base model frozen, and uses a confidence threshold to decide when a token can be predicted early; the skipped layers' key-value computations are deferred and scheduled to run in parallel with the next token, and a verification step guarantees the final output matches standard autoregressive decoding. On long-content generation tasks the paper reports up to 1.73x higher throughput with identical output, which would make long chain-of-thought and long-form generation faster without touching the model's weights.

What carries the argument

The load-bearing mechanism is a confidence-gated early exit with deferred KV-cache computation and a final verification pass. Candidate intermediate layers each carry a lightweight LM head, trained by a KL-divergence loss to match the final layer's prediction while the transformer weights stay frozen; for Llama3.1-8B these heads add roughly 48M parameters total. When a head's confidence passes a threshold, the next token's forward pass begins immediately, and the uncomputed layers of the earlier token are executed later, in parallel with the next token's layers, so the KV cache becomes complete without adding a sequential round trip. The verification step then confirms that every early prediction equals what standard autoregressive decoding would have produced.

What would settle it

Run AdaDecode and standard autoregressive decoding on the same prompts with greedy sampling and compare the emitted tokens byte-for-byte; any divergence would refute the parity claim. Separately, plot throughput against generated-sequence length: if the speedup falls toward 1x as the KV cache grows, then the deferred-layer overlap is not materializing on that hardware.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that you can 'skip' upper layers for low-information tokens and still reproduce exactly the tokens standard autoregressive decoding would produce. AdaDecode trains lightweight LM heads on frozen intermediate-layer hidden states to imitate the final layer's next-token distribution, exits early when a head's confidence is high, and then computes the skipped layers' KV cache entries lazily, in parallel with the forward passes of subsequent tokens. When all deferred work is completed, AdaDecode verifies the early predictions against the full-model result; since the full model state is eventually reconstructed, the output guarantee is exact token-level parity, not approximate agreement. The paper reports up to 1.73x speedup across generation tasks and positions the method as vertical acceleration that is orthogonal to horizontal methods such as speculative decoding.

Load-bearing premise

The speedup rests on the assumption that the postponed upper-layer work for early-predicted tokens can be scheduled alongside later tokens without materially slowing them down; if the hardware is already saturated or attention over long past text becomes the bottleneck, the wall-clock gain shrinks even though the output remains identical.

Editorial extensions

If this is right

  • If the claimed parity holds, AdaDecode can be used in quality-sensitive deployments without a correctness-versus-speed tradeoff.
  • Long chain-of-thought and other long-output workloads would gain up to roughly 1.7x more tokens per second on the same GPU, with the base model unchanged.
  • Deployment avoids the need for a separately trained draft model and the requirement that the drafter share the main model's tokenizer and vocabulary.
  • Because the method accelerates within each decoding step, it can be composed with horizontal acceleration methods such as speculative decoding for further combined gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's reported measurements, the practical speedup is likely to be largest on text with many high-confidence tokens, so the method's headroom on hard reasoning tokens is narrower; an adaptive per-task threshold is a natural extension the paper does not explore.
  • The verification step has a cost that the throughput numbers must absorb; at very low early-exit rates AdaDecode would spend extra compute on heads and verification for no gain, degenerating toward standard decoding with overhead.
  • If confirmed on other architectures, the same early-exit-plus-deferred-KV idea could apply to any transformer with accessible intermediate hidden states, including encoder-decoder and large mixture-of-experts models, though memory-bound attention behavior would have to be re-measured.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes AdaDecode, a decoding-time acceleration method for LLMs that introduces lightweight trainable LM heads at intermediate layers, uses a confidence threshold to early-exit tokens at those layers, defers the skipped upper-layer KV-cache computations, and runs them in parallel with the processing of subsequent tokens. The authors claim up to 1.73x throughput improvement while preserving exact output parity with standard autoregressive decoding, without auxiliary drafter models or modification of the base model parameters.

Significance. If the output-parity guarantee can be made rigorous, this is a practically attractive direction: it avoids the memory overhead of a separate drafter, keeps the base model frozen, adds only small trainable heads (48M parameters in their estimate), and targets vertical acceleration that is orthogonal to speculative decoding. The explicit goal of output parity, the parameter-freezing design, and the public code link are strengths. However, the central guarantee is currently not substantiated by the described algorithm, and the speedup claim depends on hardware assumptions that are not quantified.

major comments (3)
  1. [Section 2.2 and Figure 1] The claimed output parity is not established by the described procedure. The paper states that AdaDecode "predicts tokens using the hidden state at intermediate layers when confidence is high, and immediately initiates processing of the next token," and that the remaining layers' KV computations are deferred and performed in parallel. Verification happens only after those deferred computations complete. If an early prediction differs from the final-layer greedy prediction, all subsequently initiated tokens have already been conditioned on a non-autoregressive prefix. The verification step can detect this mismatch but cannot undo its effect on the already-computed continuation. No rollback, rejection, resampling, or recomputation procedure is described anywhere in Section 2. Since a finite confidence threshold on a learned intermediate head cannot make the argmax-mismatch probability zero, the abstract's claim of guaranteeing output parity is either false as stated or relies on an unspecified repair mechanism whose cost must be included in the reported speedup. Please provide the exact algorithm for the mismatch case, including what token is emitted and how the already-started continuation is recomputed.
  2. [Section 2.2 and experimental claims] The speedup claim depends on the assumption that deferred upper-layer KV computations for early-exited tokens can overlap with later-token computation without material slowdown. The paper does not describe the scheduling mechanism, the hardware assumptions (e.g., available streaming multiprocessors or memory bandwidth), or how the verification and any rollback costs are accounted for in the wall-clock time. The reported up-to-1.73x figure should be accompanied by a decomposition of time spent on early-exit computation, deferred-layer completion, verification, and any repair steps, ideally with an ablation that removes the parallelism to show the contribution of overlap.
  3. [Section 2.1, Eq. (1)] The intermediate heads are trained solely by KL divergence toward the final-layer distribution. This objective does not imply that high-confidence argmax decisions of the intermediate head agree with the final-layer argmax; the two distributions can differ on individual inputs even when the predicted probability of the top token is high. Consequently, the confidence threshold cannot by itself provide the claimed parity guarantee, and the paper must rely on the verification mechanism. Measured mismatch rates (early-exit accuracy and final-layer agreement under the chosen threshold) should be reported to quantify how often the verification step is triggered, since that frequency directly controls both parity and speedup.
minor comments (4)
  1. [Figure 1] The caption is difficult to parse: the phrase "same-colored layers run in parallel" is not defined in the text, and the repeated "Verify" labels do not indicate whether verification is performed per-token or per-batch. A pseudocode listing or a more explicit timing diagram would clarify the method.
  2. [Section 1 and Table 1] The terms "horizontal acceleration" and "vertical acceleration" are used in the introduction and Table 1, but the formal definitions appear only in a later paragraph; they should be defined at first use.
  3. [Section 2.1] The claim that the lightweight LM heads use "31x fewer parameters" than prior fully parameterized heads needs a direct citation or a concrete comparison table; as written, the reader cannot verify the baseline size being compared.
  4. [Abstract and Section 3] There are minor grammatical issues, e.g., "Experiments across diverse generation tasks shows" should be "show," and the phrase "ensuring output consistency" is used before the precise notion of parity is defined.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: both the speedup and parity claims are anchored to external benchmarks and measured quantities, not to fitted inputs or self-citation chains.

full rationale

AdaDecode's claims are (i) a wall-clock throughput speedup and (ii) exact output parity with standard autoregressive decoding. Claim (i) is a measured engineering result: throughput is benchmarked against baselines, and the early-exit rate under a confidence threshold is an empirical property of the trained heads, not a fitted quantity that is later relabeled as a prediction. Claim (ii) is anchored to an external target: the intermediate LM heads are trained by KL divergence to the final-layer distribution p*(t|h*) of the unmodified model, and parity is asserted against the standard autoregressive decoder, not against a quantity defined by AdaDecode itself. The confidence threshold is a hyperparameter and does not define the parity metric. No load-bearing self-citation chain appears; citations to prior early-exit and speculative-decoding work supply background rather than the central guarantee. One non-circular concern is present: Section 2 says "Once all KV cache computations are complete, the standard autoregressive decoding result is obtained, allowing us to verify the correctness of early predictions," but the paper does not describe a rollback or recomputation procedure if an early prediction fails verification; this is an omitted mechanism for the parity claim, not a case of the claim reducing to its own inputs by construction. Under the standard for this review (quote-based circularity, not correctness), the circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The method rests on one standard deterministic property, two empirical or hardware assumptions, and one trained add-on component. No new physical or mathematical entities are postulated; the contribution is a scheduling and training scheme, not a claim about the world outside the model.

free parameters (2)
  • Early-exit confidence threshold = Not reported in visible text
    Controls the tradeoff between early-exit rate and verification failures; the claimed speedup depends on its chosen value.
  • Candidate early-exit layer set = Not reported in visible text
    Placement and number of intermediate LM heads (48M total parameters per Table 1) are selected by the authors and affect how many tokens can exit early.
assumptions (3)
  • domain assumption Transformer forward pass is deterministic given inputs and KV cache, so deferred layers reproduce exactly the standard KV cache.
    Needed for the deferred-parallel computation to preserve output parity; implicit in the design described in Section 2.2.
  • domain assumption Intermediate-layer LM heads trained on a corpus to minimize KL divergence to the final layer generalize to the evaluation tasks.
    The early-exit acceptance rate on test data depends on this; Figure 3c is reported as qualitative evidence.
  • domain assumption The hardware has sufficient compute and memory-bandwidth headroom to overlap deferred-layer computation with subsequent-token computation.
    If the GPU is saturated or the growing KV cache dominates, the parallel execution yields little or no wall-clock benefit.
invented entities (1)
  • Intermediate-layer lightweight LM heads (theta(i))
    purpose: Produce next-token distributions from hidden states at candidate layers, enabling early prediction while the base model stays frozen.
    They are new trainable components introduced by the paper; they have no falsifiable handle outside the paper and are fitted to the model's own final-layer predictions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism." pith.science (2026). https://pith.science/paper/KIYECSHS

@misc{pith2026250603700,
  author       = {Pith},
  title        = {Pith review of: AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KIYECSHS}},
  note         = {Machine review of arXiv:2506.03700}
}
read the original abstract

Large language models (LLMs) are increasingly used for long-content generation (e.g., long Chain-of-Thought reasoning) where decoding efficiency becomes a critical bottleneck: Autoregressive decoding is inherently limited by its sequential token generation process, where each token must be generated before the next can be processed. This sequential dependency restricts the ability to fully leverage modern hardware's parallel processing capabilities. Existing methods like speculative decoding and layer skipping offer potential speedups but have notable drawbacks: speculative decoding relies on an auxiliary "drafter" model, which can be challenging to acquire and increases memory overhead, while layer skipping may introduce discrepancies in the outputs due to the missing key-value cache at skipped layers. In this work, we propose AdaDecode, which accelerates LLM decoding without requiring auxiliary models or changes to the original model parameters, while ensuring output consistency. AdaDecode leverages the insight that many tokens can accurately be generated at intermediate layers, as further layers often do not significantly alter predictions once the model reaches a certain confidence. By adaptively generating tokens at intermediate layers when confidence is high, AdaDecode enables the next token's computation to begin immediately. The remaining layer computations for early-predicted tokens are deferred and executed in parallel with subsequent tokens when needed, maximizing hardware utilization and reducing decoding latency. A final verification step ensures that early predictions match the results of standard autoregressive decoding, preserving output parity. Experiments across diverse generation tasks shows that AdaDecode consistently achieves superior decoding throughput with up to 1.73x speedup, while guaranteeing output parity with standard autoregressive decoding.

Figures

Figures reproduced from arXiv: 2506.03700 by the authors.

Figure 1
Figure 1. (Left) Speculative decoding relies on an auxiliary drafter model, leading to increased memory usage and requiring the same tokenizer and vocabulary as the main model. (Middle) Layer skipping bypasses certain layers, which results in missing KV cache at those layers and can introduce discrepancies in future token predictions. (Right) AdaDecode (Ours) accelerates decoding by adaptively predicting future tokens early b… view at source ↗
Figure 2
Figure 2. (Left): A simplified example demonstrating how early predictions enable parallelization. Here, t1 triggers the early prediction of t2 at Layer 2, and t2 triggers the early prediction of t3 at Layer 1. As a result, the Layer 2 computations of t2 and t3 can run in parallel, followed by parallelized Layer 3 computations for t1, t2, and t3. (Right): Vanilla autoregressive decoding processes tokens strictly in sequence, … view at source ↗
Figure 3
Figure 3. Probabilities of model-generated tokens predicted at the 8th, 16th, 24th, and 32nd (final) layers of the fine-tuned Llama-3.1-8B￾Instruct are shown using (a) the original last-layer language model (LM) head and (b) our newly introduced lightweight LM heads. These new LM heads are trained to minimize the KL divergence loss relative to the final layer predictions, while keeping all original model parameters frozen. Th… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Consistency ratios and speedups of different methods vs. vanilla decoding on HumanEval with CodeLlama-34B-Instruct. to higher speedup for SpecDecode compared to a moderate￾size model (e.g., CodeLlama-13B-Instruct). This is mainly because smaller drafters have fewer par…
Figure 5
Figure 5. Figure 5: Hyperparamter study of AdaDecode by varying the early prediction threshold γ. These figures present the evaluation results on HumanEval, including (a) the speedup curve, (b) the early prediction rate, and (c) the verification rejection rate. 4.3. Ablation Study As show…
Figure 6
Figure 6. Figure 6: Hyperparameter study of baseline models with CodeLlama-34B as the backbone model. (a) SpecDecode with CodeLlama-7B as drafter. (b) SpecDecode with CodeLlama-13B as drafter. (c) Self-SpecDecode optimized with Bayesian optimization (BO) search. Mathmatical reasoning. We …

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HiSpec: Hierarchical Speculative Decoding for LLMs

    cs.CL 2025-10 conditional novelty 6.0 of 10

    HiSpec uses early-exit layers to verify draft tokens midway through the model, reporting 1.28×-2.01× faster decoding over baseline speculative decoding, but the accuracy claim is not empirically tested.

  2. ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs

    cs.CV 2026-08 conditional novelty 5.0 of 10

    ParVL scales MLLM computation by running multiple prefix-conditioned ViT and LLM branches over a shared backbone, improving average benchmark scores by 0.3 to 0.9 points and showing task-dependent vision-language allocation.

Reference graph

Works this paper leans on

71 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Hydra: Sequentially-dependent draft heads for medusa decoding

    Ankner, Z., Parthasarathy, R., Nrusimha, A., Rinard, C., Ragan-Kelley, J., and Brandon, W. Hydra: Sequentially-dependent draft heads for medusa decoding. In First Conference on Language Modeling, 2024

    work page 2024

  4. [4]

    Anthropic: Introducing claude 3.5 sonnet, 2024

    Anthropic. Anthropic: Introducing claude 3.5 sonnet, 2024. URL https://www.anthropic.com/news/claude-3-5-sonnet

    work page 2024

  5. [5]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  6. [6]

    Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding

    Bae, S., Ko, J., Song, H., and Yun, S.-Y. Fast and robust early-exiting framework for autoregressive language models with synchronized parallel decoding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 5910--5924, 2023

    2023

  7. [7]

    LongWriter : Unleashing 10,000+ word generation from long context llms

    Bai, Y., Zhang, J., Lv, X., Zheng, L., Zhu, S., Hou, L., Dong, Y., Tang, J., and Li, J. LongWriter : Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055, 2024

    arXiv 2024

  8. [8]

    V., R \'e , C., and Mirhoseini, A

    Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., R \'e , C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787, 2024

    arXiv 2024

Show all 71 references
  1. [9]

    D., Chen, D., and Dao, T

    Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple LLM inference acceleration framework with multiple decoding heads. In Forty-first International Conference on Machine Learning, 2024

    2024

  2. [10]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    2023 arXiv

  3. [11]

    WAPITI : A watermark for finetuned open-source LLMs

    Chen, L., Qiu, R., Yuan, S., Liu, Z., Wei, T., Yoo, H., Zeng, Z., Yang, D., and Tong, H. WAPITI : A watermark for finetuned open-source LLMs . arXiv preprint arXiv:2410.06467, 2024

    2024 arXiv

  4. [12]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  5. [13]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  6. [14]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  7. [15]

    DeepSeek LLM : Scaling open-source language models with longtermism

    DeepSeek. DeepSeek LLM : Scaling open-source language models with longtermism. ArXiv, abs/2401.02954, 2024

    2024 arXiv

  8. [16]

    DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    Skipdecode: Autoregressive skip decoding with batching and caching for efficient LLM inference

    Del Corro, L., Del Giorno, A., Agarwal, S., Yu, B., Awadallah, A., and Mukherjee, S. Skipdecode: Autoregressive skip decoding with batching and caching for efficient LLM inference. arXiv preprint arXiv:2307.02628, 2023

    2023 arXiv

  10. [18]

    QLoRA : Efficient finetuning of quantized LLMs

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA : Efficient finetuning of quantized LLMs . In NeurIPS, 2023

    2023

  11. [19]

    Y., Karidi, T., Choshen, L., and Geva, M

    Din, A. Y., Karidi, T., Choshen, L., and Geva, M. Jump to conclusions: Short-cutting transformers with linear transformations. ArXiv, abs/2303.09435, 2023

    2023 arXiv

  12. [20]

    Glide with a cape: A low-hassle method to accelerate speculative decoding

    Du, C., Jiang, J., Xu, Y., Wu, J., Yu, S., Li, Y., Li, S., Xu, K., Nie, L., Tu, Z., and You, Y. Glide with a cape: A low-hassle method to accelerate speculative decoding. In ICML, 2024

    2024

  13. [21]

    The Llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  14. [22]

    Depth-adaptive transformer

    Elbayad, M., Gu, J., Grave, E., and Auli, M. Depth-adaptive transformer. In International Conference on Learning Representations, 2020

    2020

  15. [23]

    L ayer S kip: Enabling early exit inference and self-speculative decoding

    Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., Aly, A., Chen, B., and Wu, C.-J. L ayer S kip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of...

    2024

  16. [24]

    Break the sequential dependency of LLM inference using lookahead decoding

    Fu, Y., Bailis, P., Stoica, I., and Zhang, H. Break the sequential dependency of LLM inference using lookahead decoding. In Forty-first International Conference on Machine Learning, 2024

    2024

  17. [25]

    Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space

    Geva, M., Caciularu, A., Wang, K., and Goldberg, Y. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In EMNLP, 2022

    2022

  18. [26]

    L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M

    Guan, X., Zhang, L. L., Liu, Y., Shang, N., Sun, Y., Zhu, Y., Yang, F., and Yang, M. rStar-Math : Small LLMs can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519, 2025

    2025 arXiv

  19. [27]

    REST : Retrieval-based speculative decoding

    He, Z., Zhong, Z., Cai, T., Lee, J., and He, D. REST : Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp.\ 1582--1595, 2024

    2024

  20. [28]

    A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vin...

    2022 arXiv

  21. [29]

    The curious case of neural text degeneration

    Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In ICLR, 2020

    2020

  22. [30]

    Speed: Speculative pipelined execution for efficient decoding

    Hooper, C., Kim, S., Mohammadzadeh, H., Genc, H., Keutzer, K., Gholami, A., and Shao, S. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310.12072, 2023

    2023 arXiv

  23. [31]

    E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W

    Hu, J. E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. LoRA : Low-rank adaptation of large language models. In ICLR, 2022

    2022

  24. [32]

    Efficient test-time scaling via self-calibration

    Huang, C., Huang, L., Leng, J., Liu, J., and Huang, J. Efficient test-time scaling via self-calibration. arXiv preprint arXiv:2503.00031, 2025

    2025 arXiv

  25. [33]

    Multi-scale dense networks for resource efficient image classification

    Huang, G., Chen, D., Li, T., Wu, F., van der Maaten, L., and Weinberger, K. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations, 2018

    2018

  26. [34]

    Specdec++: Boosting speculative decoding via adaptive candidate lengths

    Huang, K., Guo, X., and Wang, M. Specdec++: Boosting speculative decoding via adaptive candidate lengths. arXiv preprint arXiv:2405.19715, 2024

    2024 arXiv

  27. [35]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ArXiv, 2023

    2023

  28. [36]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  29. [37]

    Sigsoftmax: Reanalysis of the softmax bottleneck

    Kanai, S., Fujiwara, Y., Yamanaka, Y., and Adachi, S. Sigsoftmax: Reanalysis of the softmax bottleneck. In Neural Information Processing Systems, 2018

    2018

  30. [38]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  31. [39]

    A comprehensive survey of accelerated generation techniques in large language models

    Khoshnoodi, M., Jain, V., Gao, M., Srikanth, M., and Chadha, A. A comprehensive survey of accelerated generation techniques in large language models. arXiv preprint arXiv:2405.13019, 2024

    2024 arXiv

  32. [40]

    W., Gholami, A., and Keutzer, K

    Kim, S., Mangalam, K., Moon, S., Malik, J., Mahoney, M. W., Gholami, A., and Keutzer, K. Speculative decoding with big little decoder. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  33. [41]

    Kimi K1.5 : Scaling reinforcement learning with LLMs

    Kimi, Du, A., Gao, B., Xing, B., Jiang, C., Chen, C., Li, C., Xiao, C., Du, C., Liao, C., et al. Kimi K1.5 : Scaling reinforcement learning with LLMs . arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  34. [42]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  35. [43]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  36. [44]

    EAGLE : Speculative sampling requires rethinking feature uncertainty

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE : Speculative sampling requires rethinking feature uncertainty. In Forty-first International Conference on Machine Learning, 2024 a

    2024

  37. [45]

    EAGLE -2: Faster inference of language models with dynamic draft trees

    Li, Y., Wei, F., Zhang, C., and Zhang, H. EAGLE -2: Faster inference of language models with dynamic draft trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 7421--7432, Miami, Florida, USA, November 2024 b . Association for ...

    2024

  38. [46]

    Kangaroo: Lossless self-speculative decoding via double early exiting

    Liu, F., Tang, Y., Liu, Z., Ni, Y., Han, K., and Wang, Y. Kangaroo: Lossless self-speculative decoding via double early exiting. arXiv preprint arXiv:2404.18911, 2024 a

    2024 arXiv

  39. [47]

    Speculative decoding via early-exiting for faster LLM inference with T hompson sampling control mechanism

    Liu, J., Wang, Q., Wang, J., and Cai, X. Speculative decoding via early-exiting for faster LLM inference with T hompson sampling control mechanism. In Findings of the Association for Computational Linguistics ACL 2024, pp.\ 3027--3043, Bangkok, Thailand and virtual meeting, Au...

    2024

  40. [48]

    Online speculative decoding

    Liu, X., Hu, L., Bailis, P., Stoica, I., Deng, Z., Cheung, A., and Zhang, H. Online speculative decoding. arXiv preprint arXiv:2310.07177, 2023

    2023 arXiv

  41. [49]

    A., Adkathimar, R., Wei, T., and Tong, H

    Liu, Z., Amjad, R. A., Adkathimar, R., Wei, T., and Tong, H. SelfElicit : Your language model secretly knows where is the relevant evidence. arXiv preprint arXiv:2502.08767, 2025

    2025 arXiv

  42. [50]

    Towards efficient generative large language model serving: A survey from algorithms to systems

    Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Jin, H., Chen, T., and Jia, Z. Towards efficient generative large language model serving: A survey from algorithms to systems. arXiv preprint arXiv:2312.15234, 2023 a

    2023 arXiv

  43. [51]

    Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y. Y., Zhu, A., Yang, L., Shi, X., et al. SpecInfer : Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint arXiv:2305.09781, 2023 b

    2023 arXiv

  44. [52]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.\ 1797--1807, 2018

    2018

  45. [53]

    Introducing OpenAI o1: Learning to reason with large language models, 2024

    OpenAI. Introducing OpenAI o1: Learning to reason with large language models, 2024. URL https://openai.com/index/learning-to-reason-with-llms/

    2024

  46. [54]

    M., Sun, S., and Iyyer, M

    Pham, C. M., Sun, S., and Iyyer, M. Suri: Multi-constraint instruction following for long-form text generation. arXiv preprint arXiv:2406.19371, 2024

    2024 arXiv

  47. [55]

    Optimized multi-token joint decoding with auxiliary model for LLM inference

    Qin, Z., Hu, Z., He, Z., Prakriya, N., Cong, J., and Sun, Y. Optimized multi-token joint decoding with auxiliary model for LLM inference. arXiv preprint arXiv:2407.09722, 2024

    2024 arXiv

  48. [56]

    Zero: Memory optimizations toward training trillion parameter models

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020

    2020

  49. [57]

    C., and Santoro, A

    Raposo, D., Ritter, S., Richards, B., Lillicrap, T., Humphreys, P. C., and Santoro, A. Mixture-of-Depths : Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258, 2024

    2024 arXiv

  50. [58]

    Confident adaptive language modeling

    Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., and Metzler, D. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35: 0 17456--17472, 2022

    2022

  51. [59]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024

    2024 arXiv

  52. [60]

    Blockwise parallel decoding for deep autoregressive models

    Stern, M., Shazeer, N., and Uszkoreit, J. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018

    2018

  53. [61]

    Branchynet: Fast inference via early exiting from deep neural networks

    Teerapittayanon, S., McDanel, B., and Kung, H.-T. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp.\ 2464--2469. IEEE, 2016

    2016

  54. [62]

    Accelerating Llama inference by enabling intermediate layer decoding via instruction tuning with LITE

    Varshney, N., Chatterjee, A., Parmar, M., and Baral, C. Accelerating Llama inference by enabling intermediate layer decoding via instruction tuning with LITE . arXiv preprint arXiv:2310.18581, 2023

    2023 arXiv

  55. [63]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851, 2024 a

    2024 arXiv

  56. [64]

    SWIFT : On-the-fly self-speculative decoding for LLM inference acceleration

    Xia, H., Li, Y., Zhang, J., Du, C., and Li, W. SWIFT : On-the-fly self-speculative decoding for LLM inference acceleration. In ICLR, 2025

    2025

  57. [65]

    Sheared LLaMA : Accelerating language model pre-training via structured pruning

    Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared LLaMA : Accelerating language model pre-training via structured pruning. In ICLR, 2024 b

    2024

  58. [66]

    Predictive pipelined decoding: A compute-latency trade-off for exact LLM decoding

    Yang, S., Lee, G., Cho, J., Papailiopoulos, D., and Lee, K. Predictive pipelined decoding: A compute-latency trade-off for exact LLM decoding. Transactions on Machine Learning Research, 2024. ISSN 2835-8856

    2024

  59. [67]

    Yang, Z., Dai, Z., Salakhutdinov, R., and Cohen, W. W. Breaking the softmax bottleneck: A high-rank RNN language model. In ICLR, 2018

    2018

  60. [68]

    C o S afe: Evaluating large language model safety in multi-turn dialogue coreference

    Yu, E., Li, J., Liao, M., Wang, S., Zuchen, G., Mi, F., and Hong, L. C o S afe: Evaluating large language model safety in multi-turn dialogue coreference. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 17494--17508, 2024

    2024

  61. [69]

    L., Ma, Z., Xue, Y., Zhai, J., Chen, W., Liu, Z., Zhang, P., Dong, Y., and Tang, J

    Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., Tam, W. L., Ma, Z., Xue, Y., Zhai, J., Chen, W., Liu, Z., Zhang, P., Dong, Y., and Tang, J. GLM -130b: An open bilingual pre-trained model. In The Eleventh International Conference on...

    2023

  62. [70]

    Draft & verify: Lossless large language model acceleration via self-speculative decoding

    Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape...

    2024

  63. [71]

    SafetyBench : Evaluating the safety of large language models with multiple choice questions

    Zhang, Z., Lei, L., Wu, L., Sun, R., Huang, Y., Long, C., Liu, X., Lei, X., Tang, J., and Huang, M. SafetyBench : Evaluating the safety of large language models with multiple choice questions. In ACL, 2024 b

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.