Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Reinforcement Speculative Decoding for Fast Ranking

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RSD claims RL-trained agents can approximate LLM rankings under a fixed LLM-call budget, beating single-token and speculative baselines.

desk verdict Genuinely new up-to-down speculative decoding for LLM ranking with consistent empirical gains, but the variance-reduction proof in Theorem 3 is invalid as written. read the letter →

arxiv 2505.20316 v1 pith:WCJLVRHP submitted 2025-05-23 cs.AI

Yingpeng Du , Tianjun Wei , Zhu Sun , Jie Zhang This is my paper · ORCID
classification cs.AI
keywords speculativedecodingrankingsystemsreinforcementlearninglistwiselatency-constrainedinferencelargelanguagemodelsinformationretrievalrecommender
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement Speculative Decoding (RSD) is a method for making LLM-based ranking fast enough for latency-sensitive information retrieval and recommendation systems. Instead of generating a full autoregressive ranking or settling for the first token's logits, RSD trains a lightweight agent with reinforcement learning to iteratively modify an initial ranking under a fixed budget of LLM calls. The paper claims RSD outperforms every baseline method in every setting it reports, with Kendall's tau on MS MARCO rising from 0.5801 for the best speculative baseline to 0.7169 for RSD at the same budget $T=5$. It also derives three theoretical properties: the accepted prefix grows monotonically across rounds, the ranking-tailored policy objective reduces to GRPO in a local limit, and the reference baseline yields unbiased gradient estimates with lower variance under a stated condition.

What carries the argument

The load-bearing mechanism is the up-to-down decoding loop: at round $t$, the agent computes the longest prefix $i^*_t$ of the current ranking that matches the target LLM's greedy decoding under a single encoding, keeps that prefix, and samples the remaining items from a Bradley-Terry policy over relevance scores $h_\theta(D|q,s_{<t})$. Those scores come from a Transformer that reads the history of LLM encoding matrices $S_0,\ldots,S_{t-1}$, which captures both token-level and round-level listwise dependencies. Training uses a ranking-tailored policy objective (RPO) with a KL term and an advantage computed either as a group average or against a greedy reference model; Theorem 2 connects RPO to GRPO, while Theorem 3 compares the variance of the two advantage estimators.

What would settle it

Reproduce RSD training on MS MARCO with the same budget $T=5$, and estimate the empirical variance of the reference-advantage gradient and the group-average-advantage gradient; if the reference gradient variance is not smaller whenever the reference reward noise is below the stated threshold, or if RSD's Kendall tau falls below the best baseline on any reported dataset-backbone pair, the paper's central claims would be falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that ranking can be reframed as a bounded multi-round editing problem rather than a left-to-right generation problem. Starting from a first-token ranking, the agent keeps the longest prefix that the target LLM's greedy verification accepts and proposes a new suffix drawn from a Bradley-Terry policy, so each round costs exactly one LLM encoding and the correct top part of the ranking is never discarded. The paper argues this up-to-down paradigm lets the agent exploit listwise ranking knowledge verified by the LLM across all previous rounds, which ordinary speculative decoding throws away when it drops unaccepted tokens. The theoretical claims are that the accepted prefix length strictly increases (Theorem 1), that the ranking-tailored RPO objective is equivalent to GRPO in the limit of small policy ratio changes (Theorem 2), and that a greedy reference model gives an unbiased, lower-variance gradient estimator whenever its reward noise is sufficiently small relative to trajectory noise (Theorem 3).

Load-bearing premise

The load-bearing premise is the independence of the sampled reward-minus-baseline and the policy gradient score in Theorem 3; both are computed from the same ranking trajectory, so if they are correlated, the claimed variance advantage of the reference baseline is unproven.

Editorial extensions

If this is right

  • A ranker can be served under a hard latency cap: the number of LLM encodings is fixed at $T$ instead of growing with the number of candidate items or the number of verification rounds.
  • The target LLM can stay frozen; only the small agent is trained, so RSD avoids the distribution drift that weakens fine-tuned single-token rankers when combined with speculative decoding.
  • Extra budget translates directly into guaranteed improvement in the top of the ranking, because the accepted prefix never shrinks across rounds.
  • The RPO objective's equivalence to GRPO means the training recipe can reuse standard GRPO machinery for the ranking-specific RL loop.
  • The reported RL stage needs very few training instances (under 70 per dataset in their runs), so adapting the agent to a new target LLM or dataset is cheap in data terms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The variance advantage in Theorem 3 rests on an independence assumption between the sampled return-minus-baseline and the policy gradient score; if that assumption fails on real trajectories, the reference baseline's advantage is unproven and RSD's gain would come from RPO and the listwise relevance network instead.
  • Editorial inference: The same bounded-budget edit loop could be applied to other structured outputs that a target model can verify prefix-by-prefix, such as constrained generation or multi-label prediction, not just rankings.
  • Editorial inference: The paper's adaptation experiments show cross-backbone and cross-task transfer is unreliable, so the method as presented is a tailoring plugin rather than a tuning-free universal accelerator.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Reinforcement Speculative Decoding (RSD), an agent-based up-to-down decoding scheme that iteratively modifies a ranking under a fixed LLM call budget T. A lightweight relevance network produces a Bradley–Terry policy over candidate rankings; the agent is first trained by supervised learning and then by a ranking-tailored policy optimization (RPO) claimed to be equivalent to GRPO. The paper derives three theoretical results: monotonic growth of the verified prefix (Theorem 1), equivalence of RPO to GRPO (Theorem 2), and a variance comparison between group-average and reference advantages (Theorem 3). Empirically, RSD is evaluated on MS MARCO and Quora (IR) and ML-1M and Amazon-Games (RS) under two LLM backbones, with ablations, adaptation experiments, and hyperparameter analysis, reporting large and statistically significant gains over STD and SD baselines.

Significance. If the theoretical claims and empirical results hold, RSD is a potentially valuable contribution for latency-constrained LLM-based ranking: it introduces a principled way to use a small RL-trained agent as a speculative drafter while preserving the LLM's greedy ranking as a verification target. The empirical study is broad: two tasks, two backbones, four ranking metrics, multiple SD and STD baselines, ablations, transfer experiments, and budget sensitivity. The paper also states its limitations candidly (modest K, cross-domain degradation). However, the theoretical centerpiece has a serious gap in Theorem 3, and the core equation underlying Theorem 1 appears to contain an indexing error. The empirical results may still be valid, but the advertised theoretical robustness is not currently established. No code is released, limiting reproducibility.

major comments (4)
  1. [§4.1, Eq. (1); Appendix B, Theorem 1] There is an off-by-one inconsistency between the greedy verification rule defined in §3 and the update rule in Eq. (1). In Eq. (1), the first modified token is placed at position i*_t + 1 using the context sigma_{t-1}[: i*_t] (of length i*_t). Under the verification rule in §3, however, the token at position i*_t + 1 must satisfy sigma_t[i*_t + 1] = argmax_d P_llm(d | x0, sigma_{t-1}[: i*_t + 1]), because the prefix of length i*_t + 1, including the already-verified token at position i*_t, is accepted before the next token is checked. The proof of Theorem 1 in Appendix B uses the shorter context and therefore does not establish that sigma_t[i*_t + 1] agrees with greedy decoding at that position. As printed, the monotonicity guarantee does not follow from Eq. (1); either the indexing or the algorithm must be corrected.
  2. [Appendix B, Theorem 3] The variance proof assumes two variables X and Y are independent and then sets X = R_i - B and Y = g_k. For B = mu_hat_{-i}, only the leave-one-out mean is independent of g_k; R_i itself is a deterministic function of the same sampled trajectory sigma^(i), so R_i and g_k are dependent. For B = R_ref, R_ref is independent of the trajectory, but R_i still depends on that trajectory, so R_i - R_ref remains dependent on g_k. Consequently the step E[(g_k)^2 (R_i - B)^2] = E[(g_k)^2] E[(R_i - B)^2] is unjustified, and the reduction of the gradient-variance comparison to Var[R_i - B] does not follow. Section 4.3.2 relies on this reduction when selecting the reference advantage; the w/o RA ablation provides empirical support, but the theoretical variance advantage is currently unproven.
  3. [§4.2, Theorem 2] Theorem 2 is presented as an equivalence between RPO and GRPO, but under the assumption |pi_theta / pi_theta_old - 1| < epsilon the clip and min operators are inactive by construction, so the 'equivalence' reduces to a Taylor expansion and the observation that the ratio lies inside the clipping interval. This is a consistency condition rather than an independent theoretical result. It would be more accurate to present it as a remark showing that RPO reduces to GRPO's first term when the policy ratio is close to one, rather than as a substantive equivalence theorem.
  4. [Table 2 vs. Table 5] The ablation labels in Table 2 appear to be swapped relative to Appendix Table 5. For Llama-3.2-3B on MS MARCO, Table 2 reports w/o LRK = 0.6065 KT / 55.10 FD and w/o RA = 0.6907 KT / 44.80 FD, while Table 5 reports w/o LR = 0.6907 KT / 44.804 FD and w/o RA = 0.6065 KT / 55.104 FD. The same swap appears for Qwen2.5-7B and for Amazon-Games. Since Section 5.2 draws component-level conclusions from these rows ('w/o LRK performs poorly', 'w/o RA shows performance degradation'), the labeling error changes the interpretation of the ablations. The authors must correct the labels and re-derive the ablation conclusions.
minor comments (5)
  1. [§5.1] All experimental results are reported as averages over five runs, but no standard deviations or confidence intervals are given; the significance asterisks only compare RSD against baselines. Please include variability measures so the claimed superiority can be assessed.
  2. [§5.2] Figure 4 references scenarios A, B, C, D, and E, but the caption and text do not define what these scenarios are. Please describe the adaptation settings explicitly.
  3. [Tables 3 and 4] The baseline name 'Mesuda' should be 'Medusa'.
  4. [§4.3.2, Theorem 3] The phrase 'By assigning pi_theta_ref <- pi_theta' is in tension with the subsequent choice of a greedy reference model; please clarify whether the reference is the current stochastic policy or a deterministic greedy policy derived from it, since the variance argument depends on which object sigma_delta^2 refers to.
  5. [§5.1] The source code is stated to be available only when the paper is accepted; since this is a methodology paper with several nonstandard design choices, releasing the code and trained checkpoints with the revision would materially improve reproducibility.

Circularity Check

2 steps flagged · score 4.0 of 10

Empirical core is independently grounded, but two theoretical guarantees are partly constructed: Theorem 1 is a direct consequence of Eq. (1), and Theorem 2 restates GRPO; Theorem 3 also rests on an unverified independence assumption.

  1. self definitional [Section 4.1, Eq. (1) and Theorem 1]
    "To verify the policy σt ∼ πθ(·|q, s<t) is tailored for SD, we prove ... the strict increasing monotonicity of i∗ t for t. Theorem 1. (Monotonicity.) For i∗ T < K, we have i∗ t < i∗ t+1 for all t < T. ... σt[i] = σt−1[i] if i ≤ i∗ t, argmaxPllm(d|x0, σt−1[: i∗ t ]) if i = i∗ t + 1, ˜σt[i] else"

    Equation (1) forces every position up to i∗ t to remain unchanged and places the LLM argmax conditioned on that same prefix at position i∗ t + 1. Therefore the longest greedy-consistent prefix of σt is at least i∗ t + 1 by construction; the 'monotonicity theorem' merely restates this design choice. The paper presents it as a proven guarantee that the policy is tailored for SD, but no alternative behavior is possible under Eq. (1), so the guarantee is built into the definition rather than derived from it.

  2. renaming known result [Section 4.2, RPO objective and Theorem 2]
    "Theoretically, the proposed RPO objective can be seen as the ranking-tailored GRPO objective [32] when πθ/πθold → 1 and G ∼πθold as follows: Theorem 2. If we have πθ/πθold → 1 ... we have our RPO objective is equivalent to GRPO objective (Equation (3) in [32]) in the first term w.r.t. the model parameter θ"

    The RPO objective is introduced as a new ranking-tailored policy optimization, but its functional form (probability-ratio surrogate times advantage plus KL penalty) is exactly GRPO with ranking-specific trajectories. Theorem 2's proof only Taylor-expands log(πθ/πθold) and reinserts the clip/min that are already part of GRPO, so the result is a restatement of [32] rather than an independent theoretical finding. This makes part of the advertised 'theoretical robustness and advantages' a renaming of a known method, although the ranking-specific state design and empirical adaptation do contain independent content.

full rationale

The paper's main load-bearing claim—that RSD significantly outperforms all baselines on IR and RS tasks (Section 5.2, Table 1)—is supported by self-contained experiments with standard benchmarks, multiple metrics, ablations, and statistical tests. The empirical comparisons, including the w/o RA ablation that supports the reference advantage, do not reduce to the paper's own definitions or fitted parameters; therefore the central empirical contribution is not circular. The circularity burden is concentrated in the advertised theoretical framing. Theorem 1 claims monotonic increase of the accepted prefix, but Eq. (1) already forces position i∗_t + 1 to be the LLM argmax over the unchanged prefix, so the theorem restates the construction rather than deriving a consequence. Theorem 2 presents the ranking-tailored RPO objective as a theoretical result, but L_RPO-1 is, by definition, the GRPO surrogate restricted to ranking trajectories; the proof only Taylor-expands the ratio and reinserts GRPO's clip/min. This is a restatement of DeepSeekMath's GRPO, not an independent derivation. A further non-circular but important gap appears in Theorem 3 (Appendix B): the variance comparison uses Var[XY]=E[X^2]E[Y^2]−(E[XY])^2 and assigns X=R_i−B, Y=g_k. Since R_i and g_k are both computed from the same sampled trajectory, the required independence fails for both B=µ̂−i and B=R_ref; the factorization is unjustified and the stated variance advantage is not proven. This is a correctness issue, not a circularity issue, and it does not affect the empirical ablation that independently favors the reference advantage. The paper's self-citations are to the authors' prior ranking/RS work and are used as background, not as load-bearing premises. The two definitional/renaming steps partially inflate the 'theoretical robustness' claim, but the empirical core remains independent, so the overall circularity score is 4 out of 10.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The method's learning recipe rests on standard causal-LLM verification and RLHF-style policy optimization. No new physical or conceptual entities are introduced; the agent and relevance network are learned models, not postulated entities. The main extra assumptions are the Bradley-Terry ranking model, the local ratio assumption for GRPO equivalence, and the independence plus Gaussian decomposition used in the Theorem 3 proof; the latter is not justified and is generally false as stated.

free parameters (4)
  • Budget T = 5
    Number of LLM encoding calls per query; fixed by hand to meet the latency requirement and used for all comparisons.
  • KL coefficient beta = 0.1
    Coefficient for the KL penalty in the RPO objective; selected via hyperparameter analysis (Figure 5).
  • Group size G = not reported
    Number of trajectories sampled per query in RPO; affects advantage estimation and is needed for reproduction but not specified.
  • Relevance network size = single-layer Transformer, 5 heads, hidden dimension 25
    Agent architecture chosen by hand and listed in Section 5.1.
assumptions (5)
  • domain assumption Greedy auto-regressive decoding of the target LLM defines the desired target ranking sigma_tr.
    Section 3 adopts greedy decoding 'for reproducibility'; all rewards and target labels are derived from it, so the method optimizes fidelity to greedy decoding.
  • standard math One LLM encoding of a ranking sequence provides all prefix-conditional next-item probabilities via the causal structure.
    Section 4.1 relies on this to compute Pllm(d|x0, sigma_{t-1}[:i]) for all i with a single call; true for causal attention models.
  • domain assumption The agent's ranking distribution is a Bradley-Terry factorization over pairwise preferences.
    Section 4.1 defines pi_theta(sigma|q,s) as a product of sigmoid pairwise probabilities; this restricts the policy family and may limit expressiveness.
  • ad hoc to paper The policy ratio pi_theta / pi_theta_old is close to 1 during optimization (Theorem 2).
    Used to equate RPO to GRPO via Taylor expansion; no trust-region mechanism is described that guarantees this in the algorithm.
  • ad hoc to paper Return and reference reward decompose into independent Gaussian components, and the advantage baseline is independent of the score function (Theorem 3).
    Assumed in the variance proof; the independence of R_i - B and g_k is not shown and generally fails for samples from the same trajectory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reinforcement Speculative Decoding for Fast Ranking." pith.science (2026). https://pith.science/paper/WCJLVRHP

@misc{pith2026250520316,
  author       = {Pith},
  title        = {Pith review of: Reinforcement Speculative Decoding for Fast Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WCJLVRHP}},
  note         = {Machine review of arXiv:2505.20316}
}
read the original abstract

Large Language Models (LLMs) have been widely adopted in ranking systems such as information retrieval (IR) systems and recommender systems (RSs). To alleviate the latency of auto-regressive decoding, some studies explore the single (first) token decoding for ranking approximation, but they suffer from severe degradation in tail positions. Although speculative decoding (SD) methods can be a remedy with verification at different positions, they face challenges in ranking systems due to their left-to-right decoding paradigm. Firstly, ranking systems require strict latency constraints, but verification rounds in SD methods remain agnostic; Secondly, SD methods usually discard listwise ranking knowledge about unaccepted items in previous rounds, hindering future multi-token prediction, especially when candidate tokens are the unaccepted items. In this paper, we propose a Reinforcement Speculative Decoding method for fast ranking inference of LLMs. To meet the ranking systems' latency requirement, we propose an up-to-down decoding paradigm that employs an agent to iteratively modify the ranking sequence under a constrained budget. Specifically, we design a ranking-tailored policy optimization, actively exploring optimal multi-round ranking modification policy verified by LLMs via reinforcement learning (RL). To better approximate the target LLM under the constrained budget, we trigger the agent fully utilizing the listwise ranking knowledge about all items verified by LLMs across different rounds in RL, enhancing the modification policy of the agent. More importantly, we demonstrate the theoretical robustness and advantages of our paradigm and implementation. Experiments on both IR and RS tasks show the effectiveness of our proposed method.

Figures

Figures reproduced from arXiv: 2505.20316 by the authors.

Figure 1
Figure 1. Comparison among different methods and decod￾ing strategies on MS MARCO dataset with Llama3.2-3B￾Instruct, where SD requires 5×LLM encoding [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison between existing left-to-right and our up-to-down speculative decoding. knowledge of listwise rankings verified by LLMs across different rounds in RL, capturing token￾and round-level dependency in item rankings for agent policy enhancement. More importantly, we theoretically elucidate the relationship between our method and established RL methodologies, demonstrating the theoretical robustness and advanta… view at source ↗
Figure 3
Figure 3. Performance of RSD and w/o RA in training process [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Investigate on adaptation ability of RSD. Investigate on adaptation ability of the agent to different scenarios (RQ3). In this part, we aim to investigate the adaptation ability of RSD. Specifically, we train an agent based on specific data and a backbone, evaluating i…
Figure 5
Figure 5. Figure 5: Hyper-parameters analysis on (a) KL coefficient β and (b) budget. Hyper-parameter analysis (RQ4). In this part, we investigate how hyperparameters influence the performance of the proposed RSD, including the KL coefficient β and budget T. As depicted in 10 [PITH_FULL_…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    CMA uses diffusion-generated counterfactuals and two-player Shapley values to measure whether an image or the text drives a multimodal LLM's prediction, hitting 98% on synthetic biased benchmarks.

Reference graph

Works this paper leans on

48 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    Learning to rank using gradient descent

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pages 89–96, 2005

    work page 2005

  2. [2]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. Medusa: Simple llm inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, pages 5209–5235, 2024

    work page 2024

  3. [3]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    arXiv 2023

  4. [4]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding

    Zhuoming Chen, Avner May, Ruslan Svirschevski, Yuhsun Huang, Max Ryabinin, Zhihao Jia, and Beidi Chen. Sequoia: Scalable, robust, and hardware-aware speculative decoding. arXiv preprint arXiv:2402.12374, 2024

    arXiv 2024

  5. [5]

    Cascade speculative drafting for even faster llm inference

    Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Kevin Chang, and Jie Huang. Cascade speculative drafting for even faster llm inference. Advances in Neural Information Processing Systems, 37:86226–86242, 2024

    work page 2024

  6. [6]

    Glide with a cape: a low-hassle method to accelerate speculative decoding

    Cunxiao Du, Jing Jiang, Xu Yuanchen, Jiawei Wu, Sicheng Yu, Yongqi Li, Shenggui Li, Kai Xu, Liqiang Nie, Zhaopeng Tu, et al. Glide with a cape: a low-hassle method to accelerate speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, pages 11704–11720, 2024

    work page 2024

  7. [7]

    Quasi- metric learning for bilateral person-job fit

    Yingpeng Du, Hongzhi Liu, Hengshu Zhu, Yang Song, Zhi Zheng, and Zhonghai Wu. Quasi- metric learning for bilateral person-job fit. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    work page 2025

  8. [8]

    Enhancing job recommendation through llm-based generative adversarial networks

    Yingpeng Du, Di Luo, Rui Yan, Xiaopei Wang, Hongzhi Liu, Hengshu Zhu, Yang Song, and Jie Zhang. Enhancing job recommendation through llm-based generative adversarial networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 8363–8371, 2024

    2024

Show all 48 references
  1. [9]

    Active large language model-based knowledge distillation for session-based recommendation

    Yingpeng Du, Zhu Sun, Ziyan Wang, Haoyan Chua, Jie Zhang, and Yew-Soon Ong. Active large language model-based knowledge distillation for session-based recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 11607–11615, 2025

    2025

  2. [10]

    Break the sequential dependency of llm inference using lookahead decoding

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. Break the sequential dependency of llm inference using lookahead decoding. In International Conference on Machine Learning, pages 14060–14079. PMLR, 2024

    2024

  3. [11]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [12]

    Rest: Retrieval-based speculative decoding

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. Rest: Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pag...

    2024

  5. [13]

    Speed: Speculative pipelined execution for efficient decoding

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310.12072, 2023

    2023 arXiv

  6. [14]

    Large language models are zero-shot rankers for recommender systems

    Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. Large language models are zero-shot rankers for recommender systems. In European Conference on Information Retrieval, pages 364–381. Springer, 2024. 12

    2024

  7. [15]

    Neural input search for large scale recommendation models

    Manas R Joglekar, Cong Li, Mei Chen, Taibai Xu, Xiaoming Wang, Jay K Adams, Pranav Khaitan, Jiahui Liu, and Quoc V Le. Neural input search for large scale recommendation models. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining...

    2020

  8. [16]

    Speculative decoding with big little decoder

    Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. Advances in Neural Information Processing Systems, 36:39236–39256, 2023

    2023

  9. [17]

    Ancestral gumbel-top-k sampling for sampling without replacement

    Wouter Kool, Herke Van Hoof, and Max Welling. Ancestral gumbel-top-k sampling for sampling without replacement. Journal of Machine Learning Research, 21(47):1–36, 2020

    2020

  10. [18]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023

    2023

  11. [19]

    Eagle: speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. In Proceedings of the 41st International Conference on Machine Learning, pages 28935–28948, 2024

    2024

  12. [20]

    Generalized ambiguity decomposition for ranking ensemble learning

    Hongzhi Liu, Yingpeng Du, and Zhonghai Wu. Generalized ambiguity decomposition for ranking ensemble learning. Journal of Machine Learning Research, 23(88):1–36, 2022

    2022

  13. [21]

    Online speculative decoding

    Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding. In Proceedings of the 41st International Conference on Machine Learning, pages 31131–31146, 2024

    2024

  14. [22]

    Ranked list truncation for large language model-based re-ranking

    Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, and Maarten de Rijke. Ranked list truncation for large language model-based re-ranking. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 14...

    2024

  15. [23]

    Specinfer: Accelerating large language model serving with tree-based speculative inference and verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. InProceedings of the...

    2024

  16. [24]

    Pass: Parallel speculative sampling

    Giovanni Monea, Armand Joulin, and Edouard Grave. Pass: Parallel speculative sampling. arXiv preprint arXiv:2311.13581, 2023

    2023 arXiv

  17. [25]

    Machine learning: a probabilistic perspective

    Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012

    2012

  18. [26]

    Ms marco: A human-generated machine reading comprehension dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human-generated machine reading comprehension dataset. 2016

    2016

  19. [27]

    Top-down partitioning for efficient list-wise ranking

    Andrew Parry, Sean MacAvaney, and Debasis Ganguly. Top-down partitioning for efficient list-wise ranking. arXiv preprint arXiv:2405.14589, 2024

    2024 arXiv

  20. [28]

    Rankvicuna: Zero-shot listwise document reranking with open-source large language models

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise document reranking with open-source large language models. arXiv preprint arXiv:2309.15088, 2023

    2023 arXiv

  21. [29]

    Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! arXiv preprint arXiv:2312.02724, 2023

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! arXiv preprint arXiv:2312.02724, 2023

    2023 arXiv

  22. [30]

    First: Faster improved listwise reranking with single token decoding

    Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. First: Faster improved listwise reranking with single token decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8642–865...

    2024

  23. [31]

    Accelerating transformer inference for translation via parallel decoding

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, Emanuele Rodola, et al. Accelerating transformer inference for translation via parallel decoding. In Proceedings of the 61st Annual Meeting of the Association for Computa-...

    2023

  24. [32]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  25. [33]

    Accelerating llm inference with staged speculative decoding

    Benjamin Frederick Spector and Christopher Re. Accelerating llm inference with staged speculative decoding. In Workshop on Efficient Systems for Foundation Models@ ICML2023

  26. [34]

    Blockwise parallel decoding for deep autoregressive models

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31, 2018

    2018

  27. [35]

    Spectr: Fast speculative decoding via optimal transport

    Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36:30222–30242, 2023

    2023

  28. [36]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  29. [37]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  30. [38]

    Re2llm: Reflective reinforcement large language model for session-based recommendation

    Ziyan Wang, Yingpeng Du, Zhu Sun, Haoyan Chua, Kaidong Feng, Wenya Wang, and Jie Zhang. Re2llm: Reflective reinforcement large language model for session-based recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12827–12835, 2025

    2025

  31. [39]

    A survey on large language models for recommendation

    Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, et al. A survey on large language models for recommendation. World Wide Web, 27(5):60, 2024

    2024

  32. [40]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909–3925, 2023

    2023

  33. [41]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In ACL (Findings), 2024

    2024

  34. [42]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  35. [43]

    Multi-candidate speculative decoding

    Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. Multi-candidate speculative decoding. arXiv preprint arXiv:2401.06706, 2024

    2024 arXiv

  36. [44]

    Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding.Transactions on Machine Learning Research

    Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding.Transactions on Machine Learning Research

  37. [45]

    Draft& verify: Lossless large language model acceleration via self-speculative decoding

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft& verify: Lossless large language model acceleration via self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2024

  38. [46]

    Opt: Open pre-trained transformer language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 14

    2022 arXiv

  39. [47]

    Distillspec: Improving speculative decoding via knowledge distillation

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Repres...

  40. [48]

    Large language models for information retrieval: A survey

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107, 2023. A Prompt for LLMs in IR and RS tasks Prompt ...

    1987

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.