Pith. sign in

REVIEW 5 major objections 5 minor 36 references

Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Collaborative Speculative Decoding fuses two LLMs at token level using only confidence scores, claiming accuracy gains up to 10% with near-identical latency.

desk verdict A simple, honest, training-free fusion method that works on average, but the headline gain is overstated and the hyperparameter selection is partly in-sample. read the letter →

arxiv 2502.08020 v2 pith:KCNYP4KA submitted 2025-02-11 cs.CL cs.AI

Ziyao Wang , Muneeza Azmat , Ang Li , Raya Horesh , Mikhail Yurochkin This is my paper · ORCID
classification cs.CLcs.AI
keywords collaborativespeculativedecodingLLMknowledgefusiontest-timetoken-levelconfidencedecisiontreemodelensemble
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that two large language models with complementary strengths can be fused during decoding by comparing, for each token, only the probability each model assigns to its own prediction. The method, Collaborative Speculative Decoding (CoSD), has a fast draft model generate tokens and a second assistant model verify them in parallel; when the models disagree, a simple rule or a small trained decision tree decides whether to replace the draft token with the assistant's token. The authors report that this improves average accuracy by up to 10% over existing fusion methods on five benchmarks and six model pairs, while adding almost no latency relative to standard speculative decoding. The significance is that users could combine models through APIs without retraining, hidden states, or white-box access, and the decision process stays human-readable.

What carries the argument

The load-bearing object is the per-token replacement decision. CoSD represents each disagreement by the ordered pair of probabilities $(M_p(\tilde{x}_{t+i}), M_q(\hat{x}_{t+i}))$; rule-based verification applies the threshold test with hyperparameters $\alpha$ (draft uncertainty) and $\beta$ (required confidence advantage), while tree-based verification fits a decision tree $T: \mathbb{R}^2 \to \{0,1\}$ on these pairs with ground-truth labels from a small dataset. The redraft-from-replaced-token loop is what makes the fusion and the efficiency work: only a few tokens are regenerated per iteration, and the assistant model verifies in parallel.

What would settle it

A benchmark constructed so that the assistant model is systematically overconfident on wrong answers while the draft model is underconfident but correct would falsify the core assumption: CoSD would measurably replace correct draft tokens with incorrect assistant tokens. Concretely, one could take the assistant's wrong answers from the failure mode in Table 9, group by replacement decisions, and check whether replaced tokens are correct less often than the confidence gap predicts; a dataset where replacement rate correlates with error rather than correctness would settle it.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that token-level confidence comparison is enough to fuse LLM knowledge at test time. Given a draft token $\tilde{x}_{t+i}$ from model $M_p$ and an assistant token $\hat{x}_{t+i}$ from $M_q$, CoSD replaces the draft only when the tokens differ, when $M_p(\tilde{x}_{t+i}) < \alpha$, and when $M_q(\hat{x}_{t+i}) > \beta \cdot M_p(\tilde{x}_{t+i})$; the tree-based variant replaces the threshold rule with a decision tree trained on the same two probabilities and labels derived from ground-truth tokens. When a replacement occurs, all following draft tokens are discarded and the draft model regenerates from the assistant's token, so the final sequence is a collaboration rather than a straight choice between two full outputs. The reported experiments show CoSD matching or exceeding the better model on individual tasks across complementary knowledge, catastrophic forgetting recovery, capacity imbalance, and different tokenizers.

Load-bearing premise

The whole mechanism assumes a model's confidence in a token is a trustworthy guide to whether that token is correct: when the assistant is more confident than the draft model, the draft token is replaced, so if confidence and correctness were unrelated the method would have no principled basis for choosing.

Editorial extensions

If this is right

  • CoSD can be used on a pair of models with different tokenizers: the draft text is decoded and re-encoded for the assistant, and the reported results still beat char-wise ensemble decoding.
  • The same $\alpha=0.5$, $\beta=0.5$ rule transfers across tasks and model pairs without per-task tuning, so an ordinary user could deploy it with two API endpoints and no training.
  • Tree-based CoSD transfers across domains even when trained on three out-of-distribution samples, and training a tree costs almost no compute.
  • When one model is fine-tuned and suffers catastrophic forgetting, CoSD-Rule recovers and even improves on both the base and fine-tuned scores, suggesting fusion can serve as forgetfulness repair.
  • In the capacity-imbalance case, CoSD approaches the assistant's quality while keeping the draft model's speed, so fusion and acceleration are compatible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that because the decision tree uses only two scalar inputs, one could extend it to a small set of features—context length, entropy of the next-token distribution, or the assistant's calibration error—without losing transparency; the failure mode shown in Table 9 suggests such features would help.
  • The confidence-as-correctness assumption is testable as a calibration question: applying temperature scaling or other calibration to both models before CoSD's comparison could either sharpen the reported gains or expose exactly where the proxy breaks.
  • Going beyond the paper, CoSD's replacement rule resembles a Bayes-factor test with a cutoff; if model probabilities were well calibrated, the rule-based version would be a principled model-selection procedure, and its success across benchmarks gives indirect evidence about how informative LLM confidence scores really are.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes Collaborative Speculative Decoding (CoSD), a decoding-time method for fusing two large language models. A draft model generates a block of tokens; an assistant model verifies them in parallel; a rule-based or decision-tree-based verification step decides, from the two token-level probabilities, whether to accept a draft token or replace it with the assistant's token. The method is evaluated on six model pairs spanning complementary knowledge, catastrophic forgetting recovery, capacity imbalance, and different tokenizers, across MMLU, GSM8K, HumanEval, Hellaswag, and TruthfulQA. The paper claims accuracy improvements up to 10% over existing methods, comparable efficiency to speculative decoding, transferability of hyperparameters and trained trees, and explainability of the fusion decisions.

Significance. If the claims hold, CoSD would be a simple, training-free (rule-based) or very-light-training (tree-based) way to improve accuracy by combining complementary LLM knowledge at decode time, with modest latency overhead relative to speculative decoding. The paper is honest about limitations: it explicitly acknowledges that confidence scores are imperfect proxies for token quality (Appendix A(2)) and reports a concrete failure case (Table 9). The transferability evidence is partly out-of-sample: the decision tree is trained on three AlpacaEval samples and applied to MMLU/GSM8K/HumanEval, and the rule hyperparameters selected on pairs 1-2 are applied to pairs 3-6. These are genuine strengths. However, the evaluation has important gaps: the headline 10% improvement does not appear in the tables, the core confidence-correctness assumption is untested, and results lack uncertainty quantification.

major comments (5)
  1. [Abstract / Tables 2-3] The Abstract claims accuracy improvements 'up to 10%' over existing methods, but no cell in Tables 2 or 3 shows a ten-point gain over a named baseline. The largest absolute gain I could identify is 7.84 points (Pair 5, MMLU, CoSD-Tree vs. Char-ED); most comparisons are 1-4 points. Please either identify the exact comparison that supports the 10% figure, report it explicitly in the main text, or temper the claim to match the reported results.
  2. [Section 3.2, Eqs. (4)-(6)] The verification rule replaces a draft token whenever the assistant is sufficiently more confident, which presupposes that across two different models, higher softmax probability is a reliable indicator of higher token-level correctness. The paper provides no calibration evidence for this premise: there is no analysis of P(correct | disagreement) vs. confidence for the six model pairs, no reliability curves, and no comparison of confidence scales across models. Table 9 documents a case where an overconfident incorrect assistant overwrites a correct draft, and Appendix A(2) concedes that confidence is not always aligned with token quality. Please add aggregate evidence for disagreeing tokens (e.g., accuracy in confidence bins for both models) to show that the rule operates in the regime where confidence and correctness are positively correlated.
  3. [Section 4.1 / Figure 2 / Table 2] The hyperparameters α=0.5 and β=0.5 are selected by maximizing MMLU+GSM8K accuracy on pairs 1 and 2 (Figure 2), and the same pairs and benchmarks are then reported in Table 2. The main results for pairs 1 and 2 are therefore partially tuned on the test data. The transfer to pairs 3-6 is genuine evidence of robustness, but the paper should report tuned and untuned evaluations separately and avoid presenting the pairs 1-2 results as independent confirmation of the rule's transferability.
  4. [Section 4.2, Tables 2-3] All reported scores are point estimates with no confidence intervals, standard errors, or multiple-seed runs, despite the use of tinyBenchmarks subsets where sampling noise is non-negligible. Several claimed improvements are small (e.g., Pair 4 AVG: 35.71 vs. 35.65; Pair 1 TruthfulQA: 43.65 vs. 43.92, where the baseline is higher). Please provide uncertainty estimates or a reproducibility check to support the statements that CoSD 'consistently outperformed' the baselines.
  5. [Table 6 / Abstract] The efficiency comparison is presented only against Speculative Decoding, with a latency difference of about 1 ms/token. The paper does not report total compute cost of running both models, nor does it compare to the assistant model decoding alone. The claim that CoSD 'improves inference efficiency' should be qualified as 'achieves comparable wall-clock latency to speculative decoding,' and the compute overhead of the fusion mechanism should be quantified.
minor comments (5)
  1. [Algorithm 1] The loops say 'for i in 0, 1, ..., K' and line 12 sets 't <- t + i'; the indexing appears to start at 0, which would be inconsistent with the token positions xt+1,...,xt+K in Section 3. Please fix the loop bounds and the update rule.
  2. [Throughout] The acronym is written inconsistently as 'COSD' and 'CoSD' at different points (e.g., Section 1 vs. Section 3). Please use a single spelling throughout.
  3. [Section 3.2, Eq. (7)] Training a decision tree with a cross-entropy loss (Eq. 7) is unconventional; decision trees are typically grown by greedy splitting and impurity reduction. Please clarify the actual training procedure and the relation between Eq. (7) and the fitting algorithm.
  4. [Section 3.2, Tree-Based Verification] The examples of specialized domains cite Poonia & Al-Alshaikh (2024), Amru et al. (2024), and Mazraeh et al. (2024), which appear to be non-LLM applied papers (disease diagnosis, smart home intrusion detection, groundwater modeling). These are not appropriate examples of LLM application domains; please replace them with relevant LLM references.
  5. [Section 4.1 / Table 5] The main text says the decision tree is trained on three randomly selected AlpacaEval samples, while Table 5's caption says it uses 10 MMLU samples and 3 samples of other datasets. Please clarify the exact training data size used for the main CoSD-Tree results.

Circularity Check

1 steps flagged · score 3.0 of 10

Partial evaluation circularity: Rule-Based CoSD's thresholds are tuned on the same MMLU/GSM8K pairs later reported as results, but the central transfer claims rest on out-of-sample pairs and an AlpacaEval-trained tree.

  1. fitted input called prediction [Section 4.1 (Hyperparameters), Figure 2, and Table 2 (Pairs 1-2)]
    "For Rule-Based COSD, we set α = 0.5 and β = 0.5, which were determined to be the optimal and most transferable parameters based on our analysis in Figure 2. ... The numbers in the small squares represent the sum score of MMLU and GSM8K, which reflect the overall model performance of COSD. ... with α = 0.5, 0.75 and β = 0.5, 0.75, Rule-Based COSD perform consistently well in the two model pairs."

    The α and β thresholds in the CoSD-Rule decision (Eqs. 4-6) are selected by maximizing the sum of MMLU and GSM8K accuracy on Pairs 1 and 2 in Figure 2. Those exact pairs and benchmarks then appear as evidence in Table 2 for CoSD-Rule's fusion gains, e.g., Pair 1 average 52.41 and Pair 2 average 56.77. Reporting those rows as validation is therefore partly in-sample: the thresholds were chosen to make those numbers strong. The method is not globally circular, because Pairs 3-6 and the AlpacaEval-trained Tree-Based CoSD provide independent, out-of-sample support, but the flagship complementary-fusion rows are statistically favored by the tuning procedure.

full rationale

CoSD's core derivation is mostly self-contained: the verification rule (Eqs. 4-6) is an explicit decision rule, and Tree-Based CoSD is trained on a few AlpacaEval samples with ground-truth labels and then tested on disjoint MMLU/GSM8K/HumanEval sets, so those transfer results are genuine out-of-sample predictions. The main circularity-adjacent step is Rule-Based CoSD's hyperparameter selection: Figure 2 tunes α and β on MMLU+GSM8K accuracy for Pairs 1 and 2, and Table 2 then reports the same pairs and benchmarks as evidence, making those particular rows in-sample. This inflates those specific numbers but does not force the whole claim, because Pairs 3-6 and the AlpacaEval-trained decision tree provide independent confirmation. Appendix A(2) and Table 9 concede that confidence scores are not always aligned with token quality; that is an acknowledged correctness assumption, not a circular step, and it does not raise the circularity score. The paper contains no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation; the cited works involving the authors are used as baselines, benchmarks, or related work, not as the basis for the derivation.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper contributes no new physical or mathematical entities; its components are a threshold rule and a standard decision tree over two token-probability features. The heavier baggage is procedural: two rule thresholds tuned on the evaluation benchmarks, an unspecified draft block size, and a tiny unseeded tree training set. The main conceptual axiom, that confidence implies correctness, is stated in Section 3.2 and explicitly qualified by the authors in Appendix A.

free parameters (4)
  • α (draft confidence threshold) = 0.5
    Eq. (5). Chosen by grid search over the sum of MMLU+GSM8K scores on pairs 1 and 2 (Figure 2), then applied globally in all Rule-Based CoSD experiments. Selecting it on the evaluation benchmarks makes the Table 2 flagship rows partially in-sample.
  • β (assistant margin factor) = 0.5
    Eq. (6). Selected in the same grid search as α. With β=0.5 the assistant can replace a draft token when its probability exceeds half the draft's probability, a loose margin.
  • K (draft block length) = not specified in the paper
    Algorithm 1 takes K as input but the experiments never state its value; Table 7's iteration counts at max lengths 128, 256, and 512 indirectly depend on it. The efficiency profile of CoSD depends on this unspecified parameter.
  • Decision tree training sample = 3 random AlpacaEval samples (and 10 MMLU / 3 per other dataset in Table 5)
    Section 4.1: the tree is trained on a handful of samples with no specified seed. Performance of Tree-Based CoSD depends on this choice; the paper argues the method is robust to it.
assumptions (3)
  • domain assumption A model's token probability is a reliable proxy for token quality: replacing a draft token only when the assistant's probability exceeds β times the draft's probability, and the draft's probability is below α, increases the chance the accepted token is correct.
    Grounded in Section 3.2, Eqs. (4)-(6). The authors themselves qualify it in Appendix A, point (2), and Table 9 shows the assistant's overconfidence overriding a correct draft answer, so the entire selection mechanism rests on a premise the paper admits is imperfect.
  • domain assumption The assistant model's token probabilities computed over the draft prefix (after decode/encode for differing tokenizers) are comparable to the draft model's probabilities over its own prefix.
    Eqs. (2)-(3) and the verification step treat Mp(ext+i) and Mq(xhat t+i) as comparable, but with different tokenizers the draft prefix is decoded and re-encoded (Eq. 3), changing the conditioning context before probabilities are compared.
  • domain assumption The regenerate-and-verify loop terminates and produces an acceptable output without systematic bias.
    Algorithm 1, lines 8-14: upon replacement, all subsequent draft tokens are discarded and generation resumes from the replaced token. No convergence or distributional guarantee is given for this loop, unlike standard speculative decoding's lossless acceptance rule.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding." pith.science (2026). https://pith.science/paper/KCNYP4KA

@misc{pith2026250208020,
  author       = {Pith},
  title        = {Pith review of: Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KCNYP4KA}},
  note         = {Machine review of arXiv:2502.08020}
}
read the original abstract

Large Language Models (LLMs) often excel in specific domains but fall short in others due to the limitations of their training. Thus, enabling LLMs to solve problems collaboratively by integrating their complementary knowledge promises to improve their performance across domains. To realize this potential, we introduce a novel Collaborative Speculative Decoding (CoSD) algorithm that enables efficient LLM knowledge fusion at test time without requiring additional model training. CoSD employs a draft model to generate initial sequences and an easy-to-learn rule or decision tree to decide when to invoke an assistant model to improve these drafts. CoSD not only enhances knowledge fusion but also improves inference efficiency, is transferable across domains and models, and offers greater explainability. Experimental results demonstrate that CoSD improves accuracy by up to 10\% across benchmarks compared to existing methods, providing a scalable and effective solution for LLM-based applications

Figures

Figures reproduced from arXiv: 2502.08020 by the authors.

Figure 1
Figure 1. The workflow of collaborative speculative decoding. 3.1. Generation. The generation process follows the principles of Speculative Decoding. First, the draft model Mp generates a sequence of tokens autoregressively: for i = 1 to K do xet+i ∼ Mp(x|x1, . . . , xet+i−1), (1) Here, xet+i represents the token predicted by the draft model at position i, selected as the token with the highest probabil￾ity. The sequence xet+… view at source ↗
Figure 2
Figure 2. The sum score of MMLU and GSM8K with various α, β settings on pair 1 (left figure) and pair 2 (right figure). to train the decision tree. This would result in a domain￾adapted tree that is better suited to the specific task. In addition, as mentioned in the table title, we use very few samples to train the decision tree, thus training decision trees introduces almost no additional computational overhead. Case Studie… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 4 canonical work pages

  1. [1]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [6]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  3. [7]

    Layer skip: Enabling early exit inference and self-speculative decoding

    Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, L., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710,

    arXiv

  4. [8]

    Arcee’s mergekit: A toolkit for merging large language models

    Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L., Karpukhin, V ., Benedict, B., McQuade, M., and Solawetz, J. Arcee’s mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257,

    arXiv

  5. [9]

    CharED: Character-wise Ensemble Decoding for Large Language Models

    Gu, K., Tuecke, E., Katz, D., Horesh, R., Alvarez-Melis, D., and Yurochkin, M. Chared: Character-wise ensem- ble decoding for large language models. arXiv preprint arXiv:2407.11009,

    work page Pith review arXiv

  6. [10]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y ., Li, Y ., et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196,

    arXiv

  7. [11]

    D., and He, D

    He, Z., Zhong, Z., Cai, T., Lee, J. D., and He, D. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252,

    arXiv

  8. [12]

    Measuring mas- sive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

    arXiv 2009

Show all 36 references
  1. [14]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  2. [15]

    Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling

    Kim, D., Park, C., Kim, S., Lee, W., Song, W., Kim, Y ., Kim, H., Kim, Y ., Lee, H., Kim, J., et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166,

    arXiv

  3. [16]

    Li, M., Chen, X., Holtzman, A., Chen, B., Lin, J., Yih, W.-t., and Lin, X. V . Nearest neighbor speculative de- coding for llm generation and attribution. arXiv preprint arXiv:2405.19325,

    arXiv

  4. [17]

    Truthfulqa: Measuring how models mimic human falsehoods

    Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958,

    arXiv

  5. [18]

    Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct

    Luo, H., Sun, Q., Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583,

    arXiv

  6. [19]

    Miao, X., Oliaro, G., Zhang, Z., Cheng, X., Wang, Z., Zhang, Z., Wong, R. Y . Y ., Zhu, A., Yang, L., Shi, X., et al. Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification. arXiv preprint arXiv:2305.09781,

    arXiv

  7. [20]

    E., Kadous, M

    Ong, I., Almahairi, A., Wu, V ., Chiang, W.-L., Wu, T., Gon- zalez, J. E., Kadous, M. W., and Stoica, I. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665,

    arXiv

  8. [21]

    M., Weber, L., Choshen, L., Sun, Y ., Xu, G., and Yurochkin, M

    Polo, F. M., Weber, L., Choshen, L., Sun, Y ., Xu, G., and Yurochkin, M. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.14992,

    arXiv

  9. [22]

    Z., Lang, H., Wang, B., Kim, Y ., and Sontag, D

    Shen, S. Z., Lang, H., Wang, B., Kim, Y ., and Sontag, D. Learning to decode collaboratively with multiple lan- guage models. arXiv preprint arXiv:2403.03870,

    arXiv

  10. [24]

    Knowledge fusion of large language models

    Wan, F., Huang, X., Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491,

    arXiv

  11. [25]

    Federated learning with matched averaging

    Wang, H., Yurochkin, M., Sun, Y ., Papailiopoulos, D., and Khazaeni, Y . Federated learning with matched averaging. arXiv preprint arXiv:2002.06440,

    2002 arXiv

  12. [26]

    M., Sun, Y ., Kundu, S., Xing, E., and Yurochkin, M

    Wang, H., Polo, F. M., Sun, Y ., Kundu, S., Xing, E., and Yurochkin, M. Fusing models with complementary ex- pertise. arXiv preprint arXiv:2310.01542,

    arXiv

  13. [27]

    Llama pro: Progressive llama with block expansion

    Wu, C., Gan, Y ., Ge, Y ., Lu, Z., Wang, J., Feng, Y ., Luo, P., and Shan, Y . Llama pro: Progressive llama with block expansion. arXiv preprint arXiv:2401.02415,

    arXiv

  14. [28]

    Speculative decoding: Exploiting speculative execu- tion for accelerating seq2seq generation

    Xia, H., Ge, T., Wang, P., Chen, S.-Q., Wei, F., and Sui, Z. Speculative decoding: Exploiting speculative execu- tion for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3909–3925,

    2023

  15. [29]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Xia, H., Yang, Z., Dong, Q., Wang, P., Li, Y ., Ge, T., Liu, T., Li, W., and Sui, Z. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851,

    arXiv

  16. [30]

    Openmoe: An early effort on open mixture-of-experts language models

    Xue, F., Zheng, Z., Fu, Y ., Ni, J., Zheng, Z., Zhou, W., and You, Y . Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739,

    arXiv

  17. [31]

    Predictive pipelined decoding: A compute- latency trade-off for exact llm decoding

    Yang, S., Lee, G., Cho, J., Papailiopoulos, D., and Lee, K. Predictive pipelined decoding: A compute- latency trade-off for exact llm decoding. arXiv preprint arXiv:2307.05908,

    arXiv

  18. [32]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  19. [34]

    Tinyllama: An open-source small language model

    Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385,

    arXiv

  20. [35]

    S., Menon, A

    Zhou, Y ., Lyu, K., Rawat, A. S., Menon, A. K., Ros- tamizadeh, A., Kumar, S., Kagy, J.-F., and Agarwal, R. Distillspec: Improving speculative decoding via knowl- edge distillation. arXiv preprint arXiv:2310.08461,

    arXiv

  21. [36]

    Llama-moe: Building mixture-of-experts from llama with continual pre-training

    Zhu, T., Qu, X., Dong, D., Ruan, J., Tong, J., He, C., and Cheng, Y . Llama-moe: Building mixture-of-experts from llama with continual pre-training. arXiv preprint arXiv:2406.16554,

    arXiv

  22. [2018]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288,

    arXiv

  23. [2019]

    Draft & verify: Lossless large language model acceleration via self-speculative decoding

    Zhang, J., Wang, J., Li, H., Shou, L., Chen, K., Chen, G., and Mehrotra, S. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168,

    arXiv

  24. [2020]

    Analysis of linear mode connectivity via permutation-based weight match- ing

    Ito, A., Yamada, M., and Kumagai, A. Analysis of linear mode connectivity via permutation-based weight match- ing. arXiv preprint arXiv:2402.04051,

    arXiv

  25. [2021]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

    arXiv

  26. [2022]

    J., Ganesh, E

    Amru, M., Kannan, R. J., Ganesh, E. N., Muthumarilak- shmi, S., Padmanaban, K., Jeyapriya, J., and Murugan, S. Network intrusion detection system by applying en- semble model for smart home. International Journal of Electrical & Computer Engineering (2088-8708), 14(3),

  27. [2023]

    K., Hayase, J., and Srinivasa, S

    Ainsworth, S. K., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836,

    arXiv

  28. [2024]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.