Pith. sign in

REVIEW 5 major objections 7 minor 61 references

Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proposes AdvDistill, a reward-guided dataset distillation method that trains a 1.5B student on multiple teacher responses weighted by group-relative advantages, and reports the student beating the 7B teacher on GSM-8K (91.52%…

desk verdict A sensible distillation recipe with honest reporting, but the headline result is confounded by data volume and compute; the reward mechanism isn't isolated. read the letter →

arxiv 2507.00054 v1 pith:P2JL2GZL submitted 2025-06-25 cs.AI cs.CLcs.LG

Shreyansh Padarha This is my paper
classification cs.AIcs.CLcs.LG
keywords AdvDistilldatasetdistillationknowledgereward-guidedlearninggroup-relativeadvantagesmalllanguagemodelsmathematicalreasoningQwen2.5
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a small language model learns complex reasoning better when distillation data is not a single "best" teacher answer but a batch of varied responses weighted by how much better or worse each one is than average. Its method, AdvDistill, generates eight answers per prompt, scores each with a rule-based reward, converts those scores into group-relative advantages, and uses those advantages as training weights plus a penalty for incorrect tokens. The central evidence is that a 1.5B Qwen model trained this way reaches 91.52% on GSM-8K and 69.09% on GSM-PLUS, above the 7B teacher's 88.58% and 67.83%, and roughly doubles the base 1.5B model's accuracy. If this holds, distillation can make small models more deployable while actually improving on the teacher on math-style reasoning, rather than only imitating it.

What carries the argument

The load-bearing object is the group-relative advantage weighted loss. For each prompt, the teacher samples $k=8$ responses; a rule-based reward combines a length-aware cosine score with a binary format score as $r_j = w_{\text{cosine}} \, \text{Cosine}_j + w_{\text{format}} \, \text{Format}_j$. Rewards are normalized within the prompt, $A_j=(r_j-\mu)/(\sigma+\epsilon)$, and a softmax over these advantages with temperature $\tau$ produces the per-response weights $w_i$. The student loss is $\sum_i w_i \mathcal{L}_{\text{CE}}(y_i) + \lambda_{\text{wrong}} \mathcal{L}_{\text{contrast}}(y_i)$, where the contrastive term pushes down the student's average token probability on incorrect responses. This weighting is what lets the student train on all eight responses, including wrong ones, without requiring teacher logits.

What would settle it

Run the same 1.5B fine-tuning recipe on the identical filtered eight-response data with all weights set equal (or with weights shuffled) and evaluate on GSM-8K and GSM-PLUS. If accuracy stays near 91.52% and 69.09%, the reward guidance is not the active ingredient. A second check would replace the cosine reward with a simple correctness-only reward and compare the resulting accuracy and response lengths.

Watch

Extended reading notes

Core claim

The central claim is that giving a student model not one distilled answer but a weighted bundle of eight teacher responses—where the weight of each response is its reward relative to the other responses to the same prompt—produces better reasoning than standard supervised distillation on the teacher's best output. In the paper's experiments, the AdvDistill 1.5B student scores 91.52% on GSM-8K and 69.09% on GSM-PLUS, above the 7B teacher's 88.58% and 67.83%; on OPEN-S1 and OPEN-RS it beats both the 1.5B and 3B SFT-distilled students while remaining below the teacher; and on MMLU-PRO it does not beat SFT distillation, which the paper explains as a general-knowledge task where rule-based rewards add little. The authors also report that the method changes behaviour: it shortens incorrect responses on reasoning tasks but worsens the correct-to-incorrect length ratio on simple math and hurts template adherence.

Load-bearing premise

The assumption that carries the argument is that the advantage weights, rather than the larger eight-response filtered training set, cause the student's gains; the paper reports no control with equal weights over the same responses.

Editorial extensions

If this is right

  • A 1.5B parameter student can exceed the 7B teacher's accuracy on GSM-8K (91.52% vs 88.58%) and GSM-PLUS (69.09% vs 67.83%), so on math-style benchmarks distillation can improve rather than merely approximate the teacher.
  • Training on multiple reward-weighted responses beats single-best-response SFT on the mathematical and complex-reasoning benchmarks tested, for both the 1.5B and 3B students.
  • The gains do not extend to MMLU-PRO, where SFT-distilled 1.5B scores 30.07% vs AdvDistill's 23.57%, so reward-guided distillation is domain-dependent.
  • The extra teacher sampling costs about 4.5x more than SFT ($108.75 versus $23.75 in the paper's accounting), so the accuracy gain is bought with more compute.
  • AdvDistill also changes response behaviour: it reduces the incorrect-to-correct length ratio on reasoning datasets but increases it on simple math, and lowers template adherence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not claimed by the paper: the reported gains could come from the larger filtered dataset and 4.5x compute rather than from the advantage weights, since no control trains on the same eight responses with equal weights.
  • Not claimed by the paper: varying the cosine reward's boundary values ($\eta_{\min}$, $\eta_{\max}$) should shift both accuracy and response-length ratios in a predictable direction, giving a cheap test of whether the length penalty is the active ingredient.
  • Not claimed by the paper: the same reward-weighted bundle could transfer to other verifiable domains such as code execution or formal proof, where correctness is checkable without an oracle model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes AdvDistill, a dataset distillation framework in which a teacher LLM (Qwen2.5-7B) generates k=8 responses per prompt; these responses are scored with a rule-based composite reward (length-aware cosine plus format adherence), converted to group-relative advantages, and used to weight the student's cross-entropy loss, with an additional contrastive penalty on incorrect responses. A group is retained only if at least one of its 8 responses is correct. The authors fine-tune Qwen2.5-1.5B with this loss and compare against SFT baselines that fine-tune on the teacher's single highest-advantage response per prompt, as well as against base models, on GSM-8K, OPEN-S1, MMLU-PRO and OOD sets (GSM-PLUS, OPEN-RS). Table 2 reports that the 1.5B AdvDistill model reaches 91.52% on GSM-8K and 69.09% on GSM-PLUS, outperforming the 7B teacher and all baselines on those sets, while on OPEN-S1 it is below the teacher and on MMLU-PRO it is below the SFT-distilled 1.5B model. The paper also analyzes response verbosity, token degeneracy, and template adherence across models.

Significance. If the reported gains are attributable to the reward-guided weighting mechanism, the method would be a useful off-policy distillation alternative that avoids logit matching and is simple to implement with rule-based rewards. The behavioral analysis (verbosity, degeneracy, format adherence) is a worthwhile addition and provides evidence that the method changes model behavior beyond accuracy. However, the central comparison is currently confounded by simultaneously changing the training set size, compute budget, and loss weighting, so the paper does not yet establish that advantage weighting, rather than simply seeing eight teacher responses per prompt, drives the gains. The method is not fully novel relative to the cited advantage-guided distillation works (Gao et al., 2025; Y. Zhang et al., 2025), but the specific combination of multi-response sampling, rule-based rewards, and a contrastive penalty could be a contribution if the mechanism were isolated with appropriate ablations.

major comments (5)
  1. [Section 3.1 and Table 2] The headline comparison is confounded. SFTDistilled is fine-tuned on the single highest-advantage teacher response per prompt, while AdvDistill trains on all eight responses with the Eq. (5) loss; Table 3 shows this also means 4.5x more compute. The reported gains could therefore be due to seeing eight (partly correct) responses per prompt and more training tokens, rather than to the advantage-derived weights. The authors should include a control that trains on exactly the same accepted 8-response dataset with uniform weights w_i = 1/k and no contrastive penalty (lambda_wrong = 0), holding all other hyperparameters fixed; without this control, Eq. (5) and Table 2 cannot separate the reward mechanism from scale.
  2. [Section 3] The group filtering rule, 'A group is accepted into the final dataset if at least one of the responses is correct,' is an uncontrolled variable. This rule changes the training distribution in a way that is independent of the advantage weighting and may make the 8-response set easier than the single-best-response set. The authors should ablate this filtering rule (e.g., no filtering, or filtering by a different criterion) to show that the gains are not an artifact of dataset selection.
  3. [Table 3 and Section 5] Table 3 reports compute costs for 'AdvDistill(3 Responses)' but no accuracy results are given for this variant anywhere in the paper. If the number of responses k is meant to be varied as an ablation, the corresponding performance numbers should be reported; otherwise the row is uninterpretable and gives no information about how k affects the trade-off.
  4. [Section 5] The paper reports single-run accuracy values without any statistical significance tests, confidence intervals, or multiple seeds. Given that the core claim is a comparison between two methods on several datasets, the authors should provide variance estimates across at least a few training seeds (or a paired significance test) so that the magnitude of the differences in Table 2 can be assessed.
  5. [Section 5.1] The sentence 'We observe a two-fold improvement on test and OOD sets for the 1.5B student over its base model' is imprecise. In Table 2, AdvDistill improves the 1.5B base substantially on GSM-8K and GSM-PLUS, but the improvements on OPEN-S1 and OPEN-RS are much smaller (22.77% vs 13.52% and 23.44% vs 15.18%), and on MMLU-PRO the AdvDistill model is actually worse than the SFTDistilled baseline. The claim should be restricted to the mathematical datasets or otherwise stated with the domains explicitly separated.
minor comments (7)
  1. [Section 4.1.1] There is a typo: 'represent represent' should be 'represent'.
  2. [References] Several pairs of references are listed as distinct entries but appear identical: Agarwal et al. (2024a) and (2024b), Xu et al. (2024a) and (2024b), and K. Zhang et al. (2025a) and (2025b). These should be merged or properly differentiated.
  3. [Tables 2 and 6, Figures 3 and 6] The dataset name is inconsistent: the paper uses both 'OPEN-RS' and 'OPEN-R1' for the same OOD benchmark. Please choose one name and use it consistently throughout the text, tables, and figures.
  4. [Tables 6 and 7] The table formatting is hard to read: entries such as '0.0270.019' and '552.8 552.7' appear as concatenated numbers without a visible separator. Use separate columns or parentheses for correct/incorrect response values.
  5. [Figure 1] The figure caption contains the odd string 'A+−' repeated many times, which appears to be a rendering artifact rather than a meaningful symbol; please clarify the intended notation.
  6. [Code Availability] The phrase 'The code is available at: GitHub Repository' is not a functional link; a URL or repository identifier should be provided.
  7. [Section 5.1] The sentence referencing the performance comparison says 'Table 1' but the actual comparison table is Table 2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the AdvDistill result is an empirical comparison on held-out benchmarks, not a derivation that reduces to its inputs.

full rationale

The central claim (Section 5.1, Table 2) rests on held-out accuracy on GSM-8K, GSM-PLUS, OPEN-S1, OPEN-RS, and MMLU-PRO. The training objective in Eq. (5) uses advantages computed from teacher responses and rule-based rewards (Eqs. 2-4), but nothing in those equations defines the test-set accuracies; the reported improvements are measured, not derived. No fitted parameter is renamed as a prediction, no load-bearing result depends on a self-citation, and no uniqueness theorem or ansatz is imported from the authors' prior work. The main weakness is a comparison confound: AdvDistill trains on all 8 responses with 4.5x more compute while SFTDistilled trains on one response, so the advantage-weighting mechanism is not isolated. That is a confound rather than circular reduction, and under the stated rules it does not raise the circularity score.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The method rests on a small set of hand-picked hyperparameters and domain assumptions about teacher sampling diversity, reward reliability, and the benefit of discarding all-incorrect groups. No invented entities are introduced. The central claim is empirical and does not depend on a formal derivation.

free parameters (8)
  • w_cosine = 2
    Hand-set weight for the length-aware cosine reward in the composite reward (Eq. 2).
  • w_format = 1
    Hand-set weight for the binary format reward in the composite reward (Eq. 2).
  • tau (student) = 0.5
    Temperature for softmax weighting of advantages in Eq. 7; controls how strongly high-advantage responses are favored.
  • lambda_wrong = 0.5
    Strength of contrastive penalty for incorrect responses in Eq. 5.
  • teacher temperature = 0.9
    Sampling temperature used to generate diverse teacher responses (Appendix B, Table 4).
  • k (generations per prompt) = 8
    Number of teacher responses generated per prompt for group relative advantage computation.
  • L (max token length) = 2048
    Maximum generation length used in the cosine reward formula (Eq. 3); capped length for all responses.
  • group filtering rule = at least one correct
    Groups with no correct answers are discarded; this is a data selection rule that biases training distribution.
assumptions (5)
  • domain assumption Teacher's temperature-0.9 samples provide a diverse and useful training distribution for the student.
    The entire method depends on the teacher's multiple generations being informative; stated in Section 3.
  • domain assumption Rule-based reward functions correctly identify correct/incorrect responses and reward quality.
    Correctness is judged by exact answer match; format reward checks tags; no human verification. Section 4.1.
  • ad hoc to paper Discarding groups with zero correct responses improves training and does not introduce harmful bias.
    This filtering rule is introduced specifically for this method (Section 3) and is not justified by experiments.
  • domain assumption Group relative advantage normalization (GRPO-style) is beneficial for off-policy distillation.
    Borrowed from GRPO (Shao et al., 2024) without a dedicated ablation.
  • domain assumption Contrastive penalty on incorrect responses helps without destabilizing training.
    The paper reports gradient clipping for stability but does not ablate lambda_wrong.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation." pith.science (2026). https://pith.science/paper/P2JL2GZL

@misc{pith2026250700054,
  author       = {Pith},
  title        = {Pith review of: Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P2JL2GZL}},
  note         = {Machine review of arXiv:2507.00054}
}
read the original abstract

The push to compress and impart the proficiency of Large Language Models (LLMs) into more deployable and efficient Small Language Models (SLMs) has benefited from improvements in knowledge distillation (KD) techniques. These techniques allow a smaller student model to learn from a more capable and larger teacher model's responses. However, distillation often revolves around the student model merely copying the teacher's in-distribution responses, limiting its generalisability. This limitation is amplified on reasoning tasks and can be computationally expensive. In this study, we propose AdvDistill, a reward-guided dataset distillation framework. We utilise multiple generations (responses) from a teacher for each prompt and assign rewards based on rule-based verifiers. These varying and normally distributed rewards serve as weights when training student models. Our methods and their subsequent behavioural analysis demonstrate a significant improvement in student model performance for mathematical and complex reasoning tasks, showcasing the efficacy and benefits of incorporating a rewarding mechanism in dataset distillation processes.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 40 canonical work pages

  1. [1]

    , Aneja, J

    7 APACrefauthors Abdin, M. , Aneja, J. , Awadalla, H. , Awadallah, A. , Awan, A A. , Bach, N. Zhou, X. APACrefauthors \ 2024 . Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. Phi-3 technical report: A highly capable language model locally on your phone

    work page 2024

  2. [2]

    , Vieillard, N

    gkd APACrefauthors Agarwal, R. , Vieillard, N. , Zhou, Y. , Stanczyk, P. , Garea, S R. , Geist, M. \ Bachem, O. APACrefauthors \ 2024 1 . On-policy distillation of language models: Learning from self-generated mistakes On-policy distillation of language models: Learning from self-generated mistakes . The Twelfth International Conference on Learning Repres...

    work page 2024

  3. [3]

    , Vieillard, N

    42 APACrefauthors Agarwal, R. , Vieillard, N. , Zhou, Y. , Stanczyk, P. , Garea, S R. , Geist, M. \ Bachem, O. APACrefauthors \ 2024 2 . On-policy distillation of language models: Learning from self-generated mistakes On-policy distillation of language models: Learning from self-generated mistakes . The Twelfth International Conference on Learning Represe...

    work page 2024

  4. [4]

    \ Tegmark, M

    40 APACrefauthors Baek, D D. \ Tegmark, M. APACrefauthors \ 2025 . Towards Understanding Distilled Reasoning Models: A Representational Approach Towards understanding distilled reasoning models: A representational approach . arXiv preprint arXiv:2503.03730

    arXiv 2025

  5. [5]

    , Sastre, I

    20 APACrefauthors Balad \'o n, A. , Sastre, I. , Chiruzzo, L. \ Ros \'a , A. APACrefauthors \ 2023 . RETUYT-InCo at BEA 2023 shared task: Tuning open-source LLMs for generating teacher responses Retuyt-inco at bea 2023 shared task: Tuning open-source llms for generating teacher responses . Proceedings of the 18th Workshop on Innovative Use of NLP for Buil...

    work page 2023

  6. [6]

    , Hosseini, A

    44 APACrefauthors Bansal, H. , Hosseini, A. , Agarwal, R. , Tran, V Q. \ Kazemi, M. APACrefauthors \ 2024 . Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling . arXiv preprint arXiv:2408.16737

    arXiv 2024

  7. [7]

    , Peebles, B

    6 APACrefauthors Brooks, T. , Peebles, B. , Holmes, C. , DePue, W. , Guo, Y. , Jing, L. others APACrefauthors \ 2024 . Video generation models as world simulators. 2024 Video generation models as world simulators. 2024 . URL https://openai. com/research/video-generation-models-as-world-simulators 3 1

    work page 2024

  8. [8]

    , Kosaraju, V

    50 APACrefauthors Cobbe, K. , Kosaraju, V. , Bavarian, M. , Chen, M. , Jun, H. , Kaiser, L. others APACrefauthors \ 2021 . Training verifiers to solve math word problems Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 61 references
  1. [9]

    \ Ngo, C

    52 APACrefauthors Dang, Q A. \ Ngo, C. APACrefauthors \ 2025 . Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't. Reinforcement learning for reasoning in small llms: What works and what doesn't

    2025

  2. [10]

    , See, A

    55 APACrefauthors Dathathri, S. , See, A. , Ghaisas, S. , Huang, P S. , McAdam, R. , Welbl, J. others APACrefauthors \ 2024 . Scalable watermarking for identifying large language model outputs Scalable watermarking for identifying large language model outputs . Nature 634 8035...

    2024

  3. [11]

    , Feng, B

    24 APACrefauthors DeepSeek-AI , Liu, A. , Feng, B. , Xue, B. , Wang, B. , Wu, B. Pan, Z. APACrefauthors \ 2025 . DeepSeek-V3 Technical Report. Deepseek-v3 technical report

    2025

  4. [12]

    26 APACrefauthors Feng, K. , Li, C. , Zhang, X. , Zhou, J. , Yuan, Y. \ Wang, G. APACrefauthors \ 2024 1 . Keypoint-based Progressive Chain-of-Thought Distillation for LLMs. Keypoint-based progressive chain-of-thought distillation for llms

    2024

  5. [13]

    38 APACrefauthors Feng, K. , Li, C. , Zhang, X. , Zhou, J. , Yuan, Y. \ Wang, G. APACrefauthors \ 2024 2 . Keypoint-based Progressive Chain-of-Thought Distillation for LLMs. Keypoint-based progressive chain-of-thought distillation for llms

    2024

  6. [14]

    , Wan, F

    45 APACrefauthors Gao, S. , Wan, F. , Guo, J. , Quan, X. \ Wang, Q. APACrefauthors \ 2025 . Advantage-Guided Distillation for Preference Alignment in Small Language Models Advantage-guided distillation for preference alignment in small language models . arXiv preprint arXiv:2502.17927

    2025 arXiv

  7. [15]

    , Dong, L

    minillm APACrefauthors Gu, Y. , Dong, L. , Wei, F. \ Huang, M. APACrefauthors \ 2023 . MiniLLM: Knowledge distillation of large language models Minillm: Knowledge distillation of large language models . arXiv preprint arXiv:2306.08543

    2023 arXiv

  8. [16]

    , Zhang, L L

    8 APACrefauthors Guan, X. , Zhang, L L. , Liu, Y. , Shang, N. , Sun, Y. , Zhu, Y. Yang, M. APACrefauthors \ 2025 . rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking. rstar-math: Small llms can master math reasoning with self-evolved deep thinking

    2025

  9. [17]

    , Yang, D

    11 APACrefauthors Guo, D. , Yang, D. , Zhang, H. , Song, J. , Zhang, R. , Xu, R. others APACrefauthors \ 2025 . Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning...

    2025 arXiv

  10. [18]

    , Vinyals, O

    13 APACrefauthors Hinton, G. , Vinyals, O. \ Dean, J. APACrefauthors \ 2015 . Distilling the Knowledge in a Neural Network. Distilling the knowledge in a neural network

    2015

  11. [19]

    , Borgeaud, S

    4 APACrefauthors Hoffmann, J. , Borgeaud, S. , Mensch, A. , Buchatskaya, E. , Cai, T. , Rutherford, E. Sifre, L. APACrefauthors \ 2022 . Training Compute-Optimal Large Language Models. Training compute-optimal large language models

    2022

  12. [20]

    , Li, C L

    25 APACrefauthors Hsieh, C Y. , Li, C L. , Yeh, C K. , Nakhost, H. , Fujii, Y. , Ratner, A. Pfister, T. APACrefauthors \ 2023 . Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. Distilling step-by-step! outperforming...

    2023

  13. [21]

    , Yin, Y

    30 APACrefauthors Jiao, X. , Yin, Y. , Shang, L. , Jiang, X. , Chen, X. , Li, L. Liu, Q. APACrefauthors \ 2020 . TinyBERT: Distilling BERT for Natural Language Understanding. Tinybert: Distilling bert for natural language understanding

    2020

  14. [22]

    , Li, X Z

    2 APACrefauthors Kang, J. , Li, X Z. , Chen, X. , Kazemi, A. , Sun, Q. , Chen, B. others APACrefauthors \ 2024 . Mindstar: Enhancing math reasoning in pre-trained llms at inference time Mindstar: Enhancing math reasoning in pre-trained llms at inference time . arXiv preprint a...

    2024 arXiv

  15. [23]

    \ Rush, A M

    seqKD APACrefauthors Kim, Y. \ Rush, A M. APACrefauthors \ 2016 . Sequence-level knowledge distillation Sequence-level knowledge distillation . Proceedings of the 2016 conference on empirical methods in natural language processing Proceedings of the 2016 conference on empirica...

    2016

  16. [24]

    , Chen, T

    distillm2 APACrefauthors Ko, J. , Chen, T. , Kim, S. , Ding, T. , Liang, L. , Zharkov, I. \ Yun, S Y. APACrefauthors \ 2025 . Distillm-2: A contrastive approach boosts the distillation of llms Distillm-2: A contrastive approach boosts the distillation of llms . arXiv preprint ...

    2025 arXiv

  17. [25]

    , Kim, S

    distillm APACrefauthors Ko, J. , Kim, S. , Chen, T. \ Yun, S Y. APACrefauthors \ 2024 . Distillm: Towards streamlined distillation for large language models Distillm: Towards streamlined distillation for large language models . arXiv preprint arXiv:2402.03898

    2024 arXiv

  18. [26]

    , Fang, L

    22 APACrefauthors Latif, E. , Fang, L. , Ma, P. \ Zhai, X. APACrefauthors \ 2024 . Knowledge distillation of llms for automatic scoring of science assessments Knowledge distillation of llms for automatic scoring of science assessments . International Conference on Artificial I...

    2024

  19. [27]

    , White, M

    37 APACrefauthors Lewis, A. , White, M. , Liu, J. , Koike-Akino, T. , Parsons, K. \ Wang, Y. APACrefauthors \ 2025 . Winning Big with Small Models: Knowledge Distillation vs. Self-Training for Reducing Hallucination in QA Agents Winning big with small models: Knowledge distill...

    2025 arXiv

  20. [28]

    , Cui, L

    53 APACrefauthors Li, Q. , Cui, L. , Zhao, X. , Kong, L. \ Bi, W. APACrefauthors \ 2024 . Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as math...

    2024 arXiv

  21. [29]

    , Kang, Y

    33 APACrefauthors Liu, C. , Kang, Y. , Zhao, F. , Kuang, K. , Jiang, Z. , Sun, C. \ Wu, F. APACrefauthors \ 2024 . Evolving knowledge distillation with large language models and active learning Evolving knowledge distillation with large language models and active learning . ar...

    2024 arXiv

  22. [30]

    , Chen, C

    41 APACrefauthors Liu, Z. , Chen, C. , Li, W. , Qi, P. , Pang, T. , Du, C. Lin, M. APACrefauthors \ 2025 . Understanding R1-Zero-Like Training: A Critical Perspective. Understanding r1-zero-like training: A critical perspective

    2025

  23. [31]

    , Yang, Z

    54 APACrefauthors Muennighoff, N. , Yang, Z. , Shi, W. , Li, X L. , Fei-Fei, L. , Hajishirzi, H. Hashimoto, T. APACrefauthors \ 2025 . s1: Simple test-time scaling. s1: Simple test-time scaling

    2025

  24. [32]

    , Dani, J

    16 APACrefauthors Nakka, K. , Dani, J. \ Saxena, N. APACrefauthors \ 2025 . Is On-Device AI Broken and Exploitable? Assessing the Trust and Ethics in Small Language Models. Is on-device ai broken and exploitable? assessing the trust and ethics in small language models

    2025

  25. [33]

    19 APACrefauthors Niu, S. , Ma, J. , Lin, H. , Bai, L. , Wang, Z. , Xu, Y. Yang, X. APACrefauthors \ 2025 . Knowledge-Augmented Multimodal Clinical Rationale Generation for Disease Diagnosis with Small Language Models. Knowledge-augmented multimodal clinical rationale generati...

    2025

  26. [34]

    , Lerer, A

    9 APACrefauthors OpenAI , : , Hurst, A. , Lerer, A. , Goucher, A P. , Perelman, A. Malkov, Y. APACrefauthors \ 2024 . GPT-4o System Card. Gpt-4o system card

    2024

  27. [35]

    , Liu, B

    36 APACrefauthors Panigrahi, A. , Liu, B. , Malladi, S. , Risteski, A. \ Goel, S. APACrefauthors \ 2024 . Progressive distillation induces an implicit curriculum Progressive distillation induces an implicit curriculum . arXiv preprint arXiv:2410.05464

    2024 arXiv

  28. [36]

    , Kim, D

    31 APACrefauthors Park, W. , Kim, D. , Lu, Y. \ Cho, M. APACrefauthors \ 2019 . Relational knowledge distillation Relational knowledge distillation . Proceedings of the IEEE/CVF conference on computer vision and pattern recognition Proceedings of the ieee/cvf conference on com...

    2019

  29. [37]

    \ Xie, S

    5 APACrefauthors Peebles, W. \ Xie, S. APACrefauthors \ 2023 . Scalable diffusion models with transformers Scalable diffusion models with transformers . Proceedings of the IEEE/CVF international conference on computer vision Proceedings of the ieee/cvf international conference...

    2023

  30. [38]

    , Yin, L

    21 APACrefauthors Qu, Z. , Yin, L. , Yu, Z. , Wang, W. \ . APACrefauthors \ 2024 . CourseGPT-ZH: An Educational Large Language Model Based on Knowledge Distillation Incorporating Prompt Optimization Coursegpt-zh: An educational large language model based on knowledge distillat...

    2024 arXiv

  31. [39]

    , Debut, L

    47 APACrefauthors Sanh, V. , Debut, L. , Chaumond, J. \ Wolf, T. APACrefauthors \ 2019 . DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter . arXiv preprint arXiv:1910.01108

    2019 arXiv

  32. [40]

    , Li, S S

    spur APACrefauthors Shao, R. , Li, S S. , Xin, R. , Geng, S. , Wang, Y. , Oh, S. Zettlemoyer, L. APACrefauthors \ 2025 . Spurious Rewards: Rethinking Training Signals in RLVR. Spurious rewards: Rethinking training signals in rlvr. APACrefURL https://arxiv.org/abs/2506.10947 APACrefURL

    2025 arXiv

  33. [41]

    , Wang, P

    48 APACrefauthors Shao, Z. , Wang, P. , Zhu, Q. , Xu, R. , Song, J. , Bi, X. Guo, D. APACrefauthors \ 2024 . DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    2024

  34. [42]

    , Shumaylov, Z

    29 APACrefauthors Shumailov, I. , Shumaylov, Z. , Zhao, Y. , Gal, Y. , Papernot, N. \ Anderson, R. APACrefauthors \ 2023 . The curse of recursion: Training on generated data makes models forget The curse of recursion: Training on generated data makes models forget . arXiv prep...

    2023 arXiv

  35. [43]

    , Kamath, A

    10 APACrefauthors Team, G. , Kamath, A. , Ferret, J. , Pathak, S. , Vieillard, N. , Merhej, R. others APACrefauthors \ 2025 . Gemma 3 technical report Gemma 3 technical report . arXiv preprint arXiv:2503.19786

    2025 arXiv

  36. [44]

    , Riviere, M

    23 APACrefauthors Team, G. , Riviere, M. , Pathak, S. , Sessa, P G. , Hardin, C. , Bhupatiraju, S. Andreev, A. APACrefauthors \ 2024 . Gemma 2: Improving Open Language Models at a Practical Size. Gemma 2: Improving open language models at a practical size

    2024

  37. [45]

    , Han, Y

    32 APACrefauthors Tian, Y. , Han, Y. , Chen, X. , Wang, W. \ Chawla, N V. APACrefauthors \ 2025 . Beyond answers: Transferring reasoning capabilities to smaller llms using multi-teacher knowledge distillation Beyond answers: Transferring reasoning capabilities to smaller llms ...

    2025

  38. [46]

    , Calandriello, D

    28 APACrefauthors Tiapkin, D. , Calandriello, D. , Ferret, J. , Perrin, S. , Vieillard, N. , Ram \'e , A. \ Blondel, M. APACrefauthors \ 2025 . On Teacher Hacking in Language Model Distillation On teacher hacking in language model distillation . arXiv preprint arXiv:2502.02671

    2025 arXiv

  39. [47]

    , Shaib, C

    34 APACrefauthors Wadhwa, S. , Shaib, C. , Amir, S. \ Wallace, B C. APACrefauthors \ 2025 . Who Taught You That? Tracing Teachers in Model Distillation Who taught you that? tracing teachers in model distillation . arXiv preprint arXiv:2502.06659

    2025 arXiv

  40. [48]

    , Deng, Y

    3 APACrefauthors Wang, C. , Deng, Y. , Lyu, Z. , Zeng, L. , He, J. , Yan, S. \ An, B. APACrefauthors \ 2024 . Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning. Q*: Improving multi-step reasoning for llms with deliberative planning

    2024

  41. [49]

    , Zhang, Z

    15 APACrefauthors Wang, F. , Zhang, Z. , Zhang, X. , Wu, Z. , Mo, T. , Lu, Q. others APACrefauthors \ 2024 . A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthines...

    2024 arXiv

  42. [50]

    , Zhu, J Y

    14 APACrefauthors Wang, T. , Zhu, J Y. , Torralba, A. \ Efros, A A. APACrefauthors \ 2020 . Dataset Distillation. Dataset distillation

    2020

  43. [51]

    51 APACrefauthors Wang, Y. , Ma, X. , Zhang, G. , Ni, Y. , Chandra, A. , Guo, S. others APACrefauthors \ 2024 . MMLU-Pro: A more robust and challenging multi-task language understanding benchmark Mmlu-pro: A more robust and challenging multi-task language understanding benchma...

    2024

  44. [52]

    , Wang, X

    1 APACrefauthors Wei, J. , Wang, X. , Schuurmans, D. , Bosma, M. , Xia, F. , Chi, E. others APACrefauthors \ 2022 . Chain-of-thought prompting elicits reasoning in large language models Chain-of-thought prompting elicits reasoning in large language models . Advances in neural ...

    2022

  45. [53]

    , Bai, H

    39 APACrefauthors Wu, Z. , Bai, H. , Zhang, A. , Gu, J. , Vydiswaran, V V. , Jaitly, N. \ Zhang, Y. APACrefauthors \ 2024 . Divide-or-Conquer? Which Part Should You Distill Your LLM? Divide-or-conquer? which part should you distill your llm?

    2024

  46. [55]

    , Han, R

    43 APACrefauthors Xu, W. , Han, R. , Wang, Z. , Le, L T. , Madeka, D. , Li, L. Pfister, T. APACrefauthors \ 2024 2 . Speculative knowledge distillation: Bridging the teacher-student gap through interleaved sampling Speculative knowledge distillation: Bridging the teacher-stude...

    2024 arXiv

  47. [56]

    , Yang, B

    12 APACrefauthors Yang, A. , Yang, B. , Zhang, B. , Hui, B. , Zheng, B. , Yu, B. others APACrefauthors \ 2024 . Qwen2. 5 technical report Qwen2. 5 technical report . arXiv preprint arXiv:2412.15115

    2024 arXiv

  48. [57]

    , Tong, Y

    49 APACrefauthors Yeo, E. , Tong, Y. , Niu, M. , Neubig, G. \ Yue, X. APACrefauthors \ 2025 . Demystifying Long Chain-of-Thought Reasoning in LLMs Demystifying long chain-of-thought reasoning in llms . arXiv preprint arXiv:2502.03373

    2025 arXiv

  49. [58]

    , Wang, C

    35 APACrefauthors Yue, Y. , Wang, C. , Huang, J. \ Wang, P. APACrefauthors \ 2024 . Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning. Distilling instruction-following abilities of large language models with task-aware curr...

    2024

  50. [59]

    , Zhu, R

    17 APACrefauthors Zhang, K. , Zhu, R. , Ma, S. , Xiong, J. , Kim, Y. , Murai, F. \ Liu, X. APACrefauthors \ 2025 1 . KEDRec-LM: A Knowledge-distilled Explainable Drug Recommendation Large Language Model. Kedrec-lm: A knowledge-distilled explainable drug recommendation large la...

    2025

  51. [60]

    , Zhu, R

    18 APACrefauthors Zhang, K. , Zhu, R. , Ma, S. , Xiong, J. , Kim, Y. , Murai, F. \ Liu, X. APACrefauthors \ 2025 2 . KEDRec-LM: A Knowledge-distilled Explainable Drug Recommendation Large Language Model. Kedrec-lm: A knowledge-distilled explainable drug recommendation large la...

    2025

  52. [61]

    , Shen, J

    27 APACrefauthors Zhang, R. , Shen, J. , Liu, T. , Liu, J. , Bendersky, M. , Najork, M. \ Zhang, C. APACrefauthors \ 2023 . Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation. Do not blindly imitate the teacher: Using perturbed loss for knowled...

    2023

  53. [62]

    , Wang, L

    46 APACrefauthors Zhang, Y. , Wang, L. , Fang, M. , Du, Y. , Huang, C. , Wang, J. others APACrefauthors \ 2025 . Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones? Distill not only data but also rewards: Can smaller language models surpass...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.