Pith. sign in

REVIEW 4 major objections 6 minor 23 references

Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distilling a 7B or 2.8B teacher down to 3B or 1.4B preserves over 90% of its extractive QA scores, and a single in-context example usually adds further gains—except on one split where the gain flips into a loss.

desk verdict A useful, honest scaling study of KD for QA, but the abstract overclaims one-shot gains and one key retention number is computed on an unstable split. read the letter →

arxiv 2507.07630 v1 pith:7O6P5JKE submitted 2025-07-10 cs.CL cs.LG

Joyeeta Datta , Niclas Doll , Qusai Ramadan , Zeyd Boukhers This is my paper
classification cs.CLcs.LG
keywords knowledgedistillationmodelcompressionquestionansweringfew-shotpromptingzero-shotSQuADMLQAscalinganalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how far large language models can be compressed by knowledge distillation before question-answering quality collapses. The authors distill Qwen2.5-7B and Pythia-2.8B teachers into students from 3B down to 70M parameters, then evaluate every model on SQuAD v2.0 and MLQA under zero-shot and one-shot prompts. Their central claim is that moderately sized students keep over 90% of their teacher's Exact Match and ROUGE-L scores while using as little as 43% of the teacher's parameters, and that one-shot prompting usually improves on zero-shot. They also report a case where one-shot looks better on validation but worse on the test split, which qualifies the prompting conclusion.

What carries the argument

The load-bearing object is a controlled size ladder built with DISTILLM, a distillation recipe using skew KL divergence and off-policy sampling. Each teacher is fine-tuned on a dataset and prompting configuration, then its logits are transferred to students of decreasing parameter count, and all models are evaluated with identical zero-shot and one-shot prompt templates using Exact Match and ROUGE-L averaged over five seeds. That ladder is what locates the compression point where retention starts to drop.

What would settle it

Recompute the Table 1 evaluations on a second held-out split of SQuAD and with a different one-shot demonstration. If Pythia-1.4B's one-shot test EM falls below roughly 38.9 (90% of the teacher's 43.23) while its validation one-shot score stays near 68.00, the one-shot benefit on this model is split-specific; if retention falls below 90% on the new split for either student, the central compression claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that distillation preserves task capability across a wide size range for extractive QA. Qwen2.5-3B and Pythia-1.4B retain at least 90% of the teacher scores on both EM and ROUGE-L across SQuAD and MLQA, and in several zero-shot settings the distilled 3B student matches or slightly exceeds its 7B teacher. Distilled students at a given size also outperform fine-tuned students of the same size, and one-shot prompting lifts scores for most models, with the largest gains at the 1.4B-3B scale. The paper simultaneously documents a counterexample: on SQuAD, Pythia-1.4B scores 68.00 EM with one-shot prompting on validation but 43.23 on the test split, while zero-shot drops only from 63.60 to 59.86, so the prompting advantage reverses across splits.

Load-bearing premise

The retention and prompting conclusions assume that scores on the chosen test split, measured with one fixed one-shot example and five seeds, reliably reflect what a compressed model can do; if the validation-to-test flip seen for Pythia-1.4B on SQuAD reflects broader sensitivity to splits or prompt structure, the reported ratios and prompting gains may not generalize.

Editorial extensions

If this is right

  • A distilled Qwen2.5-3B or Pythia-1.4B model can replace its 7B or 2.8B teacher on extractive QA at roughly half the parameters while keeping EM and ROUGE-L within 10% of teacher levels.
  • Distillation beats same-size fine-tuning for these models, so the practical route to small QA systems is to keep the teacher and distill rather than fine-tune a small model directly.
  • Since one-shot prompting consistently helps medium-sized students, deployment can gain accuracy at near-zero cost by prepending a single demonstration, especially on multilingual QA.
  • Because the one-shot advantage reversed between validation and test for Pythia-1.4B on SQuAD, any deployment should validate few-shot prompts on the actual evaluation split before relying on them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves open is that the headline retention ratios may be tied to a single test split; averaging over several splits or demonstration sets would give a more robust estimate of the true compression-cost curve.
  • The low-variance but low-benefit behavior of Pythia-70M suggests a capacity threshold for few-shot learning; a finer grid of student sizes could map where one-shot prompting stops helping.
  • Because results are reported as EM and ROUGE-L, the '90% retention' conclusion is safest for short span-extraction answers; generative or free-form QA needs different metrics before the same claim can be assumed.
  • The fact that one fixed demonstration can flip the conclusion on SQuAD implies that prompt selection, not model size, may dominate in some settings; a testable extension is adversarial selection of the in-context example.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies knowledge distillation (KD) for question answering (QA) by distilling Pythia-2.8B and Qwen2.5-7B teachers into smaller student models, evaluating Exact Match (EM) and ROUGE-L on SQuAD v2.0 and MLQA (English and German) under zero-shot and one-shot prompting. The authors report that larger students retain over 90% of teacher performance with up to 57.1% parameter reduction, that one-shot prompting generally improves over zero-shot, and that distilled students outperform their fine-tuned counterparts. The paper also reports a validation/test discrepancy for Pythia on SQuAD.

Significance. If the claims held as stated, the paper would provide useful empirical evidence for KD-based compression on extractive QA tasks. Strengths include the use of standard benchmarks, five-seed runs with reported variance in Table A3, and retention ratios that are direct quotients of measured scores rather than fitted parameters. However, the significance is substantially weakened by overgeneralized claims in the abstract and Section 4, a best-student-only presentation that makes the high-retention claim partly tautological, and a missing comparison to fine-tuned students that leaves a central claim unverifiable. The paper is a modest empirical contribution that needs careful qualification and additional data.

major comments (4)
  1. [Abstract and §4 (Effect of Prompting)] The abstract and Section 4 claim that 'one-shot prompting yields additional performance gains over zero-shot setups for both model families.' This is directly contradicted by Table 2, where Pythia-1.4B on SQuAD test has one-shot EM 43.23 versus zero-shot EM 59.86, and by the corresponding Table 1 row (best student one-shot EM 42.86 vs zero-shot 59.86). The paper's own Limitations section acknowledges this inconsistency. Please qualify the claim to the settings where it holds, or separate the SQuAD/Pythia case explicitly.
  2. [Table 1 and §4 (Model Size vs. Performance)] The claim that 'student models retain over 90% of their teacher models' performance' is based on Table 1, which lists only the 'Best Student' per condition. This selection-by-best makes the retention ratios high by construction and does not support a blanket statement about student models. Table A3 shows that Pythia-70M retains only about 37% EM (zero-shot) and 43% EM (one-shot) on MLQA-EN, and Qwen2.5-1.5B retains less than Qwen2.5-3B. The paper should report results for all student sizes (at least in an appendix table) and revise the claim to specify which students achieve ≥90% retention.
  3. [§4 (Distillation vs. Fine-Tuning)] The section states that distilled models 'generally' and 'substantially' outperform their fine-tuned counterparts, and gives examples such as Qwen2.5-3B and Pythia-1.4B. However, no fine-tuned student scores appear in Table 1, Table A3, or anywhere else in the paper. Without these data, the comparison cannot be verified. Please add the fine-tuned student results or remove the claim.
  4. [Table 1 and Table 2] The retention ratios in Table 1 are computed on a single test split, and Table 2 shows that the Pythia-1.4B SQuAD condition is highly split-sensitive: one-shot validation EM is 68.00 but test EM is 43.23, while zero-shot drops only from 63.60 to 59.86. The Pythia SQuAD One row in Table 1 (teacher 43.23, best student 42.86) therefore reflects both models failing on the test split, and the resulting 99% retention is not convincing evidence of preserved capability. Please report validation-side teacher/student numbers for all conditions, or otherwise demonstrate that the other rows in Table 1 are not similarly dependent on the specific test split.
minor comments (6)
  1. [§3.3] Typo: 'datsets' should be 'datasets'.
  2. [Abstract] The phrase 'reducing parameter counts by up to 57.1%' is ambiguous because the paper also evaluates students down to 7% of teacher size (92.9% reduction). Please clarify that 57.1% is the maximum reduction among the students that achieve the stated retention, and that larger compressions were studied but with lower retention.
  3. [Table 1] The caption says 'Bold values indicate where students outperform teachers,' but in the plain-text version no cells are visibly bold. Ensure the published table actually marks those cells.
  4. [Appendix A.3] Table A3 reports variance only for a few students on MLQA-EN. If the paper claims 'low variance' across all experiments, variance for the remaining models and settings should be reported.
  5. [§3.4] The DISTILLM framework is referenced but not described: no distillation loss, temperature, training steps, or compute budget are given. Some detail is needed for reproducibility, even if the focus is on the empirical comparison.
  6. [References] The reference list has broken glyphs such as 'Y .' and 'V o' in author names; please fix the LaTeX/PDF rendering.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the retention and prompting claims are arithmetic quotients of independently measured EM and ROUGE-L scores.

full rationale

The paper's central claim, that larger student models retain over 90% of teacher performance, is a direct ratio of measured EM and ROUGE-L values in Table 1. No parameter is fitted to produce the retention numbers, no quantity is defined in terms of the claimed result, and no self-citation is load-bearing. The distillation methodology is taken from DISTILLM (Ko et al., 2024), which is external to the author list and used as an implementation framework rather than as evidence for the empirical retention claim. The one-shot prompting claim is qualified by the paper's own prompt-sensitivity analysis and Limitations section, where the authors concede validation/test inconsistencies for Pythia on SQuAD; this is a threat to external validity and a correctness concern, not a circularity, because the derivation chain still reduces to measured scores rather than to the claim itself. The selection of the 'best student' row is a post-hoc descriptive choice, not a fitted input disguised as a prediction. No evidence of self-definition, renamed known results, or ansatz smuggled in via citation was found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is an empirical study with no mathematical derivation. The central claims depend on the distillation implementation being correct (DISTILLM), the chosen QA benchmarks being representative, and the one-shot prompt being a fair probe. No novel entities or fitted constants are introduced.

assumptions (3)
  • domain assumption DISTILLM's distillation procedure transfers teacher behavior to students as described in Ko et al. (2024).
    The paper applies DISTILLM without reproducing its correctness; any implementation bug or misconfiguration would propagate to all student results.
  • domain assumption SQuAD v2.0 and the English and German splits of MLQA are representative proxies for extractive QA capability.
    Only two of MLQA's seven languages are evaluated, and all conclusions about cross-lingual generalization rest on this subset.
  • domain assumption The one-shot demonstration example is a fair and representative prompt.
    A single hand-picked example is prepended to every test instance; the validation/test discrepancy on SQuAD suggests the choice of example or split affects results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks." pith.science (2026). https://pith.science/paper/7O6P5JKE

@misc{pith2026250707630,
  author       = {Pith},
  title        = {Pith review of: Exploring the Limits of Model Compression in LLMs: A Knowledge Distillation Study on QA Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7O6P5JKE}},
  note         = {Machine review of arXiv:2507.07630}
}
read the original abstract

Large Language Models (LLMs) have demonstrated outstanding performance across a range of NLP tasks, however, their computational demands hinder their deployment in real-world, resource-constrained environments. This work investigates the extent to which LLMs can be compressed using Knowledge Distillation (KD) while maintaining strong performance on Question Answering (QA) tasks. We evaluate student models distilled from the Pythia and Qwen2.5 families on two QA benchmarks, SQuAD and MLQA, under zero-shot and one-shot prompting conditions. Results show that student models retain over 90% of their teacher models' performance while reducing parameter counts by up to 57.1%. Furthermore, one-shot prompting yields additional performance gains over zero-shot setups for both model families. These findings underscore the trade-off between model efficiency and task performance, demonstrating that KD, combined with minimal prompting, can yield compact yet capable QA systems suitable for resource-constrained applications.

Figures

Figures reproduced from arXiv: 2507.07630 by the authors.

Figure 1
Figure 1. Exact Match and ROUGE-L scores for [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Exact Match and ROUGE-L scores for Pythia [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 6 canonical work pages

  1. [1]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. https://proceedings.mlr.press/v202/biderman23a.html Pythia: A Suite for Analyzing Large Language Models Across Trai...

    work page 2023

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. https://proceedings.neurips.cc/paper_fil...

    work page 2020

  3. [3]

    Kavita Ganesan. 2018. https://arxiv.org/abs/1803.01937 ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Tasks . arXiv preprint arXiv:1803.01937

    arXiv 2018

  4. [4]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the Knowledge in a Neural Network . arXiv preprint arXiv:1503.02531

    arXiv 2015

  5. [5]

    Kristen Howell, Gwen Christian, Pavel Fomitchov, Gitit Kehat, Julianne Marzulla, Leanne Rolston, Jadin Tredup, Ilana Zimmerman, Ethan Selfridge, and Joseph Bradley. 2023. https://doi.org/10.18653/v1/2023.acl-industry.24 The economic trade-offs of large language models: A case study . In Proceedings of the 61st Annual Meeting of the Association for Computa...

    work page doi:10.18653/v1/2023.acl-industry.24 2023

  6. [6]

    Yukun Huang, Yanda Chen, Zhou Yu, and Kathleen McKeown. 2022. https://arxiv.org/abs/2212.10670 In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models . arXiv preprint arXiv:2212.10670

    arXiv 2022

  7. [7]

    Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.372 TinyBERT: Distilling BERT for Natural Language Understanding . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163--4174, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.372 2020

  8. [8]

    Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. https://arxiv.org/abs/2307.10169 Challenges and Applications of Large Language Models . arXiv preprint arXiv:2307.10169

    arXiv 2023

Show all 23 references
  1. [9]

    Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.171 UNIFIEDQA: Crossing Format Boundaries with a Single QA System . In Findings of the Association for Computa...

    2020 doi

  2. [10]

    Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. https://doi.org/10.5555/3692070.3693067 DISTILLM: towards streamlined distillation for large language models . In Proceedings of the 41st International Conference on Machine Learning, pages 24872--24895

    2024

  3. [11]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020 a . https://doi.org/10.18653/v1/2020.acl-main.703 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translat...

    2020 doi

  4. [12]

    Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020 b . https://doi.org/10.18653/v1/2020.acl-main.653 MLQA: Evaluating Cross-lingual Extractive Question Answering . In Proceedings of the 58th Annual Meeting of the Association for Computational L...

    2020 doi

  5. [13]

    Fabio Petroni, Tim Rockt \"a schel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. https://doi.org/10.18653/v1/D19-1250 Language Models as Knowledge Bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language P...

    2019 doi

  6. [14]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of machine lear...

    2020

  7. [15]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know What You Don’t Know: Unanswerable Questions for SQuAD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-...

    2018 doi

  8. [16]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQuAD: 100,000+ Questions for Machine Comprehension of Text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--23...

    2016 doi

  9. [17]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. https://arxiv.org/abs/1910.01108 DistilBERT: A distilled version of BERT: smaller, faster, cheaper and lighter . arXiv preprint arXiv:1910.01108

    2020 arXiv

  10. [18]

    Anna Sauer, Shima Asaadi, and Fabian K \"u ch. 2022. https://doi.org/10.18653/v1/2022.nlp4convai-1.10 Knowledge Distillation Meets Few-Shot Learning: An Approach for Few-Shot Intent Classification Within and Across Domains . In Proceedings of the 4th Workshop on NLP for Conver...

    2022 doi

  11. [19]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. https://arxiv.org/abs/2109.01652 Finetuned Language Models Are Zero-Shot Learners . arXiv preprint arXiv:2109.01652

    2022 arXiv

  12. [20]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025. https://arxiv.org/abs/...

    2025 arXiv

  13. [21]

    Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.572 CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 716...

    2021 doi

  14. [22]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  15. [23]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.