Pith. sign in

REVIEW 5 major objections 6 minor 15 references

From Teacher to Student: Tracking Memorization Through Model Distillation

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distilling a fine-tuned teacher into a smaller student reduces verbatim memorization of the fine-tuning data while preserving task performance, suggesting knowledge distillation is also a privacy-preserving training technique.

desk verdict The distillation privacy claim is confounded by target shift; worth a referee but not a citation yet. read the letter →

arxiv 2506.16170 v2 pith:3LLMY3KP submitted 2025-06-19 cs.LG cs.AI

Simardeep Singh This is my paper
classification cs.LGcs.AI
keywords knowledgedistillationmemorizationlanguagemodelprivacycompressionsupervisedfine-tuningreverseKLdivergenceinstructiontuningGPT-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether knowledge distillation, a technique normally used to shrink models, also changes how much a fine-tuned language model memorizes its training data. Using a verbatim-reproduction definition of memorization, the author compares direct supervised fine-tuning with three distillation methods across different model sizes. The reported result is that every distilled student memorizes less training data than the same-size model trained by supervised fine-tuning, with reverse-KL distillation giving the strongest reduction while test-set quality stays roughly level. If this holds, distillation offers a dual benefit: cheaper deployment and lower risk of leaking sensitive fine-tuning text.

What carries the argument

The load-bearing object is the memorization fraction: the fraction of a sample of training prompts for which greedy decoding reproduces the gold response exactly, with a threshold set at $k=50$. The mechanism that carries the argument is teacher-softening: word-level distillation trains the student on the teacher's token-level distribution, sequence-level distillation trains it on the teacher's beam-search outputs, and reverse-KL distillation minimizes reverse KL divergence from the student to the teacher, which the author argues penalizes overconfidence and thereby discourages verbatim copying. ROUGE, an n-gram overlap score computed against both train and test targets, is the auxiliary signal that distinguishes inflated training-set overlap from genuine generalization.

What would settle it

Take the same teacher and train identical students by supervised fine-tuning with label smoothing, or with early stopping at the same token budget; if memorization fractions fall to the distilled levels, the effect is not specific to distillation. Alternatively, run membership-inference or training-data extraction attacks on the SFT and distilled students; if attacks recover fine-tuning examples at comparable rates despite lower exact-match fractions, the paper's privacy conclusion fails against realistic adversaries.

Watch

Extended reading notes

Core claim

The paper's central claim is that training a smaller student from a fine-tuned teacher's outputs, rather than directly from ground-truth labels, reduces the fraction of fine-tuning examples the student can reproduce word-for-word while preserving task performance. This reduction holds across student sizes from 120M to 760M parameters and across all three distillation variants, and it is largest for reverse-KL distillation, which reaches memorization fractions of 0.060 to 0.090 compared with 0.330 to 0.523 for direct supervised fine-tuning. The paper frames this as a dual benefit: distillation compresses the model and simultaneously lowers memorization risk, making it a candidate privacy-preserving training method for sensitive fine-tuning data.

Load-bearing premise

The privacy conclusion assumes that the exact-match memorization fraction, measured at a 50-token threshold on one dataset without reported variance, is the right target for privacy risk; if that number mostly reflects smoothed output distributions or misses what an actual attacker can extract, the claimed privacy benefit of distillation is overstated.

Editorial extensions

If this is right

  • Model builders can use distillation as a default step for fine-tuning on proprietary or sensitive data, getting smaller deployable models with a lower verbatim-leakage risk.
  • Reverse-KL distillation is the strongest tested candidate for privacy-sensitive settings, since it produces the lowest memorization fraction at every student size while keeping test ROUGE comparable.
  • The consistent size-memorization trend means larger models need explicit mitigation, and distillation provides one that applies across the tested model scale.
  • Since memorization drops while test ROUGE stays roughly flat, the reported privacy benefit does not appear to come at the cost of task performance in these experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the effect is specific to distillation would be to run supervised fine-tuning with label smoothing or early stopping at the same training budget; if memorization falls to distilled levels, the benefit is generic regularization rather than teacher-student training.
  • Exact-match memorization is only one leakage channel; students could still reveal training content through membership-inference attacks or partial reconstruction, so the reported reduction is not equivalent to formal privacy.
  • Since the same fine-tuning data are already inside the teacher, distillation may transfer less-verbatim knowledge rather than erase content; probing whether training examples can be recovered from student outputs under repeated sampling would sharpen the privacy claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies how knowledge distillation (KD) affects memorization of fine-tuning data in small language models. The authors fine-tune a GPT-2 1.5B teacher on 10,000 Dolly instruction-response pairs and distill it into smaller GPT-2 students using four methods: supervised fine-tuning (SFT), word-level KD, sequence-level KD (Seq-KD), and reverse-KL distillation (RKLD). They measure memorization as the fraction of training examples whose gold response is reproduced verbatim under greedy decoding, and also report train/test ROUGE scores. The central claim is that distillation lowers memorization relative to SFT while preserving test-set performance, giving a 'dual benefit' of model compression and privacy. The paper is a short empirical study with a single dataset, a single seed, and no statistical significance testing.

Significance. If the claimed effect is real, the finding would be practically relevant: it would suggest that KD is not only a compression tool but also a privacy-preserving training choice for fine-tuned LLMs. The paper is also one of few to measure memorization specifically in the distillation setting, which is a worthwhile question. However, the current evidence is not yet sufficient to support the strong privacy conclusion. The measurement definition is internally inconsistent, the comparison is confounded by a training-target shift between SFT and all KD methods, and there is no control for generic regularization effects or sampling variability. These issues are load-bearing because they directly affect the paper's main quantitative claim. The paper is a useful exploratory study but needs a major revision before its central conclusion can be accepted.

major comments (5)
  1. [Section 2.1, Section 3.1, Algorithm 1] The memorization criterion is defined inconsistently. Section 2.1 states that a response is memorized if it exactly matches s and 'the match persists for at least k tokens (k = length of s in our implementation)' and Algorithm 1 only checks exact_match(generated, s), never actually using k. Section 3.1, however, says 'using Algorithm 1 with k = 50'. These are different definitions: exact match over the full response is not the same as matching the first 50 tokens, and the behavior for responses shorter than 50 tokens is undefined. Since all reported memorization fractions depend on this definition, the inconsistency must be resolved and the algorithm must be implemented consistently with the stated criterion.
  2. [Section 2.2, Section 3.2, Tables 1-4] The central comparison is confounded by a training-target shift. In Eq. (1), SFT trains the student directly on the gold response s; in Eqs. (2)-(4), the KD methods train the student on teacher-generated sequences or on teacher token distributions. Yet memorization in Tables 1-4 is always measured against the original training response s_i. A student that was never trained to reproduce s_i will naturally have lower exact-match and lower train ROUGE against s_i even if it has memorized the teacher's paraphrase or the teacher's distribution over s_i. The paper does not report memorization rates against the actual teacher-generated targets used for the KD students, nor does it include an SFT control trained on those same teacher-generated targets. Without such a control, the observed gap between SFT and KD may be entirely explained by target shift rather than by a privacy-preserving property of distillation.
  3. [Section 3.1, Table 1] The results are reported without any measure of variability. A single seed is used, and no error bars, confidence intervals, or significance tests are provided. Given that the memorization fractions range from 0.060 to 0.654, the differences among KD variants (e.g., 0.140 vs 0.134 vs 0.075 for 340M) may be within sampling noise. The paper should report multiple seeds or, at minimum, bootstrap intervals over the 3,000 sampled examples, so that the claimed rank ordering of methods can be assessed.
  4. [Section 3.2, Tables 1-4] There is no regularization control. SFT is compared to KD methods that naturally smooth the training signal through soft targets or teacher-generated sequences. The lower memorization seen in KD students could be a generic consequence of output smoothing or of reduced overfitting, not a distinctive property of distillation. An SFT baseline with early stopping, label smoothing, or a matching training budget would help determine whether the observed effect is attributable to distillation per se or to any form of regularization.
  5. [Section 5, Conclusion] The conclusion that distillation 'enhanc[es] privacy' overstates what the metric measures. The paper only measures verbatim exact-match reproduction of a small subset of training responses; it does not measure extraction attacks, membership inference, or other privacy risks. The Ethical Considerations section correctly acknowledges this limitation, but the abstract and conclusion still frame the result as a general privacy benefit. The claims should be restricted to the measured memorization metric, or supported with additional privacy-focused evaluations.
minor comments (6)
  1. [Section 2.2.2, Eq. (2)] The notation in Eq. (2) is confusing: it sums over j = 1 to J, the teacher sentence length, but conditions q and p on 's, t<j', where s is defined as the student sentence length I. It is unclear whether s is the input or the target, and how teacher and student lengths relate.
  2. [Section 2.2.2] The sentence 'The student is further be trained to optimize the mixture of LWORD-KD and LWORD-NLL' has a grammatical error and does not specify the mixture coefficient or the form of the NLL term. This is necessary to reproduce the method.
  3. [Section 3.1] The experimental setup does not state the number of training epochs, learning rate, batch size, or whether the same training subset and the same random seed are used across all methods. This information is needed to interpret the comparisons.
  4. [Section 3.1, Tables 2-4] The table headers use 'KD' but Section 2.2.2 calls this method 'WL-KD'. Please use consistent terminology to avoid ambiguity.
  5. [Section 3.1] It is unclear whether the '3,000 randomly sampled examples' used for memorization measurement are a subset of the 10,000 fine-tuning examples used for all models, or whether some examples were used only for the teacher and not the students. This should be stated explicitly.
  6. [Section 6] The phrase 'implicit memorization or generalization that could be exploited by more sophisticated extraction techniques' is a useful caveat, but it would benefit from a concrete example or citation, since the paper otherwise does not discuss adversarial extraction.

Circularity Check

2 steps flagged · score 6.0 of 10

Lower KD memorization is partly by construction: students are trained on teacher outputs while memorization is measured against original gold responses.

  1. self definitional [Section 2.2 (Eqs. 2-3) vs. Section 2.1 Algorithm 1 and Section 3.1]
    "The student models are trained on the same instruction-response data, but with guidance from the teacher model rather than the gold responses directly (except in SFT). ... LSEQ-KD = − log p(t = ˆy | s) (3) Where ˆy = BeamSearch(fteacher, s) ... if exact_match(generated, s) then memorized_count ← memorized_count + 1"

    For Seq-KD (Eq. 3), the student's hard target is y_hat, the teacher's beam-search output, not the original response s. For WL-KD (Eq. 2), the target is the teacher's token distribution q, and RKLD (Eq. 4) minimizes reverse KL to the teacher. Only SFT (Eq. 1) fits s. Algorithm 1 defines memorization as exact match of the student's greedy generation to the original s. A student never trained to reproduce s therefore has a mechanically lower exact-match fraction, regardless of any privacy property of distillation. The reported KD-vs-SFT gap is thus a restatement of the training-target choice; without an SFT control trained on the same teacher-generated targets, the experiment cannot separate target shift from genuine memorization suppression.

  2. other [Section 3.2 (Results, RKLD paragraph)]
    "This is in accordance with its goal of minimizing reverse KL divergence, which penalizes overconfidence on training examples by discouraging memorization by design."

    The paper presents RKLD's low memorization fraction as evidence that distillation reduces memorization, yet the same sentence discloses that this behavior is intentionally programmed into the loss objective. The 'result' is therefore an expected consequence of the loss design, not an independent empirical discovery. Combined with the target-shift in the previous step, the headline privacy benefit reduces to the way the objectives and evaluation metric were defined, rather than being an emergent property validated against a matched control.

full rationale

Score 6: the central claim is substantially forced by the evaluation design. Section 2.2 defines all KD students' training targets as teacher outputs or teacher distributions (Eqs. 2-4), not the original gold responses, while Section 2.1 / Algorithm 1 defines memorization as exact match to those original gold responses. Consequently, lower exact-match memorization for KD students relative to SFT is a by-construction consequence of training on different targets. The paper's own Section 3.2 admits this for RKLD ('discouraging memorization by design'), and no control SFT run on teacher-generated targets is provided. The reported ROUGE train scores are computed against the original responses too, so they inherit the same confound. Additional correctness risks, not circularity, include: single seed, no variance, inconsistent thresholds (Algorithm 1 uses exact_match over len(s) while Section 3.1 says k=50), and no comparison to other regularizers like early stopping or label smoothing. I find no load-bearing self-citation: citations to Carlini, Kim & Rush, Gu et al., etc., are external and standard. The circularity is partial rather than total, because a matched-target experiment might still show some distillation benefit; however, the paper's current evidence cannot establish that claim.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a specific definition of memorization and a set of experimental choices rather than fitted parameters. The metric threshold and dataset subsampling are hand-chosen and not justified; no new entities are introduced.

free parameters (3)
  • Memorization threshold k = 50 (set in Section 3.1; Section 2.1 says k = length of response)
    Threshold for verbatim match; inconsistent definition affects the measured memorization fraction.
  • Number of training examples = 10,000
    Subsampled DollyEval for teacher fine-tuning; memorization behavior may depend on training set size.
  • Evaluation subset sizes = 3,000 for memorization, 500 for ROUGE
    Randomly sampled subsets; no variance reported across different draws.
assumptions (3)
  • domain assumption Exact match under greedy decoding with max_length = len(s) is a valid measure of memorization for instruction-following tasks.
    Adopted from Carlini et al. (2023) but adapted; no validation that this matches human-perceived memorization or extraction risk.
  • domain assumption The student models are pre-trained and then fine-tuned/distilled from the same initial checkpoint; no weight inheritance from teacher.
    Assumes that the observed differences are due to the distillation objective, not initialization or training procedure.
  • domain assumption The distillation methods (WL-KD, Seq-KD, RKLD) are correctly implemented following the cited papers.
    No code or implementation details provided; relies on the correctness of external implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Teacher to Student: Tracking Memorization Through Model Distillation." pith.science (2026). https://pith.science/paper/3LLMY3KP

@misc{pith2026250616170,
  author       = {Pith},
  title        = {Pith review of: From Teacher to Student: Tracking Memorization Through Model Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3LLMY3KP}},
  note         = {Machine review of arXiv:2506.16170}
}
read the original abstract

Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages

  1. [1]

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. https://arxiv.org/abs/2202.07646 Quantifying memorization across neural language models . Preprint, arXiv:2202.07646

    arXiv 2023

  2. [2]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. https://arxiv.org/abs/2012.07805 Extracting training data from large language models . Preprint, arXiv:2012.07805

    arXiv 2021

  3. [3]

    Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. https://arxiv.org/abs/2306.08543 Minillm: Knowledge distillation of large language models . Preprint, arXiv:2306.08543

    arXiv 2024

  4. [4]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

    arXiv 2015

  5. [5]

    Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. https://arxiv.org/abs/2205.12628 Are large pre-trained language models leaking your personal information? Preprint, arXiv:2205.12628

    arXiv 2022

  6. [6]

    Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. https://arxiv.org/abs/2401.07284 Improving domain adaptation through extended-text reading comprehension . Preprint, arXiv:2401.07284

    work page Pith review arXiv 2024

  7. [7]

    Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2023. https://arxiv.org/abs/2307.01881 Propile: Probing privacy leakage in large language models . Preprint, arXiv:2307.01881

    arXiv 2023

  8. [8]

    Yoon Kim and Alexander M. Rush. 2016. https://arxiv.org/abs/1606.07947 Sequence-level knowledge distillation . Preprint, arXiv:1606.07947

    arXiv 2016

Show all 15 references
  1. [9]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  2. [10]

    Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. https://arxiv.org/abs/2302.00539 Analyzing leakage of personally identifiable information in language models . Preprint, arXiv:2302.00539

    2023 arXiv

  3. [11]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners

    2019

  4. [12]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. https://arxiv.org/abs/1910.01108 Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter . Preprint, arXiv:1910.01108

    2020 arXiv

  5. [13]

    Xinyu Yang, Zichen Wen, Wenjie Qu, Zhaorun Chen, Zhiying Xiang, Beidi Chen, and Huaxiu Yao. 2024. https://openreview.net/forum?id=KmW8WkCKRx Memorization and privacy risks in domain-specific large language models . In ICLR 2024 Workshop on Reliable and Responsible Foundation Models

    2024

  6. [14]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  7. [15]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.