Who Taught You That? Tracing Teachers in Model Distillation

Byron C. Wallace; Chantal Shaib; Silvio Amir; Somin Wadhwa

REVIEW 3 major objections 5 minor 1 cited by

Who Taught You That? Tracing Teachers in Model Distillation

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a distilled student model's preferred part-of-speech templates mirror its teacher's, and that a logistic regression over these template indicators can identify the teacher from a closed candidate set with accuracy…

desk verdict First solid attempt at closed-set teacher attribution via PoS templates, but the 'consistent' advantage is overstated and the same-task setup leaves a format-convention confound open. read the letter →

arxiv 2502.06659 v3 pith:PS3O2MK5 submitted 2025-02-10 cs.CL

Somin Wadhwa , Chantal Shaib , Silvio Amir , Byron C. Wallace This is my paper · ORCID

classification cs.CL

keywords teacherattributionmodeldistillationpart-of-speechtemplatessyntacticfootprintsprovenanceLLMdetectionclosed-setclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces teacher attribution: given a small model trained by distillation, can you tell which large teacher model produced the training data, using only the student's outputs? It argues that naive measures—perplexity under candidate teachers, bag-of-words similarity, and BERTScore—are practically useless, but that the student's preferred part-of-speech templates carry relatively strong signal about the teacher. If this holds, an auditor could flag a deployed student model as having been distilled from a particular proprietary LLM, potentially catching terms-of-service violations without access to training data or model internals. The paper frames this as a new problem with practical implications for model provenance and compliance, and shows a simple closed-set classifier that beats both n-grams and BERT embeddings on most tasks.

What carries the argument

Part-of-speech templates: recurring sequences of part-of-speech tags (here of length 4) that a model favors when generating text, extracted with the diversity package. The paper takes the 50 most common such patterns across all candidate teachers, encodes their presence or absence in student outputs as indicator features, and trains a logistic regression classifier. The mechanism is that templates abstract away from task-specific vocabulary to syntactic habits, which are stable enough within a teacher's outputs and persistent enough in the student to carry a transferable attribution signal.

What would settle it

A concrete test: take a GPT-2 student distilled from a single teacher, then fine-tune it on an unrelated task; if the PoS-template classifier's accuracy on this student drops to the 0.2 chance level, the claimed retention of teacher syntax fails. Alternatively, run the classifier on students distilled from two candidate teachers with identical output styles; if the classifier cannot separate them despite different teachers, the footprint confounds style with teacher identity.

Watch

Extended reading notes

Core claim

Students inherit syntactic habits from their teachers: the part-of-speech template distributions in a distilled student's outputs resemble those of its teacher, and these templates function as a detectable signature. Over a closed teacher set of five models (Llama3-8B, Llama3-70B, Mistral-7B, Mixtral, Gemma2-9B) and two student models (GPT-2, Olmo-1B) across seven datasets, a logistic regression trained on binary indicators of the 50 most common length-4 PoS templates identifies the true teacher well above the 0.2 chance level on most tasks—for example, 0.71 accuracy on SumPubMed with GPT-2 students, versus 0.61 for n-grams—while perplexity, BERTScore, and bag-of-words cosine similarity hover near random discrimination.

Load-bearing premise

The student must actually keep some of the teacher's preferred sentence shapes after distillation; if later fine-tuning, data mixing, or multiple teachers erase those habits, the syntactic signal disappears.

Editorial extensions

If this is right

A closed-set classifier over PoS template indicators can attribute a distilled student to its teacher with accuracy far above chance on summarization and question-answering tasks, using only student-generated text.
Standard attribution methods—perplexity under candidate teachers, bag-of-words cosine similarity, and BERTScore—are unreliable for this purpose, so the syntactic signal is not a trivial byproduct of output similarity.
The signal strengthens with more student outputs: on SumPubMed, PoS-template accuracy rises from 0.68 at 50 support instances to 0.72 at 2000, suggesting the footprint is stable and cumulative.
The approach requires no access to the distillation data, teacher internals, or watermarking, so it could be applied to black-box deployed models to check whether they were distilled from a particular proprietary LLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the syntactic-footprint claim holds, it predicts an erosion test: students that undergo heavy post-distillation fine-tuning, data augmentation, or multi-teacher distillation should show degraded PoS-template attribution accuracy, and the paper's own limitations list these as open factors that could obscure the signal.
The Alpaca exception—where n-grams slightly beat PoS templates—suggests the reliable feature type is genre-dependent; a practical attribution system would likely combine syntactic templates with lexical features rather than relying on one.
Because the classifier operates purely on template indicator counts in text, it should transfer to API-only access settings where only generated strings are visible, making it a plausible audit tool for detecting unauthorized distillation.
The same footprint logic could extend to other syntactic abstractions, such as dependency-relation patterns or punctuation habits, which might be even more transferable across tasks than PoS n-grams.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a legitimate first step on a well-defined provenance problem, and the main result — PoS templates beat lexical similarity and perplexity for closed-set teacher attribution — holds in the experiments shown. But the authors oversell it slightly with "consistently," and they never rule out that the signal is task-format convention rather than a stable teacher style.

What's new: defining teacher attribution as a classification task, testing the obvious baselines (perplexity, BoW/BERTScore similarity), and showing those fail while PoS template indicators give 0.5–0.7 accuracy (chance 0.2) across seven tasks and two student models. The evaluation is careful: varying support from 50 to 2000 instances, two students (GPT-2 and Olmo-1B), open teachers for reproducibility, and an honest Limitations section that even says practical implications are "a bit unclear." That's good practice.

Where it's soft. First, the "consistently outperform" claim is contradicted by their own Table 2 on Alpaca, where n-grams edge out PoS (0.56 vs 0.55), and Table 3 shows a tie at higher support. Minor, but the wording should change. Second, no error bars or significance tests anywhere; 0.60 vs 0.58 on 200 instances is not obviously meaningful. Third, the bigger conceptual issue: the logistic regression is trained on teacher outputs and evaluated on student outputs from the same task, and the templates are selected from the same distribution. A classifier could be latching onto output-format conventions (e.g., "The answer is X because...") that all teachers use at different rates, which students copy as part of the distillation target. That's still useful for attribution, but it's not the same as "students inherit their teacher's syntactic style," which the abstract suggests. A cross-task transfer experiment or stripping standard prefixes would settle this. The Limitations mention additional fine-tuning and multi-teacher distillation, but not this specific confound.

Overall: the empirical core is solid for what it does, and the problem is worth studying. This deserves peer review. I'd ask for CIs, a softened "consistently," and one control that separates task format from teacher style before accepting.

Referee Report

3 major / 5 minor

Summary. The paper formalizes teacher attribution for distilled LLMs: given a student model fine-tuned on outputs from one of five teacher LLMs, the goal is to identify the teacher from a closed set of candidates. The authors test perplexity, bag-of-words/BERTScore similarity, and part-of-speech (PoS) template features. They report that perplexity and similarity are uninformative, while a logistic regression over 50 PoS templates of length 4 achieves up to 0.71-0.74 accuracy (SumPubMed) with GPT-2/Olmo students, generally exceeding n-gram and BERT baselines. The paper concludes that students retain syntactic templates from their teachers.

Significance. The problem is novel and practically relevant for detecting distillation that may violate terms of service. Strengths include the use of open models, publicly available datasets, a clear negative result on perplexity and similarity, and a reproducible setup. The main positive result, however, currently rests on an experimental design that cannot separate a stable teacher-specific syntactic signature from task-format conventions that students copy during distillation; the paper's own Limitations section concedes exactly this confound. As a result, the central claim is plausible but not yet established. The manuscript also overstates the superiority of PoS templates relative to its own tables.

major comments (3)

[Section 3.2 and Table 2] The claim that PoS templates 'consistently outperform' n-gram models is contradicted by the paper's own results. For Alpaca with GPT-2, n-grams reach 0.56 versus 0.55 for PoS templates; for Rotten Tomatoes with Olmo-1B, n-grams reach 0.48 versus 0.45 for PoS templates. Appendix Table 3 likewise shows Alpaca n-grams at 0.56 equal to PoS at 0.56 at support 2000. Please revise the wording to 'outperform in most settings' and discuss these exceptions, since the current phrasing appears in both Section 3.2 and Section 5.
[Section 3.2, experimental setup] The classifier is trained on teacher outputs and evaluated on student outputs from the same task and prompt distribution. Under this design, above-chance accuracy can come from teacher-specific rates of task-format conventions (e.g., answer-format prefixes like 'The answer is (A) because...'), which the student copies from the distillation data, rather than from a persistent teacher-specific syntactic style. This is exactly the confound acknowledged in the Limitations section ('shared footprints', 'surface output format'). To support the central mimicry claim, the authors should (i) test cross-task transfer (train on teacher outputs from one QA dataset, evaluate on student outputs from another QA dataset), (ii) strip known answer-format prefixes and prompt-specific tokens before template extraction and re-run the classifier, and (iii) report a direct per-template similarity between each student and each candidate teacher (e.g., cosine or correlation over template frequencies). Without at least one of these controls, the results establish in-distribution discriminability, not teacher-signature stability.
[Section 3.2, feature selection] The description 'finding the 50 most common PoS patterns of length 4 across all teachers, for up to 200 test instances per model' is underspecified. It is not stated whether these 200 instances are disjoint from the data used to train the logistic regression, nor whether the same instances also serve as the test sets generated by student models (which would be a different sample). If template selection is performed on the evaluation distribution, accuracies may be optimistically biased. Please specify the exact data split and, if needed, use a nested split for template selection and classifier training.

minor comments (5)

[Figure 2] The heatmaps lack a colorbar and axis labels; (b) and (c) report negative values for perplexity, so the metric should be defined (e.g., mean negative log-likelihood per token) and the caption should say what is plotted.
[Table 2 and Section 3.2] The 'BERT' baseline is never described. Please specify whether it is a fine-tuned BERT classifier and what input representation is used, or delete 'BERT' from the table if it is a feature-based approach.
[Section 3.1] 'Table 14' appears to be a typo for 'Table 1'; similarly, the cross-reference to 'Appendix Figure 5' should be re-checked.
[Section 3.2] The choice of template length 4 and 50 templates is not justified; a sentence explaining this choice or a sensitivity analysis would help.
[Abstract] The phrase 'a students' teacher' should be 'a student's teacher'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the teacher-attribution classifier is trained on teacher outputs and evaluated on held-out student outputs; self-citations to the authors' PoS-template tooling are methodological, not load-bearing.

full rationale

The paper's derivation chain is self-contained as an empirical test. In Section 3.2, PoS templates are extracted with the authors' diversity package, the 50 most common length-4 patterns are selected from teacher-generated outputs, and a logistic regression is trained with teacher labels as targets. The resulting classifier is then evaluated on student-generated outputs from the same tasks. The target quantity (which teacher distilled a student) is not used to define the features, nor is any fitted parameter renamed as a prediction. The motivating citation to Shaib et al. (2024b) is an external hypothesis about LLM syntactic preferences, not a theorem that forces the result; the diversity package is a measurement tool whose output is independently computed. The Limitations section's caveat that retention may be affected by fine-tuning, augmentation, or multi-teacher distillation concerns external validity and possible confounds, not definitional circularity. The skeptic's concern that above-chance accuracy could reflect task-format conventions rather than stable teacher signatures is a threat to generalization, not a reduction of the claimed result to its inputs. No equation or fitted quantity makes the prediction equivalent to the training data by construction.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on retention of teacher syntactic patterns in student outputs and on a closed-set candidate assumption; no new latent entities are posited. Feature choices (50 templates, length 4) are hand-selected. The paper itself acknowledges retention and closed-set as limitations.

free parameters (2)

Number of PoS template features = 50
Section 3.2: uses the 50 most common length-4 PoS patterns across all teachers; hand-selected, no sensitivity analysis.
PoS template length = 4
Section 3.2: templates are fixed to length 4, chosen without reported justification or ablation.

assumptions (3)

domain assumption Student models retain teacher-specific syntactic patterns after distillation.
Invoked in Section 3.2 ('Student models may internalize such structures') and flagged in Limitations as dependent on fine-tuning, augmentation, multi-teacher distillation.
domain assumption The true teacher is always in the closed candidate set of five models.
Stated in Section 2 and acknowledged as a limitation; if false, the classifier's softmax over candidates cannot return the correct answer.
domain assumption A single fixed POS tagger and template extraction method is sufficient to capture the teacher signal across all seven datasets.
The paper relies on the diversity package and default tagger from Shaib et al. 2024a without testing tagger sensitivity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Who Taught You That? Tracing Teachers in Model Distillation." pith.science (2026). https://pith.science/paper/PS3O2MK5

@misc{pith2026250206659,
  author       = {Pith},
  title        = {Pith review of: Who Taught You That? Tracing Teachers in Model Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PS3O2MK5}},
  note         = {Machine review of arXiv:2502.06659}
}

abstract

Model distillation -- using outputs from a large teacher model to teach a small student model -- is a practical means of creating efficient models for a particular task. We ask: Can we identify a students' teacher based on its outputs? Such "footprints" left by teacher LLMs would be interesting artifacts. Beyond this, reliable teacher inference may have practical implications as actors seek to distill specific capabilities of massive proprietary LLMs into deployed smaller LMs, potentially violating terms of service. We consider practical task distillation targets including summarization, question answering, and instruction-following. We assume a finite set of candidate teacher models, which we treat as blackboxes. We design discriminative models that operate over lexical features. We find that $n$-gram similarity alone is unreliable for identifying teachers, but part-of-speech (PoS) templates preferred by student models mimic those of their teachers.

Figures

Figures reproduced from arXiv: 2502.06659 by the authors.

**Figure 1.** We introduce the problem of teacher model attribution: Given a distilled student model (e.g., a finetuned GPT-2), determine which of a set of possible teacher models was distilled (here, Mistral). functionality (summarization, say) from a large proprietary model and using this to power a paid service. This may violate terms of service, so LLM providers might be keen to identify such cases. We consider a set of task… view at source ↗

**Figure 2.** Perplexity under teacher models of texts generated by different pupils on (a) Rotten-Tomatoes, (b) QuaRel, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** AUC-ROC curves for a one-vs-rest LR clas [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Influence of teacher models on student outputs, highlighting the retention of Part-of-Speech (PoS) [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Average cosine similarity of student outputs [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation
cs.AI 2025-06 conditional novelty 4.0 of 10

AdvDistill uses group relative advantages computed from rule-based rewards to weight teacher responses during distillation, reportedly improving a 1.5B student on math tasks beyond its 7B teacher.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 1 Pith paper

[9]

https:// github.com/tatsu-lab/stanford_alpaca

Stanford alpaca: An instruction-following llama model. https:// github.com/tatsu-lab/stanford_alpaca. Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2024a. Investigating mysteries of CoT-augmented distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 6071–6086, Miami, Florida, USA. Associ- ation ...

work page 2024
[10]

Preprint, arXiv:1910.03771

Hug- gingface’s transformers: State-of-the-art natural lan- guage processing. Preprint, arXiv:1910.03771. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou

arXiv 1910
[12]

Preprint, arXiv:1904.09675

Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. Appendix A Implementation Details We perform all experiments on two NVIDIA A100 GPUs. We use publicly available implementations of all models via the Huggingface library (Wolf et al., 2020). For all tasks, we use a learning rate of 3e−5 and a maximum input length of

arXiv 1904
[13]

is an instruction- following dataset designed to fine-tune large lan- guage models for improved task generalization. Originally containing 52,000 synthetically gener- ated instruction-response pairs from OpenAI’s text- davinci-003, this version is downsampled to 10,000 instances, excluding any that contain programming- related code for efficiency. The dat...

work page 2000
[2015]

arXiv preprint arXiv:1503.02531

Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Namgyu Ho, Laura Schmid, and Se-Young Yun

arXiv
[2017]

In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada

Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computa- tional Linguistics. Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F Siu, Byron C Wallace, and Ani Nenkova. 2024...

arXiv 2024
[2018]

In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium

Can a suit of armor conduct elec- tricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever

work page 2018
[2019]

CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. Rohan Tao...

work page 2019

Show all 13 references

[2020]

Linyang Li, Pengyu Wang, Ke Ren, Tianxiang Sun, and Xipeng Qiu

Rotten tomatoes movies and critic reviews dataset. Linyang Li, Pengyu Wang, Ke Ren, Tianxiang Sun, and Xipeng Qiu. 2023a. Origin tracing and detecting of llms. arXiv preprint arXiv:2304.14072. Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Cho...

2024 arXiv
[2021]

In Proceedings of the 2021 Conference of the Association for Com- putational Linguistics: Student Research Workshop

Sumpubmed: Summarization dataset of pubmed scientific article. In Proceedings of the 2021 Conference of the Association for Com- putational Linguistics: Student Research Workshop. Association for Computational Linguistics. Peter Hase and Mohit Bansal

2021
[2023]

In Findings of the Asso- ciation for Computational Linguistics: ACL 2023 , pages 7059–7073, Toronto, Canada

Distilling reasoning capabilities into smaller language models. In Findings of the Asso- ciation for Computational Linguistics: ACL 2023 , pages 7059–7073, Toronto, Canada. Association for Computational Linguistics. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen tau Yih, and A...

2023
[2024]

arXiv preprint arXiv:2402.13116

A survey on knowledge dis- tillation of large language models. arXiv preprint arXiv:2402.13116. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi

arXiv
[2025]

Accessed: 2025-01-

Did deepseek copy chatgpt to make new ai chatbot? AP News. Accessed: 2025-01-

2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [9]

https:// github.com/tatsu-lab/stanford_alpaca

Stanford alpaca: An instruction-following llama model. https:// github.com/tatsu-lab/stanford_alpaca. Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2024a. Investigating mysteries of CoT-augmented distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, pages 6071–6086, Miami, Florida, USA. Associ- ation ...

work page 2024

[2] [10]

Preprint, arXiv:1910.03771

Hug- gingface’s transformers: State-of-the-art natural lan- guage processing. Preprint, arXiv:1910.03771. Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou

arXiv 1910

[3] [12]

Preprint, arXiv:1904.09675

Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. Appendix A Implementation Details We perform all experiments on two NVIDIA A100 GPUs. We use publicly available implementations of all models via the Huggingface library (Wolf et al., 2020). For all tasks, we use a learning rate of 3e−5 and a maximum input length of

arXiv 1904

[4] [13]

is an instruction- following dataset designed to fine-tune large lan- guage models for improved task generalization. Originally containing 52,000 synthetically gener- ated instruction-response pairs from OpenAI’s text- davinci-003, this version is downsampled to 10,000 instances, excluding any that contain programming- related code for efficiency. The dat...

work page 2000

[5] [2015]

arXiv preprint arXiv:1503.02531

Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Namgyu Ho, Laura Schmid, and Se-Young Yun

arXiv

[6] [2017]

In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada

Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083, Vancouver, Canada. Association for Computa- tional Linguistics. Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F Siu, Byron C Wallace, and Ani Nenkova. 2024...

arXiv 2024

[7] [2018]

In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium

Can a suit of armor conduct elec- tricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever

work page 2018

[8] [2019]

CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. Rohan Tao...

work page 2019