REVIEW 4 cited by
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
read the original abstract
Knowledge distillation can be a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached. However, successfully applying this to pre-training remains largely unexplored. In this work, we prove that naive approaches for sparse knowledge distillation such as caching Top-K probabilities, while intuitive, provide biased estimates of teacher probability distribution to the student, resulting in suboptimal performance and calibration. We propose an importance-sampling-based method `Random Sampling Knowledge Distillation', which provides unbiased estimates, preserves the gradient in expectation, and requires storing significantly sparser logits. Our method enables faster training of student models with marginal overhead (<10%) compared to cross-entropy based training, while maintaining competitive performance compared to full distillation, across a range of model sizes from 300M to 3B.
Forward citations
Cited by 4 Pith papers
-
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Byte-Prefix Marginalization maps a teacher's next-token distribution onto the student's vocabulary through shared byte prefixes plus an explicit residual, giving a mass-preserving target for on-policy distillation acr...
-
When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation
Teacher top-K distillation drops the low-probability <tool call> token from the response teacher's support, creating a one-sided gradient that causally raises tool over-calling.
-
When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation
Multi-teacher on-policy distillation can cause tool over-calling due to disproportionate signals at mode-entry tokens, and a per-token divergence calibration method called Soft Clamp mitigates this shift.
-
When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation
Top-K teacher logits can preserve almost all probability mass while dropping the low-probability '<tool call>' token, biasing distilled models to over-use tools; restoring the omitted token reduces over-calling 14.2%→...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.