REVIEW 2 cited by
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Knowledge distillation addresses the problem of transferring knowledge from a teacher model to a student model. In this process, we typically have multiple types of knowledge extracted from the teacher model. The problem is to make full use of them to train the student model. Our preliminary study shows that: (1) not all of the knowledge is necessary for learning a good student model, and (2) knowledge distillation can benefit from certain knowledge at different training steps. In response to these, we propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation. In addition, we offer a refinement of the training algorithm to ease the computational burden. Experimental results on the GLUE datasets show that our method outperforms several strong knowledge distillation baselines significantly.
Forward citations
Cited by 2 Pith papers
-
Active Data Curation Effectively Distills Large-Scale Multimodal Models
Selecting training data by a reference model's loss acts as an implicit distillation, and combining it with explicit distillation yields more FLOP-efficient vision-language models that beat prior SoTA on 27 benchmarks.
-
Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models
DynSDPB fine-tunes small language models by self-distilling soft labels from the previous mini-batch, with dynamic per-sample temperature and loss weighting.
Discussion (0). Continue with ORCID to comment.