REVIEW 13 cited by
Patient Knowledge Distillation for BERT Model Compression
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Pre-trained language models such as BERT have proven to be highly effective for natural language processing (NLP) tasks. However, the high demand for computing resources in training such models hinders their application in practice. In order to alleviate this resource hunger in large-scale model training, we propose a Patient Knowledge Distillation approach to compress an original large model (teacher) into an equally-effective lightweight shallow network (student). Different from previous knowledge distillation methods, which only use the output from the last layer of the teacher network for distillation, our student model patiently learns from multiple intermediate layers of the teacher model for incremental knowledge extraction, following two strategies: ($i$) PKD-Last: learning from the last $k$ layers; and ($ii$) PKD-Skip: learning from every $k$ layers. These two patient distillation schemes enable the exploitation of rich information in the teacher's hidden layers, and encourage the student model to patiently learn from and imitate the teacher through a multi-layer distillation process. Empirically, this translates into improved results on multiple NLP tasks with significant gain in training efficiency, without sacrificing model accuracy.
Forward citations
Cited by 13 Pith papers
-
Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAG
Teacher-regularized on-policy RL with a reverse-KL anchor on student prefixes improves language consistency and evidence-grounded correctness for English-evidence cross-lingual RAG, acting as a safety net against larg...
-
NITP: Next Implicit Token Prediction for LLM Pre-training
NITP augments standard next-token prediction with implicit semantic prediction in representation space using shallow-layer self-supervision, reporting consistent downstream gains on 0.5B-9B models including 5.7% on MM...
-
MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants
Training small language models on intermediate-length reasoning chains from a merged mid-sized teacher assistant improves their math reasoning scores over direct distillation from a large teacher.
-
GenRecal: Generation after Recalibration from Large to Small Vision-Language Models
A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.
-
Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers
Squeezing-Heads Distillation mixes several teacher attention maps into one per-sample weighted map, enabling knowledge distillation between transformers with different head counts without extra parameters.
-
MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning
MaintaAvatar continually adds new appearances to a NeRF human avatar from a few images per task and retains old appearances via replay, per-appearance triplanes, and pose distillation.
-
Progressive$^2$: A Teacher-Student Progressive Co-Evolving Knowledge Distillation Method for Substantial Model Compression
Progressive^2 improves knowledge distillation under large teacher-student capacity gaps by progressively including teacher layers and gradually compressing the student through self-distillation rounds.
-
Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
Structural pruning with finetuning plus hidden-state distillation recovers most performance in multimodal LLMs, with 5% of training data sufficient at moderate compression levels.
-
AgentDistill: Training-Free Agent Distillation with Generalizable MCP Boxes
AgentDistill distills agent capabilities without any training by having a teacher generate reusable MCP tool boxes that small-model students invoke at inference time.
-
Olica: Efficient Structured Pruning of Large Language Models without Retraining
Olica prunes LLMs without retraining by low-rank decomposing attention-layer matrix products and adding small linear corrections to pruned feed-forward layers, matching or beating retraining-based methods on several b...
-
BnBERT-iPET: Sparse Few-Shot Language Modeling for Bengali via Lottery Ticket Pruning
A 90%-pruned few-shot Bengali model is reported to rival larger baselines on some tasks, but the reported F1 scores contradict the paper's own precision and recall values.
-
Enhancing deep learning models for time series classification via knowledge distillation
Knowledge distillation most benefits intermediate-complexity students for time series classification, cutting parameters sharply while matching teacher accuracy across FCN, Inception, and ConvTran on UCR.
-
TinyMusician: On-Device Music Generation with Knowledge Distillation and Mixed Precision Quantization
TinyMusician distills MusicGen and applies hand-picked mixed-precision quantization to make a 1.04 GB on-device music generator, but the headline '93% quality, 55% smaller' claims conflict with the paper's own tables.
Discussion (0). Continue with ORCID to comment.