EKD trains lightweight NMT students progressively from a chain of teachers with rising capacity, achieving BLEU scores within 0.08 of the largest teacher on IWSLT-14.
Pre-training distillation for large language models: A design space exploration
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
years
2026 3roles
background 1polarities
background 1representative citing papers
Pruning pretrained MoE models outperforms training from scratch under fixed budget, different expert compression methods converge after continued training, and progressive pruning plus multi-token KD improves the final 23A2B model.
Distillation and quantization expand the Apertus 8B LLM into a family of models up to 4B parameters with claimed strong accuracy and cost efficiency.
citing papers explorer
-
Evolving Knowledge Distillation for Lightweight Neural Machine Translation
EKD trains lightweight NMT students progressively from a chain of teachers with rising capacity, achieving BLEU scores within 0.08 of the largest teacher on IWSLT-14.
-
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
Pruning pretrained MoE models outperforms training from scratch under fixed budget, different expert compression methods converge after continued training, and progressive pruning plus multi-token KD improves the final 23A2B model.
-
Apertus LLM Family Expansion via Distillation and Quantization
Distillation and quantization expand the Apertus 8B LLM into a family of models up to 4B parameters with claimed strong accuracy and cost efficiency.