Pith. sign in

REVIEW 3 cited by

Born Again Neural Networks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1805.04770 v2 pith:UNWG4IEP submitted 2018-05-12 stat.ML cs.AIcs.LG

Tommaso Furlanello , Zachary C. Lipton , Michael Tschannen , Laurent Itti , Anima Anandkumar This is my paper · ORCID
classification stat.MLcs.AIcs.LG
keywords knowledgeteacherperformancestudentbansdistillationexperimentsmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Knowledge Distillation (KD) consists of transferring “knowledge” from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the student’s compactness, without sacrificing too much performance. We study KD from a new perspective: rather than compressing models, we train students parameterized identically to their teachers. Surprisingly, these Born-Again Networks (BANs), outperform their teachers significantly, both on computer vision and language modeling tasks. Our experiments with BANs based on DenseNets demonstrate state-of-the-art performance on the CIFAR-10 (3.5%) and CIFAR-100 (15.5%) datasets, by validation error. Additional experiments explore two distillation objectives: (i) Confidence-Weighted by Teacher Max (CWTM) and (ii) Dark Knowledge with Permuted Predictions (DKPP). Both methods elucidate the essential components of KD, demonstrating the effect of the teacher outputs on both predicted and non-predicted classes.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. How Should We Meta-Learn Reinforcement Learning Algorithms?

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A systematic comparison of black-box evolution, neural and symbolic distillation, and LLM-based proposal for meta-learning RL algorithms yields practical recommendations: warm-started LLM proposal is sample-efficient,...

  2. Optimal Self-Distillation for Rectified Flow via Linear Probing

    stat.ML 2026-07 accept novelty 4.0 of 10

    For linear rectified flow with ridge regression on fixed interpolants, optimally mixed self-distillation strictly improves velocity risk whenever the teacher is off the ridge stationary point, with a closed-form mixin...

  3. Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Data

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Smooth-Distill applies EMA parameter averaging as a self-distillation teacher for multitask HAR and placement detection, and reports consistent but modest gains over multitask baselines.

Pith tools