Pith. sign in

REVIEW 8 cited by

Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2105.08919 v1 pith:HCZDY465 submitted 2021-05-19 cs.LG cs.CV

Taehyeon Kim , Jaehoon Oh , NakYil Kim , Sangwook Cho , Se-Young Yun This is my paper · ORCID
classification cs.LGcs.CV
keywords lossmodeldivergencelogitdistillationknowledgematchingstudent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Knowledge distillation (KD), transferring knowledge from a cumbersome teacher model to a lightweight student model, has been investigated to design efficient neural architectures. Generally, the objective function of KD is the Kullback-Leibler (KL) divergence loss between the softened probability distributions of the teacher model and the student model with the temperature scaling hyperparameter tau. Despite its widespread use, few studies have discussed the influence of such softening on generalization. Here, we theoretically show that the KL divergence loss focuses on the logit matching when tau increases and the label matching when tau goes to 0 and empirically show that the logit matching is positively correlated to performance improvement in general. From this observation, we consider an intuitive KD loss function, the mean squared error (MSE) between the logit vectors, so that the student model can directly learn the logit of the teacher model. The MSE loss outperforms the KL divergence loss, explained by the difference in the penultimate layer representations between the two losses. Furthermore, we show that sequential distillation can improve performance and that KD, particularly when using the KL divergence loss with small tau, mitigates the label noise. The code to reproduce the experiments is publicly available online at https://github.com/jhoon-oh/kd_data/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Balancing Usefulness and Naturalness: An LLM-based Curation Pipeline for Code Review Comments

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Selective LLM reformulation guided by high-quality exemplars yields cleaner, more diverse code-review datasets that improve downstream comment generation and code refinement.

  2. Leveraging Reward Models for Guiding Code Review Comment Generation

    cs.SE 2025-06 conditional novelty 6.0 of 10

    CoRAL fine-tunes a language model with reinforcement learning using reward models derived from comment semantics and code refinement success, and reports improved code review comment quality.

  3. Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework

    cs.CL 2025-07 conditional novelty 5.0 of 10

    EI-BERT compresses a Chinese NLU model to 1.91 MB with competitive accuracy using attention-based vocabulary pruning, cross-distillation, and module-wise INT8 quantization, and reports deployment at Alipay.

  4. Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking

    cs.CV 2024-12 conditional novelty 5.0 of 10

    An adaptive block-activation ViT and a mutual-information multi-teacher distillation variant achieve state-of-the-art speed/accuracy trade-offs on six UAV tracking benchmarks.

  5. One-Shot Federated Clustering of Non-Independent Completely Distributed Data

    cs.LG 2026-01 reject novelty 4.0 of 10

    GOLD is a one-shot federated clustering framework that finds fine local subclusters, merges their centroids at several granularities on the server, and encodes the multi-granular partitions into an enhanced representa...

  6. Compute Can't Handle the Truth: Why Communication Tax Prioritizes Memory and Interconnects in Modern AI Infrastructure

    cs.DC 2025-07 reject novelty 4.0 of 10

    A CXL-based disaggregated memory architecture with hybrid XLink interconnects is proposed and prototyped, claiming large speedups for memory-bound AI and HPC workloads.

  7. Efficient Split Learning LSTM Models for FPGA-based Edge IoT Devices

    cs.LG 2025-02 conditional novelty 4.0 of 10

    A case study showing how knowledge distillation, pruning, and quantization let a small LSTM run on a low-end FPGA, with three split configurations trading off latency, power, and resource usage.

  8. Enhancing CLIP Conceptual Embedding through Knowledge Distillation

    cs.AI 2024-12 reject novelty 4.0 of 10

    Knowledge-CLIP distills Llama 2 embeddings into CLIP and uses k-means soft concept labels to slightly improve CLIP text and image encoder scores on three benchmarks.

Pith tools