Pith. sign in

REVIEW 5 major objections 5 minor 72 references

Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that regularizing a student with teacher logits of retrieved in-context neighbors—same-class positives pulled close, different-class negatives pushed apart—improves accuracy across offline, online, and teacher-free…

desk verdict Broad, consistent KD gains from an in-context retrieval regularizer, but the central retrieval mechanism is never ablated and the SOTA claim is too strong. read the letter →

arxiv 2501.07040 v1 pith:DUHRBIVR submitted 2025-01-13 cs.CV

Jinjing Zhu , Songze Li , Lin Wang This is my paper · ORCID
classification cs.CV
keywords knowledgedistillationin-contextsampleretrievallabelsmoothingregularizationcontrastivelearningfeaturememorybankmodelcompressionimageclassificationsemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conventional distillation asks the student to imitate the teacher's output on each individual sample. This paper argues that the knowledge worth transferring also includes how a sample relates to its in-context samples—similar examples of the same or different classes. Viewing distillation as learned label smoothing, it proposes IC-KD, which retrieves in-context samples from a teacher feature memory bank and regularizes the student with two extra losses: one aligns the student with an aggregation of same-class teacher logits, and the other repels the student from different-class teacher logits. The paper reports consistent accuracy gains over existing KD baselines on CIFAR-100 and ImageNet, plus gains on Cityscapes segmentation, across offline, online, and teacher-free KD. If the claim holds, distillation is best understood as retrieval-augmented regularization rather than pointwise imitation.

What carries the argument

The load-bearing object is the in-context sample: a set of similar training samples retrieved by cosine similarity in the teacher's feature space and separated by label into same-class positives and different-class negatives. A feature memory bank stores teacher features for the whole training set; for each query sample, Eq. (7) keeps the top-K same-class similarities, softmax-weights them, and aggregates the corresponding teacher logits into one softened target. Eq. (8), positive in-context distillation, is the KL divergence between that aggregated target and the student's prediction, and Eq. (10), negative in-context distillation, is a cosine-separation loss against different-class teacher predictions. The theoretical bridge is the label-smoothing view of distillation: if KD regularizes like label smoothing, then teacher predictions on other samples are legitimate regularizers, not noise to be ignored.

What would settle it

Replace teacher-feature retrieval with random same-class neighbor selection and re-run the IC-KD protocol on CIFAR-100; if accuracy stays the same, the retrieval mechanism is not what drives the reported gains.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that a student trained to match the teacher on a sample's retrieved in-context neighbors learns better than a student trained to match only per-sample outputs. For each sample, IC-KD builds a teacher feature memory bank, retrieves top-K same-class and top-N different-class neighbors by cosine similarity, and forms an aggregated teacher soft target from the positive neighbors. Positive in-context distillation (PICD) minimizes the KL divergence between that aggregated target and the student prediction, while negative in-context distillation (NICD) uses cosine separation to push the student away from different-class teacher predictions. The paper reports top-1 accuracy improvements over CRD and other baselines on CIFAR-100 and ImageNet, and mIoU improvements on Cityscapes, in offline, online, and teacher-free distillation settings.

Load-bearing premise

The method assumes that images judged similar by the teacher's feature space are the right images to borrow soft labels from, even after the student's own representation has drifted away from the teacher's during training.

Editorial extensions

If this is right

  • Any existing KD objective can be augmented with positive and negative in-context losses without changing the teacher or student architectures.
  • The label-smoothing framing predicts that the identity of the source of the aggregated target matters less than the target's quality, which is why the same regularizer transfers to teacher-free and online settings.
  • Retrieval from a precomputed teacher feature memory bank keeps the added cost mostly offline, so the method scales to ImageNet-sized problems.
  • Because the gains appear on classification and segmentation, in-context retrieval regularization is a general training signal rather than a task-specific trick.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same retrieval regularizer could be applied in feature space; its own limitation section identifies feature-space regularization as future work.
  • The negative loss's cosine repulsion resembles contrastive learning, so part of the gain may come from sharpening the student's representations rather than from label smoothing; a controlled study that swaps the negative loss for a plain cross-entropy term would separate these effects.
  • The fixed teacher feature bank could be replaced by a momentum-updated or student-aware bank, which may matter when teacher and student architectures differ strongly.
  • Since the method needs only a memory bank plus labels, it should transfer to semi-supervised or self-distillation settings without an external teacher, though the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes In-Context Knowledge Distillation (IC-KD), a method that reformulates the knowledge transferred in distillation as relationships between a query sample and retrieved "in-context" samples, rather than per-sample teacher outputs. A teacher feature memory bank is built, and for each training sample the method retrieves Top-K same-class neighbors and Top-N different-class neighbors by cosine similarity in the teacher feature space. Two auxiliary losses are added: Positive In-Context Distillation (PICD, Eq. 8), which matches the student's prediction to a similarity-weighted aggregate of teacher predictions over same-class neighbors, and Negative In-Context Distillation (NICD, Eq. 10), which pushes the student's prediction away from different-class teacher neighbors. The method is evaluated on CIFAR-100 and ImageNet classification, online and teacher-free KD, semantic segmentation on Cityscapes, and heterogeneous teacher-student architectures. The central claim is that this in-context-sample regularization improves student accuracy consistently across diverse KD paradigms.

Significance. If the central claim holds, the paper's main contribution is a simple, task-agnostic regularization that can be plugged into offline, online, and teacher-free KD and that improves accuracy across many teacher-student pairs. The experimental coverage is broad: 13 CIFAR-100 pairs, ImageNet, Cityscapes, online KD, teacher-free KD, and heterogeneous architectures, with consistent gains when IC-KD is added to KD and to CRD. The method is also relatively efficient, as shown by the training-time comparison in Table VII, and the proposed losses are easy to implement. The paper gives useful ablations of its loss components (Tables IV and V) and of the weighting terms (Table VI). However, the retrieval mechanism that defines the paper's novelty is not directly validated, the theoretical section is not a derivation, and the abstract's state-of-the-art claim is stronger than the tables support. No code or error bars are provided, which limits the certainty of the empirical claims.

major comments (5)
  1. [III-B, Eq. (7), and Tables I-II] The central claim that retrieval by teacher feature similarity is responsible for the observed gains is not tested. The paper never compares Top-K same-class retrieval with random same-class selection or with least-similar same-class selection. Without such a control, the accuracy improvements could come from any same-class soft-target regularization, not from the "in-context retrieval" mechanism on which the title and novelty rest. Figure 7 is qualitative and does not provide the missing control. I request an ablation on at least two representative pairs (e.g., ResNet32x4→ResNet8x4 and VGG13→VGG8) comparing (a) teacher-similarity Top-K, (b) random same-class, and (c) least-similar same-class selection, keeping all other loss terms fixed.
  2. [IV-B, Table IX (online KD)] In the online KD setting, the feature memory bank is rebuilt from student 1, which changes every epoch, rather than from the fixed teacher model. The paper's rationale that teacher feature similarity identifies useful in-context samples therefore does not directly apply, yet the method still reports gains. This is a load-bearing gap: please either justify why student-derived features can substitute for teacher features, or add an ablation comparing a fixed teacher-derived memory bank with the dynamic student-derived bank in the online setting.
  3. [III-A] The section titled "Rethinking 'Knowledge' in Distillation" is presented as a theoretical analysis, but it does not derive the central claim. Equations (1)-(4) restate the known Tf-KD result that KD is a form of learned label smoothing regularization. The subsequent statement that "the discrepancy between the teacher's predictions for in-context samples and the student's predictions for corresponding samples decreases as the similarity between these paired samples increases" is asserted, not proven, and the proposed losses in Eqs. (5)-(6) do not follow from the LSR equivalence. The authors should either provide a formal derivation under explicit assumptions (e.g., Lipschitz continuity of the logit map in the feature space) or reframe this section as motivation rather than theory.
  4. [Abstract and Tables I-III] The abstract claims that IC-KD "consistently achieves state-of-the-art performance on CIFAR-100 and ImageNet datasets." This is contradicted by the reported numbers: in Table I, DKD achieves 76.24 on WRN-40-2→WRN-16-2 while IC-KD achieves 76.21, and in Table II, several feature-based methods outperform IC-KD on multiple pairs (e.g., SimKD 78.39 vs. 77.52 on ResNet32x4→ShuffleNetV2; CAT-KD 78.41 vs. 77.52 on the same pair). Please replace "consistently achieves state-of-the-art" with a qualified claim such as "competitive with state-of-the-art methods," or specify the subset of settings for which the SOTA claim holds.
  5. [III-B, Eq. (10)] Equation (10) is ambiguous as written: L_nicd = 1 - cos(p_s_i, p_t_i) + b_i,j cos(p_s_i, p_t_j). The term b_i,j is indexed by j, but the expression has no summation or averaging over the retrieved negative set, even though the text refers to N negative in-context samples. Please clarify whether the intended loss is an average over retrieved negatives, and specify how the j index is selected in the notation. This is the definition of one of the two proposed losses, so the equation needs to be unambiguous.
minor comments (5)
  1. [Algorithm 1] Line 6 of Algorithm 1 reads "for t ← −1 to T"; this is likely a typo for "1 to T" or "0 to T−1" and should be corrected.
  2. [Table X caption] The caption lists the second teacher-student pair as "ResNet101 → ResNet18" twice; the second should refer to PSPNet-ResNet18, not ResNet101 again.
  3. [IV-A and all result tables] The paper states that each CIFAR-100 experiment is repeated three times, but no standard deviations are reported anywhere, and no code is released. Please report mean ± std or provide a reproducibility statement with code to support the claimed gains.
  4. [III-B, Eq. (7)] The aggregated teacher prediction is denoted \hat p_i^s, which is confusing because it is constructed from teacher logits and used as a target for the student. Renaming it to \hat p_i^t or another notation would improve clarity.
  5. [IV-A, teacher-free KD] In the teacher-free setting, the text says a trained baseline model serves as the teacher and an untrained model as the student; this is essentially offline KD and should be clarified, since "teacher-free" normally implies no pretrained teacher is used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central losses are regularizers validated on external test sets, not predictions constructed from their own definitions.

full rationale

IC-KD's central losses, L_picd (Eq. 8) and L_nicd (Eq. 10), are proposed training regularizers, not derived predictions; their support is external test accuracy on CIFAR-100, ImageNet, and Cityscapes. The LSR link (Eqs. 2-4) is an algebraic identity imported from the external prior work Tf-KD [28], which is not authored by the present authors, and it does not presuppose the in-context retrieval conclusion; it only recasts KD as smoothing with a learned target distribution. The in-context selection rules (Eqs. 7 and 9) are empirically motivated design choices, and no reported accuracy number is constructed from the definitions of the losses: all headline results come from held-out test sets after training. Hyperparameters are tuned on one teacher-student pair (Figs. 4-5) and then applied across other pairs, which is standard experimental practice rather than a fitted-input-called-prediction pattern. The absence of a random-retrieval ablation is a potential validity concern but not a circularity, because it does not make any equation reduce to its own input. The self-citations in the manuscript ([3], [15], [17], [26], [32]) concern other tasks and are not load-bearing for the present framework. No self-citation chain, uniqueness import, or ansatz-smuggling step sustains the central claim, so the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method introduces no new physical or architectural entities. Its free parameters are hyperparameters tuned on one CIFAR-100 pair, and its main assumptions are that teacher features are good retrieval keys and that aggregated in-context logits are useful soft targets. The LSR link is imported from prior work.

free parameters (6)
  • beta1 (similarity temperature for positive retrieval) = 1
    Selected from sensitivity study in Fig. 4 on ResNet32x4->ResNet8x4; not derived.
  • beta2 (negative retrieval temperature) = 4
    Selected from sensitivity study in Fig. 4; not derived.
  • K (number of positive in-context samples) = 100
    Tuned on one CIFAR-100 pair; performance peaks at K=100 in Fig. 4.
  • tau1 (softening temperature for PICD) = not specified
    Varies in Fig. 4; the paper reports a sensitivity study but not the final default value.
  • gamma_picd = 2
    Chosen from Fig. 5 sensitivity study on one pair.
  • gamma_nicd = 10
    Chosen from Fig. 5 sensitivity study on one pair.
assumptions (5)
  • domain assumption Knowledge distillation is a form of learned label smoothing regularization
    Used in Section III-A to justify adding extra soft targets; cited to Yuan et al. [28] and not re-derived here.
  • ad hoc to paper Aggregating teacher predictions over retrieved in-context samples benefits student training
    Eqs. (5) and (8) are proposed; no proof links them to lower generalization error. Supported only by experiments.
  • domain assumption Teacher feature cosine similarity is a valid retrieval key for in-context samples
    Eq. (7) selects neighbors by cosine similarity in teacher feature space; no theoretical justification is provided.
  • domain assumption Ground-truth labels are available to split positive and negative in-context samples
    The matrix M in Eqs. (7)-(9) uses labels; the method is supervised and cannot run label-free.
  • standard math Standard softmax, KL divergence, and cosine similarity operations are well-defined and differentiable
    These are standard building blocks used in Eq. (7), Eq. (8), and Eq. (10).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective." pith.science (2026). https://pith.science/paper/DUHRBIVR

@misc{pith2026250107040,
  author       = {Pith},
  title        = {Pith review of: Rethinking Knowledge in Distillation: An In-context Sample Retrieval Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUHRBIVR}},
  note         = {Machine review of arXiv:2501.07040}
}
read the original abstract

Conventional knowledge distillation (KD) approaches are designed for the student model to predict similar output as the teacher model for each sample. Unfortunately, the relationship across samples with same class is often neglected. In this paper, we explore to redefine the knowledge in distillation, capturing the relationship between each sample and its corresponding in-context samples (a group of similar samples with the same or different classes), and perform KD from an in-context sample retrieval perspective. As KD is a type of learned label smoothing regularization (LSR), we first conduct a theoretical analysis showing that the teacher's knowledge from the in-context samples is a crucial contributor to regularize the student training with the corresponding samples. Buttressed by the analysis, we propose a novel in-context knowledge distillation (IC-KD) framework that shows its superiority across diverse KD paradigms (offline, online, and teacher-free KD). Firstly, we construct a feature memory bank from the teacher model and retrieve in-context samples for each corresponding sample through retrieval-based learning. We then introduce Positive In-Context Distillation (PICD) to reduce the discrepancy between a sample from the student and the aggregated in-context samples with the same class from the teacher in the logit space. Moreover, Negative In-Context Distillation (NICD) is introduced to separate a sample from the student and the in-context samples with different classes from the teacher in the logit space. Extensive experiments demonstrate that IC-KD is effective across various types of KD, and consistently achieves state-of-the-art performance on CIFAR-100 and ImageNet datasets.

Figures

Figures reproduced from arXiv: 2501.07040 by the authors.

Figure 1
Figure 1. Illustration of four typical approaches: classic KD, Relation [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparative results of various teacher-student pairs [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of our framework. It consists of two feature extractors ( [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Effects of varying value of β1, K, τ1, and β2 on CIFAR-100 with ResNet32x4 → ResNet8x4. Accuracy (%) Accuracy (%) γ%"#$ γ!"#$ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Effects of varying the value of γpicd and γnicd on CIFAR-100 with ResNet32x4→ResNet8x4. TABLE VII: Training time (per epoch) vs. accuracy on CIFAR￾100 with with ResNet32x4 → ResNet8x4. Method Accuracy Time KD [14] 73.33 17.32s CRD [22] 75.51 23.34s Norm [13] 76.64 17.8…
Figure 6
Figure 6. Figure 6: The visualization of features using different models on CIFAR-100. (a) ResNet32x4, (b) ResNet8x4, (c) ResNet8x4 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Visualization of logits on CIFAR-100. The first column displays logit predictions for samples, while subsequent columns [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 39 canonical work pages

  1. [1]

    Very deep convolutional networks for large-scale image recognition,

    K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

    arXiv 2014

  2. [2]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  3. [3]

    Patch-mix transformer for unsupervised domain adaptation: A game perspective,

    J. Zhu, H. Bai, and L. Wang, “Patch-mix transformer for unsupervised domain adaptation: A game perspective,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 3561– 3571

    work page 2023

  4. [4]

    Object detection in 20 years: A survey,

    Z. Zou, K. Chen, Z. Shi, Y . Guo, and J. Ye, “Object detection in 20 years: A survey,” Proceedings of the IEEE , vol. 111, no. 3, pp. 257–276, 2023

    work page 2023

  5. [5]

    Object detection with deep learning: A review,

    Z.-Q. Zhao, P. Zheng, S.-t. Xu, and X. Wu, “Object detection with deep learning: A review,” IEEE transactions on neural networks and learning systems, vol. 30, no. 11, pp. 3212–3232, 2019

    2019

  6. [6]

    Fully convolutional networks for semantic segmentation,

    J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 3431–3440

    2015

  7. [7]

    Encoder- decoder with atrous separable convolution for semantic image segmenta- tion,

    L.-C. Chen, Y . Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder- decoder with atrous separable convolution for semantic image segmenta- tion,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 801–818

    work page 2018

  8. [8]

    Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer,

    S. Mehta and M. Rastegari, “Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer,” arXiv preprint arXiv:2110.02178, 2021

    arXiv 2021

Show all 72 references
  1. [9]

    Tinyvit: Fast pretraining distillation for small vision transformers,

    K. Wu, J. Zhang, H. Peng, M. Liu, B. Xiao, J. Fu, and L. Yuan, “Tinyvit: Fast pretraining distillation for small vision transformers,” in European Conference on Computer Vision . Springer, 2022, pp. 68–85

    2022

  2. [10]

    Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,

    L. Wang and K. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,” IEEE Trans. Pattern Anal. Mach. Intell. , vol. 44, no. 6, pp. 3048–3068, 2022

    2022

  3. [11]

    Densely guided knowledge distillation using multiple teacher assistants,

    W. Son, J. Na, J. Choi, and W. Hwang, “Densely guided knowledge distillation using multiple teacher assistants,” in 2021 IEEE/CVF Inter- national Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 . IEEE, 2021, pp. 9375–9384

    2021

  4. [12]

    Learning student- friendly teacher networks for knowledge distillation,

    D. Y . Park, M. Cha, C. Jeong, D. Kim, and B. Han, “Learning student- friendly teacher networks for knowledge distillation,” in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2...

    2021

  5. [13]

    Norm: Knowledge distillation via n-to-one representation matching,

    X. Liu, L. Li, C. Li, and A. Yao, “Norm: Knowledge distillation via n-to-one representation matching,” arXiv preprint arXiv:2305.13803 , 2023

    2023 arXiv

  6. [14]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  7. [15]

    A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation,

    J. Zhu, Y . Luo, X. Zheng, H. Wang, and L. Wang, “A good student is cooperative and reliable: Cnn-transformer collaborative learning for semantic segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 11 720–11 730

    2023

  8. [16]

    Cross- image relational knowledge distillation for semantic segmentation,

    C. Yang, H. Zhou, Z. An, X. Jiang, Y . Xu, and Q. Zhang, “Cross- image relational knowledge distillation for semantic segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 12 319–12 328

    2022

  9. [17]

    Goodsam: Bridging domain and capacity gaps via segment anything model for distortion-aware panoramic semantic segmentation,

    W. Zhang, Y . Liu, X. Zheng, and L. Wang, “Goodsam: Bridging domain and capacity gaps via segment anything model for distortion-aware panoramic semantic segmentation,” arXiv preprint arXiv:2403.16370 , 2024

    2024 arXiv

  10. [18]

    Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,

    S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,” arXiv preprint arXiv:1612.03928 , 2016

    2016 arXiv

  11. [19]

    Channel-wise knowledge distillation for dense prediction,

    C. Shu, Y . Liu, J. Gao, Z. Yan, and C. Shen, “Channel-wise knowledge distillation for dense prediction,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 5311–5320

    2021

  12. [20]

    Exploring inter-channel correlation for diversity-preserved knowledge distillation,

    L. Liu, Q. Huang, S. Lin, H. Xie, B. Wang, X. Chang, and X. Liang, “Exploring inter-channel correlation for diversity-preserved knowledge distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 8271–8280

    2021

  13. [21]

    Correlation congruence for knowledge distillation,

    B. Peng, X. Jin, J. Liu, D. Li, Y . Wu, Y . Liu, S. Zhou, and Z. Zhang, “Correlation congruence for knowledge distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019, pp. 5007–5016

    2019

  14. [22]

    Contrastive representation distillation,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive representation distillation,” arXiv preprint arXiv:1910.10699 , 2019

    1910 arXiv

  15. [23]

    Similarity-preserving knowledge distillation,

    F. Tung and G. Mori, “Similarity-preserving knowledge distillation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1365–1374

    2019

  16. [24]

    Knowledge distillation from a stronger teacher,

    T. Huang, S. You, F. Wang, C. Qian, and C. Xu, “Knowledge distillation from a stronger teacher,” Advances in Neural Information Processing Systems, vol. 35, pp. 33 716–33 727, 2022

    2022

  17. [25]

    Relational knowledge distillation,

    W. Park, D. Kim, Y . Lu, and M. Cho, “Relational knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3967–3976

    2019

  18. [26]

    Complementary relation contrastive distillation,

    J. Zhu, S. Tang, D. Chen, S. Yu, Y . Liu, M. Rong, A. Yang, and X. Wang, “Complementary relation contrastive distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 9260–9269

    2021

  19. [27]

    When does label smoothing help?

    R. Müller, S. Kornblith, and G. E. Hinton, “When does label smoothing help?” Advances in neural information processing systems , vol. 32, 2019

    2019

  20. [28]

    Revisiting knowledge distillation via label smoothing regularization,

    L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng, “Revisiting knowledge distillation via label smoothing regularization,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 3903–3911

    2020

  21. [29]

    Revisiting label smoothing and knowledge distillation compatibility: What was missing?

    K. Chandrasegaran, N.-T. Tran, Y . Zhao, and N.-M. Cheung, “Revisiting label smoothing and knowledge distillation compatibility: What was missing?” in International Conference on Machine Learning . PMLR, 2022, pp. 2890–2916

    2022

  22. [30]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,” International Journal of Computer Vision , vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  23. [31]

    Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,

    L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,” IEEE transactions on pattern analysis and machine intelligence , vol. 44, no. 6, pp. 3048–3068, 2021

    2021

  24. [32]

    Source-free cross-modal knowledge transfer by unleashing the potential of task-irrelevant data,

    J. Zhu, Y . Chen, and L. Wang, “Source-free cross-modal knowledge transfer by unleashing the potential of task-irrelevant data,” arXiv preprint arXiv:2401.05014, 2024

    2024 arXiv

  25. [33]

    Decoupled knowledge distillation,

    B. Zhao, Q. Cui, R. Song, Y . Qiu, and J. Liang, “Decoupled knowledge distillation,” in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , 2022, pp. 11 953–11 962

    2022

  26. [34]

    Structured knowledge distillation for semantic segmentation,

    Y . Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang, “Structured knowledge distillation for semantic segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2019, pp. 2604–2613

    2019

  27. [35]

    Intra-class feature variation distillation for semantic segmentation,

    Y . Wang, W. Zhou, T. Jiang, X. Bai, and Y . Xu, “Intra-class feature variation distillation for semantic segmentation,” in Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16 . Springer, 2020, pp. 346–362

    2020

  28. [36]

    Crosskd: Cross-head knowledge distillation for dense object detection,

    J. Wang, Y . Chen, Z. Zheng, X. Li, M.-M. Cheng, and Q. Hou, “Crosskd: Cross-head knowledge distillation for dense object detection,” arXiv preprint arXiv:2306.11369, 2023

    2023 arXiv

  29. [37]

    Data-free knowledge transfer: A survey,

    Y . Liu, W. Zhang, J. Wang, and J. Wang, “Data-free knowledge transfer: A survey,” arXiv preprint arXiv:2112.15278 , 2021. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

    2021 arXiv

  30. [38]

    Fitnets: Hints for thin deep nets,

    R. Adriana, B. Nicolas, K. S. Ebrahimi, C. Antoine, G. Carlo, and B. Yoshua, “Fitnets: Hints for thin deep nets,” Proc. ICLR, vol. 2, no. 3, p. 1, 2015

    2015

  31. [39]

    Knowledge distillation with the reused teacher classifier,

    D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y . Feng, and C. Chen, “Knowledge distillation with the reused teacher classifier,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 11 933–11 942

    2022

  32. [40]

    Reta-llm: A retrieval-augmented large language model toolkit,

    J. Liu, J. Jin, Z. Wang, J. Cheng, Z. Dou, and J.-R. Wen, “Reta-llm: A retrieval-augmented large language model toolkit,” arXiv preprint arXiv:2306.05212, 2023

    2023 arXiv

  33. [41]

    Retrieve anything to augment large language models,

    P. Zhang, S. Xiao, Z. Liu, Z. Dou, and J.-Y . Nie, “Retrieve anything to augment large language models,” arXiv preprint arXiv:2310.07554 , 2023

    2023

  34. [42]

    Lift yourself up: Retrieval-augmented text generation with self-memory,

    X. Cheng, D. Luo, X. Chen, L. Liu, D. Zhao, and R. Yan, “Lift yourself up: Retrieval-augmented text generation with self-memory,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  35. [43]

    Recent advances in retrieval- augmented text generation,

    D. Cai, Y . Wang, L. Liu, and S. Shi, “Recent advances in retrieval- augmented text generation,” in Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, 2022, pp. 3417–3419

    2022

  36. [44]

    Image retrieval: Ideas, influences, and trends of the new age,

    R. Datta, D. Joshi, J. Li, and J. Z. Wang, “Image retrieval: Ideas, influences, and trends of the new age,” ACM Computing Surveys (Csur) , vol. 40, no. 2, pp. 1–60, 2008

    2008

  37. [45]

    Features for image retrieval: an experimental comparison,

    T. Deselaers, D. Keysers, and H. Ney, “Features for image retrieval: an experimental comparison,” Information retrieval, vol. 11, pp. 77–107, 2008

    2008

  38. [46]

    Pic2word: Mapping pictures to words for zero-shot composed image retrieval,

    K. Saito, K. Sohn, X. Zhang, C.-L. Li, C.-Y . Lee, K. Saenko, and T. Pfister, “Pic2word: Mapping pictures to words for zero-shot composed image retrieval,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 19 305–19 314

    2023

  39. [47]

    Learning semantic relationship among instances for image-text matching,

    Z. Fu, Z. Mao, Y . Song, and Y . Zhang, “Learning semantic relationship among instances for image-text matching,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 15 159–15 168

    2023

  40. [48]

    Dynamic modality interaction modeling for image-text retrieval,

    L. Qu, M. Liu, J. Wu, Z. Gao, and L. Nie, “Dynamic modality interaction modeling for image-text retrieval,” in Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 1104–1113

    2021

  41. [49]

    Teachtext: Crossmodal generalized distillation for text-video retrieval,

    I. Croitoru, S.-V . Bogolin, M. Leordeanu, H. Jin, A. Zisserman, S. Albanie, and Y . Liu, “Teachtext: Crossmodal generalized distillation for text-video retrieval,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 11 583–11 593

    2021

  42. [50]

    Towards in-context scene understanding,

    I. Balazevic, D. Steiner, N. Parthasarathy, R. Arandjelovi´c, and O. Henaff, “Towards in-context scene understanding,” Advances in Neural Informa- tion Processing Systems , vol. 36, 2024

    2024

  43. [51]

    What makes good examples for visual in- context learning?

    Y . Zhang, K. Zhou, and Z. Liu, “What makes good examples for visual in- context learning?” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  44. [52]

    Representation learning with contrastive predictive coding,

    A. v. d. Oord, Y . Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748 , 2018

    2018 arXiv

  45. [53]

    Logit standardization in knowledge distillation,

    S. Sun, W. Ren, J. Li, R. Wang, and X. Cao, “Logit standardization in knowledge distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 15 731–15 740

    2024

  46. [54]

    Like what you like: Knowledge distill via neuron selectivity transfer,

    Z. Huang and N. Wang, “Like what you like: Knowledge distill via neuron selectivity transfer,” arXiv preprint arXiv:1707.01219 , 2017

    2017 arXiv

  47. [55]

    Paraphrasing complex network: Network compression via factor transfer,

    J. Kim, S. Park, and N. Kwak, “Paraphrasing complex network: Network compression via factor transfer,” Advances in neural information processing systems, vol. 31, 2018

    2018

  48. [56]

    Learning deep representations with probabilis- tic knowledge transfer,

    N. Passalis and A. Tefas, “Learning deep representations with probabilis- tic knowledge transfer,” in Proceedings of the European Conference on Computer Vision (ECCV) , 2018, pp. 268–284

    2018

  49. [57]

    Knowledge transfer via distillation of activation boundaries formed by hidden neurons,

    B. Heo, M. Lee, S. Yun, and J. Y . Choi, “Knowledge transfer via distillation of activation boundaries formed by hidden neurons,” in Proceedings of the AAAI conference on artificial intelligence , vol. 33, no. 01, 2019, pp. 3779–3787

    2019

  50. [58]

    Distilling knowledge via knowledge review,

    P. Chen, S. Liu, H. Zhao, and J. Jia, “Distilling knowledge via knowledge review,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 5008–5017

    2021

  51. [59]

    Class attention transfer based knowledge distillation,

    Z. Guo, H. Yan, H. Li, and X. Lin, “Class attention transfer based knowledge distillation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 11 868–11 877

    2023

  52. [60]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009

    2009

  53. [61]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition . Ieee, 2009, pp. 248–255

    2009

  54. [62]

    Going deeper with convolutions,

    C. Szegedy, W. Liu, Y . Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9

    2015

  55. [63]

    Densely connected convolutional networks,

    G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , 2017, pp. 4700–4708

    2017

  56. [64]

    Online knowledge distillation via collaborative learning,

    Q. Guo, X. Wang, Y . Wu, Z. Yu, D. Liang, X. Hu, and P. Luo, “Online knowledge distillation via collaborative learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 11 020–11 029

    2020

  57. [65]

    A comprehensive overhaul of feature distillation,

    B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y . Choi, “A comprehensive overhaul of feature distillation,” in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 1921– 1930

    2019

  58. [66]

    Knowledge distillation via softmax regression representation learning

    J. Yang, B. Martinez, A. Bulat, G. Tzimiropoulos et al., “Knowledge distillation via softmax regression representation learning.” International Conference on Learning Representations (ICLR), 2021

    2021

  59. [67]

    Cross-layer distillation with semantic calibration,

    D. Chen, J.-P. Mei, Y . Zhang, C. Wang, Z. Wang, Y . Feng, and C. Chen, “Cross-layer distillation with semantic calibration,” in Proceedings of the AAAI conference on artificial intelligence , vol. 35, no. 8, 2021, pp. 7028–7036

    2021

  60. [68]

    Distpro: Searching a fast knowledge distillation process via meta optimization,

    X. Deng, D. Sun, S. Newsam, and P. Wang, “Distpro: Searching a fast knowledge distillation process via meta optimization,” in European Conference on Computer Vision . Springer, 2022, pp. 218–235

    2022

  61. [69]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008

    2008

  62. [70]

    Deep mutual learning,

    Y . Zhang, T. Xiang, T. M. Hospedales, and H. Lu, “Deep mutual learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4320–4328

    2018

  63. [71]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022

    2021

  64. [72]

    The cityscapes dataset for semantic urban scene understanding,

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 3213–3223

    2016

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.