Pith. sign in

REVIEW 2 major objections 6 minor 70 references

Equally Critical: Samples, Targets, and Their Mappings in Datasets

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One shared soft target per image trains faster and better

desk verdict Useful taxonomy and a simple plausible trick, but the headline claim that Strategy C beats traditional knowledge distillation is untested because no standard end-to-end KD baseline is run. read the letter →

arxiv 2506.01987 v1 pith:4DZFW5FQ submitted 2025-05-17 cs.LG cs.AI

Runkang Yang , Peng Sun , Xinyi Shang , Yi Tang , Tao Lin This is my paper · ORCID
classification cs.LGcs.AI
keywords sample-to-targetmappingknowledgedistillationsofttargetsdataaugmentationtrainingefficiencydecoupledscalinglawsdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that how samples are paired with targets in a dataset—the sample-to-target mapping—is as important as the samples or targets themselves. It proposes a unified training framework that separates backbone and classifier training, and uses it to compare three mapping strategies: one-hot class targets shared by all samples in a class, a unique soft teacher target for every augmented view, and one shared soft target for all augmented views of one original image. The central discovery is that the shared-soft-target strategy combines the early-training speed of distillation with better final accuracy than either one-hot targets or per-view soft targets. If correct, this means a simple relabeling rule can extract both speed and accuracy from the same teacher, and that conclusions about when and how to use distillation depend on the mapping, not just on the targets.

What carries the argument

The central object is the sample-to-target mapping strategy ψ: X→Y, with three cases: A maps all augmented samples of a class to one one-hot target, B maps each augmented sample to its own teacher soft target, and C maps all augmented views of one original sample to that sample's single teacher soft target. The argument is carried by a decoupled loss: the backbone is trained with KL divergence to the teacher's soft targets through a softmax head, while the classifier is trained independently with cross-entropy on one-hot labels, isolating the mapping strategy from classifier effects. This machinery lets the paper attribute differences in speed and converged accuracy to the mapping itself.

What would settle it

Train a student with ordinary end-to-end knowledge distillation—classifier attached, cross-entropy plus KL divergence to teacher outputs—and compare per-view soft targets with shared-per-original-sample soft targets under identical teachers and augmentations. If the shared-target strategy does not exceed the per-view strategy in final converged accuracy while keeping an early-training speedup, the central claim fails. A simpler check is already visible in Table 1, where the standard loss makes students consistently worse than their teachers, so the claim should be re-tested with that standard loss.

Watch

Extended reading notes

Core claim

Under the paper's decoupled loss—backbone trained by KL divergence to teacher soft targets, classifier trained separately with cross-entropy on one-hot labels—the strategy that maps all augmented views of one original sample to that sample's single soft target consistently reaches higher final accuracy than the strategy that gives each augmented view its own soft target, while still accelerating early training relative to one-hot targets. The advantage is largest with weak teachers: on CIFAR-10 a teacher at 30% accuracy yields a 1.35x gain over per-view soft targets, shrinking to 1.01x for a 90% teacher. The paper also reports that stronger teachers generally help final accuracy but weaker teachers help early learning, that MixUp-trained teachers do not consistently help students, and that with enough data one-hot targets overtake soft targets.

Load-bearing premise

The strongest assumption is that training the backbone separately from the classifier with the paper's custom loss reflects what would happen under ordinary end-to-end training; Table 1 shows the loss choice changes whether a student can beat its teacher, so the reported advantage over traditional distillation could depend on this protocol.

Editorial extensions

If this is right

  • Using C, a practitioner can relabel augmented views with the teacher's soft target for the original image and obtain faster early progress without sacrificing converged accuracy.
  • Weaker teachers are sufficient for much of C's benefit, so the method lowers the cost of teacher preparation while improving final accuracy.
  • The optimal mapping depends on dataset size: soft targets help under limited data, while one-hot targets regain the advantage when samples are abundant.
  • Augmentation choices interact with the mapping: RandomResizedCrop is most effective with one-hot targets, while mix-based augmentations pair better with soft targets, and mismatched teacher-student augmentations can hurt.
  • The reported benefit of C over per-view soft targets is largest for teachers of moderate accuracy, so the choice of teacher quality should be tuned against the mapping strategy rather than assumed optimal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the shared-target idea could apply to dataset distillation, where synthetic images are currently paired with per-sample soft targets; mapping all synthetic views of one prototype to one soft target may reduce label noise there too.
  • Editorial extension: the finding that weak teachers help early learning and strong teachers help late learning implies an annealing schedule—start with weak targets and switch to strong—might dominate either fixed choice; the paper does not test this.
  • Editorial extension: since student accuracy under soft targets caps near teacher accuracy, a strategy that periodically refreshes soft targets from the student itself could break that ceiling; this is an inference, not a paper claim.
  • Editorial extension: the three-strategy comparison is a natural template for text and speech data, where augmentation is less standardized and the mapping between augmented views and targets may behave differently.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies how the mapping between training samples and their targets affects supervised learning. It introduces three mapping strategies: Strategy A (multiple augmented samples within a class share one one-hot target), Strategy B (each augmented sample receives a unique soft target from a teacher), and the proposed Strategy C (all augmented views of one original image share a single teacher soft target). To evaluate these strategies, the authors propose a decoupled loss (Eqs. 1-3) that trains the backbone with KL divergence to soft targets and trains the classifier separately with cross-entropy on hard labels. Extensive experiments on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet with ResNet, MobileNetV2, EfficientNet, and ViT backbones lead to six findings, the central one being finding (c) and Claim 1: Strategy C gives higher final converged accuracy than traditional knowledge distillation while retaining early-stage acceleration.

Significance. If the central claim holds, the paper offers a practically valuable and surprisingly simple recipe: under the proposed decoupled training protocol, mapping all augmented views of an image to a single teacher soft target (rather than one soft target per augmented view) improves both early progress and final accuracy. The taxonomy of sample-to-target mappings and the systematic empirical map of how teacher accuracy, augmentation, and sample quantity interact are useful contributions to the understanding of knowledge distillation and dataset design. The paper is strong on experimental thoroughness: detailed protocol in Appendix B, multiple datasets and architectures, standard deviations on the main comparisons, and a stated code release. These strengths support reproducibility and make the empirical claims easy to check.

major comments (2)
  1. [§3.1, Claim 1; Eqs. (1)-(3); Table 1] Finding (c) and Claim 1 claim that STRATEGY C achieves higher final accuracy than 'traditional knowledge distillation,' but the only operational baseline for STRATEGY C is STRATEGY B under the authors' decoupled loss (Eqs. 1-3). Traditional Hinton-style KD trains the student end-to-end with CE plus KL on the same logits, which is exactly what Table 1's 'Standard' row represents. Table 1 shows that the loss function alone changes whether the student surpasses its teacher, so the C-vs-B gains in Tables 3, 6, and 7 and Figures 3-4 cannot be attributed to the mapping strategy rather than to the decoupled protocol. The paper should either add a standard end-to-end KD control for STRATEGY C (and STRATEGY B) or explicitly restrict the claim to the decoupled-protocol setting. The limitations discussion in Section 5, which currently only mentions the computer-vision scope, should also state this protocol dependence.
  2. [Appendix B, 'Pre-trained Teacher Models'; §3.2, Claim 2] The teacher models with preset accuracies from 10% to 90% are obtained by saving checkpoints the first time the target accuracy is reached, meaning the 'weak teachers' are early-stopped snapshots along a single training trajectory rather than independently trained models of different strength (e.g., smaller capacity or less data). The paper repeatedly interprets results as effects of 'teacher strength' (Claim 2, Section 3.2, Table 3, Figure 5b), but the actual manipulated variable is training time, which is confounded with feature maturity, output calibration, and other properties. These claims should be reframed as being about early-stopped teachers, or supported by additional experiments with genuinely weak teachers, before the conclusions about teacher strength can be taken as general.
minor comments (6)
  1. [Abstract] The phrase 'training dynamic' should be 'training dynamics.'
  2. [§2.2.2] Calling the softmax layer g 'novel' is overstated; it is essentially a standard linear classification head used in the decoupled training procedure.
  3. [Appendix B, 'Pre-trained Teacher Models'] The sentence 'we ensure that the validation error for each teacher model with a specific accuracy is less than 1%' is unclear; it presumably means the achieved validation accuracy is within 1% of the preset accuracy, and should be rephrased.
  4. [Table 7 caption] The caption 'Strategy C consistently achieves higher final accuracy than Strategy B across all teacher models' is true only at the last reported step (250k); at 50k steps, STRATEGY B is higher for the 50% teacher on both ResNet-50 and ViT, so the caption should be qualified.
  5. [References] References [9] and [10] duplicate the same ImageNet citation, and reference [50] has a malformed author list ('and et al.'); both should be corrected.
  6. [Checklist item 8] The paper answers 'No' to providing compute-resource details; since Appendix B otherwise makes reproducibility a priority, an estimate of total GPU hours and the number of runs would complete the picture.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the central claims are empirical comparisons on external test accuracy, and no prediction reduces to a fitted input or self-citation by construction.

full rationale

The paper contains no first-principles derivation that could be circular: its claims (Claims 1-5) are empirical observations comparing Strategies A, B, and C under a fixed decoupled loss, with the reported quantity being Top-1 test accuracy on external benchmarks (CIFAR-10, CIFAR-100, Tiny-ImageNet, ImageNet). Strategy C is defined by a mapping rule, psi_C(x_i^(j)) = y_i, and its reported advantage is a measured result, not an algebraic consequence of the loss or of the definition. The unified loss in Eqs. (1)-(3) is a training protocol used to isolate mapping strategies, and comparing strategies under the same protocol does not reduce the outcome to the protocol. The self-citations [40] and [41] are used only to motivate the sample-target framing and do not carry the load of any empirical claim; removing them would not change the reported accuracies. The legitimate concern that the headline comparison to 'traditional knowledge distillation' lacks a standard end-to-end KD control for Strategy C is an external-validity or experimental-design issue, not a circularity: no equation in the paper makes the claim true by definition, and no fitted parameter is renamed as a prediction. Therefore no circular step is identified.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities and fits no numerical constants to data. The central experimental conclusions rest on two unevaluated proxy assumptions: linear-probe evaluation with a decoupled classifier is treated as a faithful measure of mapping quality, and teacher models with preset accuracies are treated as genuinely different teacher strengths even though they are early-stopped checkpoints. The soft-label temperature 2.0 is a hand-set hyperparameter that affects all soft-target comparisons.

free parameters (1)
  • Soft-label temperature tau = 2.0
    Chosen by hand for all STRATEGY B and STRATEGY C experiments (Appendix B). The soft-target comparisons and conclusions could shift if this temperature were tuned.
assumptions (4)
  • domain assumption Backbone representational ability is faithfully measured by a separately trained linear or CE classifier on frozen features (linear probing protocol).
    Section 2.2.1 and 2.2.2: the paper argues that a coupled CE loss obscures the backbone, and all accuracy comparisons use this decoupled proxy.
  • ad hoc to paper Teacher models of different accuracies are comparable proxies for teachers of different strengths, even though they are obtained by early stopping at preset validation accuracies.
    Appendix B: the teacher model is saved the first time it reaches a preset accuracy. This confounds teacher strength with training progress and is not validated.
  • domain assumption KL divergence to soft targets plus a separate CE classifier is a suitable unified loss for comparing mapping strategies.
    Section 2.2.2, Eqs. (1)-(3): no comparison to the standard combined hard and soft knowledge distillation loss is provided.
  • domain assumption The datasets and augmentation families used are representative enough for general conclusions about targets, samples, and mappings.
    Appendix B: only computer vision datasets are used; the authors acknowledge the limited scope in Section 5 future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Equally Critical: Samples, Targets, and Their Mappings in Datasets." pith.science (2026). https://pith.science/paper/4DZFW5FQ

@misc{pith2026250601987,
  author       = {Pith},
  title        = {Pith review of: Equally Critical: Samples, Targets, and Their Mappings in Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DZFW5FQ}},
  note         = {Machine review of arXiv:2506.01987}
}
read the original abstract

Data inherently possesses dual attributes: samples and targets. For targets, knowledge distillation has been widely employed to accelerate model convergence, primarily relying on teacher-generated soft target supervision. Conversely, recent advancements in data-efficient learning have emphasized sample optimization techniques, such as dataset distillation, while neglected the critical role of target. This dichotomy motivates our investigation into understanding how both sample and target collectively influence training dynamic. To address this gap, we first establish a taxonomy of existing paradigms through the lens of sample-target interactions, categorizing them into distinct sample-to-target mapping strategies. Building upon this foundation, we then propose a novel unified loss framework to assess their impact on training efficiency. Through extensive empirical studies on our proposed strategies, we comprehensively analyze how variations in target and sample types, quantities, and qualities influence model training, providing six key insights to enhance training efficacy.

Figures

Figures reproduced from arXiv: 2506.01987 by the authors.

Figure 1
Figure 1. The key difference between hard and soft [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Three different sample-to-target mapping strategies. STRATEGY A: Multiple augmented samples within the same class are mapped to one same one-hot target. STRATEGY B: Each augmented sample is mapped to a unique soft target. STRATEGY C: Multiple augmented views of a sample are mapped to one same soft target. Blue and red colors denote class 1 and class 2, respectively. For each strategy, x1 and x2 means original sample… view at source ↗
Figure 3
Figure 3. Student training performance under three strategies using ResNet50 (Left) and ViT (Right) as backbone on ImageNet. Under different backbones, STRATEGY A shows long-term advantages, while STRATEGY B exhibits short-term benefits. And the advantages of STRATEGY C become increasingly prominent when applied to weaker teacher models. 2 Preliminary This section begins by formally defining three mapping strategies between s… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Comparison of STRATEGY B and STRATEGY C. Left: STRATEGY C demonstrates superior performance in long-term training. Even when using a randomly initialized teacher model with 10% accuracy, the student model trained with STRATEGY C outperforms the student model trained un…
Figure 5
Figure 5. Figure 5: STRATEGY B accelerates early-stage training but achieves inferior final performance. Left: Comparison of STRATEGY A and STRATEGY B that employs teacher models of differing accuracies. Right: Weaker teacher models can facilitate early-stage learning. (dataset: CIFAR-10,…
Figure 6
Figure 6. Figure 6: Left: Teacher models trained with various augmentation strategies. Teacher models with high accuracy, trained using MixUp, can not help student model training. Right: Applying various augmentations to both teacher and student models. RandomResizedCrop significantly red…
Figure 7
Figure 7. Figure 7: Left: Student model accuracy scaling with increasing Images Per Class (IPC). With limited IPC, STRATEGY B deviates from the typical power-law scaling compared with other strategies, whereas STRATEGY A exhibits notable advantages when IPC is sufficient. Right: Strategy …
Figure 8
Figure 8. Figure 8: Scaling behavior of models using various data augmentation methods across three strategies. For STRATEGY A, RandomResizedCrop enhances model performance significantly. In STRATEGY B and STRATEGY C, augmentation strategies surpass scaling law limitations when samples ar…
Figure 9
Figure 9. Figure 9: Comparison of STRATEGY B and STRATEGY C under more different teacher models with different accuracies. It further shows that as the accuracy of the teacher model increases, the advantage of using STRATEGY C over STRATEGY B becomes less apparent. D Additional Dataset We…
Figure 10
Figure 10. Figure 10: Ablation experiments of training the student model with 30% (Left) and 50% (Right) teacher models in the main body. It is evident that using the RandomResizedCrop strategy has certain disadvantages for the student model. Additionally, applying the MixUp strategy to te…
Figure 11
Figure 11. Figure 11: The scaling behavior across different teachers (by changing the number of image per class) under STRATEGY B (left) and STRATEGY C (right). In STRATEGY B, different teacher models exhibit similar patterns of scaling, whereas in STRATEGY C, the performance of the studen…
Figure 12
Figure 12. Figure 12: The scaling behavior across different teacher using different data augmentation under STRAT￾EGY B. From left to right, the teacher models have accuracies of 10%, 50%, and 90%, respectively. For a teacher model with moderate or high accuracy, applying different data au…
Figure 13
Figure 13. Figure 13: The comparison of strategies STRATEGY A and STRATEGY B under different learning rate and batch size settings. All results show that the STRATEGY B has a short term advantage while STRATEGY A have a long term advantage [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Comparison of STRATEGY A and STRATEGY B using different teachers on CIFAR100 (Left) and TinyImageNet (Right). Both show the long-term advantages of STRATEGY A, and the weaker teacher models exhibit more significant improvements in the performance of the student models…
Figure 15
Figure 15. Figure 15: The different behavior across teachers in different early-stage training steps on CIFAR100 (Left) and TinyImageNet (Right). For a teacher model with moderate accuracy, there is an advantage in the early stages of training the student model. 0 1000 2000 3000 4000 5000 …
Figure 16
Figure 16. Figure 16: The different impacts on student model training when applying different image augmentation strategies to the teacher model under STRATEGY B, with the left figure representing CIFAR100 and the right figure representing TinyImageNet. As the accuracy of the teacher model…
Figure 17
Figure 17. Figure 17: The scaling behavior using different strategy for training on CIFAR100 (Left) and TinyImageNet (Right). In both datasets, STRATEGY B consistently shows a greater advantage, STRATEGY C has some advantage when the sample size is small. And for CIFAR100, when the sample …
Figure 18
Figure 18. Figure 18: The scaling behavior of the student model using different image augmentation methods under the three strategies. The three images above show the results of CIFAR100, while the three images below show the results of TinyImageNet, all of which demonstrate the advantages…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 37 canonical work pages

  1. [1]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  2. [2]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021

    2021

  3. [3]

    Semi- supervised knowledge distillation for model compression

    Mingxing Chen, Vijay Badrinarayanan, Chung-Ching Lee, and Andrew Rabinovich. Semi- supervised knowledge distillation for model compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

    work page 2020

  4. [4]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020

    2020

  5. [5]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640–9649, 2021

    2021

  6. [6]

    A survey of model compression and acceleration for deep neural networks

    Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arXiv:1710.09282, 2017

    arXiv 2017

  7. [7]

    Robust locally weighted regression and smoothing scatterplots

    William S Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74(368):829–836, 1979

    work page 1979

  8. [8]

    Dc-bench: Dataset condensation bench- mark

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Dc-bench: Dataset condensation bench- mark. Advances in Neural Information Processing Systems, 35:810–822, 2022

    work page 2022

Show all 70 references
  1. [9]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  2. [10]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. IEEE, 2009

    2009

  3. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Confer...

    2021

  4. [12]

    Scaling laws of synthetic images for model training

    Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7382–7392, 2024

    2024

  5. [13]

    How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization

    Jonas Geiping, Micah Goldblum, Gowthami Somepalli, Ravid Shwartz-Ziv, Tom Goldstein, and Andrew Gordon Wilson. How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. arXiv preprint arXiv:2210.06441, 2022

    arXiv

  6. [14]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819, 2021

    2021

  7. [15]

    Bootstrap your own latent-a new approach to self-supervised learning

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural i...

    2020

  8. [16]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  9. [17]

    To- wards lossless dataset distillation via difficulty-aligned trajectory matching

    Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. To- wards lossless dataset distillation via difficulty-aligned trajectory matching. In International Conference on Learning Representations, 2024

    2024

  10. [18]

    Clip and complementary methods

    Markus Hafner, Maria Katsantoni, Tino Köster, James Marks, Joyita Mukherjee, Dorothee Staiger, Jernej Ule, and Mihaela Zavolan. Clip and complementary methods. Nature Reviews Methods Primers, 1(1):1–23, 2021. 10

    2021

  11. [19]

    Momentum contrast for unsupervised visual representation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020

    2020

  12. [20]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE, 2016

    2016

  13. [21]

    Knowledge adaptation: Teaching to adapt

    Zeyi He, Huaxiong Li, Xin Liu, and Steven C Hoi. Knowledge adaptation: Teaching to adapt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 384–393, 2020

    2020

  14. [22]

    Deep learning scaling is predictable, empirically

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017

    2017 arXiv

  15. [23]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015

    2015

  16. [24]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  17. [25]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  18. [26]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673, 2020

    2020

  19. [27]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  20. [28]

    Tiny imagenet visual recognition challenge

    Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015

    2015

  21. [29]

    A survey on text classification: From traditional to deep learning

    Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, and Lifang He. A survey on text classification: From traditional to deep learning. ACM Transactions on Intelligent Systems and Technology (TIST), 13(2):1–41, 2022

    2022

  22. [30]

    Image segmentation using deep learning: A survey

    Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3523–3542, 2021

    2021

  23. [31]

    Deep learning on a data diet: Finding important examples early in training

    Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34:20596–20607, 2021

    2021

  24. [32]

    The effectiveness of data augmentation in image classification using deep learning

    Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621, 2017

    2017 arXiv

  25. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  26. [34]

    Data efficient learning of molecular slow modes from nonequi- librium metadynamics

    Debasis Ray and Jakob Hanni. Data efficient learning of molecular slow modes from nonequi- librium metadynamics. ChemRxiv, 2025

    2025

  27. [35]

    Deep clustering: A comprehensive survey

    Yazhou Ren, Jingyu Pu, Zhimeng Yang, Jie Xu, Guofeng Li, Xiaorong Pu, S Yu Philip, and Lifang He. Deep clustering: A comprehensive survey. IEEE Transactions on Neural Networks and Learning Systems, 2024

    2024

  28. [36]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018

    2018

  29. [37]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  30. [38]

    A survey on image data augmentation for deep learning

    Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019. 11

    2019

  31. [39]

    Beyond neu- ral scaling laws: beating power law scaling via data pruning

    Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523–19536, 2022

    2022

  32. [40]

    Efficiency for free: Ideal data are transportable representations

    Peng Sun, Yi Jiang, and Tao Lin. Efficiency for free: Ideal data are transportable representations. arXiv preprint arXiv:2405.14669, 2024

    2024 arXiv

  33. [41]

    On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm

    Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  34. [42]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR, 2019

    2019

  35. [43]

    Cafe: Learning to condense dataset by aligning features

    Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12196–12205, 2022

    2022

  36. [44]

    Knowledge distillation meets self-supervision

    Runjian Xu, Jiaming Zhang, Xiaoyang Hu, and Dahua Lin. Knowledge distillation meets self-supervision. arXiv preprint arXiv:2006.07114, 2020

    2006 arXiv

  37. [45]

    A gift from knowledge distillation: Fast optimization, network minimization and transfer learning

    Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4133–4141, 2017

    2017

  38. [46]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. arXiv preprint arXiv:2306.13092, 2023

    2023 arXiv

  39. [47]

    Cutmix: Regularization strategy to train strong classifiers with localizable features

    Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6023–6032. I...

    2019

  40. [48]

    Barlow twins: Self- supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International conference on machine learning, pages 12310–12320. PMLR, 2021

    2021

  41. [49]

    Network representation learning: A survey

    Daokun Zhang, Jie Yin, Xingquan Zhu, and Chengqi Zhang. Network representation learning: A survey. IEEE transactions on Big Data, 6(1):3–28, 2018

    2018

  42. [50]

    Mixup: Beyond empirical risk minimization

    Hongyi Zhang, Yu Zhong, Iwao Sato, and et al. Mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018

    2018

  43. [51]

    Dataset condensation with distribution matching

    Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6514–6523, 2023

    2023

  44. [52]

    Dataset condensation with gradient matching

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929, 2020

    2006 arXiv

  45. [53]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022

    2022

  46. [54]

    Data-efficient learning with active label cleaning and dynamic curriculum

    Yuanlong Zhu et al. Data-efficient learning with active label cleaning and dynamic curriculum. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 12 A Related Work To assess the impact of data—consisting of samplesX and targetsY —on the...

    2023

  47. [55]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We show them in Abstract and Section 1. Guidelines: • The answer NA means that the abstract and introduction do not in...

  48. [56]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We show them in Section 5. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitation...

  49. [57]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 24 Justification: The paper does not include theoretical results. Guidelines: • The answer NA means that the...

  50. [58]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  51. [59]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 25 Answer: [Yes] Justification: We use the public datase...

  52. [60]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: We clearly state our experime...

  53. [61]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: We show them in the tables. Guidelines: • The answ...

  54. [62]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [No] Justification: We provide most of them ...

  55. [63]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: This research fully complies with the NeurIPS Code of Ethics. Guidelines: • The...

  56. [64]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: The paper does not provide discussion of potential societal impacts. Guidelines: • The answer NA means tha...

  57. [65]

    • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: The paper does ...

  58. [66]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All code, data, ...

  59. [67]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 28 Answer: [NA] Justification: The paper does not release new assets. Guidelines: • The answer NA means that the paper does not release new asset...

  60. [68]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  61. [69]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  62. [70]

    Answer: [NA] 29 Justification: LLM is used only for editing

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.