REVIEW 4 major objections 4 minor 25 references
Self Distillation via Iterative Constructive Perturbations
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper proposes a cyclic optimization scheme in which inputs are iteratively perturbed to reduce the current task loss and then used as self-distillation teachers for their own features, and reports large accuracy gains on CIFAR-100…
desk verdict Plausible new self-distillation trick, but the central 'constructive' claim is untested and the evidence table is too thin to support the headline 19-point gain. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Iterative Constructive Perturbation (ICP) update, a gradient-descent step on the input instead of on the weights: $x_t = x_{t-1} - \epsilon \nabla_{x_{t-1}} J(\theta, x_{t-1}, y)$. This reverses FGSM's logic in two ways: it uses the full gradient rather than its sign, and it minimizes the loss rather than maximizing it, over $T$ refinement steps. The refined input is then passed through the model a second time, and the mean-squared error between original and refined feature maps at selected layers, $\mathrm{MSE}(F_i, F'_i)$, is added to the task loss with a cosine-decayed mixing weight $\alpha_e$. The paper also makes the perturbation step optimizer-dependent, replacing plain SGD with Adam-style moment estimates (Adam-ICP) and with AdEMAMix's extra momentum term (AdEMAMix-ICP), and reports that the choice of this inner optimizer changes the results enough to matter.
What would settle it
Run the exact k=25, T=5, weighted-feature protocol on CIFAR-100, but replace the ICP update with (a) a fixed random perturbation of matched norm per step, or (b) an i-FGSM adversarial perturbation of matched magnitude. If either replacement reproduces the 41.99% accuracy, the constructive (loss-minimizing) direction of ICP is not what carries the benefit; if neither does, the gradient-descent direction is doing the work. A second check is to record the task loss on the perturbed input across the T steps and verify it actually decreases as the constructive claim requires.
Extended reading notes
Core claim
The paper's central claim is that features extracted from a loss-minimizing perturbed version of an input are better supervision targets than the original input's own features, and that aligning the two with a layer-wise distillation loss is enough to improve task performance. Concretely, after k baseline epochs the combined training loss becomes $L_{\mathrm{total}} = \alpha_e L_{\mathrm{task}} + (1-\alpha_e) \sum_i \mathrm{MSE}(F_i, F'_i)$, where $F_i$ are features of the original input and $F'_i$ are features of the ICP-refined input, with $\alpha_e$ cosine-decaying from 1 to 0 over the remaining epochs. The authors report that on CIFAR-100 the best configuration (k=25, T=5, weighted feature maps) lifts accuracy from 22.93% (100-epoch control) to 41.99% for AdEMAMix-ICP, a 19.06-percentage-point gain, with F1 rising from 0.229 to 0.414. In their constrained VAE experiment on CUB, the same configuration improves SSIM from 0.2580 to 0.3893 and FID from 161.830 to 157.604. The paper interprets these results as evidence that alternating optimization of weights and inputs reduces the gap between fitting and generalization.
Load-bearing premise
The method assumes that the input produced by T steps of gradient descent on the model's task loss is a genuinely better view of the same input, with richer or more stable features, rather than just a sample that happens to lower the current loss; if that premise fails, the distillation term aligns the model with its own noise instead of with better supervision.
Editorial extensions
If this is right
- With k=25 and T=5, AdEMAMix-ICP improves CIFAR-100 accuracy by 19.06 percentage points over a 100-epoch control run while keeping the same total epoch budget.
- Weighting deeper layers more heavily in the feature-alignment loss yields better classification results than uniform weighting in the paper's ablation.
- The same hyperparameter configuration transfers to image generation, improving both SSIM and FID for a VAE trained on CUB.
- The method needs no teacher model and no extra parameters; the distillation target is produced by the same network from a perturbed copy of the input.
Reading between the lines
- A natural reading the paper does not spell out is that ICP's main effect is to act as a data-dependent regularizer: tying original features to features of a loss-minimizing perturbation discourages the model from relying on directions that are unstable under small input shifts.
- A direct way to test this reading is to ablate the distillation term while keeping the ICP-refined input as the only change, or to replace the ICP direction with a random or adversarial perturbation of the same magnitude; the reported table does not isolate the distillation contribution on its own.
- Because the authors fix the total epoch budget at 100, a comparison on larger models and datasets, for instance transformer-based vision models or ImageNet-scale inputs, would show whether the gain survives outside the small-model, small-data regime they tested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Iterative Constructive Perturbation (ICP), which applies several gradient-descent steps to the input in the direction that minimizes the model's task loss, then feeds the perturbed input through the network and uses the resulting intermediate feature maps as self-distillation targets for the original input's features. The total loss is a cosine-decayed mixture of the task loss and layer-wise MSE feature-alignment losses. Experiments on CIFAR-100 classification with a modified ResNet20 and on CUB image generation with a small VAE report consistent improvements over a 100-epoch control, with the best configuration (k=25, T=5, Weighted=True, AdEMAMix-ICP) achieving 41.99% accuracy versus 22.93% for the control and SSIM 0.3893 versus 0.2580 on CUB.
Significance. The proposed idea—using input-space gradient descent to create feature targets for self-distillation—is potentially interesting and would be a useful addition to the self-distillation literature if the mechanism were convincingly established. The paper deserves credit for comparing three input optimizers (SGD-, Adam-, and AdEMAMix-ICP), reporting wall-clock times, and for attempting a transfer to a generative task. However, the current evidence does not isolate the proposed mechanism from several confounds: the teacher features are generated by the same network on its own gradient-adjusted inputs, hyperparameters are selected on the test set, every reported number is a single run with no error bars, and Table 1 contains a duplicated value. As written, the central claim that minimizing the task loss on inputs produces better supervision targets is not supported. The results are promising enough to warrant a major revision with controlled ablations and more rigorous evaluation, rather than outright rejection.
major comments (4)
- [Section 3.2.2, Eq. (3)] The central mechanism is not independently verified. The teacher features F'_i are computed by the same network on an input I' that has been moved along the network's own task-loss gradient, so the distillation targets are self-generated rather than externally validated. The only support offered is Figure 1, which is a toy two-dimensional MLP example on isotropic Gaussian clusters and does not measure feature quality or transfer to the CIFAR-100/ResNet20 setting. Please add ablations that replace the ICP perturbation with (i) random Gaussian input noise, (ii) adversarial (loss-increasing) perturbations, and (iii) a different self-distillation target such as features from a differently augmented view of the same input or from a stale copy of the network, while keeping the loss schedule and feature-alignment loss identical. Without these controls, the observed improvement could be due to the extra input-gradient computation or to the MSE feature-alignment regularizer rather than to 'constructive' perturbations.
- [Section 4.2, Table 1] The main experimental table does not support the headline claim because hyperparameters are selected on the test set. The text states that the optimal configuration (k=25, T=5, Weighted=True) was determined from Table 1, but no validation split is described; this makes 41.99% a test-set-selected result and risks overfitting to the test set. Please use a separate validation split for selecting k, T, and the Weighted flag, and report test accuracy with means and standard deviations over at least three seeds. In addition, Table 1 contains duplicated entries: AdEMAMix-ICP with k=50, T=10, Weighted=True and with k=75, T=5, Weighted=False both report 26.27% accuracy and 0.249 F1. This strongly suggests a data-entry error and must be corrected before the table can be used as evidence.
- [Section 3.3, Eqs. (4)-(5)] The comparison against the 100-epoch control confounds the method with the training schedule. The control is trained with alpha_e = 1 for all 100 epochs, while every ICP run trains for k baseline epochs and then switches to the cosine-decayed combination of task loss and feature-alignment loss. The reported improvement over the control could therefore be caused by the change in the objective after epoch k (including the MSE alignment term or the cosine schedule) rather than by the input perturbation itself. Please include a baseline that uses the same alpha_e schedule and the same feature-alignment loss but with the perturbed input replaced by an unperturbed or randomly perturbed input, and report that comparison. This is the minimal experiment needed to attribute the observed gain specifically to ICP.
- [Sections 4.1 and 4.3] Key implementation details needed for reproducibility are missing. The ICP step size epsilon for the CIFAR-100 experiments is never given (epsilon=0.002 is mentioned only for the toy example in Figure 1), the identity and number of distilled layers are unspecified, the 'modified ResNet20' architecture is not described, and the VAE architecture and its loss weighting are not provided. Without these details, the experiments cannot be reproduced, and the sensitivity of the method to epsilon and layer choice remains unknown. Please report all hyperparameters, specify the network layers used for distillation, describe the VAE architecture, and ideally release code. This is load-bearing because the method's only empirical support is these experiments.
minor comments (4)
- [Section 4.2] The phrase '19.06% more than the control baseline' is misleading; the reported numbers are percentage points (41.99% minus 22.93% equals 19.06 percentage points), not a relative 19.06% improvement.
- [Eq. (1)] There is a missing space in 'andsgn' after 'and'.
- [Introduction and References] The citations for BYOT and Born Again Neural Networks appear mismatched: reference [25] is cited as BYOT but is actually 'Deep Mutual Learning', while 'Born Again Neural Networks' is reference [4]. Please correct the citation mapping.
- [Eqs. (8) and (11)] The Adam-ICP and AdEMAMix-ICP update rules are typeset ambiguously; the bias-correction factors and the division by sqrt(v) should be written with explicit fractions so the update equations are unambiguous.
Circularity Check
No significant circularity: the self-generated distillation targets are self-referential by design, but the paper's empirical claims are checked against an external control and a transferred-task evaluation.
full rationale
The derivation chain is not circular. The paper defines the ICP input I' by gradient descent on the current model's task loss (Eq. 2) and then aligns original features F_i to the features F'_i of the perturbed input (Eq. 3). Because F'_i is produced by the same network on an input moved along the model's own loss gradient, calling these 'improved intermediate features' is an assumption about target quality, not a consequence derived from the equations; however, the paper does not claim to derive a prediction from this assumption. The reported outcomes—CIFAR-100 accuracy and F1, and CUB SSIM/FID—are compared against an independently trained 100-epoch control and, in the VAE experiment, on a different task with hyperparameters fixed from the classification experiments, so the main empirical claims are externally falsifiable. There are no load-bearing self-citations; the cited works on FGSM, AdEMAMix, and self-distillation theory are external. The lack of a proof that ICP features are better teachers is a causal-attribution weakness, not a circular reduction. Other concerns, such as selection of hyperparameters on the test set and the duplicated 26.27 table entry, are experimental-reporting issues outside the scope of circularity.
Assumptions & free parameters
free parameters (5)
- k (baseline epochs) =
25
- T (ICP iterations) =
5
- Weighted flag =
True
- epsilon (ICP step size) =
not reported for experiments (0.002 used in Figure 1)
- alpha_e schedule parameters (E, k) =
E=100, k=25
assumptions (3)
- domain assumption Negative gradient steps on the input improve the input representation
- domain assumption Feature maps of the perturbed input are valid distillation targets
- ad hoc to paper Driving the task-loss weight to zero at the final epoch is safe
Cite this review
Pith. "Pith review of Self Distillation via Iterative Constructive Perturbations." pith.science (2026). https://pith.science/paper/FXKJ2HLM
@misc{pith2026250514751,
author = {Pith},
title = {Pith review of: Self Distillation via Iterative Constructive Perturbations},
year = {2026},
howpublished = {\url{https://pith.science/paper/FXKJ2HLM}},
note = {Machine review of arXiv:2505.14751}
}
read the original abstract
Deep Neural Networks have achieved remarkable achievements across various domains, however balancing performance and generalization still remains a challenge while training these networks. In this paper, we propose a novel framework that uses a cyclic optimization strategy to concurrently optimize the model and its input data for better training, rethinking the traditional training paradigm. Central to our approach is Iterative Constructive Perturbation (ICP), which leverages the model's loss to iteratively perturb the input, progressively constructing an enhanced representation over some refinement steps. This ICP input is then fed back into the model to produce improved intermediate features, which serve as a target in a self-distillation framework against the original features. By alternately altering the model's parameters to the data and the data to the model, our method effectively addresses the gap between fitting and generalization, leading to enhanced performance. Extensive experiments demonstrate that our approach not only mitigates common performance bottlenecks in neural networks but also demonstrates significant improvements across training variations.
Figures
Reference graph
Works this paper leans on
-
[1]
An empirical study of training self-supervised vision transformers
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. CoRR, abs/2104.02057, 2021. 1
arXiv 2021
-
[2]
Bert: Pre-training of deep bidirectional trans- formers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human language tech- nologies, volume 1 (long and short papers) , pages 4171– 4186, 2019. 1
2019
-
[3]
Multi-task Self-Supervised Visual Learning
Carl Doersch and Andrew Zisserman. Multi-task self- supervised visual learning. CoRR, abs/1708.07860, 2017. 1
work page Pith review arXiv 2017
-
[4]
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pages 1607–1616. PMLR, 2018. 1
work page 2018
-
[5]
Un- supervised representation learning by predicting image rota- tions
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Un- supervised representation learning by predicting image rota- tions. arXiv preprint arXiv:1803.07728, 2018. 1
arXiv 2018
-
[6]
Goodfellow, Jonathon Shlens, and Christian Szegedy
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. 2
2015
-
[7]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 1, 5
work page 2016
-
[8]
Gans trained by a two time-scale update rule converge to a local nash equilib- rium
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilib- rium. Advances in neural information processing systems , 30, 2017. 5
2017
Show all 25 references
-
[9]
Distill- ing the knowledge in a neural network
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 1
2015 arXiv
-
[10]
Batch normalization: Accelerating deep network training by reducing internal co- variate shift
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal co- variate shift. In International conference on machine learn- ing, pages 448–456. pmlr, 2015. 1
2015
-
[11]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 ,
-
[12]
Auto-encoding vari- ational bayes, 2013
Diederik P Kingma, Max Welling, et al. Auto-encoding vari- ational bayes, 2013. 5
2013
-
[13]
Learning multiple layers of features from tiny images.(2009), 2009
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.(2009), 2009. 5 6
2009
-
[14]
Imagenet classification with deep convolutional neural net- works
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 1
2012
-
[15]
Simple and scalable predictive uncertainty estima- tion using deep ensembles
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estima- tion using deep ensembles. Advances in neural information processing systems, 30, 2017. 1
2017
-
[16]
Deep learning
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436–444, 2015. 1
2015
-
[17]
Self-distillation amplifies regularization in hilbert space
Hossein Mobahi, Mehrdad Farajtabar, and Peter Bartlett. Self-distillation amplifies regularization in hilbert space. In Advances in Neural Information Processing Systems , pages 3351–3361. Curran Associates, Inc., 2020. 1
2020
-
[18]
The ademamix optimizer: Better, faster, older
Matteo Pagliardini, Pierre Ablin, and David Grangier. The ademamix optimizer: Better, faster, older. arXiv preprint arXiv:2409.03137, 2024. 5
2024 arXiv
-
[19]
A bayesian perspective on generalization and stochastic gradient descent
Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017. 1
2017 arXiv
-
[20]
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Be- longie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technol- ogy, 2011. 5
2011
-
[21]
Esrgan: En- hanced super-resolution generative adversarial networks
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: En- hanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018. 1
2018
-
[22]
Chenglin Yang, Lingxi Xie, Chi Su, and Alan L. Yuille. Snapshot distillation: Teacher-student optimization in one generation. CoRR, abs/1812.00123, 2018. 1
2018 arXiv
-
[23]
Regularizing class-wise predictions via self-knowledge dis- tillation
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge dis- tillation. CoRR, abs/2003.13964, 2020. 1
2003 arXiv
-
[24]
Understanding deep learning (still) requires rethinking generalization
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107–115, 2021. 1
2021
-
[25]
Hospedales, and Huchuan Lu
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4320–4328, 2018. 1 7
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.