An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning

Fatemeh Ghofrani; Pooyan Jamshidi

REVIEW 4 major objections 5 minor 29 references

An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adversarial self-supervised training can be cut from hundreds of epochs to ten by using many crops per image and free adversarial training, without losing clean accuracy or robustness.

desk verdict Free adversarial training on multi-crop EMP-SSL is a sensible and potentially useful combination, but the headline speedup is confounded and the tables don't all agree. read the letter →

arxiv 2501.03507 v1 pith:VQFDN5DZ submitted 2025-01-07 cs.CV cs.LG

Fatemeh Ghofrani , Pooyan Jamshidi This is my paper · ORCID

classification cs.CVcs.LG

keywords self-supervisedlearningadversarialtrainingrobustnessmulti-cropaugmentationfreeEMP-SSLefficiencyCIFAR-10

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that adversarial training for self-supervised visual representations does not require hundreds of epochs: the number of crops per image can substitute for training time, and a replay-based adversarial trainer can shrink the schedule to about ten epochs. If the claim holds, adversarial SSL becomes practical on a single GPU and the usual clean-accuracy-versus-robustness tradeoff is softened instead of accepted. On CIFAR-10, the introduced CF-AMC-SSL method reaches 75.78% clean and 33.34% PGD-8 accuracy in 97 minutes, compared with robust SimCLR's 72.86% clean and 16.81% PGD-8 after 500 epochs and 934 minutes. The paper also argues that central-crop linear evaluation gives a cheaper and better view of the clean/robust tradeoff than multi-crop embedding aggregation.

What carries the argument

The load-bearing object is CF-AMC-SSL, a training loop that combines the EMP-SSL loss with free adversarial training. The loss sums over $C$ crops an invariance term $D(Z_i,\bar{Z}) = \mathrm{Tr}(Z_i^T \bar{Z})$, which aligns each embedding $Z_i$ with the average embedding $\bar{Z}$, and a redundancy-reduction term $R(Z_i) = \frac{1}{2}\log\det\left(I + \frac{d}{b\epsilon^2} Z_i Z_i^T\right)$. Free adversarial training repeats each minibatch step $m$ times and uses the same gradient of this loss to update both the perturbation and the network weights, which is what lets the method run with 16 multi-scale crops for as few as 6 to 10 epochs. A central-crop linear probe turns the resulting representation into clean and adversarial classification accuracy.

What would settle it

Run a 16-crop SimCLR variant with free adversarial training for 10 epochs on CIFAR-10 under the same augmentation ranges and perturbation sizes, and measure clean and PGD-8 accuracy and wall-clock time. If it matches CF-AMC-SSL's 75.78% clean and 33.34% PGD-8 in comparable time, the gain is crop diversity plus replay rather than the EMP-SSL loss; if it falls short, the EMP-SSL objective is the essential ingredient.

Watch

Extended reading notes

Core claim

The central claim is that crop diversity is a first-class substitute for epoch count in adversarially trained SSL. Using the EMP-SSL objective—an invariance term that aligns each crop embedding with the mean embedding plus a log-determinant regularization term that penalizes redundancy—the authors' robust crop-based EMP-SSL trains in 30 epochs and beats the 500-epoch robust SimCLR baseline on both clean accuracy and PGD robustness. Adding free adversarial training with minibatch replays yields CF-AMC-SSL, which reaches 75.78% clean and 33.34% PGD-8 on CIFAR-10 in 10 epochs and 97 minutes, compared with the 530-minute robust crop-based EMP-SSL at 76.55% clean and 28.49% PGD-8. The authors interpret the gain as the model seeing many perturbed crops of each image, learning content over style, and clustering adversarial examples around their average embedding.

Load-bearing premise

The efficiency claim rests on comparing EMP-SSL and CF-AMC-SSL at 10 to 30 epochs against SimCLR at 500 epochs with hyperparameters otherwise fixed, without testing whether SimCLR given 16 crops or a smaller epoch budget would close the gap.

Editorial extensions

If this is right

Adversarial self-supervised pretraining can be completed in minutes rather than hours: CF-AMC-SSL's 10-epoch CIFAR-10 run takes 97 minutes versus 530 minutes for robust crop-based EMP-SSL and 934 minutes for robust SimCLR.
Increasing the number of crops per image is an effective lever for reducing the epoch requirement without the usual clean-accuracy penalty.
Robust linear evaluation with a single central crop is both cheaper and a better indicator of the clean/robust tradeoff than multi-crop embedding aggregation, so evaluation protocols built around aggregation may understate robustness.
Free adversarial training transfers from supervised learning to self-supervised objectives and remains effective even when the epoch count is drastically reduced.
The pattern extends beyond CIFAR-10: similar trends on CIFAR-100 and ImageNet-100, and with ResNet-50, suggest the recipe generalizes across datasets and architectures.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step the paper leaves implicit is an iso-compute analysis: for a fixed wall-clock budget there is an optimal combination of crop count, epoch count, and replay count, and the reported 97-minute figure is one point on that frontier.
Because free adversarial training derives its speedup from reusing gradients inside a minibatch, the wall-clock advantage is likely sensitive to batch size, GPU memory, and the replay count $m$; practitioners would need to re-measure on their own hardware.
The comparison does not rule out that a contrastive method like SimCLR, given 16 crops and free adversarial training at a 10-epoch budget, would close the gap; if it did, the active ingredient would be crop diversity plus replay rather than the EMP-SSL loss.
The central-crop-versus-aggregation result suggests reported robustness numbers in SSL are partly a property of the probe protocol, so downstream tasks should evaluate both single-view and aggregated-view classifiers before concluding which representation is more robust.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper does something useful, combining Shafahi-style free adversarial training with EMP-SSL's multi-crop non-contrastive objective, and reports an order-of-magnitude training-time reduction that, if it holds up, makes robust SSL practical on a single GPU. But the central comparison is confounded, and a few table entries are inconsistent enough that the numbers need a careful cleanup before I'd trust them.

What's genuinely new: the specific stack of free adversarial training on top of 16-crop EMP-SSL, plus a systematic crop-based versus patch-based augmentation comparison in the adversarial SSL setting. That combination is not in the cited prior work, and the public code means someone can reproduce it. The result — roughly 76% clean and 33% PGD-8 on CIFAR-10 in 97 minutes on one A6000 — is eye-catching, and the ResNet-50 and ImageNet-100 experiments give some evidence it's not a fluke of one architecture.

The soft spots are real but manageable. First, the headline efficiency claim compares CF-AMC-SSL (16 crops, 10 epochs) against robust SimCLR (2 crops, 500 epochs). That changes crop count, epoch count, and training objective simultaneously. The paper never trains SimCLR with 16 crops or EMP-SSL with 2 crops, so the conclusion "multi-crop augmentations compensate for fewer epochs" is not isolated from the choice of method. The SimCLR-FreeAdv baseline at 167 epochs helps, but it still uses 2 crops. Second, the internal inconsistencies matter: clean accuracy for the same configuration is 75.88 in Table 1 and 75.78 in Tables 4 and 6; Table 6's AutoAttack column for crop-based EMP-SSL is non-monotonic in epsilon (23.93 at 4/255 versus 26.57 at 8/255) and its CIFAR-100 entries exactly match the PGD numbers in Table 4, which looks like copy-paste. Third, there are no error bars or multiple seeds, and the abstract promises theoretical analysis that never appears. These aren't fatal to the core idea, but they're exactly the details that determine whether the 97-minute number is real.

Bottom line: the method is clearly specified and the empirical direction is consistent with prior work on multi-crop SSL and free adversarial training. The paper deserves a serious referee — it's the kind of practical result people would try to reproduce — but it needs major revision: fix the tables, add ablations that hold crop count fixed, run multiple seeds, and either deliver the theory or drop the claim. I'd send it to review, not desk-reject it.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical study of robust self-supervised learning on CIFAR-10, CIFAR-100, and ImageNet-100. It builds on EMP-SSL, which uses many crops or patches per image, and combines it with PGD-based adversarial training or free adversarial training, the latter yielding a method the authors call CF-AMC-SSL. The central claims are that multi-crop augmentation can compensate for reduced training epochs, that crop-based robust EMP-SSL achieves a better clean-accuracy/robustness balance than robust SimCLR, and that CF-AMC-SSL reduces training time while improving both clean accuracy and adversarial robustness. Evaluation is performed with linear probing, multi-crop embedding aggregation, PGD attacks, and AutoAttack.

Significance. If the claims were supported by controlled experiments and trustworthy measurements, the paper would make a practical contribution by showing that multi-crop SSL with free adversarial training can substantially reduce training cost while retaining robustness. The paper has useful strengths: public code is provided, the ablation space is reasonably broad (crop/patch counts, m values, ResNet-50, ImageNet-100), and the authors attempt to confirm PGD results with AutoAttack. However, the headline comparison does not isolate the variables claimed to drive the efficiency and robustness gains, and several table entries appear internally inconsistent. The significance of the reported findings therefore cannot be assessed until these issues are resolved.

major comments (4)

[Section 3.1 and Table 1] The central comparison changes method, crop count, and epoch budget simultaneously. SimCLR is evaluated at 2 crops and 500 epochs, while crop-based EMP-SSL and CF-AMC-SSL use 16 crops and 30 or 10 epochs. The claim that "increasing the number of multi-scale crops effectively offsets fewer training epochs" is therefore not isolated: the paper never trains SimCLR with 16 crops nor EMP-SSL with 2 crops. Please add experiments that vary the crop count within each method (e.g., SimCLR with 16 crops and reduced epochs, or EMP-SSL with 2 crops) or restrict the causal language accordingly.
[Tables 1, 4, and 6] The reported numbers contain inconsistencies that prevent the reader from trusting the measurements. The CF-AMC-SSL (16 crops, m=3, 10 epochs) clean accuracy is 75.88 in Table 1 but 75.78 in Tables 4 and 6. In Table 6, the CIFAR-100 AutoAttack values for crop-based EMP-SSL (33.88, 19.35, 4.92) are identical to the PGD values in Table 1, and the CIFAR-10 AA(8/255) value (26.57) is larger than the AA(4/255) value (23.93). These entries need to be corrected or explained; as printed, the AutoAttack section does not support the paper's claim that the findings are confirmed.
[Section 3.1 and Tables 1-6] All results are reported from single runs without error bars or seed information. Several headline comparisons (e.g., 75.78 vs 72.86 clean accuracy, or 33.34 vs 16.81 PGD-8 accuracy) are between configurations that differ in many hyperparameters; without run-to-run variance it is impossible to tell whether the differences are meaningful. Please report means and standard deviations over at least three independent training runs, or provide a clear justification for reporting single runs.
[Section 3.5 and Table 4] The efficiency claim is also confounded by the comparison budget. Table 4 compares CF-AMC-SSL at 10 epochs with robust EMP-SSL at 30 epochs and robust SimCLR at 500 epochs, but the methods differ in the training objective (free vs PGD adversarial training) and in the number of crops. The paper does not test whether robust EMP-SSL trained with free adversarial training at 30 epochs, or CF-AMC-SSL with 2 crops, would close the reported gap. Please include such controls to support the statement that multi-crop augmentation, rather than the free-training formulation, is responsible for the efficiency gain.

minor comments (5)

[Section 3.3] The text states that EMP-SSL generates 40 random patches/crops, while Tables 1, 4, and 6 use 16 crops or patches; please clarify the default configuration and reconcile the text with the reported runs.
[Algorithm 1] The outer loop is labeled "for epoch = 1 to Nep/m" but then iterates over the entire dataset inside each loop, which is not the standard meaning of an epoch. Please rename the loop variable or explain the relationship between Nep, m, and total data passes.
[Abstract and Introduction] The abstract and introduction mention "combining theoretical analysis with comprehensive experimental evaluation," but no theoretical results appear in the paper; please remove this claim or add the missing analysis.
[Table 4] The row for CF-AMC-SSL (16 crops, m=5, 10 epochs) has dashes in the CIFAR-10 and CIFAR-100 columns and ImageNet-100 entries; please clarify that this configuration was run only on ImageNet-100, or move it to a separate table.
[Figure 2] The legend labels "Baseline: patch-based EMP-SSL" and "Baseline: Crop-based SimCLR" are not tied to the scale/ratio tuples used in the experiments; adding a reference to the exact configurations would improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: CF-AMC-SSL's claims are empirical and independently benchmarked; only minor background self-citations appear.

full rationale

This is an empirical study of training-time efficiency and accuracy-robustness tradeoffs; there is no derivation chain whose conclusions are equivalent to its inputs by construction. The headline results (Tables 1, 4, and 6) compare CF-AMC-SSL against external baselines and building blocks: SimCLR, EMP-SSL [23], and free adversarial training [22]. The paper's own prior work [12] is cited only as background for the known vulnerability of SSL to adversarial attacks, and that vulnerability is independently demonstrated in Table 2 and in external work [17, 24]. No parameter is fitted from the target result; no predicted quantity is defined in terms of the claimed outcome; and no uniqueness theorem is invoked to force a choice. The skeptical concern that the comparison varies crop count, epoch budget, and objective simultaneously is a threat to experimental validity, but changing multiple variables together does not make a result true by definition, so it is not circularity. The table inconsistencies (e.g., CF-AMC-SSL clean accuracy 75.88 vs. 75.78 across tables; non-monotonic AutoAttack values in Table 6) are reporting and correctness concerns and do not by themselves show that any claim reduces to its inputs. No passage asserts a limitation or omitted proof that bears on circularity; the methodological footnotes discuss evaluation choices only. Accordingly, no circular step is identified; the score of 2 reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The analysis is empirical, so there are no fitted parameters in the derivation sense. The listed free parameters are hand-chosen experimental configs that the conclusions depend on. The axioms are standard SSL and adversarial-training assumptions plus two unvalidated design choices: per-crop perturbations and the asymmetric epoch budgets. The paper introduces no new physical or conceptual entities.

free parameters (7)

Number of crops/patches C = 16 (main results); 4 and 40 in ablations
The number of augmentations per image is a central knob. The paper shows 16 crops gives a good tradeoff (Figure 5) but this is a hand-chosen value; the conclusion that crops compensate for epochs depends on it.
Minibatch replays m = 3, 5, 7, 12
Free adversarial training replays each minibatch m times. Main CF-AMC-SSL results use m=3 or m=5; larger m increases robustness but also runtime (Table 4).
Training epochs = 10 for CF-AMC-SSL, 30 for EMP-SSL, 167 or 500 for SimCLR
The efficiency comparison fixes different epoch budgets per method. The claimed speedup is directly a function of these chosen budgets.
PGD steps and epsilon = 5-step PGD with epsilon 8/255 for training; 20-step PGD with epsilons 4, 8, 16 for evaluation
These attack parameters define the threat model and determine all robustness numbers. The paper does not justify why 5-step training and 20-step evaluation are the right choices.
Crop scale/ratio parameters (S,R) = S=(0.08,1.0), R=(0.75,1.3) for crops; S=(0.25,0.25), R=(1,1) for patches
The crop-versus-patch distinction is entirely determined by these RandomResizedCrop settings. Different scale ranges could change the ranking.
EMP-SSL regularization constants b, d = not reported
The objective includes a log-det regularizer with constants b and d (Section 2.1.2). Their values are not given, but they control the strength of the redundancy penalty.
Optimizer hyperparameters (learning rate, weight decay, batch size) = not reported
The experimental setup does not list optimizer settings. These choices can significantly affect convergence speed and final accuracy, especially when comparing algorithms with different epoch counts.

assumptions (5)

domain assumption EMP-SSL's invariance and regularization losses are effective for multi-crop SSL and prevent collapse.
These losses are taken unchanged from Tong et al. [23]. The paper does not re-derive or justify them, but the central results depend on this objective.
domain assumption Adversarial training with PGD and evaluation with PGD/AutoAttack is a valid measure of adversarial robustness.
The paper uses 5-step PGD for training and 20-step PGD plus AutoAttack for evaluation as the definition of robustness. This is standard in the field but still a modeling choice about the threat model.
domain assumption Linear probing on a frozen encoder is a meaningful measure of representation quality for SSL.
All accuracy and robustness numbers come from linear classifiers trained on top of the frozen encoder. The paper does not test fine-tuning or transfer.
ad hoc to paper Independent adversarial perturbations per crop are more effective than a shared perturbation.
Section 2.2.2 states this as a design choice without proof or ablation. The robustness gains could be due to this independence or to other factors.
ad hoc to paper Training SimCLR for 500 epochs is the correct baseline for the efficiency comparison.
Section 3.1 assigns 500 epochs to SimCLR and 10-30 to EMP-SSL/CF-AMC-SSL. This choice is not justified by a sensitivity analysis and may exaggerate the speedup.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning." pith.science (2026). https://pith.science/paper/VQFDN5DZ

@misc{pith2026250103507,
  author       = {Pith},
  title        = {Pith review of: An Empirical Study of Accuracy-Robustness Tradeoff and Training Efficiency in Self-Supervised Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQFDN5DZ}},
  note         = {Machine review of arXiv:2501.03507}
}

read the original abstract

Self-supervised learning (SSL) has significantly advanced image representation learning, yet efficiency challenges persist, particularly with adversarial training. Many SSL methods require extensive epochs to achieve convergence, a demand further amplified in adversarial settings. To address this inefficiency, we revisit the robust EMP-SSL framework, emphasizing the importance of increasing the number of crops per image to accelerate learning. Unlike traditional contrastive learning, robust EMP-SSL leverages multi-crop sampling, integrates an invariance term and regularization, and reduces training epochs, enhancing time efficiency. Evaluated with both standard linear classifiers and multi-patch embedding aggregation, robust EMP-SSL provides new insights into SSL evaluation strategies. Our results show that robust crop-based EMP-SSL not only accelerates convergence but also achieves a superior balance between clean accuracy and adversarial robustness, outperforming multi-crop embedding aggregation. Additionally, we extend this approach with free adversarial training in Multi-Crop SSL, introducing the Cost-Free Adversarial Multi-Crop Self-Supervised Learning (CF-AMC-SSL) method. CF-AMC-SSL demonstrates the effectiveness of free adversarial training in reducing training time while simultaneously improving clean accuracy and adversarial robustness. These findings underscore the potential of CF-AMC-SSL for practical SSL applications. Our code is publicly available at https://github.com/softsys4ai/CF-AMC-SSL.

Figures

Figures reproduced from arXiv: 2501.03507 by the authors.

**Figure 1.** Illustration of workflow comparison {xˆ1, xˆ2, . . . , xˆC }. These patches are independently augmented and passed through a shared encoder h(·) and a projection head g(·) to produce embeddings: Zi = g(h(ˆxi)), where Zi is the embedding of the i-th patch. • Invariance Term: To encourage consistency among embeddings, an invariance term D(Zi ,Z¯) aligns each patch’s embedding Zi with the average embedding Z¯: Z¯ = 1 … view at source ↗

**Figure 2.** The key findings include: • Training base encoders with SimCLR using only two augmentations per image results in a notable tradeoff between clean accuracy and robustness. • Crop-based EMP-SSL demonstrates greater robustness against adversarial attacks, whereas patchbased EMP-SSL excels on clean data. • Central cropping is computationally efficient and achieves strong clean accuracy and robustness, particularly w… view at source ↗

**Figure 2.** Evaluation of robustness against PGD attacks through adversarial pretraining on CIFAR-10 and CIFAR100 datasets. We compare the performance of robust SimCLR and robust EMP-SSL with central crop evaluation under different training configurations. Our analysis includes the evaluation of patch-based SimCLR with varying patch sizes and baseline SimCLR, revealing a noticeable trade-off between clean accuracy and robustne… view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Evaluating the robustness against PGD attacks through adversarial pretraining on CIFAR-10 and CIFAR100 datasets, we compare the performance of patch-based SimCLR (with various patch sizes) to that of baseline SimCLR. Our findings reveal a noticeable trade-off between …

**Figure 4.** Figure 4: Evaluating the robustness against PGD attacks through adversarial pretraining on CIFAR-10 and CIFAR100 datasets, we compare the performance of crop-based EMP-SSL (with various crop sizes) to that of baseline EMP-SSL. Our analysis reveals that the crop-based approach i…

**Figure 5.** Figure 5: Evaluation of robust EMP-SSL across different patch (crop) sizes on CIFAR-10 and CIFAR-100 datasets: Our results emphasize that, when employing the patch-based EMP-SSL method with multi-patch aggregation during evaluation, a significant augmentation in the number of pa…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 20 canonical work pages

[1]

Balestriero, M

R. Balestriero, M. Ibrahim, V . Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mi- alon, Y . Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210 , 2023

arXiv 2023
[2]

Bardes, J

A. Bardes, J. Ponce, and Y . LeCun. Vicreg: Variance-invariance-covariance regularization for self- supervised learning. arXiv preprint arXiv:2105.04906, 2021

arXiv 2021
[3]

Bromley, I

J. Bromley, I. Guyon, Y . LeCun, E. S ¨ackinger, and R. Shah. Signature verification using a ”siamese” time delay neural network. Advances in Neural Information Processing Systems, 6, 1993

work page 1993
[4]

Brown, B

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

work page 1901
[5]

Caron, I

M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912–9924, 2020

work page 2020
[6]

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Ma- chine Learning, pages 1597–1607. PMLR, 2020a

work page
[7]

T. Chen, S. Liu, S. Chang, Y . Cheng, L. Amini, and Z. Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 699–708, 2020b

work page
[9]

Croce and M

F. Croce and M. Hein. Reliable evaluation of adversar- ial robustness with an ensemble of diverse parameter- free attacks. In International Conference on Machine Learning, pages 2206–2216. PMLR, 2020

work page 2020

Show all 29 references

[10]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transform- ers for language understanding. arXiv preprint arXiv:1810.04805, 2018

2018 arXiv
[11]

L. Fan, S. Liu, P.-Y . Chen, G. Zhang, and C. Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in Neural Information Processing Systems , 34:21480– 21492, 2021

2021
[12]

Ghofrani, M

F. Ghofrani, M. Yaghouti, and P. Jamshidi. Rethinking robust contrastive learning from the adversarial per- spective. In The Second Workshop on New Frontiers in Adversarial Machine Learning , 2023

2023
[13]

Gowal, P.-S

S. Gowal, P.-S. Huang, A. van den Oord, T. Mann, and P. Kohli. Self-supervised adversarial robustness for the low-label, high-data regime. In International Conference on Learning Representations , 2021

2021
[14]

Grill, F

J.-B. Grill, F. Strub, F. Altch´e, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Advances in Neural Information Processing Systems , 33:21271– 21284, 2020

2020
[15]

K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9729–9738, 2020

2020
[16]

Jiang, T

Z. Jiang, T. Chen, T. Chen, and Z. Wang. Robust pre- training by adversarial contrastive learning. Advances in Neural Information Processing Systems , 33:16199– 16210, 2020

2020
[17]

M. Kim, J. Tack, and S. J. Hwang. Adversarial self- supervised contrastive learning. Advances in Neural Information Processing Systems, 33:2983–2994, 2020

2020
[18]

Krizhevsky, G

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

2009
[19]

Z. Li, Y . Chen, Y . LeCun, and F. T. Sommer. Neural manifold clustering and embedding. arXiv preprint arXiv:2201.10000, 2022

2022 arXiv
[20]

Madry, A

A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017

2017 arXiv
[21]

Moshavash, M

M. Moshavash, M. Eftekhari, and K. Bahraman. Mo- mentum contrast self-supervised based training for adversarial robustness. Journal of Computing and Security, 8(1):33–43, 2021

2021
[22]

Shafahi, M

A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dicker- son, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems , 32, 2019

2019
[23]

S. Tong, Y . Chen, Y . Ma, and Y . LeCun. EMP-SSL: Towards self-supervised learning in one training epoch. arXiv preprint arXiv:2304.03977 , 2023

2023 arXiv
[24]

Wahed, A

M. Wahed, A. Tabassum, and I. Lourentzou. Adver- sarial contrastive learning by permuting cluster assign- ments. arXiv preprint arXiv:2204.10314 , 2022

2022 arXiv
[25]

Z. Wu, Y . Xiong, S. X. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrim- ination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733– 3742, 2018

2018
[26]

Zbontar, L

J. Zbontar, L. Jing, I. Misra, Y . LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310–12320. PMLR, 2021

2021
[27]

Intra-instance VICReg: Bag of self- supervised image patch embedding

Yubei Chen, Adrien Bardes, Zengyi Li, and Yann LeCun. Intra-instance VICReg: Bag of self- supervised image patch embedding. arXiv preprint arXiv:2206.08954, vol. 2, 2022. Appendix

2022 arXiv
[28]

These methods involve aggregating embeddings from multiple fixed-size patches during evaluation

Evaluation with 32- and 64-Patch Aggregation In addition to central cropping, we evaluated the robust base encoders using multi-patch aggregation with 32 and 64 patches. These methods involve aggregating embeddings from multiple fixed-size patches during evaluation. While this...
[30]

The results for CIFAR10 and CIFAR100 are shown in Figure 5

Detailed Ablation Study of Robust EMP-SSL The ablation study analyzed the impact of varying the number of patches (crops) used for adversarial training in the EMP-SSL framework. The results for CIFAR10 and CIFAR100 are shown in Figure 5. Findings include: • Increasing the numb...
[64]

• However, the computational cost increases signifi- cantly, making central cropping more practical for resource-constrained settings

is used. • However, the computational cost increases signifi- cantly, making central cropping more practical for resource-constrained settings

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Balestriero, M

R. Balestriero, M. Ibrahim, V . Sobal, A. Morcos, S. Shekhar, T. Goldstein, F. Bordes, A. Bardes, G. Mi- alon, Y . Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210 , 2023

arXiv 2023

[2] [2]

Bardes, J

A. Bardes, J. Ponce, and Y . LeCun. Vicreg: Variance-invariance-covariance regularization for self- supervised learning. arXiv preprint arXiv:2105.04906, 2021

arXiv 2021

[3] [3]

Bromley, I

J. Bromley, I. Guyon, Y . LeCun, E. S ¨ackinger, and R. Shah. Signature verification using a ”siamese” time delay neural network. Advances in Neural Information Processing Systems, 6, 1993

work page 1993

[4] [4]

Brown, B

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

work page 1901

[5] [5]

Caron, I

M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912–9924, 2020

work page 2020

[6] [6]

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Ma- chine Learning, pages 1597–1607. PMLR, 2020a

work page

[7] [7]

T. Chen, S. Liu, S. Chang, Y . Cheng, L. Amini, and Z. Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 699–708, 2020b

work page

[8] [9]

Croce and M

F. Croce and M. Hein. Reliable evaluation of adversar- ial robustness with an ensemble of diverse parameter- free attacks. In International Conference on Machine Learning, pages 2206–2216. PMLR, 2020

work page 2020