{"id":"6c55ae80-0913-43e2-9c06-6ffd07fd0ba9","arxiv_id":"2501.01529","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fine-tuning only the few transformer layers with the highest sharpness, using SAM while freezing the rest, improves clean and adversarial accuracy over full-model adversarial training baselines.","lead":"SAFER is a fine-tuning recipe that makes vision transformers harder to fool by adversarial attacks: it finds the layers most likely to overfit during adversarial training and updates only those layers with a sharpness-aware optimizer. The authors report consistent gains of about 5% in clean and adversarial accuracy on several transformer models, with some gains as high as 20%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 contradicts the 'consistently enhances' claim: [30]+SAFER on DeiT-Ti/Imagenette drops PGD-20 from 57.24 to 50.80.","rationale":"I reviewed the paper in good faith. The central claim is that SAFER consistently improves clean and adversarial accuracy. The reader's weakest assumption concerned the sharpness proxy in Eq (6). I partially agree: the proxy is a gradient-norm heuristic, and the SAM theory in Eq (1) does not directly justify per-layer adversarial sharpness. However, Table 6 provides an empirical control showing sharpness-selected layers outperform randomly selected layers, so the proxy carries at least some signal. The more serious issue is the paper's own Table 1, where applying SAFER to the Tian et al. baseline reduces PGD-20 on Imagenette from 57.24 to 50.80. This is a direct counterexample to the consistency claim. The paper does not address it. The reader's verdict of CONDITIONAL is appropriate, but the conditions should explicitly require resolving this inconsistency. I therefore keep the verdict unchanged.","tokens_in":15077,"tokens_out":5560,"duration_ms":48288,"concrete_test":"Reproduce the [30]+SAFER configuration exactly as described for DeiT-Ti on Imagenette (PGD-AT pretraining, SAFER fine-tuning, PGD-20 evaluation, ε=0.03, α=0.007, 20 steps) and compare to the Tian et al. baseline. If the PGD-20 drop is reproducible, the claim must be qualified to exclude this baseline. If it was a typo, correct the table. Also compute the distribution of per-baseline deltas for all rows in Table 1 and report how many configurations improve vs. degrade.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SAFER 'consistently enhances both clean and adversarial accuracy over baseline approaches' (Abstract). This is contradicted by the paper's own Table 1. In the DeiT-Ti row on Imagenette, the baseline 'Tian et al.' achieves PGD-20 of 57.24, while 'Tian et al. + SAFER' (i.e., [30] + SAFER) achieves 50.80, a drop of 6.44 points. Clean accuracy improves by only 0.91, and AA improves by 1.49. The authors include this row as evidence of SAFER 'over SOTA techniques,' but the PGD-20 degradation is never discussed. If the claim is meant to hold for every baseline listed, the data refute it. If it is meant only for the PGD-AT baselines, the paper must say so and remove the misleading SOTA comparison. This internal inconsistency is more directly load-bearing than the sharpness-proxy assumption: even if the proxy is heuristic, Table 6 shows it beats random layer selection, whereas Table 1 contains a quantitative failure of the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAFER, a layer-selective fine-tuning method for vision transformers that estimates a per-layer sharpness of the adversarial loss using a gradient-norm proxy (Eq. (6)), selects the top-K sharpest layers, and applies sharpness-aware minimization only to those layers while freezing the rest. Experiments across CIFAR-10, CIFAR-100, and Imagenette on ViT, DeiT, ConViT, and Swin variants compare SAFER against PGD-AT with SGD/SAM optimizers, ARD+PRM, and the method of Tian et al., and also include black-box evaluations and PEFT variants with LoRA/DoRA. The central claim is that SAFER consistently improves both clean and adversarial accuracy over baselines, with typical gains around 5% and peaks up to 20%.","tokens_in":15299,"tokens_out":5762,"duration_ms":56756,"significance":"If the reported results are reliable, SAFER is a practically useful and computationally cheap method: it shows that adversarial overfitting in ViTs may be concentrated in a small number of layers and that targeted SAM fine-tuning can mitigate it. The paper's strengths are its breadth of architectures and datasets, the direct comparison against layer-selective baselines (RiFT, CLAT), the ablation of dynamic versus fixed layer selection, the random-layer control in Table 6, and the overhead analysis in Section 4.4. However, the headline consistency claim is contradicted by one of the paper's own table rows, and the theoretical motivation for the sharpness proxy is heuristic; these issues need to be addressed before the paper can be accepted.","major_comments":[{"comment":"The Abstract claims SAFER 'consistently enhances both clean and adversarial accuracy over baseline approaches.' Table 1 contains a direct counterexample: for DeiT-Ti on Imagenette, 'Tian et al.' obtains PGD-20=57.24 and AA=54.89, while '[30]+SAFER' obtains PGD-20=50.80 and AA=56.38. PGD-20 drops by 6.44 points, even though clean accuracy improves by 0.91 points. The text in Sec. 4.2.1 presents this row as evidence of SAFER 'over SOTA techniques' without discussing the regression. If the consistency claim is intended only for the PGD-AT baselines, that scope must be stated explicitly; as written, the claim is falsified by the paper's own table.","section":"Sec. 4.2.1, Table 1"},{"comment":"The derivation from Eq. (4) to Eq. (6) reduces the sharpness measure to a gradient norm: for each x, max_{||epsilon||<=rho} epsilon^T grad = rho * ||grad||, so the maximum over the ball contains no second-order curvature information. More importantly, Eq. (1) is a bound on the clean training loss LS, whereas Eq. (4) uses the adversarial loss Ladv; the transfer of the SAM generalization bound to per-layer adversarial sharpness is not justified. Because SAFER's layer selection depends entirely on this proxy, the paper should either validate the proxy directly (e.g., by showing that the Eq. (6) ranking correlates with per-layer fine-tuning gains) or explicitly present the proxy as a heuristic, rather than as a consequence of the SAM theory.","section":"Sec. 3.1, Eq. (6)"},{"comment":"Several free hyperparameters—K (about 5% of layers), the 10-epoch sharpness re-measurement interval, and the number of initial adversarial pretraining epochs—are selected via hyperparameter optimization on the same CIFAR-10 and Imagenette benchmarks that appear in the main tables. No held-out validation split is described, and the ablations in Sec. 4.3.2 and Appendices B and C are also run on these datasets. Consequently, the reported gains could be partly due to tuning these hyperparameters to the test distributions. A cross-dataset sensitivity check or an explicit statement of this risk is needed to support the paper's generalization claims.","section":"Sec. 4.1, Sec. 4.3"},{"comment":"The manuscript reports 'the lowest observed accuracies' from at least 10 runs, but no standard deviations or confidence intervals are given anywhere. Several reported improvements are small (e.g., Swin-Ti on Imagenette: PGD-20 from 74.40 to 75.45 and AA from 71.20 to 72.69 in Table 1), and without variance estimates it is impossible to judge whether these differences are meaningful. Reporting means and standard deviations for the main comparisons in Table 1 would materially strengthen the claims.","section":"Sec. 4.1"}],"minor_comments":[{"comment":"In Eq. (7), the SAM perturbation epsilon is defined in terms of the gradient of Ladv with respect to w_i, but the notation leaves implicit that epsilon depends on the current w_i and on the adversarial example; a more explicit definition would improve reproducibility.","section":"Sec. 3.2, Eq. (7)"},{"comment":"Several table headers and cells contain typographical artifacts such as 'C IFAR -10', 'ARD + PRM', and 'TIAN ET AL .'. These should be cleaned before publication.","section":"Table 1 and throughout"},{"comment":"The table reports SAFER improvements as '+3.36' and '+2.61' for FAB and StAdv, but the absolute baseline values are not shown for FAB, StAdv, and PIXEL. Please provide the absolute numbers for both baseline and SAFER models.","section":"Appendix D.2, Table 11"},{"comment":"The statement that 'incorporating SAFER at any stage of training results in higher clean accuracy and robustness at convergence' is supported only for the 70-epoch budget in Figure 2. The extended 150-epoch curves in Appendix E compare SAFER starting at epoch 50 with SAFER from scratch, not with SAFER starting at other stages, so the phrase 'any stage' is too broad and should be qualified.","section":"Sec. 4.3.1, Figure 2"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the paper has a solid empirical core and a useful practical idea, but the current form overstates the consistency of the results. The specific regression in Table 1 for DeiT-Ti on Imagenette under the [30]+SAFER comparison is directly load-bearing because it contradicts the Abstract's headline claim. I also share the concern that the sharpness proxy in Eq. (6) is heuristic; this is acceptable if framed as an empirical technique, but the theoretical language in Sec. 3.1 should be softened or supported with a direct validation. I would like to see the authors either explain the Table 1 regression (e.g., as a variance artifact, with error bars) or narrow the claimed scope, and report error bars for the main results. If these issues are fixed, the paper could be publishable; in its current form it needs a major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: SAFER is a sensible, cheap idea—measure per-layer sharpness (which after a Taylor expansion is just the gradient norm), fine-tune the top-K layers with SAM, freeze everything else. On most reported settings it does help: clean and PGD-20/AA numbers go up over PGD-AT(SGD/SAM) and ARD+PRM, and the LoRA/DoRA extension plus the dynamic-selection ablation add value. I'd credit the overhead analysis (0.2% for sharpness estimation, ~3% per epoch) and the worst-of-10 reporting. The soft spot is not the heuristic—it's the abstract. 'Consistently enhances both clean and adversarial accuracy' is false as stated. Look at Table 1, DeiT-Ti on Imagenette: 'Tian et al.' has PGD-20 of 57.24, and 'Tian et al. + SAFER' gets 50.80, a 6.44-point drop. Clean improves by 0.91 and AA by 1.49, but the adversarial number that matters for the claim collapses. That contradiction is load-bearing. Also, Eq (6) reduces sharpness to a gradient norm, so calling it a 'novel sharpness metric' oversells it. The SAM bound in Eq (1) is for clean loss; applying it to adversarial loss is hand-waving. There are no error bars, and K and the re-selection interval are tuned on the same benchmarks, so the exact magnitude of the 5% gain is uncertain. None of this makes the method useless. The empirical pattern—targeted SAM on a few layers beats full-model SAM and random layer selection (Table 6)—is a real finding. But the paper needs major revision: soften the consistency claim, report per-configuration gains honestly, add variance, release code, and ideally validate K on held-out data. Recommendation: send to peer review, with the caveat that the authors must address the Table 1 contradiction and the overbroad claim. I'd want a referee to check whether the [30]+SAFER drop is a fluke or a sign that SAFER can hurt when the base model is already strong on PGD.","headline":"The paper's own Table 1 contradicts its headline 'consistently enhances' claim, and that needs fixing before I'd trust the rest.","tokens_in":15851,"tokens_out":2673,"would_cite":false,"duration_ms":25166,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sharpness-aware fine-tuning of a few layers lifts ViT defense by 5–20%.","keywords":["adversarial robustness","vision transformers","sharpness-aware minimization","layer-selective fine-tuning","adversarial overfitting","parameter-efficient fine-tuning","PGD adversarial training","AutoAttack"],"falsifier":"Compute the full SAM sharpness (Eq. 4, with the maximization over $\\epsilon$ solved iteratively rather than linearized) for every layer of a pretrained DeiT-Ti, then fine-tune each single layer with SAM in turn and record the resulting clean and PGD-20 accuracy; if the layers with the highest full sharpness are not the ones whose individual fine-tuning yields the largest improvements, or if the linear proxy of Eq. (6) disagrees with the full sharpness ranking, the selection mechanism is not the cause of SAFER's gains.","tokens_in":14855,"feed_emoji":"🛡️","tokens_out":6348,"duration_ms":48883,"temperature":0.7,"pith_summary":"This paper argues that adversarial overfitting in vision transformers is concentrated in a small number of layers, and that sharpness-aware minimization (SAM) applied only to those layers—while freezing the rest—simultaneously improves clean accuracy and adversarial robustness. The proposed method, SAFER, measures each layer's tendency to overfit by a sharpness proxy computed from the norm of the adversarial-loss gradient with respect to that layer's weights, selects the top few layers, and fine-tunes them with the SAM objective, periodically re-selecting layers. Across ViT, DeiT, ConViT, and Swin on CIFAR-10, CIFAR-100, and Imagenette, SAFER reports typical gains of about 5% and peak gains near 20% over strong baselines, and it also improves results when combined with LoRA and DoRA. If correct, the paper shows that a global robustness problem can be solved locally: a small subset of layers carries most of the overfitting burden, and targeted smoothing of that subset suffices.","feed_headline":"Tune only the sharpest layers: ViT accuracy and defense rise 5–20%","feed_subtitle":"SAFER locates the few layers that overfit during adversarial training and smooths just those, beating full-model baselines.","key_machinery":"The load-bearing object is the first-order Taylor sharpness metric of Eq. (6): for each layer $i$, $\\gamma_i \\approx \\sum_{x\\in B} \\max_{\\|\\epsilon\\|_2\\le\\rho} \\epsilon^T \\partial \\mathcal{L}_{\\mathrm{adv}}(w_i,x)/\\partial w_i \\propto \\sum_{x\\in B} \\|\\partial \\mathcal{L}_{\\mathrm{adv}}(w_i,x)/\\partial w_i\\|_2$. This single-backward-pass proxy estimates how much the adversarial loss changes when that layer's weights are perturbed, ranking layers by susceptibility to overfitting. SAFER then fine-tunes the top-$K$ layers with the SAM objective, perturbing only those weights in the direction of the adversarial-loss gradient, while freezing the remaining layers; the selection is refreshed every ten epochs to track the layers that currently overfit.","core_discovery":"The central claim is that the per-layer sharpness of the adversarial loss, approximated by $\\gamma_i \\propto \\sum_{x\\in B} \\|\\partial \\mathcal{L}_{\\mathrm{adv}}(w_i, x)/\\partial w_i\\|_2$, reliably identifies which transformer layers overfit during adversarial training, and that applying SAM only to those layers (about 5% of the model's layers) while freezing the others yields models that are both more accurate on clean data and more resistant to white-box and black-box attacks. The authors state that this layer-selective application resolves the convergence problems that arise when SAM is applied to the entire transformer, and that it outperforms layer-selective methods designed for CNNs, such as CLAT and RiFT, which degrade ViT robustness.","pith_inferences":["If the sharpness proxy genuinely localizes overfitting, the same principle might apply beyond adversarial training—for example, to standard fine-tuning of large models where a few layers dominate catastrophic forgetting or distribution shift.","The stability of sharp layers across datasets raises the question of whether specific transformer modules (attention vs. MLP, early vs. late blocks) are structurally prone to non-robust features; identifying that mechanism could inspire architectural changes rather than post-hoc fine-tuning.","A natural next test is scaling: if SAFER's benefit grows with model size and parameter count, it could become a cheap default for robustly adapting large pretrained transformers, where full-model SAM is prohibitively expensive.","Because SAFER uses only gradient norms, it may extend to architectures where feature-based criticality indices (like CLAT's) are incomparable across layers, such as mixed CNN-transformer or multimodal backbones."],"forward_implications":["SAFER removes the need for early stopping: models keep improving in clean and adversarial accuracy through extended adversarial training, whereas PGD-AT and PGD-AT with SAM decline.","Robustness transfers to unseen attacks: SAFER improves accuracy under AutoAttack, FAB, StAdv, and Pixel attacks, even though only PGD attacks are seen during training.","The method composes with parameter-efficient fine-tuning: adding SAFER to LoRA and DoRA improves both clean and PGD-20 accuracy over the corresponding PGD-AT baselines.","Sharpness rankings are stable across datasets and batch samples, indicating that certain layers have architecture-inherent tendencies to overfit, and dynamic re-selection is needed because the set of overfitting layers changes during fine-tuning.","The overhead is small: layer sharpness can be estimated from 0.001% of the training data with about 0.2% extra time, and a SAFER fine-tuning epoch adds roughly 3% over SGD."],"supporting_citations":[{"why":"Supplies the sharpness-aware minimization objective and the generalization bound (Eq. 1) that motivates measuring per-layer sharpness.","marker":"[11]"},{"why":"Defines the PGD adversarial training objective and attack used to train and evaluate the baseline models.","marker":"[22]"},{"why":"Introduces critical-layer selection for adversarial fine-tuning; SAFER compares against it and claims its feature-based index fails on transformers.","marker":"[13]"},{"why":"Proposes robust critical fine-tuning (RiFT), a layer-redundancy-based selective method used as a second baseline in Table 2.","marker":"[40]"},{"why":"Provides the ARD+PRM adversarial training recipe for vision transformers, the strongest baseline SAFER is compared against.","marker":"[23]"},{"why":"Sets the transformer adversarial training protocol and best practices that the experiments follow.","marker":"[24]"},{"why":"Supplies AutoAttack, the ensemble of parameter-free attacks used to evaluate white-box and black-box robustness.","marker":"[8]"},{"why":"Links sharpness-aware minimization to adversarial training, motivating the use of SAM in the adversarial setting.","marker":"[39]"}],"fun_headline_variants":["SAFER: tune only the sharpest ViT layers for 5-20% robustness boost","Layer-selective sharpness tuning lifts ViT accuracy and defense by up to 20%","Identify overfitting layers, apply SAM to just 5% for ViT gains up to 20%","Sharpness-aware layer selection: 5% of ViT layers yields 5-20% accuracy boost","Tune just the sharpest layers: SAFER lifts ViT clean and adversarial accuracy 5-20%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's success rests on the empirical claim that the first-order Taylor sharpness of Eq. (6), a gradient-norm proxy computed on the adversarial loss, ranks layers in the same order as their actual contribution to whole-model overfitting; if that proxy misranks layers, SAFER degenerates into fine-tuning random layers.","fun_headline_variants_meta":{"raw":{"variants":["SAFER: tune only the sharpest ViT layers for 5-20% robustness boost","Layer-selective sharpness tuning lifts ViT accuracy and defense by up to 20%","Identify overfitting layers, apply SAM to just 5% for ViT gains up to 20%","Sharpness-aware layer selection: 5% of ViT layers yields 5-20% accuracy boost","Tune just the sharpest layers: SAFER lifts ViT clean and adversarial accuracy 5-20%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001065,"raw_usage":{"total_tokens":4427,"prompt_tokens":870,"completion_tokens":3557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":3430}},"tokens_in":486,"tokens_out":3557,"duration_ms":26848,"temperature":1.0,"reasoning_tokens":3430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:27:03.057697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the full SAM sharpness (Eq. 4, with the maximization over $\\epsilon$ solved iteratively rather than linearized) for every layer of a pretrained DeiT-Ti, then fine-tune each single layer with SAM in turn and record the resulting clean and PGD-20 accuracy; if the layers with the highest full sharpness are not the ones whose individual fine-tuning yields the largest improvements, or if the linear proxy of Eq. (6) disagrees with the full sharpness ranking, the selection mechanism is not the cause of SAFER's gains.","supporting_citations":[{"cited_title":"Sharpness-aware minimization for efficiently improving generalization, 2021","cited_arxiv_id":null,"evidence_quote":"Supplies the sharpness-aware minimization objective and the generalization bound (Eq. 1) that motivates measuring per-layer sharpness."},{"cited_title":"Towards deep learning models resistant to adversarial attacks, 2019","cited_arxiv_id":null,"evidence_quote":"Defines the PGD adversarial training objective and attack used to train and evaluate the baseline models."},{"cited_title":"Criticality leveraged adversarial train- ing (clat) for boosted performance via parameter efficiency,","cited_arxiv_id":null,"evidence_quote":"Introduces critical-layer selection for adversarial fine-tuning; SAFER compares against it and claims its feature-based index fails on transformers."},{"cited_title":"Improving generalization of adversarial training via robust critical fine-tuning, 2023","cited_arxiv_id":null,"evidence_quote":"Proposes robust critical fine-tuning (RiFT), a layer-redundancy-based selective method used as a second baseline in Table 2."},{"cited_title":"When adversarial training meets vision trans- formers: Recipes from training to architecture, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the ARD+PRM adversarial training recipe for vision transformers, the strongest baseline SAFER is compared against."},{"cited_title":"When adversarial training meets vision trans- formers: Recipes from training to architecture","cited_arxiv_id":null,"evidence_quote":"Sets the transformer adversarial training protocol and best practices that the experiments follow."},{"cited_title":"Reliable evalua- tion of adversarial robustness with an ensemble of diverse parameter-free attacks, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies AutoAttack, the ensemble of parameter-free attacks used to evaluate white-box and black-box robustness."}],"review_version":1}