REVIEW 6 cited by
Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Overparametrized Deep Neural Networks (DNNs) often achieve astounding performances, but may potentially result in severe generalization error. Recently, the relation between the sharpness of the loss landscape and the generalization error has been established by Foret et al. (2020), in which the Sharpness Aware Minimizer (SAM) was proposed to mitigate the degradation of the generalization. Unfortunately, SAM s computational cost is roughly double that of base optimizers, such as Stochastic Gradient Descent (SGD). This paper thus proposes Efficient Sharpness Aware Minimizer (ESAM), which boosts SAM s efficiency at no cost to its generalization performance. ESAM includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. In the former, the sharpness measure is approximated by perturbing a stochastically chosen set of weights in each iteration; in the latter, the SAM loss is optimized using only a judiciously selected subset of data that is sensitive to the sharpness. We provide theoretical explanations as to why these strategies perform well. We also show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis base optimizers, while test accuracies are preserved or even improved.
Forward citations
Cited by 6 Pith papers
-
Flat Minima and Generalization: Insights from Stochastic Convex Optimization
In smooth stochastic convex optimization, flat empirical minima can incur constant population risk while sharp minima generalize optimally, and sharpness-aware algorithms can converge to such bad flat minima.
-
Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
GEAR-SAM re-allocates SAM's fixed perturbation radius across network blocks in proportion to an EMA of squared block-gradient norms, improving generalization on CIFAR, transfer, and label-noise benchmarks.
-
We-Math 2.0: A Versatile MathBook System for Incentivizing Visual Mathematical Reasoning
The abstract claims a new multimodal math reasoning system, but the body is an unrelated tensor-optimization paper, leaving every claim unverifiable.
-
Domain-Generalization to Improve Learning in Meta-Learning Algorithms
DGS-MAML layers gradient matching onto SharpMAML and claims O(1/T) convergence and tighter PAC-Bayes bounds, but the displayed theorems give O(1/sqrt T) under the paper's own parameter choices.
-
Attributing Data for Sharpness-Aware Minimization
SAM-HIF and SAM-GIF are proposed as data attribution scores for SAM-trained models, but SAM-GIF is TracIn with SAM gradients and SAM-HIF's derivation contains a load-bearing error.
-
FEDTAIL: Federated Long-Tailed Domain Generalization with Sharpness-Guided Gradient Matching
FedTAIL combines sharpness-aware minimization, gradient coherence, and curvature-aware class weighting, and reports improved domain generalization accuracy on PACS, OfficeHome, Digits-DG, and mini-DomainNet.
Discussion (0). Sign in to comment.