REVIEW 7 cited by
Surrogate Gap Minimization Improves Sharpness-Aware Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recently proposed Sharpness-Aware Minimization (SAM) improves generalization by minimizing a \textit{perturbed loss} defined as the maximum loss within a neighborhood in the parameter space. However, we show that both sharp and flat minima can have a low perturbed loss, implying that SAM does not always prefer flat minima. Instead, we define a \textit{surrogate gap}, a measure equivalent to the dominant eigenvalue of Hessian at a local minimum when the radius of the neighborhood (to derive the perturbed loss) is small. The surrogate gap is easy to compute and feasible for direct minimization during training. Based on the above observations, we propose Surrogate \textbf{G}ap Guided \textbf{S}harpness-\textbf{A}ware \textbf{M}inimization (GSAM), a novel improvement over SAM with negligible computation overhead. Conceptually, GSAM consists of two steps: 1) a gradient descent like SAM to minimize the perturbed loss, and 2) an \textit{ascent} step in the \textit{orthogonal} direction (after gradient decomposition) to minimize the surrogate gap and yet not affect the perturbed loss. GSAM seeks a region with both small loss (by step 1) and low sharpness (by step 2), giving rise to a model with high generalization capabilities. Theoretically, we show the convergence of GSAM and provably better generalization than SAM. Empirically, GSAM consistently improves generalization (e.g., +3.2\% over SAM and +5.4\% over AdamW on ImageNet top-1 accuracy for ViT-B/32). Code is released at \url{ https://sites.google.com/view/gsam-iclr22/home}.
Forward citations
Cited by 7 Pith papers
-
OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.
-
LightSAM: Parameter-Agnostic Sharpness-Aware Minimization
An adaptive SAM variant using AdaGrad and Adam steps for both perturbation and update is claimed to converge at O(ln T / T^{1/4}) without tuning, but the Adam version still needs decaying hyperparameters and the proof...
-
Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
GEAR-SAM re-allocates SAM's fixed perturbation radius across network blocks in proportion to an EMA of squared block-gradient norms, improving generalization on CIFAR, transfer, and label-noise benchmarks.
-
GLAD: Generalizable Tuning for Vision-Language Models
GLAD improves few-shot CLIP generalization by fusing original and sharpness-aware gradients during LoRA tuning and adding image-conditional text alignment.
-
Adversarial Data Augmentation for Single Domain Generalization via Lyapunov Exponent-Guided Optimization
LEAwareSGD modulates the learning rate with a Lyapunov exponent estimate and claims state-of-the-art accuracy on three single-domain generalization benchmarks, but the method is underspecified and its hyperparameters ...
-
C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning
Adding zeroth- and first-order flatness penalties to continual learning losses yields small consistent accuracy gains across seven methods, with the gated C-Flat++ variant at roughly 30% of the update cost.
-
Domain-Generalization to Improve Learning in Meta-Learning Algorithms
DGS-MAML layers gradient matching onto SharpMAML and claims O(1/T) convergence and tighter PAC-Bayes bounds, but the displayed theorems give O(1/sqrt T) under the paper's own parameter choices.
Discussion (0). Continue with ORCID to comment.