Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

In smooth convex stochastic optimization, a flat empirical minimum can have constant population risk while a sharp empirical minimum of the same problem has zero population risk; the sharpness-aware algorithms SA-GD and SAM can also be driv

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 23:52 UTC pith:WVRY2F6X

load-bearing objection The central lower-bound theorems have a probability miscalculation: with d=2n the stated constant-probability Ω(1) results fail, and the construction actually needs d≈2^n. the 3 major comments →

arxiv 2511.03548 v2 pith:WVRY2F6X submitted 2025-11-05 cs.LG

Flat Minima and Generalization: Insights from Stochastic Convex Optimization

Matan Schliserman , Shira Vansover-Hager , Tomer Koren This is my paper
classification cs.LG MSC 68Q3290C2568T05
keywords flat minimasharpness-aware minimizationgeneralizationstochastic convex optimizationpopulation riskalgorithmic stabilitylower boundssmooth convex optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests the widely held belief that flat minima generalize better. It defines flatness in the strongest way — the loss is exactly zero on a ball of radius ρ around the minimizer — and shows that even in the simplest convex, smooth, realizable setting, a flat empirical-risk minimizer can have constant (Ω(1)) population risk while a sharp empirical minimizer of the same problem has zero population risk. The same failure is proved for the trajectories of two sharpness-aware algorithms, SA-GD and SAM: both converge quickly in empirical risk yet can land on solutions with Ω(1) population risk. The paper also proves stability-based upper bounds showing that when the perturbation radius is kept close to the flatness radius, both algorithms generalize at the optimal O(1/n) rate. The upshot is that flatness alone is not a sufficient explanation of good generalization, even in the regime most favorable to it.

Core claim

A flat empirical minimum is not a reliable indicator of small population risk, even when flatness means the loss is identically zero on a ball of radius ρ and the losses are non-negative β-smooth convex. The paper builds a distribution whose empirical risk has two global minimizers: a perfectly flat one that minimizes the sharpness-aware risk for every r, and a sharp one whose empirical risk rises quadratically in every neighborhood. The flat one has population risk Ω(1); the sharp one has population risk 0. The construction is adapted to SA-GD and SAM: both can be steered to the bad flat minimizer and suffer Ω(1) population risk while still optimizing the empirical risk at rate O(1/T). Stab

What carries the argument

The machinery is the sharpness-aware empirical risk F_S^r(w)=max_{||v||≤r} F_S(w+v) paired with a strong flatness condition: a minimizer where the loss is exactly zero on a ball of radius ρ. Lower bounds combine a one-dimensional 'flat hinge' with a classical stochastic-convex optimization construction that contains a spurious empirical minimizer; the proof then chains T orthogonal copies of the construction so that each iteration's worst-case perturbation points one step deeper into the bad direction. For SAM, the normalized ascent step is exploited to amplify a small initial gradient into a deterministic chain across subspaces. Upper bounds use leave-one-out on-average stability of the per

Load-bearing premise

The lower-bound proofs assume the sharpness oracle (for SA-GD) or the normalized-gradient step (for SAM) picks the particular perturbation direction that steers the trajectory toward the spurious flat minimizer; if ties are broken differently, the trajectory need not reach the bad minimum.

What would settle it

Run SA-GD on the paper's hard T-subspace instance with the maximization oracle modified to break ties toward the good minimizer; if the population risk drops below Ω(1), the theorem's conclusion depends on the oracle's tie-breaking rather than on flatness. For SAM, add a tiny random perturbation to the normalized ascent direction while keeping all other parameters fixed and check whether the chained steps to the bad minimizer break.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • SA-GD and SAM achieve a fast, dimension-independent O(1/T) empirical-risk rate on general smooth convex objectives, even though the sharpness-aware objective is non-smooth.
  • The empirical minimizer found by SA-GD can be flat, yet still have Ω(1) population risk; flatness of the output does not certify generalization.
  • The lower bounds hold in the realizable (zero-noise) and smooth regime, so the failure is not caused by label noise or non-smoothness.
  • With T=n, η≈1/β, and r≲ρ+1/√T, both SA-GD and SAM attain O(1/n + r^2 n) population risk, matching the optimal rate when r is small.
  • A single empirical problem can contain flat minima with maximal population risk and sharp minima with zero population risk, so the selection mechanism (which minimizer the algorithm finds) is part of the generalization story.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This suggests that if the maximization oracle is allowed to break ties arbitrarily, the SA-GD lower bound may be driven by tie-breaking rather than by flatness; a randomized or averaged oracle might escape the bad minimizer, which would separate the algorithm from the landscape property.
  • A testable extension is that SAM's behavior depends on the exact normalized direction: variants that use a different norm or a stochastic perturbation should generalize differently on tasks with many near-tied worst-case directions.
  • The constructions suggest that flatness measures based only on the worst-case direction can be fooled; evaluating the loss along several random directions, or using Hessian information, may give a more reliable predictor of population risk.
  • For theory, the results imply that any positive flatness-generalization theorem needs an additional condition on how the flat minimum is selected or on the structure of the loss landscape around it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the flatness–generalization link in smooth stochastic convex optimization (SCO). It introduces a strong ρ-flatness condition (perfectly flat minimizer of radius ρ) and analyzes three sharpness-aware objects: SA-ERM, SA-GD, and SAM. On the positive side, it proves fast O(1/T) empirical-risk convergence for SA-GD and SAM when the perturbation radius r is tuned near ρ, and dimension-independent population-risk upper bounds via algorithmic stability. The central negative claims are: (i) in dimension d=2n+1 there is a convex, 1-smooth, 1-Lipschitz, ρ-flat SCO instance where a flat empirical-risk minimizer has population risk Ω(1) while a sharp empirical-risk minimizer has zero risk (Theorem 1); (ii) SA-GD (Theorem 3) and SAM (Theorem 6) can also incur Ω(1) population risk in this setting; and (iii) SAM can converge to a sharp empirical minimum (Theorem 5).

Significance. If the negative results were established as stated, the paper would make a substantial contribution: it would challenge the flatness–generalization conjecture in the simplest convex smooth realizable setting, give the first global convergence rates for normalized SAM on general smooth convex objectives, and provide dimension-independent stability guarantees. The upper-bound machinery is a genuine strength, and the paper is careful to state its strong notion of flatness. However, the lower-bound proofs contain load-bearing probabilistic and oracle-dependence errors, so the central negative claims are not currently established. The paper is likely salvageable by exponential-dimension constructions and a precise specification of the tie-breaking behavior of SA-GD/SAM, but the revised results would be less striking.

major comments (3)
  1. [B.1, Theorem 1; also B.3, Theorem 3; B.6, Theorem 6] The probability estimates in the three main lower-bound proofs are incorrect. In Theorem 1, with S uniform over {0,1}^{2n}, the probability that some coordinate is 0 on all n samples is 1−(1−2^{−n})^{2n}, which for n=4 is ≈0.399 < 1/2 and decays like 2n/2^n. The claimed >1−e^{−1}>1/2 requires the number of coordinates to be exponential in n, i.e., d≈2^n, not d=2n+1. The same binomial miscalculation appears in Theorem 3 (all-ones coordinate) and Theorem 6 (exactly one all-zero coordinate: for n=6 the probability is ≈0.157, not ≥1/3). This directly invalidates the stated Ω(1) population-risk lower bounds at the claimed dimensions and probabilities.
  2. [B.3, Theorem 3] The SA-GD lower bound depends on a particular tie-breaking rule for the sharpness oracle. At w_1=0, and at every step on the constructed trajectory, v = r e_d is also a valid maximizer of F_S(w_t+v) with value (1/2)(r−ρ)^2, because w_t(d)=0 and the construction's earlier negative coordinates do not affect the e_d direction. The proof simply selects v_t = r e_{I_t} from the argmax set. A different tie-breaking rule can keep SA-GD away from the spurious empirical minimum, so the theorem does not hold for an arbitrary valid run of SA-GD. The construction needs a unique maximizer (or the oracle choice must be specified as part of the algorithm, with the theorem stated accordingly).
  3. [B.6, Theorem 6] The dimension used in the construction is inconsistent with the theorem statement. Theorem 6 states d=2nT, while the proof builds d=T·2n+1 blocks (there are 2n coordinates in Z plus an extra coordinate). Even with the +1 correction, the probability argument has the exponential-dimension problem described above: the 'exactly one all-zero coordinate' event has probability 2n·2^{−n}(1−2^{−n})^{2n−1}, which is below the claimed 1/e for n≥6. To obtain constant probability one needs d=T·2^n+1. This is load-bearing for the SAM Ω(1) lower bound.
minor comments (5)
  1. [B.1, proof of Theorem 1] The final display writes F(ρ e_I)−F(w⋆) where the preceding construction defines w^(2)=ρ e_d; this should be F(ρ e_d)−F(w⋆).
  2. [Theorem 4] The text says 'produced by SAM for T steps (Eq.(4))' but SAM is defined in Eq. (5); Eq. (4) is SA-GD.
  3. [Theorem 5] The SAM update uses the normalized gradient ∥∇F_S(w_t)∥ in the denominator; at w_1=0 the gradient is zero and the update is undefined. A convention for the zero-gradient case (e.g., v_t=0) should be stated.
  4. [B.6.1, Lemma 8] There are small typos in the induction algebra, e.g., 'δ_t u3_{t−1}(t−1)' in the u_t(t−1) display should presumably be 'δ_t u_{t−1}(t−1)'.
  5. [Theorem 3 statement] There is an extra comma in 'For every n, T∈N, , η >0'; the condition ρ < r(1−3/(3+η√T)) should be stated with clear parentheses.

Circularity Check

0 steps flagged

No circularity: lower bounds are explicit constructions and upper bounds rely on independent stability lemmas.

full rationale

The paper's main negative results are existence constructions, not predictions from fitted parameters. Theorem 1 explicitly exhibits an empirical risk minimizer w^(1)=e_I with zero empirical loss on the sample (because coordinate I is zero on all training points) and positive population risk; this is a direct probabilistic construction, not a quantity fit from the data and then renamed. Theorems 3 and 6 similarly construct explicit smooth functions and track the dynamics of SA-GD/SAM; the only freedom is the adversary's choice of the argmax oracle or the normalization chain, which is an assumption of the lower bound rather than a fitted input. The upper bounds (Theorems 7 and 8) are derived from on-average leave-one-out stability (Lemma 3) plus a stability-to-generalization transfer (Lemma 2); the latter is cited from the authors' earlier work [54], but it is a parameter-free general lemma whose assumptions do not include the present flatness claims, so it is independent support and not circular. The flatness radius ρ is a property of the constructed loss, not estimated from data. Self-citations to prior SCO hard-instance constructions ([55], [36], [63]) are used as building blocks and are not the sole justification for the paper's central claim. The probability miscalculation suggested in review (d=2n vs d=2^n) would be a correctness flaw, not a circularity. Accordingly, no derivation step reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The central results are existential constructions, so the free parameters are the flatness radius and the perturbation radius. The SAM construction introduces many carefully chosen constants, but they are internal to the construction, not fitted to external data. No new physical or metaphysical entities are introduced.

free parameters (3)
  • flatness radius ρ = varies per theorem; assumed known
    The definition of ρ-flatness is the key modeling assumption. It is a free parameter of the problem instance, not fitted to data. Lower bounds are proven for all ρ in certain ranges; upper bounds depend on knowing ρ.
  • perturbation radius r = algorithm hyperparameter, not fitted
    In the lower bounds, r is chosen in specific ranges (e.g., r ≈ ρ + 1/sqrt(T)). It is a hyperparameter of the algorithm, not fitted, but it is essential to the constructions.
  • SAM construction parameters δ_j, σ, γ, λ, δ_1 = chosen by existence arguments in Lemma 8 and surrounding text
    In the SAM lower bound (Theorem 6), the loss function is parameterized by δ_j and other constants that are chosen sequentially to force the SAM dynamics. These are not fitted to data but are hand-crafted to make the proof work, and the proof of Lemma 8 establishes their existence.
axioms (3)
  • domain assumption For every z, f(w, z) is convex, non-negative, β-smooth, and admits a ρ-flat minimizer (Definition 1).
    This is the problem setting. It is a strong assumption used in all theorems; flagged by the authors as 'strong' in the text.
  • domain assumption The SAER maximization argmax over the ball is exactly computable (SA-GD) and in the adversarial cases it returns a specific tie-breaking vector.
    In the SA-GD lower bound, the proof relies on the argmax returning the coordinate direction e_{I_t}. The paper assumes that the algorithm uses such a tie-breaking; this is an assumption about the oracle and the algorithm's behavior.
  • domain assumption The population distribution D is uniform over an exponentially large instance set Z, and the model class is all of R^d with d = 2n+1 or 2nT.
    The constructions define the instances; they are valid SCO instances but they show the existence of worst-case distributions, not typical high-dimensional learning.

pith-pipeline@v1.3.0-alltime-deepseek · 26568 in / 7100 out tokens · 51882 ms · 2026-08-03T23:52:17.722011+00:00 · methodology

0 comments
read the original abstract

Understanding the generalization behavior of learning algorithms is a central goal of learning theory. A recently emerging explanation is that learning algorithms are successful in practice because they converge to flat minima, which have been consistently associated with improved generalization performance. In this work, we study the link between flat minima and generalization in the canonical setting of stochastic convex optimization with a non-negative, $\beta$-smooth objective. Our first finding is that, even in this fundamental and well-studied setting, flat empirical minima may incur trivial $\Omega(1)$ population risk while sharp minima generalizes optimally. Then, we show that this poor generalization behavior extends to two natural ''sharpness-aware'' algorithms originally proposed by Foret et al. (2021), designed to bias optimization toward flat solutions: Sharpness-Aware Gradient Descent (SA-GD) and Sharpness-Aware Minimization (SAM). For SA-GD, which performs gradient steps on the maximal loss in a predefined neighborhood, we prove that while it successfully converges to a flat minimum at a fast rate, the population risk of the solution can still be as large as $\Omega(1)$, indicating that even flat minima found algorithmically using a sharpness-aware gradient method might generalize poorly. For SAM, a computationally efficient approximation of SA-GD based on normalized ascent steps, we show that although it minimizes the empirical loss, it may converge to a sharp minimum and also incur population risk $\Omega(1)$. Finally, we establish population risk upper bounds for both SA-GD and SAM using algorithmic stability techniques.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

    cs.LG 2026-07 conditional novelty 5.0

    EISAM adds an extragradient-inspired prediction step to SAM, yielding better generalization and reduced sensitivity to the perturbation radius across image, text, and detection tasks.

Reference graph

Works this paper leans on

76 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Agarwala and Y

    A. Agarwala and Y. Dauphin. SAM operates far from home: eigenvalue regularization as a dynamical phenomenon. InICML, 2023

    2023

  2. [2]

    I. Amir, T. Koren, and R. Livni. SGD generalizes better than gd (and regularization doesn’t help). InConference on Learning Theory, pages 63–92. PMLR, 2021

    2021

  3. [3]

    Andriushchenko and N

    M. Andriushchenko and N. Flammarion. Towards Understanding Sharpness-Aware Minimiza- tion. InICML, 2022

    2022

  4. [4]

    Andriushchenko, D

    M. Andriushchenko, D. Bahri, H. Mobahi, and N. Flammarion. Sharpness-aware minimization leads to low-rank features. InNeurIPS, 2023

    2023

  5. [5]

    Andriushchenko, F

    M. Andriushchenko, F. Croce, M. M¨ uller, M. Hein, and N. Flammarion. A modern look at the relationship between sharpness and generalization.arXiv preprint arXiv:2302.07011, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Attia, M

    A. Attia, M. Schliserman, U. Sherman, and T. Koren. Fast last-iterate convergence of sgd in the smooth interpolation regime.arXiv preprint arXiv:2507.11274, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Bahri, H

    D. Bahri, H. Mobahi, and Y. Tay. Sharpness-aware minimization improves language model generalization.arXiv preprint arXiv:2110.08529, 2021

    Pith/arXiv arXiv 2021

  8. [8]

    P. L. Bartlett, P. M. Long, and O. Bousquet. The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima.Journal of Machine Learning Research, 24(316):1–36, 2023

    2023

  9. [9]

    Behdin and R

    K. Behdin and R. Mazumder. Sharpness-aware minimization: An implicit regularization per- spective.arXiv preprint arXiv:2302.11836, 2023

    Pith/arXiv arXiv 2023

  10. [10]

    Blanc, N

    G. Blanc, N. Gupta, G. Valiant, and P. Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. InConference on learning theory, pages 483–513. PMLR, 2020

    2020

  11. [11]

    Bousquet and A

    O. Bousquet and A. Elisseeff. Stability and generalization.The Journal of Machine Learning Research, 2:499–526, 2002

    2002

  12. [12]

    Carmon, R

    D. Carmon, R. Livni, and A. Yehudayoff. The sample complexity of ERMs in stochastic convex optimization.arXiv preprint arXiv:2311.05398, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Chen, C.-J

    X. Chen, C.-J. Hsieh, and B. Gong. When vision transformers outperform resnets without pre-training or strong data augmentations.arXiv preprint arXiv:2106.01548, 2021

    Pith/arXiv arXiv 2021

  14. [14]

    Z. Chen, J. Zhang, Y. Kou, X. Chen, C.-J. Hsieh, and Q. Gu. Why does sharpness-aware minimization generalize better than sgd? InNeurIPS, 2024

    2024

  15. [15]

    Y. Dai, K. Ahn, and S. Sra. The crucial role of normalization in sharpness-aware minimization. InNeurIPS, 2023. 10

    2023

  16. [16]

    Damian, T

    A. Damian, T. Ma, and J. D. Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449–27461, 2021

    2021

  17. [17]

    L. Ding, D. Drusvyatskiy, M. Fazel, and Z. Harchaoui. Flat minima generalize for low-rank matrix recovery.Information and Inference: A Journal of the IMA, 13(2):iaae009, 2024

    2024

  18. [18]

    L. Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp minima can generalize for deep nets. InInternational Conference on Machine Learning, pages 1019–1028. PMLR, 2017

    2017

  19. [19]

    J. Du, H. Yan, J. Feng, J. T. Zhou, L. Zhen, R. S. M. Goh, and V. Y. Tan. Efficient sharpness- aware minimization for improved training of neural networks.arXiv preprint arXiv:2110.03141, 2021

    Pith/arXiv arXiv 2021

  20. [20]

    J. Du, D. Zhou, J. Feng, V. Tan, and J. T. Zhou. Sharpness-aware training for free.Advances in Neural Information Processing Systems, 35:23439–23451, 2022

    2022

  21. [21]

    G. K. Dziugaite and D. Roy. Entropy-sgd optimizes the prior of a pac-bayes bound: General- ization properties of entropy-sgd and data-dependent priors. InICML, 2018

    2018

  22. [22]

    G. K. Dziugaite and D. M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data.arXiv preprint arXiv:1703.11008, 2017

    Pith/arXiv arXiv 2017

  23. [23]

    Evron, R

    I. Evron, R. Levinstein, M. Schliserman, U. Sherman, T. Koren, D. Soudry, and N. Sre- bro. Better rates for random task orderings in continual linear models.arXiv preprint arXiv:2504.04579, 2025

    arXiv 2025

  24. [24]

    V. Feldman. Generalization of ERM in stochastic convex optimization: The dimension strikes back. InAdvances in Neural Information Processing Systems, volume 29, 2016

    2016

  25. [25]

    Foret, A

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for effi- ciently improving generalization. InICLR, 2021

    2021

  26. [26]

    J. Z. HaoChen, C. Wei, J. Lee, and T. Ma. Shape matters: Understanding the implicit bias of the noise covariance. InConference on Learning Theory, pages 2315–2357. PMLR, 2021

    2021

  27. [27]

    Hardt, B

    M. Hardt, B. Recht, and Y. Singer. Train Faster, Generalize Better: Stability of Stochastic Gradient Descent. InInternational Conference on Machine Learning, pages 1225–1234. PMLR, 2016

    2016

  28. [28]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Flat minima.Neural computation, 9(1):1–42, 1997

    1997

  29. [29]

    Jastrzebski, Z

    S. Jastrzebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey. Three Factors Influencing Minima in SGD. InInternational Conference of Artificial Neural Networks (ICANN), 2018

    2018

  30. [30]

    Jiang and S

    X. Jiang and S. U. Stich. Adaptive SGD with polyak stepsize and line-search: Robust conver- gence and variance reduction. InNeurIPS, 2023

    2023

  31. [31]

    Jiang, B

    Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. Fantastic generalization measures and where to find them. InICLR, 2019

    2019

  32. [32]

    Kaddour, L

    J. Kaddour, L. Liu, R. Silva, and M. J. Kusner. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577–16595, 2022. 11

    2022

  33. [33]

    N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. InICLR, 2016

    2016

  34. [34]

    H. Kim, J. Park, Y. Choi, and J. Lee. Stability analysis of sharpness-aware minimization. arXiv preprint arXiv:2301.06308, 2023

    Pith/arXiv arXiv 2023

  35. [35]

    M. Kim, D. Li, S. X. Hu, and T. Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. InInternational Conference on Machine Learning, pages 11148–11161. PMLR, 2022

    2022

  36. [36]

    Koren, R

    T. Koren, R. Livni, Y. Mansour, and U. Sherman. Benign underfitting of stochastic gradient descent.Advances in Neural Information Processing Systems, 35:19605–19617, 2022

    2022

  37. [37]

    J. Kwon, J. Kim, H. Park, and I. K. Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. InInternational conference on machine learning, pages 5905–5914. PMLR, 2021

    2021

  38. [38]

    H. Lee, H. Cho, H. Kim, D. Gwak, J. Kim, J. Choo, S.-Y. Yun, and C. Yun. Plastic: Improving input and label plasticity for sample efficient reinforcement learning.Advances in Neural Information Processing Systems, 36:62270–62295, 2023

    2023

  39. [39]

    Lei and Y

    Y. Lei and Y. Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. InInternational Conference on Machine Learning, pages 5809–5819. PMLR, 2020

    2020

  40. [40]

    Li and G

    B. Li and G. Giannakis. Enhancing sharpness-aware optimization through variance suppres- sion. InNeurIPS, 2023

    2023

  41. [41]

    T. Li, P. Zhou, Z. He, X. Cheng, and X. Huang. Friendly sharpness-aware minimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5631–5640, 2024

    2024

  42. [42]

    Z. Li, T. Wang, and S. Arora. What happens after sgd reaches zero loss?–a mathematical framework.arXiv preprint arXiv:2110.06914, 2021

    Pith/arXiv arXiv 2021

  43. [43]

    Y. Liu, S. Mai, X. Chen, C.-J. Hsieh, and Y. You. Towards efficient and scalable sharpness- aware minimization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12360–12370, 2022

    2022

  44. [44]

    R. Livni. The sample complexity of gradient descent in stochastic convex optimization.arXiv preprint arXiv:2404.04931, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    K. Lyu, Z. Li, and S. Arora. Understanding the generalization benefit of normalization layers: Sharpness reduction.Advances in Neural Information Processing Systems, 35:34689–34708, 2022

    2022

  46. [46]

    Ma and L

    C. Ma and L. Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems, 34:16805–16817, 2021

    2021

  47. [47]

    P. Mi, L. Shen, T. Ren, Y. Zhou, X. Sun, R. Ji, and D. Tao. Make sharpness-aware minimiza- tion stronger: A sparsified perturbation approach. InNeurIPS, 2022

    2022

  48. [48]

    M. S. Nacson, K. Ravichandran, N. Srebro, and D. Soudry. Implicit bias of the step size in linear diagonal neural networks. InInternational Conference on Machine Learning, pages 16270–16295. PMLR, 2022. 12

    2022

  49. [49]

    Neyshabur, R

    B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning.arXiv preprint arXiv:1412.6614, 2014

    Pith/arXiv arXiv 2014

  50. [50]

    Neyshabur, S

    B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring generalization in deep learning.Advances in neural information processing systems, 30, 2017

    2017

  51. [51]

    K. E. Nikolakakis, F. Haddadpour, A. Karbasi, and D. S. Kalogerias. Beyond lipschitz: Sharp generalization and excess risk bounds for full-batch gd.arXiv preprint arXiv:2204.12446, 2022

    Pith/arXiv arXiv 2022

  52. [52]

    M. D. Norton and J. O. Royset. Diametrical risk minimization: Theory and computations. Machine Learning, 112(8):2933–2951, 2023

    2023

  53. [53]

    Ramasinghe, L

    S. Ramasinghe, L. E. MacDonald, M. Farazi, H. Saratchandran, and S. Lucey. How much does initialization affect generalization? InInternational Conference on Machine Learning, pages 28637–28655. PMLR, 2023

    2023

  54. [54]

    Schliserman and T

    M. Schliserman and T. Koren. Stability vs implicit bias of gradient methods on separable data and beyond. In P.-L. Loh and M. Raginsky, editors,Proceedings of Thirty Fifth Conference on Learning Theory, volume 178 ofProceedings of Machine Learning Research, pages 3380–3394. PMLR, 02–05 Jul 2022

    2022

  55. [55]

    Schliserman, U

    M. Schliserman, U. Sherman, and T. Koren. The dimension strikes back with gradients: Generalization of gradient methods in stochastic convex optimization. InAlgorithmic Learning Theory, pages 1041–1107. PMLR, 2025

    2025

  56. [56]

    Shalev-Shwartz, O

    S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Learnability, stability and uniform convergence.The Journal of Machine Learning Research, 11:2635–2670, 2010

    2010

  57. [57]

    Si and C

    D. Si and C. Yun. Practical sharpness-aware minimization cannot converge all the way to optima. InNeurIPS, 2023

    2023

  58. [58]

    S. P. Singh, H. Mobahi, A. Agarwala, and Y. Dauphin. Avoiding spurious sharpness mini- mization broadens applicability of sam.arXiv preprint arXiv:2502.02407, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Srebro, K

    N. Srebro, K. Sridharan, and A. Tewari. Smoothness, low noise and fast rates.Advances in neural information processing systems, 23, 2010

    2010

  60. [60]

    H. Sun, L. Shen, Q. Zhong, L. Ding, S. Chen, J. Sun, J. Li, G. Sun, and D. Tao. Adasam: Boosting sharpness-aware minimization with adaptive learning rate and momentum for train- ing deep neural networks.Neural Networks, 169:506–519, 2024. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2023.10.044

    work page doi:10.1016/j.neunet.2023.10.044 2024

  61. [61]

    Tahmasebi, A

    B. Tahmasebi, A. Soleymani, D. Bahri, S. Jegelka, and P. Jaillet. A universal class of sharpness- aware minimization algorithms. InICML, 2024

    2024

  62. [62]

    C. Tan, J. Zhang, J. Liu, Y. Wang, and Y. Hao. Stabilizing sharpness-aware minimization through a simple renormalization strategy.Journal of Machine Learning Research, 26(68): 1–35, 2025

    2025

  63. [63]

    Vansover-Hager, T

    S. Vansover-Hager, T. Koren, and R. Livni. Rapid overfitting of multi-pass stochastic gradient descent in stochastic convex optimization.arXiv preprint arXiv:2505.08306, 2025

    Pith/arXiv arXiv 2025

  64. [64]

    Wei and T

    C. Wei and T. Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation.Advances in neural information processing systems, 32, 2019. 13

    2019

  65. [65]

    Wei and T

    C. Wei and T. Ma. Improved sample complexities for deep networks and robust classification via an all-layer margin.arXiv preprint arXiv:1910.04284, 2019

    Pith/arXiv arXiv 1910

  66. [66]

    C. Wei, Y. Chen, and T. Ma. Statistically meaningful approximation: a case study on ap- proximating turing machines with transformers.Advances in Neural Information Processing Systems, 35:12071–12083, 2022

    2022

  67. [67]

    K. Wen, T. Ma, and Z. Li. How does sharpness-aware minimization minimize sharpness?arXiv preprint arXiv:2211.05729, 2022

    Pith/arXiv arXiv 2022

  68. [68]

    K. Wen, Z. Li, and T. Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization.Advances in Neural Information Processing Systems, 36: 1024–1035, 2023

    2023

  69. [69]

    Wu, S.-T

    D. Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. NeurIPS, 2020

    2020

  70. [70]

    Wu and W

    L. Wu and W. J. Su. The implicit regularization of dynamical stability in stochastic gradient descent. InInternational Conference on Machine Learning, pages 37656–37684. PMLR, 2023

    2023

  71. [71]

    L. Wu, C. Ma, et al. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective.Advances in Neural Information Processing Systems, 31, 2018

    2018

  72. [72]

    W. Xie, T. Pethick, and V. Cevher. Sampa: Sharpness-aware minimization parallelized. In NeurIPS, 2024

    2024

  73. [73]

    Zhang, S

    C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In5th International Conference on Learning Representations, ICLR 2017, 2017

    2017

  74. [74]

    Y. Zhao, H. Zhang, and X. Hu. Randomized sharpness-aware training for boosting computa- tional efficiency in deep learning.arXiv preprint arXiv:2203.09962, 2022

    Pith/arXiv arXiv 2022

  75. [75]

    Zheng, R

    Y. Zheng, R. Zhang, and Y. Mao. Regularizing neural networks via adversarial model per- turbation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8156–8165, 2021

    2021

  76. [76]

    Zhuang, B

    J. Zhuang, B. Gong, L. Yuan, Y. Cui, H. Adam, N. C. Dvornek, S. Tatikonda, J. S. Duncan, and T. Liu. Surrogate gap minimization improves sharpness-aware training. InICLR, 2022. 14 A Stability and Generalization of SA-GD and SAM In this section we show upper bounds for the population loss achieved by SA-GD and SAM. For SA-GD, we prove the following theorem...

    2022