Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SAM's implicit bias includes a sharpness-reduction term once perturbation aligns with the top Hessian eigenvector, and Eigen-SAM enforces that alignment.

desk verdict A useful third-order SDE for SAM and a simple, empirically supported Eigen-SAM tweak, but the headline eigenvalue-descent corollary is asserted rather than proved; fixable, not fatal. read the letter →

arxiv 2501.12666 v1 pith:B2OPJMLC submitted 2025-01-22 cs.LG cs.CV

Haocheng Luo , Tuan Truong , Tung Pham , Mehrtash Harandi , Dinh Phung , Trung Le This is my paper · ORCID
classification cs.LGcs.CV MSC 68T07
keywords Sharpness-AwareMinimizationthird-orderSDEtopHessianeigenvalueimplicitregularizationperturbation-eigenvectoralignmentEigen-SAMgeneralizationboundflatminima
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to explain what Sharpness-Aware Minimization (SAM) actually does to the loss landscape, beyond the standard story that it penalizes gradient norm. It derives a third-order stochastic differential equation for SAM's discrete updates and shows that, when the perturbation direction lines up with the Hessian's leading eigenvector, SAM's drift contains an explicit gradient of the top eigenvalue, so the method implicitly minimizes sharpness. The paper then reports that this alignment is poor in practice and proposes Eigen-SAM, which periodically estimates the leading eigenvector and adds its gradient-orthogonal component to the perturbation to force the alignment. If the derivation holds, Eigen-SAM is a direct way to turn SAM's implicit sharpness bias into an explicit one, and the experiments support that with consistent accuracy gains over SAM on several image benchmarks.

What carries the argument

The load-bearing object is the third-order stochastic differential equation (Theorem 4.1): an order-1 weak approximation of discrete SAM with drift $-\nabla\tilde f^{\mathrm{SAM}}$ and curvature-dependent diffusion $\Sigma^{\mathrm{SAM}}$. The measure of sharpness is the largest Hessian eigenvalue $\lambda_1(\nabla^2 f(x))$, and the bridge from the cubic term to eigenvalue descent is the eigenvalue-differentiation identity $\nabla^2 f_\gamma(x)(v_1,v_1) = \nabla \lambda_1(\nabla^2 f_\gamma(x))$ (Magnus, 1985), valid under an eigenvalue-gap assumption. The alignment measure $\mathrm{Align}(\epsilon, v_1) := 1 - \min_{s\in\{\pm1\}}\|\epsilon/\|\epsilon\| - s v_1\|$ quantifies when the perturbation points along the top eigenvector; Corollary 4.1.1 turns that alignment into the explicit eigenvalue-gradient drift.

What would settle it

Train a network with SAM while logging $\mathrm{Align}(\epsilon^{\mathrm{SAM}}_\gamma, v_1)$ and the projection of the SDE's cubic drift term onto $\nabla\lambda_1$. The central claim predicts that whenever alignment is $1-O(\rho)$, that projection equals $\frac{\rho^2}{2}\|\nabla\lambda_1\|^2$ up to $O(\rho^3)$; observing a mismatch growing like $\rho^2$ at such alignments would falsify Corollary 4.1.1. A complementary check: compare Eigen-SAM to SAM with the top-eigenvalue component removed from the perturbation; if accuracy and final $\lambda_1$ do not separate, the explicit regularization is not carrying the gain.

Watch

Extended reading notes

Core claim

The central discovery is that SAM's training dynamics, modeled to third order, are driven by the effective objective $\tilde f^{\mathrm{SAM}}(x) = f(x) + \rho\mathbb{E}\|\nabla f_\gamma(x)\| + \frac{\rho^2}{2}\mathbb{E}\frac{\nabla f_\gamma^\top \nabla^2 f_\gamma(x)\nabla f_\gamma}{\|\nabla f_\gamma\|^2}$, with the cubic term acting as a gradient of the top Hessian eigenvalue whenever the perturbation $\nabla f_\gamma/\|\nabla f_\gamma\|$ is aligned with the leading eigenvector $v_1(\nabla^2 f_\gamma(x))$. Corollary 4.1.1 states that if alignment is at least $1-O(\rho)$, the SDE drift becomes $\nabla f + \rho\nabla\mathbb{E}\|\nabla f_\gamma\| + \frac{\rho^2}{2}\nabla\mathbb{E}\lambda_1(\nabla^2 f_\gamma)$, so SAM implicitly regularizes both gradient norm and sharpness. On the practical side, the paper shows the alignment is typically insufficient in real networks and introduces Eigen-SAM, which adds $\alpha\,\mathrm{sign}(\langle\nabla f_\gamma,\hat v\rangle)\hat v_\perp$ to the SAM perturbation, where $\hat v$ is a power-method estimate of the top eigenvector.

Load-bearing premise

The derivation treats the SAM perturbation direction as frozen, independent of $x$, when forming the SDE drift; if the true dependence of that direction on $x$ contributes at the same order as the cubic term, the drift is not a true gradient and the eigenvalue-descent reading collapses.

Editorial extensions

If this is right

  • SAM can be understood as jointly descending loss, gradient norm, and top Hessian eigenvalue, with the three contributions at scales $1$, $\rho$, and $\rho^2/2$, respectively, provided alignment holds.
  • Measured alignment in practice is poor, so standard SAM does not efficiently minimize $\lambda_1$; this explains why second-order SDE theories that only see gradient-norm regularization miss part of SAM's behavior.
  • Eigen-SAM's perturbation, formed from the top-eigenvector estimate's gradient-orthogonal component, provably improves alignment for a wide range of $\alpha$ (Proposition D.1) and reaches the same order of convergence as SAM on smooth non-convex objectives (Theorem D.2).
  • With eigenvector re-estimation every $p=100$ steps and $q=5$ power iterations, the added cost is roughly $q/p$ times a gradient, and increasing $p$ to 1000 keeps most of the test-accuracy gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if the third-order SDE is the right model, SAM's benefit over SGD should be reproducible by any optimizer that penalizes $\lambda_1$ directly; Eigen-SAM is one such optimizer, and the paper's Hessian spectrum plots predict that flatter spectra, not just lower loss, carry the generalization gain.
  • Beyond the paper, the frozen-perturbation step that makes the cubic term a gradient suggests a boundary: for large $\rho$, the true $x$-dependence of $\nabla f_\gamma/\|\nabla f_\gamma\|$ should show up as a drift that is not a gradient, so sharpness descent should degrade; measuring the projection of the residual onto $\nabla\lambda_1$ as $\rho$ grows would locate that boundary.
  • Beyond the paper, the eigenvalue-gap assumption limits the corollary to regions where $v_1$ is well-defined; near eigenvalue crossings the alignment measure loses meaning, and Eigen-SAM's eigenvector estimate may flip, so performance gains may depend on the training trajectory staying away from degeneracy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper analyzes Sharpness-Aware Minimization (SAM) through a third-order stochastic differential equation (SDE). The main theoretical claim is that, when the SAM perturbation vector aligns with the top Hessian eigenvector, the SDE drift explicitly contains the gradient of the top eigenvalue, implying that SAM implicitly regularizes sharpness. The paper further proposes Eigen-SAM, which adds the gradient-orthogonal component of an estimated top eigenvector to the perturbation, and reports experiments on CIFAR, Fashion-MNIST, SVHN, and fine-tuning tasks where Eigen-SAM consistently outperforms SAM. The paper also proves a PAC-Bayes generalization bound involving the top eigenvalue and proves a convergence rate for Eigen-SAM.

Significance. If the theoretical claims are correct, the paper provides a more refined continuous-time model of SAM than existing second-order SDEs and identifies perturbation-eigenvector alignment as the key mechanism, leading to a novel and simple algorithm. The empirical contribution is solid: experiments are repeated with confidence intervals, sensitivity analysis is provided, Hessian spectra are shown, and code is available, which makes the observed gains of Eigen-SAM over SAM credible. The theoretical part, however, has load-bearing gaps: the proof of Corollary 4.1.1 does not establish the required vector drift bound, and the SDE drift is presented as the gradient of a scalar while the derivation uses a frozen-perturbation gradient that differs from the full gradient. These issues are fixable but must be addressed for the paper's central interpretation to be rigorous.

major comments (3)
  1. [Theorem B.4 (Eq. 29) and Lemma B.3 (Eq. 16)] The SDE drift is defined as -∇f̃_SAM, where f̃_SAM is a scalar function, but Lemma B.3 computes the discrete one-step mean using the gradient of the perturbed loss with the perturbation direction held fixed, yielding the frozen-gradient vector ∂e_i f̃_SAM. These two vector fields are not equal in general; the full gradient of f̃_SAM contains additional terms from differentiating ∇fγ/‖∇fγ‖ with respect to x. The proof of Theorem B.4 compares the SDE one-step mean to the discrete one-step mean without accounting for this difference. Under the alignment assumption the difference is O(ρ³), which is O(η) when ρ=O(η^{1/3}) and therefore still compatible with an order-1 weak approximation, but the statement of the theorem as an exact SDE with a scalar potential is not justified. Please either define the SDE drift explicitly as the frozen-gradient vector field (which may not be a gradient) or state and prove the O(ρ³) error in the theorem statement.
  2. [Corollary 4.1.1, Appendix B (Eqs. 40-43)] The proof of the corollary bounds scalar differences of Rayleigh quotients, such as ‖E[uᵀ∇²fγu] - E[v1ᵀ∇²fγv1]‖ = O(ρ³) in Eq. (40). However, the corollary claims a replacement in the drift vector field: the term ρ²/2 E[∇³fγ(u,u)] must be replaced by ρ²/2 ∇Eλ1(∇²fγ). This requires a vector bound on ‖E[∇³fγ(u,u)] - E∇λ1(∇²fγ)‖, which is not shown. While Eqs. (42)-(43) provide a vector bound for the first-order term, no such vector bound is given for the third-order term. The missing bound can be supplied using the eigengap (Assumption B.2) and Lipschitz continuity of the third-order tensor, but as written the eigenvalue-descent interpretation in Eqs. (5)-(6) is asserted rather than derived. The corollary should also explicitly state that the replacement holds up to O(ρ³) (or O(ρ⁴) for the stronger alignment).
  3. [Section 4.1, Eq. (4)] The equality ρ²/2 E[∇³fγ(∇fγ,∇fγ)/‖∇fγ‖²] = ρ²/2 ∇E[∇fγᵀ∇²fγ∇fγ/‖∇fγ‖²] is stated to hold because ∇fγ is treated as a perturbation vector independent of Xt. This equality is correct only if the gradient on the right-hand side does not differentiate through the normalized perturbation. In standard calculus, the right-hand side would contain additional terms involving ∂(∇fγ/‖∇fγ‖)/∂x. The paper should make this frozen-gradient convention explicit in the main text, in the definition of f̃_SAM in Theorem B.4, and in the proof of Lemma B.3, since the subsequent interpretation of the drift as a gradient of a scalar potential relies on this convention.
minor comments (5)
  1. [Theorem 3.1 (main text)] In the statement of Theorem 3.1, the term L/(2√n)√(d log(1+‖x‖²/(dσ²))) + O(1) + 2 log(1/δ) + 4 log(n+d) is typeset so that the square root appears to cover only d log(...), whereas the proof in Appendix C places the O(1)+... terms inside the square root. Please make the formula consistent.
  2. [Theorem D.2 (Appendix D)] In the proof of Theorem D.2, the mini-batch variance bound σ² is used to bound E‖∇f(xt+ρϵ)-∇f(xt)‖², but this is not the variance of the mini-batch gradient. The correct bound follows from β-smoothness: E‖∇f(xt+ρϵ)-∇f(xt)‖² ≤ β²ρ²E‖ϵ‖² = β²ρ²(1+α²). The final bound should therefore contain β²ρ²(1+α²) rather than β²(ρ²+α²).
  3. [Table 1 and Appendix E] Table 1 states that training was conducted for 200 epochs, but Appendix E says that SGD was allowed to train for twice the number of epochs. Please clarify the exact training protocol for each method.
  4. [Section 6.1] The numerical simulation of the third-order SDE is not fully described. Please specify the discretization scheme (e.g., Euler-Maruyama), the number of trajectories used, and how the diffusion term was simulated so that the results are reproducible.
  5. [Tables 1 and 2] The claim of 'state-of-the-art' performance is too strong given that comparisons are made only against SGD and SAM, not against other SAM variants such as GSAM or ASAM. I suggest rephrasing to 'outperforms SAM and SGD on the tested benchmarks.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SDE is derived by Taylor expansion and moment matching, and the alignment corollary is a conditional mathematical consequence, not a fitted or self-cited result.

full rationale

The paper's derivation chain is self-contained. SAM's update (Eq. 1) is expanded in ρ with the perturbation direction held fixed, which is the stated SAM implementation convention; this yields the drift terms in Lemma B.3 (Eqs. 19–22). The third-order SDE (Theorem 4.1/B.4) is then obtained by matching the first two conditional moments of the discrete process to a continuous SDE in the standard Li et al. (2017)/Mil'shtein framework. Corollary 4.1.1 is conditional: if the alignment between the perturbation and the top Hessian eigenvector is 1−O(ρ), the cubic Taylor term becomes ρ²/2 ∇E[λ₁], using the eigenvalue-differentiation identity of Magnus (1985). This is a mathematical implication, not a quantity fitted to make the conclusion come out. Eigen-SAM's design goal is explicitly to improve that alignment, and Proposition D.1 proves the alignment improvement by a trigonometric inequality; its reported accuracy gains are external benchmark outcomes, not parameters fitted to the theory. There are no load-bearing self-citations by the present authors. The main caveat is a proof gap rather than circularity: the proof of Corollary 4.1.1 bounds the scalar Rayleigh-quotient error (Eqs. 40–43) instead of the vector error of the drift, so the eigenvalue-descent reading is asserted modulo a missing but plausibly fixable Lipschitz bound. That is a correctness issue, not an equivalence-to-input by construction. The frozen-perturbation approximation is also explicit and not concealed, so it does not constitute circular reasoning.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central theory rests on standard smoothness assumptions plus a paper-specific frozen-perturbation approximation and a positive eigenvalue gap. The algorithm introduces three hyperparameters (alpha, p, q) but no new physical or mathematical entities. No new particles, forces, dimensions, or conserved quantities are postulated.

free parameters (3)
  • alpha (Eigen-SAM perturbation strength) = 0.2
    Controls the weight of the gradient-orthogonal eigenvector component in the perturbation. Tuned over {0.05, 0.1, 0.2} on 10% of the training set; 0.2 was used for all reported experiments.
  • p (eigenvector re-estimation interval) = 100
    The top eigenvector is estimated once every p mini-batch steps. p=100 is the default; Appendix F shows p=1000 retains most of the accuracy gain.
  • q (power method iterations) = 5
    Number of Hessian-vector products per eigenvector estimate. Figure 5 suggests a few iterations suffice.
assumptions (6)
  • ad hoc to paper The perturbation direction grad f_gamma / ||grad f_gamma|| is treated as independent of x when forming the SDE drift (frozen perturbation).
    Section 4.1 and Lemma B.3 use this to convert the third-order Taylor term into grad E[grad f_gamma^T Hessian f_gamma grad f_gamma / ||grad f_gamma||^2]. This is the key move enabling the eigenvalue-descent interpretation, but it is not a standard gradient of the displayed scalar function.
  • domain assumption Loss and derivatives up to order 7 exist, are bounded, Lipschitz, and of polynomial growth, and ||grad f_gamma|| is treated as smooth despite non-differentiability at zero gradient.
    Assumption B.1 and Lemma B.3 require these regularity conditions for the SDE approximation. The norm term is not differentiable at grad f_gamma = 0, which is not addressed.
  • domain assumption Positive eigenvalue gap: lambda_1 > lambda_2 along the trajectory (Assumption B.2).
    Needed to differentiate the top eigenvalue and to approximate the third-order term by grad E lambda_1. Common in sharpness literature but not verified for the networks tested.
  • domain assumption The true loss is no larger than its Gaussian-smoothed version: f_D(x) <= E_{epsilon~N(0,sigma^2 I)} f_D(x+epsilon).
    Assumed in Theorem 3.1, inherited from Foret et al. (2021). Strong at arbitrary points and not empirically checked.
  • domain assumption Loss is bounded by L and third derivatives are bounded by C in Theorem 3.1.
    Used for the Hoeffding bound and Taylor expansion in the PAC-Bayes proof. Reasonable for bounded losses but restrictive for unbounded training losses.
  • domain assumption The power-method estimate of the top eigenvector is accurate enough with q=5 iterations every p=100 steps.
    Eigen-SAM's alignment improvement relies on this estimate. Figure 5 shows rapid convergence in the tested settings, but there is no formal guarantee under the actual training distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization." pith.science (2026). https://pith.science/paper/B2OPJMLC

@misc{pith2026250112666,
  author       = {Pith},
  title        = {Pith review of: Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2OPJMLC}},
  note         = {Machine review of arXiv:2501.12666}
}
read the original abstract

Sharpness-Aware Minimization (SAM) has attracted significant attention for its effectiveness in improving generalization across various tasks. However, its underlying principles remain poorly understood. In this work, we analyze SAM's training dynamics using the maximum eigenvalue of the Hessian as a measure of sharpness, and propose a third-order stochastic differential equation (SDE), which reveals that the dynamics are driven by a complex mixture of second- and third-order terms. We show that alignment between the perturbation vector and the top eigenvector is crucial for SAM's effectiveness in regularizing sharpness, but find that this alignment is often inadequate in practice, limiting SAM's efficiency. Building on these insights, we introduce Eigen-SAM, an algorithm that explicitly aims to regularize the top Hessian eigenvalue by aligning the perturbation vector with the leading eigenvector. We validate the effectiveness of our theory and the practical advantages of our proposed approach through comprehensive experiments. Code is available at https://github.com/RitianLuo/EigenSAM.

Figures

Figures reproduced from arXiv: 2501.12666 by the authors.

Figure 1
Figure 1. Alignment and top eigenvalue for a 6-layer CNN model trained on CIFAR-10. The left panel [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Training dynamics of discrete SAM, second-order SDE, and third-order SDE during [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. We observed that the test accuracy peaks at [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of training loss and test loss metrics across algorithms. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: The effect of the number of Hessian-vector product steps in Algorithm 1 (power iteration) [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 26 canonical work pages

  1. [1]

    Ahn, K., Jadbabaie, A., and Sra, S. (2023). How to escape sharp minima with random perturbations. arXiv preprint arXiv:2305.15659

    arXiv 2023

  2. [2]

    Alquier, P., Ridgway, J., and Chopin, N. (2016). On the properties of variational approximations of gibbs posteriors. Journal of Machine Learning Research , 17(236)

    work page 2016

  3. [3]

    and Flammarion, N

    Andriushchenko, M. and Flammarion, N. (2022). Towards understanding sharpness-aware minimization. In International Conference on Machine Learning , pages 639--668. PMLR

    2022

  4. [4]

    Arora, S., Li, Z., and Panigrahi, A. (2022). Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning , pages 948--1024. PMLR

    2022

  5. [5]

    Barrett, D. G. and Dherin, B. (2020). Implicit gradient regularization. arXiv preprint arXiv:2009.11162

    arXiv 2020

  6. [6]

    L., Long, P

    Bartlett, P. L., Long, P. M., and Bousquet, O. (2023). The dynamics of sharpness-aware minimization: Bouncing across ravines and drifting towards wide minima. Journal of Machine Learning Research , 24(316):1--36

    work page 2023

  7. [7]

    Blanc, G., Gupta, N., Valiant, G., and Valiant, P. (2020). Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In Conference on learning theory , pages 483--513. PMLR

    work page 2020

  8. [8]

    S., Pinto, J., and Prabhu, T

    Borkar, V. S., Pinto, J., and Prabhu, T. (2009). A new learning algorithm for optimal stopping. Discrete Event Dynamic Systems , 19:91--113

    work page 2009

Show all 46 references
  1. [9]

    M., Biggio, L., Orvieto, A., Proske, F

    Compagnoni, E. M., Biggio, L., Orvieto, A., Proske, F. N., Kersting, H., and Lucchi, A. (2023). An sde for modeling sam: Theory and insights. In International Conference on Machine Learning , pages 25209--25253. PMLR

    2023

  2. [10]

    Dagréou, M., Ablin, P., Vaiter, S., and Moreau, T. (2024). How to compute hessian-vector products? In ICLR Blogposts 2024 . https://iclr-blogposts.github.io/2024/blog/bench-hvp/

    2024

  3. [11]

    Damian, A., Nichani, E., and Lee, J. D. (2022). Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594

    2022 arXiv

  4. [12]

    Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141--142

    2012

  5. [13]

    Deng, Z., Sun, R., Xue, M., Wen, S., Camtepe, S., Nepal, S., and Xiang, Y. (2024). Leakage-resilient and carbon-neutral aggregation featuring the federated ai-enabled critical infrastructure. arXiv preprint arXiv:2405.15258

    2024 arXiv

  6. [14]

    Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. (2017). Sharp minima can generalize for deep nets. In International Conference on Machine Learning , pages 1019--1028. PMLR

    2017

  7. [15]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    2020 arXiv

  8. [16]

    and Le Cun, Y

    Drucker, H. and Le Cun, Y. (1991). Double backpropagation increasing generalization performance. In IJCNN-91-Seattle International Joint Conference on Neural Networks , volume 2, pages 145--150. IEEE

    1991

  9. [17]

    Duchi, J. C. and Ruan, F. (2018). Stochastic methods for composite and weakly convex optimization problems. SIAM Journal on Optimization , 28(4):3229--3259

    2018

  10. [18]

    Dziugaite, G. K. and Roy, D. M. (2017). Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008

    2017 arXiv

  11. [19]

    Folland, G. B. (2005). Higher-order derivatives and taylor’s formula in several variables. Preprint , pages 1--4

    2005

  12. [20]

    Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. (2021). Sharpness-aware minimization for efficiently improving generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net

    2021

  13. [21]

    Gatmiry, K., Li, Z., Ma, T., Reddi, S., Jegelka, S., and Chuang, C.-Y. (2024a). What is the inductive bias of flatness regularization? a study of deep matrix factorization models. Advances in Neural Information Processing Systems , 36

    2024

  14. [22]

    J., and Jegelka, S

    Gatmiry, K., Li, Z., Ruiz, L., Reddi, S. J., and Jegelka, S. (2024b). Simplicity bias of SGD via sharpness minimization

    2024

  15. [23]

    Harold, J., Kushner, G., and Yin, G. (1997). Stochastic approximation and recursive algorithm and applications. Application of Mathematics , 35(10)

    1997

  16. [24]

    He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770--778

    2016

  17. [25]

    and Schmidhuber, J

    Hochreiter, S. and Schmidhuber, J. (1997). Flat minima. Neural computation , 9(1):1--42

    1997

  18. [26]

    B., Wang, H., Xiong, C., Socher, R., Cho, K., and Geras, K

    Jastrzebski, S., Arpit, D., Astrand, O., Kerg, G. B., Wang, H., Xiong, C., Socher, R., Cho, K., and Geras, K. J. (2021). Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In International Conference on Machine Learning , pages 4772--4784. PMLR

    2021

  19. [27]

    Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. (2019). Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178

    2019 arXiv

  20. [28]

    Kaddour, J., Liu, L., Silva, R., and Kusner, M. J. (2022). When do flat minima optimizers work? Advances in Neural Information Processing Systems , 35:16577--16595

    2022

  21. [29]

    S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P

    Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. (2016). On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836

    2016 arXiv

  22. [30]

    Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images

    2009

  23. [31]

    Li, Q., Tai, C., and Weinan, E. (2017). Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning , pages 2101--2110. PMLR

    2017

  24. [32]

    Li, Z., Malladi, S., and Arora, S. (2021). On the validity of modeling sgd with stochastic differential equations (sdes). Advances in Neural Information Processing Systems , 34:12712--12725

    2021

  25. [33]

    Liu, H., Li, Z., Hall, D., Liang, P., and Ma, T. (2023a). Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342

    2023 arXiv

  26. [34]

    M., Li, Z., and Ma, T

    Liu, H., Xie, S. M., Li, Z., and Ma, T. (2023b). Same pre-training loss, better downstream: Implicit bias matters for language models. In International Conference on Machine Learning , pages 22188--22214. PMLR

    2023

  27. [35]

    Lyu, K., Li, Z., and Arora, S. (2022). Understanding the generalization benefit of normalization layers: Sharpness reduction. Advances in Neural Information Processing Systems , 35:34689--34708

    2022

  28. [36]

    Magnus, J. R. (1985). On differentiating eigenvalues and eigenvectors. Econometric theory , 1(2):179--191

    1985

  29. [37]

    Mil’shtein, G. (1986). Weak approximation of solutions of systems of stochastic differential equations. Theory of Probability & Its Applications , 30(4):750--766

    1986

  30. [38]

    Y., et al

    Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. (2011). Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , volume 2011, page 7. Granada, Spain

    2011

  31. [39]

    and Yun, C

    Si, D. and Yun, C. (2024). Practical sharpness-aware minimization cannot converge all the way to optima. Advances in Neural Information Processing Systems , 36

    2024

  32. [40]

    Song, M., Ahn, K., and Yun, C. (2024). Does sgd really happen in tiny subspaces?

    2024

  33. [41]

    Tahmasebi, B., Soleymani, A., Bahri, D., Jegelka, S., and Jaillet, P. (2024). A universal class of sharpness-aware minimization algorithms. arXiv preprint arXiv:2406.03682

    2024

  34. [42]

    Wen, K., Ma, T., and Li, Z. (2022). How does sharpness-aware minimization minimize sharpness? CoRR , abs/2211.05729

    2022 arXiv

  35. [43]

    Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747

    2017 arXiv

  36. [44]

    Xie, Z., Sato, I., and Sugiyama, M. (2020). A diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. arXiv preprint arXiv:2002.03495

    2020 arXiv

  37. [45]

    and Komodakis, N

    Zagoruyko, S. and Komodakis, N. (2016). Wide residual networks. arXiv preprint arXiv:1605.07146

    2016 arXiv

  38. [46]

    Zhuang, J., Gong, B., Yuan, L., Cui, Y., Adam, H., Dvornek, N., Tatikonda, S., Duncan, J., and Liu, T. (2022). Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065

    2022 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.