Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Avoiding spurious sharpness minimization broadens applicability of SAM

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that SAM's failure in language modeling comes from its descent being dominated by logit sharpness, and that Functional-SAM, which perturbs only the function Jacobian, improves validation loss over AdamW and SAM across…

desk verdict Genuinely useful decomposition of SAM's sharpness gradient, but the causal story about logit-path dominance is not actually tested in the paper—preconditioned SAM recovers most of the gain and tau is never measured on the proposed algorithms. read the letter →

arxiv 2502.02407 v1 pith:E67FCAEP submitted 2025-02-04 cs.LG cs.CLstat.ML

Sidak Pal Singh , Hossein Mobahi , Atish Agarwala , Yann Dauphin This is my paper · ORCID
classification cs.LGcs.CLstat.ML
keywords Sharpness-AwareMinimizationlanguagemodelinggeneralizationHessiandecompositionGauss-Newtonfunctionalpreconditioninglargemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sharpness-Aware Minimization (SAM), a curvature regularizer that helps vision models, consistently hurts language modeling. This paper argues the reason is that in language models SAM mostly reduces sharpness through the logits—making the network overconfident—rather than through the function the network computes, and calls that reduction spurious. To test the diagnosis, the authors derive a decomposition of SAM's sharpness gradient into a logit path and a functional path, measure that the logit path dominates in language while the two are balanced in vision, and then remove the logit path in a new update rule, Functional-SAM. Combined with a preconditioned perturbation that uses AdamW's second-moment statistics, the method improves validation loss over both AdamW and SAM in fixed-length and compute-optimal training at scales up to about 1.2 billion parameters. If correct, this makes sharpness regularization usable for large language models, not just vision.

What carries the argument

The central object is the decomposition of the sharpness gradient into two additive routes, $\delta_{\mathrm{logit}}$ and $\delta_{\mathrm{func}}$, obtained by applying the product rule to $\nabla_\theta\|\nabla_\theta\mathcal{L}\|$ and identifying the two terms with the GGN and functional Hessian components of the loss Hessian. The paper measures their normalized squared norms ($\tau_{\mathrm{logit}}$ and $\tau_{\mathrm{func}}$) to diagnose which route dominates, and builds Functional-SAM by keeping only the $\delta_{\mathrm{func}}$ route in the finite-difference perturbation update. Preconditioned SAM modifies the perturbation to $\rho M^{-1}\epsilon^*$, using Adam's second-moment estimate to reduce the logit route.

What would settle it

Train a language model with a deliberately logit-path variant (Angle-SAM with $\phi=\pi/2$, i.e., Logit-SAM) at the same compute; if that variant matches or beats AdamW's validation loss, the claim that logit sharpness reduction is spurious is refuted. A complementary check is to measure $\tau_{\mathrm{logit}}$ and $\tau_{\mathrm{func}}$ during actual SAM and Functional-SAM training rather than only along AdamW trajectories; if the functional path does not become dominant during Functional-SAM training, the proposed mechanism is not what drives its gains.

Watch

Extended reading notes

Core claim

Starting from the penalty form of SAM, $\min_\theta \mathcal{L}(\theta)+\rho\|\nabla_\theta\mathcal{L}(\theta)\|$, the paper uses the product rule and the Gauss-Newton decomposition of the Hessian, $\nabla_\theta \mathrm{SP}=\delta_{\mathrm{logit}}+\delta_{\mathrm{func}}=H_G\,\epsilon^*+H_F\,\epsilon^*=H_L\,\epsilon^*$, to split the sharpness-reduction signal into a logit path (through the generalized Gauss-Newton (GGN) term $H_G$, reducible by making softmax outputs more one-hot) and a functional path (through the functional Hessian $H_F$, tied to the geometry of the network function). Measuring the normalized contributions $\tau_{\mathrm{logit}}$, $\tau_{\mathrm{func}}$, and $\tau_{\mathrm{cross}}$ on Transformers, the paper finds that in vision the two paths are comparable, while in language modeling $\tau_{\mathrm{logit}}\approx 1$ throughout training, so SAM's updates mostly shrink logit sharpness rather than flattening the function. The proposed remedy, Functional-SAM, computes $g_{\mathrm{Func\text{-}SAM}}=-\nabla_\theta F(\theta+\rho\epsilon^*)\cdot \nabla_F \mathcal{L}(\theta)$, perturbing only the Jacobian of the function and discarding the logit contribution, and the authors argue and show that preconditioning the perturbation with the AdamW preconditioner $M^{-1}$ further favors the functional path. The paper's central claim is that this distinction between logit and functional sharpness explains SAM's failure in NLP and that emphasizing the functional path makes sharpness minimization work for language modeling.

Load-bearing premise

The load-bearing premise is that functional sharpness, not logit sharpness, is what drives generalization in language modeling—and that the sharpness split measured along AdamW trajectories is representative of SAM's own updates; if either gives way, the diagnosis of spurious minimization collapses.

Editorial extensions

If this is right

  • In language modeling, SAM's poor validation loss is attributable to the logit path: at equal compute the baseline SAM underperforms AdamW, while Functional-SAM outperforms both in the same fixed step budget.
  • Preconditioning the SAM perturbation with the optimizer's own second-moment statistics is enough to shift the balance toward the functional path and improve validation loss, so the inner perturbation and outer optimizer geometries should match.
  • Functional-SAM and its preconditioned version reach flatter solutions (lower maximum Hessian eigenvalue and trace) than AdamW, and those lower-curvature solutions are the ones with better validation loss.
  • The combination works across model sizes from 23.9M to 1.2B parameters and in both fixed-length and Chinchilla-style training, so curvature regularization can scale to large language models.
  • The decomposition into logit and functional paths suggests an interpolation (Angle-SAM) between Functional-SAM and Logit-SAM, which could be tuned per domain instead of always using the SAM default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: The causal direction may be weaker than the experiments suggest: the paper measures $\tau$ values along AdamW trajectories and hypothesizes that functional sharpness is the beneficial object, so the gains of Functional-SAM support the hypothesis without independently proving the mechanism.
  • Inference: The logit-path diagnosis may also explain why SAM helps in fine-tuning settings where parameters move less and logits are constrained, and would predict that Logit-SAM should hurt pre-training while possibly helping fine-tuning; this is testable.
  • Inference: The same decomposition could be applied to other sharpness variants and to other output-rich tasks such as speech or multi-label classification to decide whether those settings are logit-dominated before running expensive SAM experiments.
  • Inference: Because preconditioned Functional-SAM also avoids the numerical instabilities that plain SAM shows at billion-parameter scale, the method may be of practical value even if the sharpness-decomposition narrative were later revised.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies why Sharpness Aware Minimization (SAM) underperforms in language modeling. It decomposes the SAM sharpness gradient into a logit path (through the loss derivative with respect to the logits) and a functional path (through the function Jacobian), defines normalized contributions tau_logit, tau_func, and tau_cross, and measures them on Transformer language models and vision Transformers. The measurements show that tau_logit is close to 1 in language modeling, whereas the two paths are more balanced in vision. Motivated by the hypothesis that logit-path sharpness reduction is spurious, the authors propose Functional-SAM, which perturbs only the Jacobian, and preconditioned SAM, which applies the AdamW second-moment preconditioner to the SAM perturbation. Experiments on C4 with Nanodo models from 2M to 1.2B parameters report consistent validation-loss improvements over AdamW and SAM in fixed-length and Chinchilla-style training regimes, together with Hessian-trace and one-shot pruning analyses.

Significance. If the causal story holds, the paper offers a useful explanation of why naive SAM fails in NLP and a practical modification that makes sharpness regularization viable for language modeling at scale. The mathematical decomposition in Section 3 is clean, the provided JAX snippets match the described update rules, and the empirical comparisons include a wide range of model sizes, including a 1.2B-parameter run. The equal-step gains over AdamW are modest but consistent across scales and training regimes, and the paper goes beyond validation loss with Hessian and pruning evidence. The main weakness is that the central causal attribution is not directly tested, which is why the paper needs revision rather than acceptance in its current form.

major comments (4)
  1. [Section 3.3, Figures 2 and 4; Section 5.3, Tables 2 and 3] The causal claim that SAM fails because the logit path dominates sharpness reduction is not tested on the trajectories of the compared methods. The tau measurements are taken while training with AdamW, not while training with SAM, preconditioned SAM, or Functional-SAM. At the same time, preconditioned SAM, which does not remove the logit-path term and only changes the perturbation geometry, already captures most of the gain over AdamW (e.g., 3.55 versus 3.57 at 23.9M and 3.27 versus 3.28 at 117.9M in Table 2). The authors should either measure tau_logit and tau_func along the trajectories of the proposed methods, or provide a controlled ablation that isolates the logit path while holding the perturbation geometry fixed, for example the Angle-SAM variant with phi=pi/2 discussed in Appendix D. Without one of these, the improved results are also consistent with an Adam-geometry mismatch and do not establish the title's claim of avoiding spurious sharpness minimization.
  2. [Section 5.5, Table 4] The proposed algorithms are meant to reduce functional sharpness, but the reported Hessian metrics do not isolate the functional component. Table 4 reports lambda_max(H_L), tr(H_L), and tr(H_G), but not tr(H_F) or lambda_max(H_F). Moreover, the reported lambda_max(H_L) is larger for preconditioned Functional-SAM (6.20) than for SAM (2.71), so the maximum eigenvalue of the full Hessian does not support the claim that the proposed method finds a flatter solution under the metric that the paper itself emphasizes. The authors should report the functional-Hessian counterparts and state explicitly which flatness measure Functional-SAM is designed to reduce.
  3. [Section 5.2 and Section 5.3, Tables 1-3] The main empirical results are presented as single numbers without error bars or repeated seeds. The improvements over AdamW are 0.01 to 0.04 in validation loss, which is within the range where run-to-run variability matters. The statement that standard deviations are at the fourth decimal is made for the 2M prototype in Section 5.2 and cannot automatically be assumed for the larger models in Tables 2 and 3. The paper should report multiple seeds, error bars, or an equivalent statistical analysis for the final tables, or explicitly state the computational constraints that prevent this and hedge the claims accordingly.
  4. [Appendix B.1, Eqs. (13)-(17)] The random-matrix argument for preconditioning assumes free independence of A and B and an isotropic random vector v, whereas in the actual network setting the relevant vector is the training gradient and the matrices H_G and H_F are related to that gradient. The appendix already says the conclusion is a suspicion, but the main text in Section 4.2 presents preconditioned SAM as an indirect way to promote the functional path. Either soften this motivation or add a direct empirical check, such as measuring the norms of H_G M^{-1} g and H_F M^{-1} g during training, to show that the proposed preconditioner indeed upweights the functional component relative to the logit component.
minor comments (5)
  1. [Appendix C, Listing 1] The code snippet contains a typo: 'hax_aux' should be 'has_aux'; the text in Listing 1 also has 'dF_dtheta_fn' defined with a function that takes only theta, which is correct but should be aligned with the surrounding notation.
  2. [Abstract and Section 1] The PDF text contains visible spacing artifacts such as 'ofSA M' and 'SA M' that should be fixed in the final version.
  3. [Section 3.2, Eq. (9)] The definitions of tau_logit, tau_func, and tau_cross have a denominator of ||nabla_theta SP||^2, which is undefined if the sharpness gradient is exactly zero; a short remark on this degenerate case would be helpful.
  4. [Section 5.3, Tables 2 and 3] At the 1.2B scale the tables show NaN for SAM and preconditioned SAM; the warm-up mitigation numbers (3.07 and 3.08) are mentioned only in the text. These values should be reported in the main tables, or the tables should clearly state that the proposed method comparison at this scale is only preconditioned Functional-SAM.
  5. [Section 4.2] The relationship to FisherSAM and to Zhang et al. (2025) is described only briefly; a sentence stating the precise difference between the proposed preconditioning and those alternatives would help the reader judge novelty.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Functional-SAM and preconditioned SAM are explicit algebraic/geometric modifications of SAM's update, and the tau measurements motivate but do not define them.

full rationale

The paper's central diagnostic is the algebraic decomposition in Eq. (6)-(9): the sharpness-penalty gradient is split as delta_logit + delta_func = (H_G + H_F) * epsilon*, and the measured quantities tau_logit, tau_func, and tau_cross are ratios of norms of these algebraically defined terms. This decomposition follows from the chain rule, the product rule, and the standard Gauss-Newton split of the Hessian; it is re-derived in the paper rather than imported as a fitted result. Functional-SAM in Eq. (11) is obtained from Eq. (10) by deleting the first-order delta_logit term from the SAM update, so the algorithm is a direct algebraic modification of SAM and does not depend numerically on the measured tau values. Preconditioned SAM in Eq. (12) is similarly an explicit change of perturbation geometry, and the claim that inverse preconditioning promotes the functional path is presented as a random-matrix heuristic in Appendix B.1, explicitly ending in 'we suspect that generically preconditioning by H_G^{-1} will downweigh H_G epsilon* compared to H_F epsilon*'; it is not a fitted input and not a definitional identity. The self-citations to Singh et al. (2021, 2023) and Dauphin et al. (2024) supply terminology for the functional Hessian and a caveat about penalty-SAM robustness, but the load-bearing decomposition is present in the manuscript's own equations, so those citations are not the source of the central claim. The empirical improvements in Tables 1-3 over AdamW and SAM are measured validation losses, not quantities that the algorithms were constructed to match by fitting. The skeptic's concern that tau is never measured on Functional-SAM or preconditioned-SAM trajectories is a legitimate evidential limitation about whether the proposed logit-path mechanism is the true cause of the gains, but it is not circularity: the algorithms are well-defined without the tau measurements, and their improved performance is an independent empirical outcome. No step in the derivation reduces to its own inputs by construction, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a tunable perturbation radius rho, on standard SAM and Gauss-Newton mathematics, and on two domain assumptions: the desirability of functional sharpness over logit sharpness, and the representativeness of tau measurements made along the AdamW trajectory. The random-matrix preconditioning argument adds a further assumption of approximate freeness of H_G and H_F. No new physical entities are introduced.

free parameters (1)
  • perturbation radius rho = Tuned per method and model size (e.g., ~0.1 for SAM at 23.9M; larger for precond Func-SAM; exact values for larger…
    The reported gains depend on per-method, per-size tuning of rho. Section 5.4 notes SAM's best value is the smallest nonzero rho tested, while precond Functional-SAM favors larger rho. No explicit list of rho values for all runs is provided.
assumptions (6)
  • standard math First-order approximation of SAM's inner maximization: epsilon* = rho * grad L / ||grad L|| (Eq. 1).
    Adopted from Foret et al. (2020) as the starting point for the decomposition.
  • standard math Gauss-Newton decomposition of the loss Hessian into GGN (H_G) and functional Hessian (H_F) (Eq. 8).
    Used to identify delta_logit with H_G and delta_func with H_F; attributed to Schraudolph (2002) and Singh et al. (2021).
  • domain assumption The hypothesis that functional sharpness reduction is desirable and logit sharpness reduction via overconfidence is spurious.
    Stated in Section 3.3 as a 'simple but spurious way' to decrease logit sharpness; this hypothesis motivates Functional-SAM but is not proven.
  • domain assumption Measuring sharpness gradient composition along the AdamW trajectory is representative of SAM's behavior during SAM training.
    Figures 2 and 4 measure tau_logit and tau_func while training with AdamW, not with SAM; the diagnosis assumes these compositions characterize the geometry SAM acts on.
  • domain assumption AdamW's diagonal second-moment statistic M is a sufficient proxy for the GGN H_G for preconditioning purposes.
    Section 4.2 states 'AdamW already gives us a diagonal estimator in its own preconditioner M^-1 at no additional cost'; the theory in Appendix B.1 argues for H_G^-1 preconditioning.
  • domain assumption Random matrix model: H_G and H_F are freely independent (Appendix B.1).
    Used in the heuristic argument that inverse preconditioning upweighs the functional path; the paper notes the exact calculations 'don't hold' in neural settings and only 'suspects' generic behavior.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Avoiding spurious sharpness minimization broadens applicability of SAM." pith.science (2026). https://pith.science/paper/E67FCAEP

@misc{pith2026250202407,
  author       = {Pith},
  title        = {Pith review of: Avoiding spurious sharpness minimization broadens applicability of SAM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E67FCAEP}},
  note         = {Machine review of arXiv:2502.02407}
}
read the original abstract

Curvature regularization techniques like Sharpness Aware Minimization (SAM) have shown great promise in improving generalization on vision tasks. However, we find that SAM performs poorly in domains like natural language processing (NLP), often degrading performance -- even with twice the compute budget. We investigate the discrepancy across domains and find that in the NLP setting, SAM is dominated by regularization of the logit statistics -- instead of improving the geometry of the function itself. We use this observation to develop an alternative algorithm we call Functional-SAM, which regularizes curvature only through modification of the statistics of the overall function implemented by the neural network, and avoids spurious minimization through logit manipulation. Furthermore, we argue that preconditioning the SAM perturbation also prevents spurious minimization, and when combined with Functional-SAM, it gives further improvements. Our proposed algorithms show improved performance over AdamW and SAM baselines when trained for an equal number of steps, in both fixed-length and Chinchilla-style training settings, at various model scales (including billion-parameter scale). On the whole, our work highlights the importance of more precise characterizations of sharpness in broadening the applicability of curvature regularization to large language models (LLMs).

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Flat Minima and Generalization: Insights from Stochastic Convex Optimization

    cs.LG 2025-11 conditional novelty 7.0 of 10

    In smooth stochastic convex optimization, flat empirical minima can incur constant population risk while sharp minima generalize optimally, and sharpness-aware algorithms can converge to such bad flat minima.

Reference graph

Works this paper leans on

62 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Andriushchenko and N

    M. Andriushchenko and N. Flammarion. Towards understanding sharpness-aware minimization, 2022. URL https://arxiv.org/abs/2206.06232

    arXiv 2022

  2. [2]

    Bahri, H

    D. Bahri, H. Mobahi, and Y. Tay. Sharpness-aware minimization improves language model generalization, 2022. URL https://arxiv.org/abs/2110.08529

    arXiv 2022

  3. [3]

    Becker, F

    M. Becker, F. Altrock, and B. Risse. Momentum-sam: Sharpness aware minimization without computational overhead, 2024. URL https://arxiv.org/abs/2401.12033

    arXiv 2024

  4. [4]

    Bradbury, R

    J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. Vander P las, S. Wanderman- M ilne, and Q. Zhang. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

  5. [5]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amod...

    arXiv 2020

  6. [6]

    Chaudhari, A

    P. Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. Chayes, L. Sagun, and R. Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys, 2017. URL https://arxiv.org/abs/1611.01838

    arXiv 2017

  7. [7]

    Ciregan, U

    D. Ciregan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In 2012 IEEE conference on computer vision and pattern recognition, pages 3642--3649. IEEE, 2012

    work page 2012

  8. [8]

    Y. N. Dauphin, A. Agarwala, and H. Mobahi. Neglected hessian component explains mysteries in sharpness regularization, 2024

    work page 2024

Show all 62 references
  1. [9]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/2010.11929

    2021 arXiv

  2. [10]

    J. Du, D. Zhou, J. Feng, V. Y. Tan, and J. T. Zhou. Sharpness-aware training for free. arXiv preprint arXiv:2205.14083, 2022

    2022 arXiv

  3. [11]

    Everett, L

    K. Everett, L. Xiao, M. Wortsman, A. A. Alemi, R. Novak, P. J. Liu, I. Gur, J. Sohl-Dickstein, L. P. Kaelbling, J. Lee, et al. Scaling exponents across parameterizations and optimizers. arXiv preprint arXiv:2407.05872, 2024

    2024 arXiv

  4. [12]

    Foret, A

    P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020

    2010 arXiv

  5. [13]

    Gur-Ari, D

    G. Gur-Ari, D. A. Roberts, and E. Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018

    2018 arXiv

  6. [14]

    M. Haas, J. Xu, V. Cevher, and L. C. Vankadara. P^2 : Effective sharpness aware minimization requires layerwise perturbation scaling, 2024. URL https://arxiv.org/abs/2411.00075

    2024 arXiv

  7. [15]

    J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Z ee. F lax: A neural network library and ecosystem for JAX , 2024. URL http://github.com/google/flax

    2024

  8. [16]

    G. E. Hinton and D. Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5--13, 1993

    1993

  9. [17]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Flat minima. Neural Computation, 9: 0 1--42, 1997. URL https://api.semanticscholar.org/CorpusID:733161

    1997

  10. [18]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifr...

    2022 arXiv

  11. [19]

    Jacot, F

    A. Jacot, F. Gabriel, and C. Hongler. The asymptotic spectrum of the hessian of dnn throughout training, 2020. URL https://arxiv.org/abs/1910.02875

    2020 arXiv

  12. [20]

    Jiang, D

    K. Jiang, D. Malik, and Y. Li. How does adaptive optimization impact local neural network geometry? Advances in Neural Information Processing Systems, 36, 2024

    2024

  13. [21]

    Jiang, B

    Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. Fantastic generalization measures and where to find them, 2019. URL https://arxiv.org/abs/1912.02178

    2019 arXiv

  14. [22]

    Jiang, P

    Y. Jiang, P. Foret, S. Yak, D. M. Roy, H. Mobahi, G. K. Dziugaite, S. Bengio, S. Gunasekar, I. Guyon, and B. Neyshabur. Neurips 2020 competition: Predicting generalization in deep learning, 2020. URL https://arxiv.org/abs/2012.07976

    2020 arXiv

  15. [23]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020. URL https://arxiv.org/abs/2001.08361

    2020 arXiv

  16. [24]

    N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima, 2017. URL https://arxiv.org/abs/1609.04836

    2017 arXiv

  17. [25]

    M. Kim, D. Li, S. X. Hu, and T. Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pages 11148--11161. PMLR, 2022

    2022

  18. [26]

    D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    2017 arXiv

  19. [27]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

    2012

  20. [28]

    Krogh and J

    A. Krogh and J. Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991

    1991

  21. [29]

    J. Kwon, J. Kim, H. Park, and I. K. Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 5905--5914. PMLR, 2021

    2021

  22. [30]

    Li and G

    B. Li and G. Giannakis. Enhancing sharpness-aware optimization through variance suppression. Advances in Neural Information Processing Systems, 36, 2024

    2024

  23. [31]

    H. Liu, S. M. Xie, Z. Li, and T. Ma. Same pre-training loss, better downstream: Implicit bias matters for language models, 2023. URL https://openreview.net/forum?id=F5uYcwABMu

    2023

  24. [32]

    L. Liu, X. Liu, J. Gao, W. Chen, and J. Han. Understanding the difficulty of training transformers. arXiv preprint arXiv:2004.08249, 2020

    2004 arXiv

  25. [33]

    P. J. Liu, R. Novak, J. Lee, M. Wortsman, L. Xiao, K. Everett, A. A. Alemi, M. Kurzeja, P. Marcenac, I. Gur, S. Kornblith, K. Xu, G. Elsayed, I. Fischer, J. Pennington, B. Adlam, and J.-S. Dickstein. Nanodo: A minimal transformer decoder-only language model implementation in J...

    2024

  26. [34]

    Y. Liu, S. Mai, X. Chen, C.-J. Hsieh, and Y. You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12360--12370, 2022

    2022

  27. [35]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  28. [36]

    C. Na, S. V. Mehta, and E. Strubell. Train flat, then compress: Sharpness-aware minimization learns more compressible models. In Findings of the Association for Computational Linguistics: EMNLP 2022, page 4909–4936. Association for Computational Linguistics, 2022. doi:10.18653...

    2022 doi

  29. [37]

    L. Noci, S. Anagnostidis, L. Biggio, A. Orvieto, S. P. Singh, and A. Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35: 0 27198--27211, 2022

    2022

  30. [38]

    Ormaniec, F

    W. Ormaniec, F. Dangel, and S. P. Singh. What does it mean to be a transformer? insights from a theoretical hessian analysis, 2024. URL https://arxiv.org/abs/2410.10986

    2024 arXiv

  31. [39]

    Pan and Y

    Y. Pan and Y. Li. Toward understanding why adam converges faster than sgd for transformers. arXiv preprint arXiv:2306.00204, 2023

    2023 arXiv

  32. [40]

    V. Papyan. The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size, 2019. URL https://arxiv.org/abs/1811.07062

    2019 arXiv

  33. [41]

    Pennington and Y

    J. Pennington and Y. Bahri. Geometry of neural network loss surfaces via random matrix theory. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2798--2806. PM...

    2017

  34. [42]

    Pittorino, C

    F. Pittorino, C. Lucibello, C. Feinauer, E. M. Malatesta, G. Perugini, C. Baldassi, M. Negri, E. Demyanenko, and R. Zecchina. Entropic gradient descent algorithms and wide flat minima. CoRR, abs/2006.07897, 2020. URL https://arxiv.org/abs/2006.07897

    2006 arXiv

  35. [43]

    Radford, J

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  36. [44]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  37. [45]

    Ridnik, E

    T. Ridnik, E. Ben-Baruch, A. Noy, and L. Zelnik-Manor. Imagenet-21k pretraining for the masses, 2021. URL https://arxiv.org/abs/2104.10972

    2021 arXiv

  38. [46]

    Rissanen

    J. Rissanen. Modeling by shortest data description. Automatica, 14 0 (5): 0 465--471, 1978

    1978

  39. [47]

    Sagun, U

    L. Sagun, U. Evci, V. U. Guney, Y. Dauphin, and L. Bottou. Empirical analysis of the hessian of over-parametrized neural networks, 2018. URL https://arxiv.org/abs/1706.04454

    2018 arXiv

  40. [48]

    N. N. Schraudolph. Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14: 0 1723--1738, 2002

    2002

  41. [49]

    Sherborne, N

    T. Sherborne, N. Saphra, P. Dasigi, and H. Peng. Tram: Bridging trust regions and sharpness aware minimization, 2024. URL https://arxiv.org/abs/2310.03646

    2024 arXiv

  42. [50]

    S. P. Singh, G. Bachmann, and T. Hofmann. Analytic insights into structure and rank of neural network hessian maps. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id...

    2021

  43. [51]

    S. P. Singh, T. Hofmann, and B. Sch\" o lkopf. The hessian perspective into the nature of convolutional neural networks. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023

    2023

  44. [52]

    Srivastava, G

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 0 (1): 0 1929--1958, 2014

    1929

  45. [53]

    C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era, 2017. URL https://arxiv.org/abs/1707.02968

    2017 arXiv

  46. [54]

    Tahmasebi, A

    B. Tahmasebi, A. Soleymani, D. Bahri, S. Jegelka, and P. Jaillet. A universal class of sharpness-aware minimization algorithms, 2024. URL https://arxiv.org/abs/2406.03682

    2024

  47. [55]

    V. Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems, 4, 1991

    1991

  48. [56]

    A. Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  49. [57]

    N. Vyas, D. Morwani, R. Zhao, I. Shapira, D. Brandfonbrener, L. Janson, and S. Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    2024 arXiv

  50. [58]

    D. Wu, S. tao Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization, 2020. URL https://arxiv.org/abs/2004.05884

    2020 arXiv

  51. [59]

    W. Xie, T. Pethick, and V. Cevher. Sampa: Sharpness-aware minimization parallelized, 2024. URL https://arxiv.org/abs/2410.10683

    2024 arXiv

  52. [60]

    H. Zhang. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017

    2017 arXiv

  53. [61]

    Zhang, C

    Y. Zhang, C. Chen, T. Ding, Z. Li, R. Sun, and Z.-Q. Luo. Why transformers need adam: A hessian perspective, 2024. URL https://arxiv.org/abs/2402.16788

    2024 arXiv

  54. [62]

    Zhang, B

    Y. Zhang, B. Li, and G. B. Giannakis. Preconditioned sharpness-aware minimization: Unifying analysis and a novel learning algorithm. arXiv preprint arXiv:2501.06603, 2025

    2025 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.