Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations

Dongwook Shin; Junho So

REVIEW 3 major objections 4 minor 40 references

Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations

T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mini-batch noise lowers top-k sharpness by fluctuating in the dominant Hessian subspace, and a deterministic correction term reproduces this effect.

desk verdict A clean local derivation of a sharpness-correction term, with convincing controlled experiments; the core global surrogate step is asserted and not proven, but the work is worth refereeing. read the letter →

arxiv 2607.23012 v1 pith:AI4GPNVH submitted 2026-07-25 cs.LG

Junho So , Dongwook Shin This is my paper · ORCID

classification cs.LG MSC 68T07

keywords mini-batchnoisesharpnessdominantsubspaceHessianSGDimplicitregularizationcorrectionflatminima

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the dominant subspace, the span of the top-k Hessian eigenvectors, is not where SGD reduces loss but where mini-batch noise acts to lower sharpness. Concretely, the authors show that mean-zero fluctuations confined to this subspace reduce top-k sharpness, and that averaging the gradient over such fluctuations leaves a deterministic correction term, one half the gradient of Tr(H C) with C the displacement covariance. They derive the displacement covariance created by mini-batch noise through a local Lyapunov recursion, and show that adding the resulting correction to full-batch GD brings its sharpness trajectory close to that of mini-batch SGD on MLP, CNN, and Transformer. The value is in identifying a mechanism, dominant-subspace fluctuations, rather than merely a correlation, and a term that could be used to design flat-minima optimizers.

What carries the argument

The central mechanism is the sharpness correction term that survives gradient averaging over dominant-subspace fluctuations. For a mean-zero displacement delta supported on the dominant subspace with covariance C, the second-order term (1/2) Grad(delta^T H delta) averages to (1/2) Grad Tr(H C), acting to lower covariance-weighted curvature. Combined with the mini-batch-induced displacement covariance C_mb, obtained as the stationary solution of the Lyapunov recursion C_{s+1} = A_c C_s A_c^T + eta^2 Sigma_dom, this becomes a deterministic correction that mimics SGD's sharpness-reducing effect.

What would settle it

Inject mean-zero noise of controlled covariance C into full-batch GD along the dominant subspace at several noise scales and measure the change in top-k sharpness; if the sharpness drop deviates from (1/2) Grad Tr(H C) beyond third-order terms, the correction term is wrong. Alternatively, along a real SGD run, estimate C_mb and check whether the effective drift matches the correction term by tracking the dominant-subspace covariance evolution.

Watch

Extended reading notes

Core claim

The central claim is that mini-batch SGD's tendency to find lower top-k sharpness is driven by fluctuations within the dominant subspace (the span of the top-k Hessian eigenvectors), not by the bulk directions that carry most loss reduction. Averaging the gradient over mean-zero displacements confined to this subspace yields a sharpness correction term (1/2) Grad Tr(H(theta) C), where C is the displacement covariance. The paper derives C for mini-batch noise as the stationary covariance of a local recursion, C_mb = eta^2 sum_{ell=0}^{infinity} A_c^ell Sigma_dom (A_c^T)^ell, with A_c = P_c (I - eta H_c) P_c. Adding the resulting correction to full-batch GD makes its sharpness evolution approx

Load-bearing premise

The local recursion freezes the reference point and the dominant subspace, and assumes that the stationary covariance of that frozen recursion equals the covariance along the actual SGD trajectory; if the dominant subspace rotates too fast or bulk-direction noise is not negligible, the derived correction term may not describe SGD's sharpness dynamics.

Editorial extensions

If this is right

If the correction term is the right explanation, full-batch GD with the correction provides a cheap proxy for SGD's sharpness dynamics, avoiding the cost of sampling noise.
Batch-size scaling of sharpness follows from the covariance recursion: larger batches shrink Sigma_dom and move SGD's sharpness toward GD, matching the paper's batch-size sweep.
The mechanism implies that any noise source whose covariance aligns with high-curvature directions will lower top-k sharpness, not just mini-batch sampling noise.
Because the correction is the gradient of Tr(H C), it can be combined with any base optimizer to steer training toward flatter regions of the loss landscape.
The dominant/bulk decomposition suggests that deep-network optimization can be separated into loss-reduction directions and sharpness-shaping directions, potentially informing projected-update training methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The derivation predicts a quantitative relation: injecting mean-zero dominant-aligned noise of covariance C into full-batch GD should lower sharpness by (1/2) Grad Tr(H C); a clean experiment could test this amplitude directly.
If the correction term is accurate, it suggests a principled way to compensate for large-batch training by adding a deterministic dominant-subspace correction, rather than relying on inherent noise.
The frozen-subspace approximation implies the theory should break down when the top-k eigenspace rotates rapidly, such as near saddle points; probing that regime could delineate the validity boundary.
The stability condition lambda_1 < 2/eta used for the covariance recursion hints that the correction term may change character at the edge of stability, a regime the authors do not analyze—an extension left implicit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the one thing: this paper gives a concrete, testable mechanism for why mini-batch SGD lands in flatter minima than full-batch GD, and it derives a correction term that, added to GD, reproduces SGD-like sharpness dynamics. That is a real contribution. The local math is clean: the Taylor expansion of the averaged gradient (Eq. 2) and the Lyapunov recursion for displacement covariance (Eq. 17) are standard, and under Assumptions 1–4 the algebra checks out. The controlled perturbation experiment in Figure 3 is also well designed: matching rank and variance between dominant and random perturbations isolates the dominant subspace as the relevant one. Credit where due: the paper is honest that C_mb is not generally isotropic, so the correction is not simply proportional to ∇S_k; that restraint is good.

The soft spot is exactly the one the stress-test note pins: the step from the frozen-reference local recursion in Appendix C.3 to the global SGD trajectory is a surrogate assumption. Equation (21) substitutes the stationary covariance of a linear recursion with fixed P_c and θ_c into the local averaging identity, and then b_corr is used over 100-step intervals. For that chain to be the actual mechanism, the bulk component of the noise has to be negligible for top-k sharpness, and the dominant subspace and noise covariance have to vary slowly. Neither is established. The perturbation experiment uses isotropic noise in a fixed random subspace as the 'random' control; it does not inject the true bulk mini-batch noise, so it can't rule out the possibility that bulk noise matters. And the main validation—Figure 4—shows closeness, but without error bars, multiple seeds, code, or quantitative gap measures. A skeptic could say the corrected GD is just a differently-perturbed GD that lowers sharpness for another reason.

That said, none of these concerns is fatal. They are addressable, and they are the natural next set of experiments. The derivation is mathematically sound under stated assumptions; what's missing is evidence that those assumptions govern real SGD. I'd send this to a serious referee. The right reviewer will check whether the frozen-reference covariance can be justified (e.g. via timescale separation) and whether the bulk-noise contribution is actually small. This is a workshop paper that deserves to be developed into a full submission.

Referee Report

3 major / 4 minor

Summary. The paper studies the role of the dominant Hessian subspace (top-k eigenspace) in top-k sharpness S_k under mini-batch SGD. Based on projected-update experiments (Dom-SGD vs Bulk-SGD) and controlled perturbation experiments (GD+Dom vs GD+Rand), it argues that stochastic motion within the dominant subspace lowers S_k even though it does not reduce the training loss. The authors derive a sharpness correction term: averaging the gradient over a mean-zero displacement δ with covariance C yields an additional term (1/2)∇Tr(H(θ_c)C). For mini-batch noise, they introduce a local recursion δ_{s+1} = A_c δ_s − η P_c ξ_B(θ_c), whose stationary covariance C_mb is substituted into the correction, giving b_corr = −(1/2)∇Tr(H C_mb). Adding this correction to full-batch GD is reported to bring its S_k trajectory closer to mini-batch SGD on MLP, CNN, and Transformer benchmarks, with a batch-size sweep and a correction-direction sanity check in the appendix.

Significance. The local derivation in Section 4 and Appendix C is mathematically clean: the second-order Taylor expansion, the covariance recursion, and the discrete Lyapunov solution are correct under Assumptions 1–4. The paper also makes a useful conceptual contribution by separating the loss-reduction and sharpness-reduction roles of the dominant subspace, and it provides a falsifiable experimental design (corrected GD) with encouraging results across three architectures. The controlled perturbation experiment (Figure 3) and the correction-direction experiment (Appendix E.2) are good checks. However, the central bridge from the frozen-reference local recursion to the global SGD trajectory is not established. If that bridge can be justified quantitatively, the result would be a meaningful mechanism for SGD's implicit regularization; in the current form, the manuscript presents a promising hypothesis supported by local analysis and indirect experiments.

major comments (3)

[Section 4 / Appendix C.3, Eq. (21)] The load-bearing step is the substitution of the stationary covariance C_mb of the frozen-reference local recursion (14) into the local averaging identity (5) to obtain the effective drift of global SGD. This requires at least three conditions: (i) the actual SGD-minus-GD displacement is effectively confined to E_dom(θ_c) on the averaging timescale; (ii) the bulk component of mini-batch noise has negligible effect on S_k; and (iii) P_c and Σ_dom(θ_c) vary slowly enough that a checkpoint-frozen C_mb remains valid over the 100-step intervals used in §D.4. None of these is established. The paper should either prove these from verifiable assumptions or measure them directly: compare the empirical covariance of P_c(θ_{t+1}^{SGD} − θ_{t+1}^{GD}) over an interval with C_mb; report ∥P_bulk(θ_{t+1}^{SGD} − θ_{t+1}^{GD})∥ and its contribution to S_k; and report the principal-angle rotation of P_do
[Section 3 / Figure 3 and Eq. (11)] The perturbation experiment uses isotropic Gaussian noise in a fixed random subspace (GD+Rand) and compares it with isotropic noise in the current dominant subspace (GD+Dom). This does not test whether the bulk component of real mini-batch noise is negligible. The derivation discards the bulk component by defining δ_{s+1} through the projection P_dom(θ_c) in Eq. (11), but no estimate of the bulk component's magnitude or its effect on S_k is provided. A direct experiment adding the actual bulk projection of mini-batch noise to GD, or measuring the bulk component of the SGD-minus-GD displacement, would be needed to rule out a significant bulk mechanism.
[Section 4, Eqs. (5), (8), Appendix C.2 vs C.3] The theoretical connection to top-k sharpness is established only in the isotropic case C = σ² P_dom(θ_c), where the correction becomes (σ²/2)∇S_k. For the general mini-batch-induced covariance C_mb, the correction is (1/2)∇Tr(H C_mb), which need not be aligned with ∇S_k; no theorem or condition shows that this term reduces S_k. The abstract and title claim that dominant-subspace fluctuations lower top-k sharpness, but the derivation only shows they lower the covariance-weighted curvature Tr(HC_mb). The empirical Figure 4 is the only current evidence for the S_k claim in the mini-batch case. The authors should either provide a structural condition on C_mb (e.g., approximate isotropy on the dominant subspace, or dominance of the top-k trace) or temper the claim to the quantity actually derived.

minor comments (4)

[Section 5, formula for b_corr] The displayed definition b_corr(θ) := −(1/2)∇_θ Tr(H(θ) C_mb(θ))|_{θ=θ} is ambiguous: taken literally it differentiates C_mb(θ) as well. The derivation in Appendix C.1 explicitly holds C fixed and uses an auxiliary variable ϑ. Please write b_corr(θ_c) = −(1/2)∇_ϑ Tr(H(ϑ) C_mb(θ_c))|_{ϑ=θ_c} to match the derivation and the third-derivative implementation described in §D.4.
[Figures 4–7, Appendix D.5] The main comparisons are single trajectories with no error bars or multiple seeds. Given that the sharpness gaps in Figures 4 and 5 are sometimes small, reporting the mean and standard deviation over at least three seeds (or stating that only one seed is used) would make the empirical claims easier to evaluate.
[Section 3, Figure 3 caption] The caption says the perturbation runs are 'matched in rank and noise scale (k=10, ρ=0.1)', but the random projection P_rand is fixed at initialization while P_dom(θ_t) changes. Since the dominant subspace rotates, the two perturbations are matched only in rank and instantaneous variance, not in their geometric relationship to the Hessian. Please state this explicitly.
[Appendix D.3] The phrase 'the stable learning-rate (GF) regime of Song et al. [29]' is not defined. A one-sentence description of how the stable/GF regime is identified (e.g., λ_1 < 2/η or the criterion from [29]) would help readers assess whether Assumption 4 is met in the experiments.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the correction term is derived from measured noise covariance and Hessian, not fitted to sharpness.

full rationale

The paper's central derivation is not circular. The sharpness correction term in Eq. (2)/(5) is obtained from a Taylor expansion of the gradient averaged over a zero-mean displacement; this is a mathematical identity. The mini-batch-induced covariance C_mb is computed from a local recursion (Eqs. 14 and 17) whose inputs are the measured dominant-projected noise covariance Sigma_dom and the local Hessian; it is not fitted to the sharpness curve S_k. The paper explicitly notes (Appendix C.3) that for general C_mb the correction is not proportional to grad S_k, so the result does not reduce to the target by construction. The experimental validation in Figure 4 compares corrected GD against an external benchmark, mini-batch SGD, and includes a sign-reversal control (Appendix E.2), so the sharpness reduction is not an artifact of a fitted parameter. The dominant-subspace restriction is a stated modeling assumption with explicit local approximations (frozen reference, projected recursion, discarded bulk noise); its validity is an approximation risk, not a circularity. No load-bearing self-citations or imported uniqueness claims appear: the cited prior work [29,33] is external and used only to motivate the setup. Overall, the derivation is self-contained and the empirical test is meaningful.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The theoretical derivation relies on standard smoothness/eigengap/stability assumptions plus a load-bearing ad hoc surrogate assumption connecting the local frozen-subspace recursion to global SGD dynamics. No new physical entities are introduced. The main empirical inputs are the measured noise covariance and Hessian eigenspace.

free parameters (3)

top-k dimension k = k=10 (MLP, CNN), k=2 (Transformer)
The dominant subspace, S_k, and all derived correction terms depend on the hand-chosen k. Results may be sensitive to this choice; no sweep over k is reported.
perturbation scale rho^2 = 0.1
In the controlled noise-injection experiment (Section 3), the variance rho^2=0.1 of the mean-zero perturbations is chosen by hand; it sets the magnitude of the sharpness reduction shown in Figure 3.
learning rate eta = 0.01 (MLP), 0.001 (CNN/Transformer)
The learning rate is a training hyperparameter taken from prior work; it enters the stability condition lambda_1<2/eta and the local map A_c. It is not fitted to the sharpness result, but it is a chosen input.

assumptions (6)

standard math Local Taylor regularity: L is locally C^4 (Assumption 1).
Used to expand the gradient to second order and control the O(E||delta||^3) remainder in Eq. (1)-(2).
domain assumption Eigengap: lambda_k(theta_c) > lambda_{k+1}(theta_c) (Assumption 2).
Needed for differentiability of the top-k eigenspace and for the identity grad Tr(H P_dom) = grad S_k in Appendix C.2.
domain assumption Mini-batch noise is mean-zero with finite covariance and is frozen locally: xi_B(theta_c+delta_s) ≈ xi_B(theta_c) (Assumption 3).
A standard noise-freezing approximation; it makes the local recursion linear in delta_s. The error term r_{xi,s} is dropped without a quantitative bound.
domain assumption Stable learning-rate regime: 0 < lambda_k <= lambda_1 < 2/eta (Assumption 4).
Ensures rho(A_c)<1 so the Lyapunov recursion converges to a finite stationary covariance C_mb.
ad hoc to paper The local recursion with frozen P_c and frozen noise covariance is a valid surrogate for global SGD's dominant-subspace fluctuations; bulk noise components are negligible for top-k sharpness.
This is the key bridging assumption from the local derivation to the corrected-GD experiments. It is not listed among Assumptions 1-4 and is not proven; if the subspace rotates significantly or bulk noise matters, the surrogate fails.
domain assumption Empirically estimated Sigma_dom from 100 mini-batches at sparse checkpoints accurately represents the true noise covariance.
Corrected GD uses Sigma_dom estimated at checkpoints every 100 steps and holds the correction fixed in between; this assumes the covariance changes slowly.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations." pith.science (2026). https://pith.science/paper/AI4GPNVH

@misc{pith2026260723012,
  author       = {Pith},
  title        = {Pith review of: Mini-batch Noise Lowers Sharpness via Dominant-Subspace Fluctuations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AI4GPNVH}},
  note         = {Machine review of arXiv:2607.23012}
}

abstract

During SGD training, the gradients often align strongly with the dominant subspace spanned by the top-$k$ eigenvectors of the Hessian of the loss. While this seems to naturally imply that loss reduction mainly occurs within this space, prior work has shown that updates within this dominant subspace make no meaningful progress in reducing the loss. In this work, we argue that the dominant subspace is better understood not as the main space for loss reduction, but as a key subspace for explaining the sharpness dynamics of mini-batch SGD. To explain the role of the dominant subspace in reducing top-$k$ sharpness, we show how the averaged gradient over fluctuations in the dominant directions produces a sharpness correction term, and derive a sharpness correction term induced by mini-batch noise in the dominant directions. Experimental results show that adding the derived correction term to GD brings the sharpness evolution of GD closer to that of SGD.

Figures

Figures reproduced from arXiv: 2607.23012 by the authors.

**Figure 1.** The dominant component does not reduce loss but reduces sharpness. (a) Training gradients align with the dominant subspace. (b) Dom-SGD fails to reduce loss, while Bulk-SGD continues to train. (c) Dom-SGD lowers 𝑆𝑘, whereas Bulk-SGD maintains higher 𝑆𝑘 than SGD. SGD as 𝜃𝑡+1 = 𝜃𝑡−𝜂(∇𝐿(𝜃𝑡)+𝜉𝐵𝑡 (𝜃𝑡)), and full-batch GD as 𝜃𝑡+1 = 𝜃𝑡−𝜂∇𝐿(𝜃𝑡). Let (𝜆𝑖(𝜃), 𝑒𝑖(𝜃)) be the eigenpairs of 𝐻(𝜃) = ∇ 2𝐿(𝜃), ordered so that 𝜆1 (𝜃) … view at source ↗

**Figure 2.** River-valley intuition. However, as shown in [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Dominant perturbations reduce top-𝑘 sharpness. We compare GD, GD with dominant mean-zero perturbations (GD+Dom), and GD with random mean-zero perturbations (GD+Rand). The two perturbation runs are matched in rank and noise scale (𝑘 = 10, 𝜌 = 0.1). (a) Dominant perturbations induce strong dominant alignment. (b) Only dominant perturbations reduce 𝑆𝑘. (c) Both perturbed runs leave the GD loss curve nearly unchanged. A… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: The sharpness correction term brings GD closer to mini-batch SGD in top-𝑘 sharpness. Across MLP, CNN, and Transformer, full-batch GD reaches higher top-𝑘 sharpness 𝑆𝑘, whereas adding the derived sharpness correction term to GD brings its top-𝑘 sharpness curve closer t…

**Figure 5.** Figure 5: Corrected GD across batch sizes. (a) MLP on MNIST-5k, (b) CNN on CIFAR10-5k, and (c) Transformer on SST2-1k. Solid lines denote mini-batch SGD, and dashed lines denote corrected GD. As the batch size increases, the 𝑆𝑘 curve of SGD moves closer to full-batch GD, and cor…

**Figure 6.** Figure 6: Effect of the correction direction on MLP / MNIST-5k. We compare GD, SGD, corrected GD, and GD with the reversed correction. (a) Top-𝑘 sharpness 𝑆𝑘. (b) Training loss. The derived correction lowers sharpness relative to GD, whereas reversing its direction increases sh…

**Figure 7.** Figure 7: Effect of the correction direction on CNN / CIFAR10-5k. We compare GD, SGD, corrected GD, and GD with the reversed correction. (a) Top-𝑘 sharpness 𝑆𝑘. (b) Training loss. The derived correction lowers sharpness relative to GD, whereas reversing its direction increases s…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 6 linked inside Pith

[1]

Towards understanding sharpness-aware minimization

Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. InKamalikaChaudhuri,StefanieJegelka,LeSong,CsabaSzepesvari,GangNiu, andSivanSabato,editors, Proceedings of the 39th International Conference on Machine Learn- ing,volume162of Proceedings of Machine Learning Research,pages639–668.PMLR,17–23 Jul 2022. URLhttps:...

2022
[2]

Bartlett, Philip M

Peter L. Bartlett, Philip M. Long, and Olivier Bousquet. The dynamics of sharpness-aware minimization: Bouncingacrossravinesanddriftingtowardswideminima. Journal of Machine Learning Research, 24(316):1–36, 2023. URL http://jmlr.org/papers/v24/23-043. html

2023
[3]

Gradient descent on neural networks typically occurs at the edge of stability

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InInternational Con- ference on Learning Representations , 2021. URL https://openreview.net/forum?id= jh-rTtvkGeM

2021
[4]

Jeremy Cohen, Alex Damian, Ameet Talwalkar, J Zico Kolter, and Jason D. Lee. Understand- ing optimization in deep learning with central flows. InThe Thirteenth International Con- ference on Learning Representations , 2025. URL https://openreview.net/forum?id= sIE2rI3ZPs

2025
[5]

Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gra- dient descent at the edge of stability, 2023. URLhttps://arxiv.org/abs/2209.15594

arXiv 2023
[6]

Suspicious alignment of sgd: A fine-grained step size condition analysis, 2026

Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, and Yaoqing Yang. Suspicious alignment of sgd: A fine-grained step size condition analysis, 2026. URLhttps: //arxiv.org/abs/2601.11789

arXiv 2026
[7]

Sharp minima can gen- eralize for deep nets

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can gen- eralize for deep nets. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learn- ing Research, pages 1019–1028. PMLR, 06–11 Aug 2017. URL https://proceedings. mlr.press/v70/dinh17b.html

2017
[8]

Sharpness-aware min- imization for efficiently improving generalization

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum?id=6Tm1mposlrM

2021

Show all 40 references

[9]

An investigation into neural net opti- mization via hessian eigenvalue density

Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Procee...

2019
[10]

Roberts, and Ethan Dyer

Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace,
[11]

Flat minima.Neural Computation, 9(1):1–42, 01

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima.Neural Computation, 9(1):1–42, 01
[12]

Three factors influencing minima in SGD, 2018

StanisławJastrzębski,ZacKenton,DevanshArpit,NicolasBallas,AsjaFischer,AmosStorkey, and Yoshua Bengio. Three factors influencing minima in SGD, 2018. URL https:// openreview.net/forum?id=rJma2bZCW

2018
[13]

Fan- tasticgeneralizationmeasuresandwheretofindthem

YidingJiang*, Behnam Neyshabur*, Hossein Mobahi, Dilip Krishnan, and SamyBengio. Fan- tasticgeneralizationmeasuresandwheretofindthem. In International Conference on Learning Representations, 2020. URLhttps://openreview.net/forum?id=SJgIPJBFvH

2020
[14]

On large-batch training for deep learning: Generalization gap and sharp minima

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations , 2017. URL https: //openreview.net/f...

2017
[15]

Andrew V. Knyazev. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM Journal on Scientific Computing , 23(2): 517–541, 2001. doi: 10.1137/S1064827500366124. URL https://doi.org/10.1137/ S1064827500366124

2001 doi
[16]

Learning multiple layers of features from tiny images

Alex Krizhevsky. Learning multiple layers of features from tiny images. Techni- cal report, University of Toronto, 2009. URL https://www.cs.toronto.edu/~kriz/ learning-features-2009-TR.pdf

2009
[17]

Asam: Adaptivesharpness- aware minimization for scale-invariant learning of deep neural networks

JungminKwon,JeongseopKim,HyunseoPark,andInKwonChoi. Asam: Adaptivesharpness- aware minimization for scale-invariant learning of deep neural networks. In Marina Meila and Tong Zhang, editors,Proceedings of the 38th International Conference on Machine Learning , volume 139 ofPro...

2021
[18]

Lecun, L

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791

1998 doi
[19]

Stochastic modified equations and adaptive stochastic gradient algorithms

Qianxiao Li, Cheng Tai, and Weinan E. Stochastic modified equations and adaptive stochastic gradient algorithms. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th In- ternational Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Resear...

2017
[20]

Stochasticmodifiedequationsanddynamicsofstochas- tic gradient algorithms i: Mathematical foundations.Journal of Machine Learning Research , 20(40):1–47, 2019

QianxiaoLi,ChengTai,andWeinanE. Stochasticmodifiedequationsanddynamicsofstochas- tic gradient algorithms i: Mathematical foundations.Journal of Machine Learning Research , 20(40):1–47, 2019. URLhttp://jmlr.org/papers/v20/17-526.html

2019
[21]

On the validity of modeling SGD with stochastic differential equations (SDEs)

Zhiyuan Li, Sadhika Malladi, and Sanjeev Arora. On the validity of modeling SGD with stochastic differential equations (SDEs). In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wort- man Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://open...

2021
[22]

What happens after SGD reaches zero loss? –a mathematical framework

Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after SGD reaches zero loss? –a mathematical framework. InInternational Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=siCt4xZn5Ve

2022
[23]

Hoffman, and David M

Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate bayesian inference, 2018. URLhttps://arxiv.org/abs/1704.04289

2018 arXiv
[24]

Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians

Vardan Papyan. Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 ofPro- ceedings of M...

2019
[25]

Traces of class/cross-class structure pervade deep learning spectra.Journal of Machine Learning Research, 21(252):1–64, 2020

Vardan Papyan. Traces of class/cross-class structure pervade deep learning spectra.Journal of Machine Learning Research, 21(252):1–64, 2020. URLhttp://jmlr.org/papers/v21/ 20-933.html

2020
[26]

Eigenvalues of the hessian in deep learning: Singularity and beyond, 2017

Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond, 2017. URLhttps://openreview.net/forum?id=B186cP9gx

2017
[27]

Ugur Güney, Yann N

Levent Sagun, Utku Evci, V. Ugur Güney, Yann N. Dauphin, and Léon Bottou. Empirical analysis of the hessian of over-parametrized neural networks.CoRR, abs/1706.04454, 2017. URL http://arxiv.org/abs/1706.04454

2017 arXiv
[28]

Recursive deep models for semantic compositionality over a sentiment treebank

RichardSocher,AlexPerelygin,JeanWu,JasonChuang,ChristopherD.Manning,AndrewNg, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Pr...

2013
[29]

Does SGD really happen in tiny subspaces? In The Thirteenth International Conference on Learning Representations, 2025

Minhak Song, Kwangjun Ahn, and Chulhee Yun. Does SGD really happen in tiny subspaces? In The Thirteenth International Conference on Learning Representations, 2025. URLhttps: //openreview.net/forum?id=v6iLQBoIJw

2025
[30]

On the interplay between noise and curvature and its effect on optimizationandgeneralization

Valentin Thomas, Fabian Pedregosa, Bart van Merriënboer, Pierre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimizationandgeneralization. InSilviaChiappaandRobertoCalandra, editors, Proceedings of the Tw...
[31]

Attention is all you need.Advances in neural information processing systems, 30, 2017

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

2017
[32]

How sharpness-aware minimization minimizes sharpness? In The Eleventh International Conference on Learning Representations , 2023

Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How sharpness-aware minimization minimizes sharpness? In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=5spDgWmpY6x. 8 DOMINANT FLUCTUATIONS LOWER SHARPNESS

2023
[33]

Wang, David Leo Wright Hall, Percy Liang, and Tengyu Ma

Kaiyue Wen, Zhiyuan Li, Jason S. Wang, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape view. In The Thirteenth International Conference on Learning Representations, 2025. URLhttps: //openreview....

2025
[34]

An empirical study of stochastic gradient descent with structured covariance noise

Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An empirical study of stochastic gradient descent with structured covariance noise. In Silvia Chi- appa and Roberto Calandra, editors,Proceedings of the Twenty Third International Confer- ence on A...

2020
[35]

The alignment property of sgd noise and how it helps select flat minima: A stability analysis

Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: A stability analysis. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pa...
[36]

Accelerating neural network training along sharp and flat directions, 2025

Daniyar Zakarin and Sidak Pal Singh. Accelerating neural network training along sharp and flat directions, 2025. URLhttps://arxiv.org/abs/2505.11972

2025 arXiv
[37]

The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects

Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,Proceedings of the 36th International...

2000
[1997]

doi: 10.1162/neco.1997.9.1.1

ISSN 0899-7667. doi: 10.1162/neco.1997.9.1.1. URLhttps://doi.org/10.1162/ neco.1997.9.1.1

1997 doi
[2018]

6 DOMINANT FLUCTUATIONS LOWER SHARPNESS

URL https://arxiv.org/abs/1812.04754. 6 DOMINANT FLUCTUATIONS LOWER SHARPNESS

arXiv
[2022]

URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 1e55c38dd7d465c2526ae29d7ec85861-Paper-Conference.pdf

2022

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

Towards understanding sharpness-aware minimization

Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. InKamalikaChaudhuri,StefanieJegelka,LeSong,CsabaSzepesvari,GangNiu, andSivanSabato,editors, Proceedings of the 39th International Conference on Machine Learn- ing,volume162of Proceedings of Machine Learning Research,pages639–668.PMLR,17–23 Jul 2022. URLhttps:...

2022

[2] [2]

Bartlett, Philip M

Peter L. Bartlett, Philip M. Long, and Olivier Bousquet. The dynamics of sharpness-aware minimization: Bouncingacrossravinesanddriftingtowardswideminima. Journal of Machine Learning Research, 24(316):1–36, 2023. URL http://jmlr.org/papers/v24/23-043. html

2023

[3] [3]

Gradient descent on neural networks typically occurs at the edge of stability

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InInternational Con- ference on Learning Representations , 2021. URL https://openreview.net/forum?id= jh-rTtvkGeM

2021

[4] [4]

Jeremy Cohen, Alex Damian, Ameet Talwalkar, J Zico Kolter, and Jason D. Lee. Understand- ing optimization in deep learning with central flows. InThe Thirteenth International Con- ference on Learning Representations , 2025. URL https://openreview.net/forum?id= sIE2rI3ZPs

2025

[5] [5]

Alex Damian, Eshaan Nichani, and Jason D. Lee. Self-stabilization: The implicit bias of gra- dient descent at the edge of stability, 2023. URLhttps://arxiv.org/abs/2209.15594

arXiv 2023

[6] [6]

Suspicious alignment of sgd: A fine-grained step size condition analysis, 2026

Shenyang Deng, Boyao Liao, Zhuoli Ouyang, Tianyu Pang, Minhak Song, and Yaoqing Yang. Suspicious alignment of sgd: A fine-grained step size condition analysis, 2026. URLhttps: //arxiv.org/abs/2601.11789

arXiv 2026

[7] [7]

Sharp minima can gen- eralize for deep nets

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can gen- eralize for deep nets. In Doina Precup and Yee Whye Teh, editors,Proceedings of the 34th International Conference on Machine Learning, volume 70 ofProceedings of Machine Learn- ing Research, pages 1019–1028. PMLR, 06–11 Aug 2017. URL https://proceedings. mlr.press/v70/dinh17b.html

2017

[8] [8]

Sharpness-aware min- imization for efficiently improving generalization

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware min- imization for efficiently improving generalization. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum?id=6Tm1mposlrM

2021