Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature

T0 review · 3 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that SGD noise covariance is proportional to the expected squared per-sample Hessian, so noise and curvature connect superlinearly with a power-law exponent between 1 and 2.

desk verdict The central derivation assumes an isotropy that its own rank-1 AWD lemma rules out; the empirical CE/MSE comparison still has value. read the letter →

arxiv 2602.05600 v2 pith:W5B4B4TE submitted 2026-02-05 cs.LG

Yikuan Zhang , Ning Yang , Yuhai Tu This is my paper · ORCID
classification cs.LG MSC 68T07
keywords SGDnoisecovarianceHessianper-sampleActivity-WeightDualitypower-lawscalingFisherinformationapproximationflatminimacross-entropyloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to replace the common assumption that SGD noise covariance C is proportional to the loss Hessian H, an equivalence inherited from the Fisher information matrix. Using Activity–Weight Duality, it derives a loss-agnostic formula: C ∝ E_p[h_p²], the second moment of per-sample Hessians. Since H is the first moment, C and H share an approximate eigensystem but do not coincide, and their diagonals follow C_ii ∝ H_ii^γ with provable bounds 1 ≤ γ ≤ 2. This explains why cross-entropy training shows superlinear scaling (γ ≈ 1.3–1.5) while mean-squared error stays near γ ≈ 1, offering a more faithful picture of the geometry that biases SGD toward flat minima.

What carries the argument

Activity–Weight Duality (AWD): a mapping from minibatch activity fluctuations to minimal-norm weight perturbations, whose explicit form is a rank-1 outer product ΔW* = (WΔa)aᵀ/‖a‖². The argument's engine is a spectral decomposition of the resulting covariance in the global Hessian eigenbasis, which assumes that the weight perturbations are isotropic in each sample's local Hessian eigenbasis (M_p ≈ σ_w²I). That assumption turns the noise into E_p[Σ_m (κ_m^(p))² (u_m^(p)·v_i)²], isolating curvature magnitudes from their alignment with global directions and yielding the 1–2 exponent bounds.

What would settle it

Compute M_p = E[Δw_p Δw_pᵀ] for a single fully connected layer at convergence; because ΔW* is a rank-1 outer product, M_p has rank at most d_out, which in the paper's 50×50 layer is only 50 out of 2500 dimensions, so it cannot be a scalar multiple of the identity. If the measured M_p violates isotropy, the spectral formula in Theorem 3.4 loses its foundation, and with it the universal exponent bound.

Watch

Extended reading notes

Core claim

The central discovery is that the SGD noise covariance is governed by the squared per-sample curvature: C ≈ (σ_w²/2B) E_p[h_p²], derived without assuming the model distribution equals the data distribution. Because the Hessian H = E_p[h_p], the noise amplifies high-curvature directions more than the Hessian itself, producing a power-law C_ii ∝ H_ii^γ with 1 ≤ γ ≤ 2. The paper proves the bound from Cauchy–Schwarz and bounded per-sample eigenvalues, and confirms it empirically on MLPs and CNNs across MNIST and CIFAR-10 with both cross-entropy and mean-squared-error losses.

Load-bearing premise

The load-bearing premise is that per-sample weight perturbations from Activity–Weight Duality have isotropic covariance in each sample's local Hessian eigenbasis (M_p ≈ σ_w²I, uniform across samples); since the paper's own solution ΔW* is rank 1, this isotropy cannot hold exactly, and if it fails the derived formula for C and the 1–2 exponent bound collapse.

Editorial extensions

If this is right

  • The Fisher-approximation identity C ∝ H is generically false for cross-entropy losses; algorithms and theories built on it inherit a misleading equivalence.
  • C and H approximately commute, so SGD noise analysis can treat them as simultaneously diagonalizable in the Hessian eigenbasis.
  • The exponent γ measures how strongly per-sample curvature fluctuations align with global Hessian directions, giving a quantitative diagnostic of landscape geometry.
  • Because γ exceeds 1 for cross-entropy, noise anisotropically weights high-curvature directions more than H alone, reinforcing curvature-driven escape from sharp minima.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is to verify C ∝ E_p[h_p²] directly in Transformer blocks, where per-sample Hessians are approximated; the paper only analyzes fully connected layers.
  • The local isotropy assumption is contradicted by the paper's own rank-1 perturbation formula; replacing it with a realistic rank-deficient covariance model may change the derived exponent bound, so the 1–2 interval should be tested against that weaker premise.
  • A practical consequence the authors leave implicit: if γ is tuned by decoupling curvature magnitude from eigendirection alignment (the X–Y correlation), it could offer a new knob for controlling flat-minima selection without changing the loss landscape.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes a general relationship between SGD noise covariance C and the loss Hessian H, claiming that under an Activity-Weight Duality (AWD) analysis C is proportional to the second moment of per-sample Hessians, E_p[h_p^2], rather than to H itself. From this it derives approximate commutativity of C and H and a universal superlinear power-law exponent 1≤γ≤2 for diagonal entries in the Hessian eigenbasis. The claims are tested on MLP and CNN models with cross-entropy (CE) and mean-squared-error (MSE) losses, finding γ>1 for CE and γ≈1 for MSE, and are supplemented by a suppression experiment isolating the role of the leading per-sample eigenvalues.

Significance. If the central derivation were sound, the paper would make a substantial contribution: it would replace the restrictive Fisher-information equivalence with a loss-agnostic formula and explain the empirical CE/MSE distinction in a unified way. The dimensional argument in Section 2.2 and the suppression experiment are interesting, and the paper includes an anonymized code release and fairly extensive experiments. However, the theoretical core is invalid: the isotropy assumption in Theorem 3.4 is structurally incompatible with the paper's own closed-form AWD perturbation from Lemma 3.2. Because the main quantitative claims (Eq. (15), Eq. (17), Theorem 5.1, and the abstract's C∝E_p[h_p^2]) all rest on that assumption, the significance of the paper is not realized as written.

major comments (3)
  1. [Theorem 3.4 / Eq. (14)] The isotropy assumption M_{p,mn} ≈ σ_w² δ_mn in Eq. (14) contradicts Lemma 3.2. Lemma 3.2 gives ΔW = (WΔa)a^T/||a||^2, a rank-1 matrix. For fixed input activity a, the vectorized perturbation Δw = vec(ΔW) lies in the d_out-dimensional subspace {a⊗u : u∈R^{d_out}}. Therefore M_p = E[Δw Δw^T] has rank at most d_out (50 in the D=2500 experiments), so in any basis it has at most d_out nonzero eigenvalues and cannot equal σ_w² I. Consequently, Eq. (15), the abstract's C∝E_p[h_p^2], and the derived bounds in Theorem 5.1 are not established. The paper explicitly acknowledges this assumption in the Conclusion as 'insensitive to detailed architectural choices', but the rank argument shows it is not merely a choice of prefactor; it is structurally impossible.
  2. [Appendix E.2.3, Figures 4-6 and 10-12] The numerical evidence cited for Eq. (14) is incommensurate with the claim. Figures 4-6(h) compare diagonal magnitudes of C_hh, C_hh,SD, C_hh,SD,WD in the global Hessian basis, and Figures 10-12 show log-log diagonal scalings of C and H. None of these displays the matrix M_p in the per-sample Hessian eigenbasis, nor its off-diagonal entries. Thus the statement that 'M_{p,mn} ≈ σ_w² δ_mn' is not empirically supported by the presented figures, and cannot rescue the internal contradiction.
  3. [Section 4 / Table 1] The paper describes γ_AWD as a 'prediction' and γ_emp as an independent measurement, but γ_AWD is computed from Eq. (15) using the same per-sample Hessians that appear in the theoretical relation, and γ_emp is computed from the same model and dataset. The agreement is therefore a consistency check of Eq. (15), not an out-of-sample or independent validation. While no free parameter is fit to force the slope, the reuse of the same spectral information substantially weakens the evidential value. Given that Eq. (15) relies on the invalid assumption (14), the agreement in Table 1 cannot by itself support the proposed C∝E_p[h_p^2] law.
minor comments (3)
  1. [Remark 3.5] The claim that the projections (u_m^{(p)}·v_i) and (u_m^{(p)}·v_j) are 'statistically independent' is not justified. For a random unit vector, these dot products with orthogonal directions are uncorrelated, but not independent in general. To conclude that off-diagonal elements C_ij vanish, one needs a variance or concentration argument, not merely zero mean.
  2. [Theorem 5.1 / Eq. (20)] The derivation of 1≤γ≤2 assumes an exact power law C_ii = c H_ii^γ over a range where H_ii can be arbitrarily small. The experiments only fit the top 1000 eigenvalues, so the logical step from Eq. (19) to Eq. (20) is more delicate than stated, even apart from the failure of Eq. (15).
  3. [Appendix D] The p-value is reported as 'p≪10^{-50}' after a Z-score of about -3378. This is reasonable, but the exact tail probability is machine zero; the notation is informal. Also, the effective number of spikes M≈20 is estimated from the same data used for the baseline, so the test is partly circular.

Circularity Check

2 steps flagged · score 4.0 of 10

Central C∝E_p[h_p^2] result is the Eq. (14) isotropy assumption restated; the AWD 'prediction' is an in-sample consistency check.

  1. self definitional [Theorem 3.4, Eqs. (13)-(15); Appendix E.2.3; abstract]
    "if the covariance of per-sample weight perturbations in its local eigenbasis is isotropic and uniform across different samples: M_{p,mn}=E_μ[(u_m^{(p)})^T Δw_p^μ (Δw_p^μ)^T u_n^{(p)}]≈σ_w^2 δ_{mn} ... C_{ii}≈σ_w^2/(2B) E_p[ Σ_m (κ_m^{(p)})^2 (u_m^{(p)}·v_i)^2 ]"

    The abstract's headline 'showing that C ∝ E_p[h_p^2]' is obtained by substituting Eq. (14) into the already-derived quadratic form Eq. (13), C≈(1/2B)E_p[h_p M_p h_p]. With M_p≈σ_w^2 I, Eq. (13) immediately becomes C∝E_p[h_p^2]; no step beyond the isotropy ansatz produces the claimed second-moment law. The paper's own Lemma 3.2 gives ΔW*=(WΔa)a^T/||a||^2, a rank-1 matrix, so M_p is at most rank d_out in a D=2500 weight space and cannot be isotropic; hence the central 'finding' is an internally inconsistent ansatz restated as a theorem.

  2. fitted input called prediction [Table 1 and Section 4, Figure 2(b)]
    "For the CE loss, the AWD-predicted exponent γ_AWD closely matches the empirical exponent γ_emp ... As shown in Figure 2(b) ... the empirical power-law relationship, C_{ii} ∝ H_{ii}^γ, also applies to our theoretical results based on AWD with similar exponents."

    γ_AWD is computed from Eq. (15), which is built from the same per-sample Hessians h_p that define H and are used to fit γ_emp. Thus the comparison in Table 1 is a self-consistency check between two quantities constructed from the same training checkpoints, not an out-of-sample prediction. No free parameter is fit to force the slope, which lowers the severity, but labeling γ_AWD a 'prediction' overstates its independence from the empirical input.

full rationale

The paper is mostly self-contained and transparent: the conditional derivation of Eq. (15) from Eq. (13) plus Eq. (14) is explicit, and the empirical superlinear exponent for CE (γ>1) is a real observation that does not require the isotropy assumption. However, the abstract and contributions present 'C ∝ E_p[h_p^2]' and the 1≤γ≤2 bounds as found results, when the core of that result is the Eq. (14) assumption restated through Eq. (13). The AWD 'prediction' of γ is also evaluated on the same per-sample Hessians used to measure γ_emp, making it a consistency check. Because the empirical C-H scaling and the CE/MSE distinction remain independent content, and because the AWD citation (Feng et al., 2023) is externally published and the key lemma is proven in the appendix, the circularity is partial rather than total.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities. The AWD and Random Shifts Model come from prior work. The free parameters are fit exponents, truncation levels, and thresholds used to support the empirical scaling law.

free parameters (4)
  • scaling exponent γ_emp = e.g., 1.14±0.01 (MNIST MLP CE, C=10)
    Slope of log C_ii vs log H_ii over top-K eigenvalues; the paper's central claim of 1 ≤ γ ≤ 2 is supported by fitting these values.
  • eigenvalue truncation N = 300–1500 depending on setting
    Number of top Hessian eigenvalues used for the power-law fit; the fit and the bounds depend on this choice.
  • suppression threshold θ = 0.001 or 0.05
    Hand-chosen threshold in the suppression experiment to define stiff samples; used to establish the CE/MSE explanation.
  • effective spikes M = ≈20
    Estimated from the Hessian spectrum to compute the random-baseline μ_rand ≈ 0.8/√M and the p-value.
assumptions (6)
  • domain assumption AWD framework (Feng et al. 2023) is valid and applicable
    Section 3 uses the Minimal Activity-Weight Duality as a black box; if AWD fails for deep nets, the derivation collapses.
  • domain assumption Near a local minimum, per-sample gradients vanish so Term II in Eq. (12) can be dropped
    Lemma 3.3 and Appendix E.2.1 justify this by vanishing gradients; the paper validates empirically near convergence.
  • domain assumption Cross-terms between distinct samples vanish (Eq. 63)
    Assumed for i.i.d. sampling; used to reduce the double sum in Theorem 3.4.
  • ad hoc to paper Local isotropy M_p ≈ σ_w^2 I (Eq. 14)
    This is the load-bearing assumption; it is contradicted by the rank-1 form of ΔW from Lemma 3.2.
  • domain assumption Per-sample Hessians are PSD with bounded eigenvalues κ_max
    Required for Theorem 5.1's bounds; only plausible at high-quality local minima, not throughout training.
  • ad hoc to paper Random projection independence for off-diagonal terms (Remark 3.5)
    Used to assert [C,H] ≈ 0; no rigorous proof, relies on high-dimensional randomness of eigenvectors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature." pith.science (2026). https://pith.science/paper/W5B4B4TE

@misc{pith2026260205600,
  author       = {Pith},
  title        = {Pith review of: On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W5B4B4TE}},
  note         = {Machine review of arXiv:2602.05600}
}
abstract

Stochastic Gradient Descent (SGD) introduces anisotropic noise that is correlated with the local curvature of the loss landscape, thereby biasing optimization toward flat minima. Prior work often assumes an equivalence between the Fisher Information Matrix and the Hessian for negative log-likelihood losses, leading to the claim that the SGD noise covariance $\mathbf{C}$ is proportional to the Hessian $\mathbf{H}$. We show that this assumption holds only under restrictive conditions that are typically violated in deep neural networks. Using the recently discovered Activity--Weight Duality, we find a more general relationship agnostic to the specific loss formulation, showing that $\mathbf{C} \propto \mathbb{E}_p[\mathbf{h}_p^2]$, where $\mathbf{h}_p$ denotes the per-sample Hessian with $\mathbf{H} = \mathbb{E}_p[\mathbf{h}_p]$. As a consequence, $\mathbf{C}$ and $\mathbf{H}$ commute approximately rather than coincide exactly. We further find that, within the analyzed fully connected layers, their diagonal elements follow per-layer empirical power laws $C_{ii} \propto H_{ii}^{\gamma}$, with layer-dependent fitted exponents bounded by $1 \leq \gamma \leq 2$. Experiments across datasets, architectures, and loss functions support the resulting layerwise bounds, providing a unified characterization of the noise-curvature relationship in deep learning.

Figures

Figures reproduced from arXiv: 2602.05600 by the authors.

Figure 1
Figure 1. Noise–curvature alignment for a CNN trained on CIFAR-10 with cross-entropy loss (top 100 eigen-directions). (a) The empirical covariance matrix C represented in the Hessian eigenbasis. (b) The scale-invariant correlation matrix Rreal, normalized by diagonal elements. (c) The randomized Baseline Rrand, constructed by randomly rotating C while preserving its eigenvalue spectrum. See Appendix Figures 7 to 9 for more de… view at source ↗
Figure 2
Figure 2. Log-log plot of diagonal elements using the top 1000 eigenvalues for models trained to convergence (100% training accuracy for CE and > 95% for MSE). Data points are mean￾centered and vertically shifted for visualization; solid lines denote linear fits. (a) Empirical noise covariance versus the Hessian. (b) AWD-derived noise covariance (Eq. 15) versus the Hessian. The dotted and dashed lines correspond to slope 1 an… view at source ↗
Figure 3
Figure 3. Log-log plots of the diagonal elements of the result￾ing Covariance by “suppression experiment” versus the original Covariance.(a, b) Covariance derived from per-sample Hessians retaining only the dominant eigenvalues. (c, d) Covariance derived after replacing the dominant eigenvalues with their mean value. Columns correspond to distinct models: (a, c) MLP on CIFAR-10 (CE loss, γ ≈ 1.4) and (b, d) MLP on MNIST (MSE … view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Comprehensive Analysis of SGD Noise Structure and Approximations (MLP on MNIST, CE Loss). (a) Evolution of the scaling exponent γ. The exponent γ remains robustly within the interval [1, 2] throughout training, it gradually increases as training progress and tends to d…
Figure 5
Figure 5. Figure 5: Comprehensive Analysis of SGD Noise Structure and Approximations (MLP on MNIST, MSE Loss) (a) Different from the case with CE loss, the exponent γ remains 1 when the model is near the global minimum. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_5.png]
Figure 6
Figure 6. Figure 6: Comprehensive Analysis of SGD Noise Structure and Approximations (CNN on CIFAR-10, CE Loss) (a) The exponent γ gradually increases as the training progress and shows significant deviation from the lower bound 1 when the model is near the global minimum. 28 [PITH_FULL_…
Figure 7
Figure 7. Figure 7: Visualization of Noise-Curvature Alignment (CNN on CIFAR-10 with CE loss, trained to convergence with 100% training accuracy). The first column shows the Empirical Covariance Matrix Covar, AWD-derived CAWD,raw, Chh,SD, Chh,SD,WD, Chh and 2C/σ2 w in the Hessian eigenbas…
Figure 8
Figure 8. Figure 8: Visualization of Noise-Curvature Alignment (MLP on MNIST with CE loss, trained to convergence with 100% training accuracy). 30 [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: Visualization of Noise-Curvature Alignment (MLP on MNIST with MSE loss, trained to approach ∼ 95% training accuracy). The dark block in the second column originates from the degenerated spectrum where the eigenvalues are too small for these flatter direction. 31 [PITH…
Figure 10
Figure 10. Figure 10: Log-log scaling analysis of diagonal elements (Top N = 1500 eigenvalues) for models trained to convergence 100% training accuracy (CNN on CIFAR-10 with CE loss). To isolate the scaling exponent γ (slope) from magnitude differences, data (dots) are mean-centered; solid…
Figure 11
Figure 11. Figure 11: Log-log scaling analysis of diagonal elements (Top N = 1500 eigenvalues) for models trained to convergence 100% training accuracy (MLP on MNIST with CE loss). 33 [PITH_FULL_IMAGE:figures/full_fig_p033_11.png]
Figure 12
Figure 12. Figure 12: Log-log scaling analysis of diagonal elements (Top N = 1500 eigenvalues) for models trained to convergence 100% training accuracy (MLP on MNIST with MSE loss). 34 [PITH_FULL_IMAGE:figures/full_fig_p034_12.png]
Figure 13
Figure 13. Figure 13: “Suppression experiment” for MLP on CIFAR-10 with CE loss (γ ∼ 1.4 significantly larger than 1). The 1st column shows the log-log plot of diagonals between 2Cii/σ2 w = Ep P m(κ (p) m ) 2  u (p) m · vi 2  and Hii = Ep P m(κ (p) m )  u (p) m · vi 2  , while each…
Figure 14
Figure 14. Figure 14: “Suppression experiment” for MLP on MNIST with MSE loss (γ ∼ 1.05 near to 1). Similar to [PITH_FULL_IMAGE:figures/full_fig_p036_14.png]
Figure 15
Figure 15. Figure 15: “Suppression experiment” for MLP on MNIST with CE loss (γ ∼ 1.1 slightly larger than 1). The threshold θ = 0.05 and keeps 20 stiff samples out of 200. Compared to the raw data, cooked 2Cii/σ2 w and Hii changes to some extend. 37 [PITH_FULL_IMAGE:figures/full_fig_p037…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Worker Disagreement Reveals Sharp Directions in Local SGD

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Worker-average gaps in Local SGD serve as a Hessian-free estimator of the dominant sharp subspace by capturing gradient alignment with high-curvature directions.

Reference graph

Works this paper leans on

28 extracted references · 6 linked inside Pith · cited by 1 Pith paper

  1. [1]

    suppression experiment

    and M represents the Max pooling (with kernel size 2 and stride 2), followed by the adaptive average pooling with out put dimension 128×1×1 . These feature maps are flattened into a vector of dimension 128 and fed into a classifier consisting of a hidden fully connected layer and an out put layer ( 128→20→10 ). We use the ReLu activation without bias. To ...

  2. [2]

    Identification of Stiff Samples.We set a relative threshold τ based on a percentage θ (e.g., 10%) of the maximum principal eigenvalue observed in the batch: τ=θ·max p∈{1...B} κ(p) 1 .(24) The set of stiff samples is defined asS stiff ={p|κ (p) 1 > τ}

  3. [3]

    To ensure rank-1 dominance, we suppress their tail modes (m >1) by a factor ofϵ tail = 10−5

    Homogenization and Suppression.We construct a modified curvature profile˜κ(p) m for each sample: • Stiff Sample Homogenization:For stiff samples ( p∈ Sstiff), we replace their dominant eigenvalue κ(p) 1 with the group mean ¯κstiff to remove magnitude variance. To ensure rank-1 dominance, we suppress their tail modes (m >1) by a factor ofϵ tail = 10−5. ¯κs...

  4. [4]

    heavy-tailed

    Global Reconstruction.Finally, we reconstruct the global Hessian ˜Husing the modified spectral components: ˜Hii = 1 B BX p=1 DX m=1 ˜κ(p) m u(p) m ·v i 2 .(27) We then compute the eigenvalues of this reconstructed global matrix ˜H and compare its spectrum to the original H. We also reconstruct the covariance2 ˜Cii/σ2 w = 1 B PB p=1 PD m=1(˜κ(p) m )2 u(p) ...

  5. [5]

    Wu, L., Zhu, Z., and E, W

    URL https://proceedings.mlr.press/ v119/tsuzuku20a.html. Wu, L., Zhu, Z., and E, W. Towards Understanding Gen- eralization of Deep Learning: Perspective of Loss Land- scapes.arXiv:1706.10239 [cs, stat], November 2017. Wu, L., Ma, C., and E, W. How SGD selects the global minima in over-parameterized learning: A dynamical stability perspective. InNeurIPS, p...

    arXiv 2017

  6. [6]

    cc/paper_files/paper/2018/file/ 9c19a2aa1d84e04b0bd4bc888792bd1e-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 9c19a2aa1d84e04b0bd4bc888792bd1e-Paper. pdf. Yang, N., Tang, C., and Tu, Y . Stochastic Gradient Descent Introduces an Effective Landscape-Dependent Regular- ization Favoring Flat Solutions.Physical Review Letters, 130(23):237101, June 2023. ISSN 0031-9007, 1079-7114. doi: 10.1103/PhysRevLet...

    arXiv 2018

  7. [7]

    true parameter

    URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ 102f0bb6efb3a6128a3c750dd16729be-Paper. pdf. Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Pyhessian: Neural networks through the lens of the hessian.CoRR, abs/1912.07145, 2019. URL http: //arxiv.org/abs/1912.07145. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Underst...

    arXiv 2018

  8. [9]

    Per-Sample Decomposition.For a given mini-batch of size B, we compute the per-sample Hessian hp for each sample p∈ {1, . . . , B}. We perform a full eigen-decomposition for each sample: hp = DX m=1 κ(p) m u(p) m (u(p) m )⊤,(23) where eigenvalues are sorted in descending order κ(p) 1 ≥κ (p) 2 ≥. . .. Since the Hessian spectrum is heavily dominated by the t...

Show all 28 references
  1. [13]

    The Denominator (Diagonal Elements).The diagonal elements represent the energy projected onto each dimension: Cii ≈σ 2 MX k=1 Q2 ik.(32) Since Q2 ik follows a scaled Chi-squared distribution with 1 degree of freedom, its expectation is E[Q2 ik] = 1 D . By the Law of Large Numb...

  2. [14]

    SinceQ ik andQ jk are independent (fori̸=j),E[X k] = 0

    The Numerator (Off-Diagonal Elements).Fori̸=j, the off-diagonal element is a sum of products: Cij ≈σ 2 MX k=1 QikQjk .(35) LetX k =Q ikQjk. SinceQ ik andQ jk are independent (fori̸=j),E[X k] = 0. The variance of each term is: Var(Xk) =E[Q 2 ik]E[Q2 jk ] = 1 D · 1 D = 1 D2 .(36...

  3. [15]

    random noise floor

    Effective Dimension Collapse:In a spiked model, the “random noise floor” is determined not by the total dimension N, but by the effective dimension (number of spikes)M

  4. [16]

    High Background Noise:For a typical deep learning setting where the spectrum is sharp (e.g., M≈20−50 ), the random baseline is relatively high (0.11−0.18). Therefore, an observed coupling value significantly lower than 0.8√ M indicates astatistically significant structural ali...

    2000

  5. [17]

    Theoretical Mean (µH0).As derived in (40), for a spectrum dominated byMeffective spikes: µH0 ≈ r 2 π 1√ M .(43)

  6. [18]

    Standard Error (SE).The variance of a single absolute off-diagonal element |Rij| (which follows a Folded Normal distribution derived from a variable with varianceτ 2 ≈1/M) is: σ2 single =τ 2 1− 2 π = 1 M 1− 2 π .(44) The Standard Error of the mean overK≈D 2 terms is: SE= σsing...

  7. [19]

    Numerical Evaluation We apply this framework to our experimental observations (CNN on CIFAR-10) with the following parameters: • Matrix Dimension:D≈2560

    Calculation of Z-Statistic.The Z-score measures how many standard deviations the observed value deviates from the random baseline: Z= µobs −µ H0 SE .(46) D.3. Numerical Evaluation We apply this framework to our experimental observations (CNN on CIFAR-10) with the following par...

    2023

  8. [20]

    As the model approaches a global minimum (or a high-quality local minimum), the per-sample gradients for correctly classified examples tend to vanish (i.e., ∥∇ℓ∥ →0)

    Vanishing Gradients near Global Minima:Term II involves the gradient of the loss ∇ℓ, which acts as a scaling factor. As the model approaches a global minimum (or a high-quality local minimum), the per-sample gradients for correctly classified examples tend to vanish (i.e., ∥∇ℓ...

  9. [21]

    Empirical Verification:As demonstrated in our numerical experiments (referenced in Figures 4 to 6 ( f&g )), specifi- cally in the regime near the global minimum, the magnitude of Term I is observed to be orders of magnitude larger than that of Term II. The contribution of the ...

    2000

  10. [22]

    The Upper Bound ( 2B σ2w Cii ≤κ maxHii): Assume the per-sample Hessian eigenvalues are bounded by κmax, i.e., 0≤κ (p) m ≤κ max for all m, p. We can bound the second moment termB (p) i : B(p) i = X m κ(p) m ·κ (p) m (u(p) m ·v i)2 ≤κ max X m κ(p) m (u(p) m ·v i)2 =κ maxA(p) i ....

  11. [23]

    23 On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature

    Spectral Dominance:The per-sample Hessian spectrum is locally dominated by the top n eigenvalues with magnitude κ(p) m ≈¯κ(p) ≫κ rest form≤n. 23 On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature

  12. [24]

    This tiny fluctuations contribute to the bulk eigenvalues of Global Hessian which is the source of heavy-tailed spectrum

    Local Alignment:The per-sample dominant eigen-directions (u(p) m ·v i)2 are aligned across different samples p, but with tiny fluctuation so that (u(p) m ·v i)2 →δ mi for i≤n while (u(p) m ·v i)2 remains finite for i > n. This tiny fluctuations contribute to the bulk eigenvalu...

  13. [25]

    Statistical Homogeneous Magnitude: κ(p) m for m≤n are identically distributed (i.i.d.) variables drawn from a distribution characteristic of the loss basin. We define the first and second moments as characteristic constants: Ep[κ(p) m ] =µ κ,E p[(κ(p) m )2] =µ κ2 =µ 2 κ +σ 2 κ...

  14. [26]

    The Physical Prediction (RSM):The Random Shifts Model with isotropic sampling noise yields a noise covariance strictly quadratic in the Hessian eigenvalues: CRSM ii =σ 2(Hii)2 =σ 2λ2 i ,(106) whereλ i denotes the eigenvalues of the global Hessian

  15. [27]

    The Proposition 5.3:Under the alignment assumption from Section E.4.1, the noise covariance is determined by the second moment of the local curvatureκ (p) i : CSMR ii ∝E p[(κ(p) i )2].(107) Applying the bias-variance decompositionE[X 2] =Var(X) + (E[X]) 2 and noting thatE p[κ(...

  16. [28]

    Vanishing Gradients

    Establishing Equivalence:Matching the physical prediction with the statistical definition yields the constraint: σ2λ2 i ∝Var p(κ(p) i ) +λ 2 i .(109) 25 On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature Rearranging terms reveals the impl...

    1934

  17. [2016]

    URL https://proceedings.mlr

    PMLR. URL https://proceedings.mlr. press/v48/hardt16.html. Hochreiter, S. and Schmidhuber, J. Flat minima.Neural Computation, 9(1):1–42, 1997. doi: 10.1162/NECO.1997. 9.1.1. Hoffer, E., Hubara, I., and Soudry, D. Train longer, gen- eralize better: Closing the generalization ga...

    1997 arXiv

  18. [2017]

    cc/paper_files/paper/2017/file/ a96b65a721e561e1e3de768ac819ffbb-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2017/file/ a96b65a721e561e1e3de768ac819ffbb-Paper. pdf. Li, Z., Malladi, S., and Arora, S. On the validity of modeling sgd with stochastic differential equations (sdes), 2021. URLhttps://arxiv.org/abs/2102.12470. Liu, C., Z...

    2017 arXiv

  19. [2018]

    Sun, R., Li, D., Liang, S., Ding, T., and Srikant, R

    URL https://openreview.net/forum? id=r1q7n9gAb. Sun, R., Li, D., Liang, S., Ding, T., and Srikant, R. The Global Landscape of Neural Networks: An Overview.IEEE Signal Processing Magazine, 37(5): 95–108, September 2020. ISSN 1053-5888, 1558-0792. doi: 10.1109/MSP.2020.3004124. ...

    2020

  20. [2020]

    Jastrzebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y ., and Storkey, A

    URL https://openreview.net/forum? id=SkgscaNYPS. Jastrzebski, S., Kenton, Z., Arpit, D., Ballas, N., Fischer, A., Bengio, Y ., and Storkey, A. Three factors influencing minima in SGD. InInternational Conference on Artificial Neural Networks (ICANN), pp. 169–180. Springer, 2018...

    2018 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.