pith. sign in

arxiv: 2606.02596 · v1 · pith:KKK2ZMSOnew · submitted 2026-05-22 · 💻 cs.LG

Spectral Asymptotics of Neural Network Loss Landscapes: An Exact Decomposition of the Curvature Exponent

Anherutowa Calvo This is my paper

Pith reviewed 2026-06-30 15:35 UTC · model grok-4.3

classification 💻 cs.LG
keywords spectral asymptoticsneural network loss landscapesHessian eigenvaluesgradient singular valuescurvature exponentspectral alignmentpreconditioningHessian decay
0
0 comments X

The pith

The curvature exponent α in neural network Hessians equals 2 plus the log-log derivative of an alignment measure between Kronecker eigenbases and gradient singular directions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that the scaling exponent α relating Hessian eigenvalues h_k to gradient singular values σ_k decomposes exactly as α = 2 + d log Φ_k / d log σ_k, where Φ_k quantifies alignment of Kronecker factor eigenbases with gradient singular vectors. This turns the observed layer-type differences in α (near 2 for convolutions, near 1 for attention, below 1 for some MLPs) into a concrete geometric question about eigenbasis alignment, which the authors resolve for LayerNorm, residuals, and softmax. The same decomposition yields an algebraic spectral transfer identity s = α γ that links curvature exponent, gradient rank decay γ, and Hessian decay exponent s. When α and γ are obtained from independent measurements (Hessian-vector products versus SVD), the identity recovers observed s values to roughly 2 percent median error over 93 layers, five architectures, and three datasets with no adjustable parameters. A zeta-function bound on participation ratio further shows that curvature concentrates onto effectively one direction per layer.

Core claim

The central claim is the Spectral Alignment Decomposition α = 2 + d log Φ_k / d log σ_k, where Φ_k measures alignment between Kronecker factor eigenbases and gradient singular directions. This identity is algebraic and directly implies the transfer relation s = α γ. When α is fit from Hessian-vector products and γ from SVD on independent data, the product recovers the directly measured Hessian decay exponent s to about 2 percent median error across dozens of layers and multiple models with zero free parameters. The same geometry explains why α takes different characteristic values in convolutions, attention blocks, and MLP projections.

What carries the argument

The Spectral Alignment Decomposition, which writes the curvature exponent α exactly as 2 plus the logarithmic derivative of the alignment measure Φ_k between Kronecker eigenbases and gradient singular vectors.

If this is right

  • Layer-type differences in α reduce to differences in how Kronecker eigenbases align with gradient singular vectors for LayerNorm, residual connections, and softmax heads.
  • The algebraic identity s = α γ allows prediction of full Hessian spectra from gradient singular-value data alone.
  • Curvature per layer concentrates onto effectively one dominant direction, as bounded by the participation-ratio zeta function.
  • An architecture-dependent preconditioner T(σ; α) constructed in the gradient singular basis can be implemented as Spectral Newton and tested against AdamW on vision tasks where α is near 2.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If Φ_k can be approximated from cheaper statistics than full SVD, the decomposition could enable on-the-fly curvature estimates in very large models.
  • The same alignment geometry may appear in other non-neural optimization problems that possess Kronecker-structured Hessians.
  • Direct tests of the preconditioner on attention layers where α is near 1 would clarify whether performance gains are tied to the α ≈ 2 regime.
  • Random-matrix models of eigenbasis misalignment could predict the typical numerical values of the derivative term for different layer classes.

Load-bearing premise

The alignment measure Φ_k and the independent fits of α and γ draw from data with no hidden shared dependence on the same curvature information, so that the transfer identity s = α γ constitutes a genuine prediction rather than a tautology.

What would settle it

Compute α from Hessian-vector products and γ from SVD on the same set of trained networks, then check whether their product deviates by more than a few percent from directly measured Hessian eigenvalue decay exponents across multiple architectures and datasets.

Figures

Figures reproduced from arXiv: 2606.02596 by Anherutowa Calvo.

Figure 1
Figure 1. Figure 1: Exact HVP curvature hk vs. gradient singular value σk (log–log). (a–d) Representative layers: convolutions follow α ≈ 2; FC and transformer MLP layers deviate. Fit exponents shown in-panel; dashed grid omitted on log axes for clarity [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Per-layer curvature exponent α vs. depth index across ResNet-18, VGG-11, and GPT-2 (CIFAR-10). Shaded band: α ∈ [1.85, 2.15]. Interior convolutions cluster near α = 2; transformers and boundary layers deviate. the Hessian decay exponent is the product of a curvature–gradient alignment exponent α (which we trace to Kronecker factor eigenbases, Theorem 1) and an effective gradient rank-decay exponent γ. In s… view at source ↗
Figure 3
Figure 3. Figure 3: Measured Hessian decay s vs. predicted s = αγ across 93 layers and five architec￾tures/datasets. Points near the diagonal validate the spectral transfer identity; shaded band is ±10%. Color indicates data source. 6.5 Refined prediction: LayerNorm is necessary but not sufficient ViT-Tiny on CIFAR-10 has LayerNorm yet α ≈ 2. Controlled ablations (Appendix C) show that large output dimension creates sparse, a… view at source ↗
Figure 4
Figure 4. Figure 4: Gradient singular value rank profiles on ResNet-18 with BIC-selected models. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Kronecker factorization quality vs. |α − 2| on ResNet-18 conv layers (+ FC control). (a) Deviation of concentration ratio from 1 (marker size ∝ accuracy). (b) R2 of hk ∝ σ α k ; dotted line at 0.97 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Fraction of VGG-16 layers whose top-k gradient spectrum is best fit by each BIC model vs. training epoch (Tiny-ImageNet). Log-normal dominance at init gives way to power-law and exponential wins—the spectral phase transition. Reproducibility Statement All empirical claims are reproduced from scripts that measure exact Hessian-vector products per layer. The full reproducibility package is available at https… view at source ↗
read the original abstract

The curvature exponent $\alpha$ in $h_k \propto \sigma_k^\alpha$ -- governing how Hessian eigenvalues scale with gradient singular values -- varies systematically across layer types ($\alpha \approx 2$ for convolutions, $\approx 1$ for transformer attention, $< 1$ for MLP up-projections). Why? We prove the Spectral Alignment Decomposition: $\alpha = 2 + d\log\Phi_k / d\log\sigma_k$, where $\Phi_k$ measures alignment between Kronecker factor eigenbases and gradient singular directions. This reduces "why does $\alpha$ vary?" to a geometric question we answer for LayerNorm, residual connections, and softmax heads. The decomposition implies a spectral transfer identity $s = \alpha\gamma$ linking curvature exponent, effective gradient rank-decay $\gamma$, and Hessian decay exponent $s$. The identity is algebraic; its empirical content is that $\alpha$ and $\gamma$, fit on independent data (HVPs vs. SVD), recover $s$ to ~2% median error across 93 layers, five architectures, and three datasets -- with no free parameters. A zeta-function bound on participation ratio shows curvature concentrates onto effectively one direction per layer. As a proof of concept, we derive the architecture-adaptive preconditioner $T(\sigma;\alpha)$ and show that Spectral Newton -- implementing $T$ in the gradient singular basis -- outperforms AdamW on vision benchmarks where $\alpha \approx 2$.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims to prove the Spectral Alignment Decomposition α = 2 + dlogΦ_k / dlogσ_k, where Φ_k measures alignment between Kronecker factor eigenbases and gradient singular directions. This explains layer-type variation in the curvature exponent α (≈2 for convolutions, ≈1 for attention, <1 for MLP up-projections). The decomposition yields the algebraic spectral transfer identity s = αγ linking curvature exponent α, gradient rank-decay γ, and Hessian decay exponent s. Empirically, α (fit via HVPs) and γ (fit via SVD) on independent data recover s to ~2% median error across 93 layers, five architectures, and three datasets with no free parameters. Additional results include a zeta-function bound on participation ratio and a Spectral Newton preconditioner T(σ;α) outperforming AdamW on vision tasks where α≈2.

Significance. If the algebraic decomposition and independence of measurements hold, the work supplies a geometric account of why α varies systematically by architecture component and demonstrates that the transfer identity functions as a non-trivial, parameter-free prediction rather than a tautology. The explicit handling of LayerNorm, residuals, and softmax heads, combined with the empirical recovery across diverse models, would constitute a substantive advance in understanding neural loss landscape spectra. The proof-of-concept optimizer further indicates downstream utility.

major comments (2)
  1. [Abstract] Abstract: The central empirical claim—that α (HVP) and γ (SVD) are measured on independent data so that s = αγ recovers the directly measured Hessian exponent s to ~2% median error—requires an explicit demonstration that no shared curvature information (mini-batches, activations, or random seeds) enters both the HVP fits for α and the SVD/spectrum measurements for s and γ. Without this check the reported recovery risks being a consistency verification rather than an out-of-sample test of the decomposition.
  2. [Proof of decomposition] Proof section (decomposition): The manuscript asserts an algebraic proof of α = 2 + dlogΦ_k / dlogσ_k but the abstract supplies neither the full derivation steps nor the explicit definition of the alignment measure Φ_k. These must be provided in sufficient detail to confirm that the identity does not embed hidden dependencies that would render the subsequent transfer identity s = αγ tautological by construction.
minor comments (1)
  1. [Abstract] Notation for Φ_k and the logarithmic derivatives should be introduced with a self-contained definition before the decomposition is stated, to aid readability for readers outside the immediate subfield.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The two major comments identify areas where additional explicitness will strengthen the manuscript. We address each below and have prepared revisions to incorporate the requested clarifications.

read point-by-point responses

  1. Referee: [Abstract] The central empirical claim—that α (HVP) and γ (SVD) are measured on independent data so that s = αγ recovers the directly measured Hessian exponent s to ~2% median error—requires an explicit demonstration that no shared curvature information (mini-batches, activations, or random seeds) enters both the HVP fits for α and the SVD/spectrum measurements for s and γ. Without this check the reported recovery risks being a consistency verification rather than an out-of-sample test of the decomposition.

    Authors: We agree that an explicit protocol is required to establish out-of-sample status. The original manuscript states that α and γ are obtained from HVPs versus SVD on independent data, but does not detail the separation. In the revision we add Appendix B, which specifies: (i) disjoint mini-batches drawn from the same training distribution but with distinct random seeds, (ii) no shared activations or forward passes between the two computations, and (iii) separate Hessian-vector products performed only on the HVP subset. With this addition the recovery test is unambiguously out-of-sample. revision: yes

  2. Referee: [Proof of decomposition] Proof section (decomposition): The manuscript asserts an algebraic proof of α = 2 + dlogΦ_k / dlogσ_k but the abstract supplies neither the full derivation steps nor the explicit definition of the alignment measure Φ_k. These must be provided in sufficient detail to confirm that the identity does not embed hidden dependencies that would render the subsequent transfer identity s = αγ tautological by construction.

    Authors: The full algebraic derivation appears in Section 3, beginning from the Kronecker factorization of the layer Hessian and the definition Φ_k := |U_k^T V_k| where U_k and V_k are the eigenbases of the Kronecker factors and the gradient singular vectors, respectively. The step α = 2 + dlogΦ_k / dlogσ_k follows directly from the chain rule applied to the eigenvalue scaling without presupposing the transfer identity. To address the referee’s concern we will (a) insert the explicit definition of Φ_k into the abstract and (b) add a one-paragraph outline of the derivation immediately after the abstract. These changes make the logical independence of the decomposition from s = αγ transparent. revision: yes

Circularity Check

0 steps flagged

No significant circularity; algebraic identity with independent empirical validation

full rationale

The paper states the Spectral Alignment Decomposition as a proven algebraic result α = 2 + dlogΦ_k / dlogσ_k from which the transfer identity s = αγ follows directly. It explicitly describes the identity as algebraic and positions the ~2% recovery of s as an empirical test using α (via HVPs) and γ (via SVD) on independent data with no free parameters. No quoted equations or steps reduce a claimed prediction to a fitted input by construction, nor rely on self-citation chains or ansatzes smuggled via prior work. The derivation chain is self-contained against the stated independent measurements.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on standard linear-algebra properties of Kronecker factorizations and singular-value decompositions together with the newly introduced alignment measure Φ_k; no free parameters are reported.

axioms (1)
  • standard math Hessian and gradient matrices in neural networks admit Kronecker factorizations whose eigenbases and singular vectors are well-defined and comparable.
    Invoked to define the alignment Φ_k that appears in the decomposition of α.
invented entities (1)
  • Φ_k no independent evidence
    purpose: Quantifies alignment between Kronecker factor eigenbases and gradient singular directions to decompose the curvature exponent.
    New geometric quantity introduced to convert the 'why α varies' question into a measurable alignment problem.

pith-pipeline@v0.9.1-grok · 5795 in / 1685 out tokens · 73258 ms · 2026-06-30T15:35:50.363580+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [1]

    Investigating the Overlooked Hessian Structure: From CNNs to LLMs,

    Q.-Y . Tang et al., “Investigating the Overlooked Hessian Structure: From CNNs to LLMs,” ICML, 2025

    2025

  2. [2]

    Implicit Self-Regularization in Deep Neural Networks,

    C. H. Martin, M. W. Mahoney, “Implicit Self-Regularization in Deep Neural Networks,” JMLR, 2021

    2021

  3. [3]

    Optimizing Neural Networks with Kronecker-factored Approximate Curvature,

    J. Martens, R. Grosse, “Optimizing Neural Networks with Kronecker-factored Approximate Curvature,” ICML, 2015

    2015

  4. [4]

    A Kronecker-factored approximate Fisher matrix for convolution layers,

    R. Grosse, J. Martens, “A Kronecker-factored approximate Fisher matrix for convolution layers,” ICML, 2016

    2016

  5. [5]

    Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks,

    Y . Wu et al., “Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks,” arXiv:2010.04261, 2022

    work page arXiv 2010

  6. [6]

    Limit of the smallest eigenvalue of a large dimensional sample covariance matrix,

    Z. Bai, Y . Yin, “Limit of the smallest eigenvalue of a large dimensional sample covariance matrix,” Ann. Probab., 1993. 9

    1993

  7. [7]

    Products of Many Large Random Matrices and Gradients in Deep Neural Networks,

    B. Hanin, M. Nica, “Products of Many Large Random Matrices and Gradients in Deep Neural Networks,” arXiv:1812.05994, 2018

    work page arXiv 2018

  8. [8]

    Muon: An optimizer for hidden layers in neural networks,

    K. Jordan, “Muon: An optimizer for hidden layers in neural networks,” GitHub, 2024

    2024

  9. [9]

    Sharpness-aware minimization,

    P. Foret et al., “Sharpness-aware minimization,” ICLR, 2021

    2021

  10. [10]

    Old optimizer, new norm: An anthology,

    J. Bernstein, “Old optimizer, new norm: An anthology,” arXiv, 2024

    2024

  11. [11]

    Analysis of the limiting spectral measure of large random matrices of the separable covariance type,

    R. Couillet, W. Hachem, “Analysis of the limiting spectral measure of large random matrices of the separable covariance type,” Random Matrices: Theory Appl., 2014

    2014

  12. [12]

    Cracking the Hessian: Closed-Form Hessian Spectra for Fundamental Neural Networks,

    S. P. Singh, W. Ormaniec, T. Hofmann, “Cracking the Hessian: Closed-Form Hessian Spectra for Fundamental Neural Networks,” under review, 2026

    2026

  13. [13]

    On the Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning,

    Y . Zhang et al., “On the Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning,” ICML, 2025

    2025

  14. [14]

    Shampoo: Preconditioned stochastic tensor optimization,

    V . Gupta et al., “Shampoo: Preconditioned stochastic tensor optimization,” ICML, 2018

    2018

  15. [15]

    ESO: Efficient Spectral Preconditioner for Second-Order Optimization,

    Y . Chen, J. He, Z. Zheng, J. Lavaei, “ESO: Efficient Spectral Preconditioner for Second-Order Optimization,” 2026. A HVP Validation Table LayerR 2(h, σ2)R 2(hGN, σ2)R 2(h, hGN) conv1 0.999 0.925 0.929 layer2.0.conv1 0.998 0.998 0.997 layer4.0.conv2 0.995 0.995 0.998 fc (10 classes) 0.694 0.869 0.786 Median (21 layers) 0.998 0.992 0.996 B Participation Ra...

    2026