Spectral-factorized Positive-definite Curvature Learning for NN Training

Felix Dangel; Juhan Bae; Richard E. Turner; Roger B. Grosse; Runa Eschenhagen; Wu Lin

REVIEW 4 major objections 5 minor 72 references

Spectral-factorized Positive-definite Curvature Learning for NN Training

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes that a curvature matrix can be learned in spectral-factorized form while matching the trajectory of full-matrix AdaGrad and RMSprop to first order, so matrix roots become scalar operations.

desk verdict Genuinely useful spectral-factorized optimizer with a real but patchable gap in its central equivalence at isotropic initialization. read the letter →

arxiv 2502.06268 v3 pith:7FSFJDXK submitted 2025-02-10 stat.ML cs.LG

Wu Lin , Felix Dangel , Runa Eschenhagen , Juhan Bae , Richard E. Turner , Roger B. Grosse This is my paper · ORCID

classification stat.MLcs.LG MSC 65K1068T07

keywords spectralfactorizationpositive-definitecurvaturelearningadaptiveoptimizerRiemanniangradientdescentmatrixrootswithoutdecompositionlow-precisiontrainingKronecker-factoredpreconditionerAdaGradandRMSprop

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a way to learn a positive-definite curvature matrix for neural-network training without ever computing a matrix square root. It keeps the curvature as a spectral factorization $S = B\mathrm{Diag}(d)B^\top$, with $B$ orthogonal and $d$ positive, and updates $B$ and $d$ directly through elementwise operations and the Cayley map. The central claim is that this factored update is equivalent, up to first order in $\beta_2$, to the standard full-matrix update $S \leftarrow (1-\beta_2\gamma)S + \beta_2 g g^\top$ whenever $d$ has no repeated entries. If true, preconditioned training gains arbitrary fractional roots on $S$ at scalar cost, generic curvature information, and numerical stability in half precision, because roots and inverses act on the scalar eigenvalues $d_i$ rather than on the whole matrix.

What carries the argument

The central object is the spectral parameterization $S = B\,\mathrm{Diag}(d)B^\top$ with $B$ orthogonal and $d$ positive, learned through Riemannian gradient descent in a local coordinate system that diagonalizes the Fisher-Rao metric at the origin (Claims 4 and 6). The update of $B$ uses the Cayley map $\mathrm{Cayley}(N) = (I+N)(I-N)^{-1}$ with a skew-symmetric generator $N = \mathrm{Skew}(\mathrm{Tril}(U))$; U's entries $[U]_{ij} = -[B^\top g g^\top B]_{ij}/(d_i-d_j)$ are the ratio identity that makes the factored update cancel to the full-matrix update to first order. In the Kronecker case, the factorization $S^{(C)} \otimes S^{(K)}$ is made unique by $\det(\mathrm{Diag}(d^{(l)})) = 1$ and a learnable scale $\alpha$, which removes the metric singularity caused by Kronecker ambiguity.

What would settle it

Choose a fixed sequence of gradients and initialize $S_0$ with no repeated eigenvalues; run the default full-matrix update and the spectral update with the same $\beta_2$, and plot $\|B_k\mathrm{Diag}(d_k)B_k^\top - S_k^{\mathrm{default}}\|_{F}$ as a function of $\beta_2$. If Claim 2 holds, this residual should remain $O(\beta_2^2)$; a visible $O(\beta_2)$ gap at small $\beta_2$ would disprove it. A second probe is to deliberately initialize with two equal eigenvalues, let the updates run, and observe whether the equality of the resulting $S$ trajectories is also maintained outside the stated regime.

Watch

Extended reading notes

Core claim

The paper discovers that the spectral-factorized update in its Fig. 1 --- $d \leftarrow d \odot \exp\{\beta_2 d^{-1} \odot [-\gamma d + \mathrm{diag}(B^\top g g^\top B)]\}$ and $B \leftarrow B\,\mathrm{Cayley}(\beta_2/2\,\mathrm{Skew}(\mathrm{Tril}(U)))$, with $[U]_{ij} = -[B^\top g g^\top B]_{ij}/(d_i-d_j)$ when $d_i \neq d_j$ and $0$ otherwise --- reproduces the classical full-matrix update $S \leftarrow (1-\beta_2\gamma)S + \beta_2 g g^\top$ up to first order in $\beta_2$. This means the factored trajectory follows the same curvature path as full-matrix AdaGrad or RMSprop while maintaining $S$ in factored form, so a fractional root $S^{-1/p} = B\,\mathrm{Diag}(d^{-1/p})B^\top$ is computed by elementwise powers on $d$. The same construction is extended to Kronecker-structured curvature $S = \alpha(S^{(C)} \otimes S^{(K)})$, where a determinant constraint on each factor plus the learnable scalar $\alpha$ makes the representation unique and allows a block-diagonal metric inversion. Experiments on SPD matrix optimization, gradient-free evolution strategies, and half-precision vision-transformer training support the claimed flexibility.

Load-bearing premise

The load-bearing premise of the central equivalence claim is that the eigenvalues in $d$ are all distinct; when $d_i = d_j$ or they are close enough to make the division in $[U]_{ij}$ numerically unstable, the proof's cancellation stops, the update of $B$ freezes for those directions, and the claimed first-order match to full-matrix AdaGrad is no longer guaranteed.

Editorial extensions

If this is right

Any fractional root $p$ can be applied to the preconditioner at the cost of elementwise powers on $d$, removing the matrix-decomposition bottleneck that restricts Shampoo-style methods to fixed roots.
Because no eigendecomposition is involved, the whole update can run in half precision; the paper demonstrates vision-transformer training in BFP16 where SOAP must fall back to FP32.
In the diagonal limit the spectral scheme reduces to AdaGrad/RMSprop with $p=2$ and to fractional diagonal methods with $p=4$, so it generalizes those widely used updates rather than replacing them.
The Kronecker version keeps memory low, one small factor per matrix dimension, while matching the curvature estimates of an impractical full-matrix projection method in the paper's fixed-point and iterate-matching tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the scalable bottleneck for a practical spectral optimizer is the eigenvalue-gap regime; if trained networks drive $d_i$ close together, the Moore-Penrose freezing of $B$ effectively switches the optimizer to a block-diagonal preconditioner, so monitoring $\min_{i\neq j}|d_i-d_j|$ during training is the cheapest diagnostic for when the equivalence guarantee degrades.
Editorial inference: because the derivation only needs any positive-semidefinite curvature oracle in place of $g g^\top$, the same factored update should carry over to Hessian, Fisher, and Gauss-Newton curvature estimates; a direct test is to feed a full Hessian-vector product and compare with standard Newton-like Riemannian gradient descent in low precision.
Editorial inference: the truncated Cayley map with a Neumann series opens a continuous accuracy-cost trade-off, suggesting that the truncation order can be tuned per layer; one could test whether order-2 truncation changes ImageNet-level training curves relative to the exact Cayley map.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. First, the paper delivers something real: a way to maintain a spectral factorization B Diag(d) B^T of a preconditioner without ever computing a matrix root or eigendecomposition, so arbitrary fractional roots and half-precision arithmetic become cheap. The Kronecker extension with determinant-normalized factors is a clean answer to the non-uniqueness ambiguity in Kronecker products. Second, the central equivalence between their update and full-matrix RMSprop/AdaGrad is only guaranteed when d has no repeated eigenvalues, and the standard isotropic initialization S0 = I violates exactly that. At the first update, the B-rotation is frozen, the updated S stays diagonal, and the deviation from the default scheme is beta2(H - diag(H)), which is order beta2, not beta2^2. That is not a fatal flaw — the method still trains well — but the paper needs to own this regime rather than leaving it as an unanalyzed edge case.

What is good: the proof of Claim 2 for distinct d is detailed, and the synthetic iterate-matching experiments line up with it. The Kronecker uniqueness result (Claim 5) is simple and correct. The low-precision ViT plots show the promised wall-clock advantage over SOAP and parity with Muon; the truncated Cayley comparison in Fig 10 is a useful sanity check.

Soft spots, in proportion: (1) the degenerate-eigenvalue gap above, plus the practical 'near zero' threshold in Fig 1 which has no error bound; (2) Claims 4 and 6 are asserted with closed-form expressions but the derivations are not actually carried out in the appendix, so a reader must trust a lot of algebra; (3) no multiple seeds or error bars in Fig 4, which makes the comparison look thin; (4) no code release at the time of writing. These are conditions for acceptance, not reasons to reject.

For optimizer researchers this is worth a serious referee: the central idea is likely correct, the empirical evidence is suggestive, and the missing analysis is patchable. I would accept for peer review with a clear request to address the isotropic initialization, give the metric derivation, and report seeds.

Referee Report

4 major / 5 minor

Summary. The paper proposes a spectral-factorized parameterization of SPD curvature matrices, S = B Diag(d) B^T, with on-the-fly Riemannian updates of the factors d and B, and extends the idea to Kronecker-structured preconditioners by introducing a scalar α and determinant constraints. The central theoretical claim is that the spectral update is equivalent to the full-matrix AdaGrad/RMSprop update S ← (1−β2γ)S + β2 gg^T up to first order in β2 when d has distinct entries (Claim 2, Appendix G). The paper also provides local-coordinate derivations for the Fisher-Rao metric (Claims 4 and 6), empirical validation on synthetic fixed-point/iterate matching, SPD matrix optimization, gradient-free optimization, and half-precision vision-transformer training comparing against AdamW, SOAP, and Muon.

Significance. If the equivalence and numerical-stability claims hold, the contribution is valuable: it offers a decomposition-free way to apply arbitrary fractional roots and to train with low-precision arithmetic, which is a real bottleneck for Shampoo-style methods. The algebraic proof of Claim 2 is detailed and the synthetic matching experiments are an appropriate check of the claimed equivalence. The Kronecker determinant-constraint factorization (Claim 5) is a useful device for resolving the ambiguity of Kronecker representations. However, the significance is tempered by the fact that the central equivalence is proved only for non-repeated eigenvalues, while the natural isotropic initialization S0=I is the excluded case, and the practical scheme further zeroes entries near degeneracy with an unquantified threshold. The NN experiments also lack seed variance, so the headline empirical claim is not yet fully supported.

major comments (4)

[Section 3.1, Claim 2, Appendices E and G] The first-order equivalence to Eq. (3) is proved only under the assumption that d has no repeated entries, but the natural spectral initialization S0=I corresponds to B0=I and d0=(1,...,1). At the first iteration with H=gg^T, the B update freezes because U=0, the spectral update produces a diagonal matrix, and the deviation from the default scheme equals β2(H−Diag(diag(H))), which is O(β2), not O(β2^2). The manuscript does not state the initialization used in the NN experiments and does not analyze this degenerate regime despite introducing a Moore-Penrose rule in Appendix E. This is load-bearing because the paper's central claim is that the spectral scheme tracks the default curvature trajectory.
[Figure 1 caption and Section 3.4] The practical scheme sets [U]_ij=0 whenever |d_i−d_j| is 'near 0' for numerical stability, but no threshold is specified and no bound is given on the resulting error relative to Eq. (3). In near-degenerate situations this heuristic reintroduces an uncontrolled O(β2) error, so the algorithm that is actually run in the experiments is not exactly the one covered by Claim 2.
[Appendices I and K] Claim 4 (closed-form metric diagonalization) and Claim 6 (closed-form metric block diagonalization) are central to the derivation of the update rules, but they are asserted rather than proved. Appendix I simply states the closed form and says 'we can analytically compute', and Appendix K gives the block structure without deriving the inverse F_mm^{-1}. Since the RGD simplification in Appendix L relies on these claims, the derivation of the update schemes is incomplete.
[Section 4.4, Figures 4, 10-12] The headline empirical claim that the proposed method matches Muon in half-precision ViT training is based on single-run curves without error bars, confidence intervals, or multiple seeds. The exact-versus-truncated Cayley comparison in Figure 10 is also reported on only one dataset and without seed variance. The random hyperparameter search is useful, but it does not substitute for reporting run-to-run variability of the final training curves.

minor comments (5)

[Section 3.1] There is a typo: 'Notatbly' should be 'Notably'.
[Appendix G] The induction hypothesis in the proof of Claim 2 writes ar S_k = S_k + O(β2), but the conclusion requires the stronger ar S_k = S_k + O(β2^2) to propagate the first-order equivalence; this should be corrected for clarity.
[Appendix I] The heading says 'Proof of Claim 4' but the statement inside is numbered 'Claim 5'; the claim numbering should be made consistent throughout the appendix.
[Appendix J] In the uniqueness proof, the step from I = ((S(C)_1)^{-1}S(C)_2) ⊗ ((S(K)_1)^{-1}S(K)_2) to each factor being the identity should mention the general scalar ambiguity A=cI, B=c^{-1}I and explain that the determinant constraints force c=1; as written the argument is incomplete.
[Section 4.4] The notation for half precision is inconsistent: the text uses both BFP-16 and BFP16; please standardize.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Claim 2 is a proven algebraic equivalence, and the method's components are derived from RGD rather than fitted to the target update.

full rationale

The paper's central claim (Claim 2, Sec. 3.1, proof in Appx. G) asserts that the spectral update d <- d ⊙ exp{β2 d^{-1} ⊙ [-γd + diag(B^T H B)]}, B <- B Cayley(β2/2 Skew(Tril(U))) with U_ij = -[B^T H B]_ij/(d_i-d_j), matches S <- (1-β2γ)S + β2 gg^T up to O(β2^2) when d has no repeated entries. This is not circular: the proof expands exp and Cayley to first order and uses the definition of U to obtain the cancellation ND - DN = Q - Diag(diag(Q)) (Observation 2), which is an algebraic identity derived from the update rule, not an assumption of the conclusion. The U update itself is obtained in Appx. L as the natural gradient [F^{-1} g_M]_ij = (B^T g_{S^-1} B)_ij/(d_i-d_j), i.e., from the inverse Fisher-Rao metric in local coordinates, not by fitting to AdaGrad/RMSprop. The paper's citations to Lin et al. (2024) and other previous work supply the Gaussian variational reformulation and GOP Hessian approximation, but those are background assumptions also supported by Opper & Archambeau (2009) and Duchi et al. (2011); the paper's contribution—spectral factorization updated without matrix decomposition—does not reduce to those citations. No fitted parameter is renamed as a prediction; the empirical comparisons use externally generated gradient sequences. The repeated-eigenvalue restriction in Claim 2 and the Moore-Penrose handling in Appx. E are a robustness/correctness caveat (e.g., the isotropic initialization S0=I is outside the claim's stated hypothesis), but this is not circularity: the paper explicitly states the condition rather than smuggling in the conclusion. Hence a non-finding is appropriate.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. It relies on standard Gaussian information geometry and a novel uniqueness normalization for Kronecker factors. The main free parameters are standard optimizer hyperparameters, tuned per experiment; the theoretical equivalence claim is parameter-free except for the assumption of non-repeated eigenvalues.

free parameters (6)

β2 (curvature learning rate) = tuned via random search
Controls the update size of d and B; appears in the first-order equivalence, which assumes small β2.
β1 (step size) = tuned via random search
Learning rate for the weight update in Fig 1.
λ (damping term) = adaptive formula λ k(l) mean(d^{(C)})^{-1} mean(d^{(K)})^{-1} / mean(d^{(l)})^{-1}
Introduced to keep d positive in truncated updates (Fig 6); not discussed in the theoretical equivalence.
p (fractional root) = 1, 2, 4 in experiments
Choice of root; the method supports arbitrary p, but the experiments only test p=1 and p=2 for the proposed method.
γ = 0 or 1
Selects AdaGrad (γ=0) or RMSprop (γ=1) behavior.
β̄2 (scale for truncated Cayley step) = tuned via random search
Used only in the truncated Cayley map to satisfy Neumann series convergence; introduces a gradient-dependent step size that breaks the exact first-order equivalence.

assumptions (7)

domain assumption The original loss can be reformulated as a variational Gaussian problem (Eq 4) with entropy regularizer.
This is the starting point of the RGD derivation, inherited from Lin et al. 2024.
domain assumption The gradient outer product H = g g^T is used as a Hessian approximation (Eq 7).
This approximation is used to obtain the concrete update rules; if it fails, the method's curvature estimate may not match the Hessian.
standard math Stein's estimator and the delta evaluation at the mean are valid (Eq 7).
These are standard identities used to simplify the RGD update.
standard math The Cayley map properties (orthogonality, injectivity) hold (Claim 9).
Used for B updates and proved in Appx L.
domain assumption The Fisher-Rao metric at the origin is diagonal with the claimed closed form (Claim 4).
The simplification of RGD relies on this; the appendix only sketches the computation.
ad hoc to paper A Kronecker SPD matrix has a unique representation with det factors = 1 and α > 0 (Claim 5).
This normalization is introduced in this paper to handle the non-uniqueness of Kronecker factorization.
domain assumption The Neumann series for the Cayley inverse converges when ||βN||_Frob < 1, and the nonconstant step keeps it in this regime.
Required for the truncated Cayley map used in low-precision training (Sec 3.4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Spectral-factorized Positive-definite Curvature Learning for NN Training." pith.science (2026). https://pith.science/paper/7FSFJDXK

@misc{pith2026250206268,
  author       = {Pith},
  title        = {Pith review of: Spectral-factorized Positive-definite Curvature Learning for NN Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7FSFJDXK}},
  note         = {Machine review of arXiv:2502.06268}
}

read the original abstract

Many training methods, such as Adam(W) and Shampoo, learn a positive-definite curvature matrix and apply an inverse root before preconditioning. Recently, non-diagonal training methods, such as Shampoo, have gained significant attention; however, they remain computationally inefficient and are limited to specific types of curvature information due to the costly matrix root computation via matrix decomposition. To address this, we propose a Riemannian optimization approach that dynamically adapts spectral-factorized positive-definite curvature estimates, enabling the efficient application of arbitrary matrix roots and generic curvature learning. We demonstrate the efficacy and versatility of our approach in positive-definite matrix optimization and covariance adaptation for gradient-free optimization, as well as its efficiency in curvature learning for neural net training.

Figures

Figures reproduced from arXiv: 2502.06268 by the authors.

**Figure 2.** Empirical validation of our update schemes for SPD curvature learning. Full-matrix Scheme: The first plot on the left shows that the scheme converges to a fixed-point solution as fast as the default scheme in Eq. (3) (with γ = 1) to update S ∈ R 100×100 and the Cholesky-based scheme. The second plot illustrates how closely our scheme matches the iterates generated by the default update scheme at each iteration. Kron… view at source ↗

**Figure 3.** Experiments showcase the efficacy and versatility of our approach for generic curvature learning. Our update scheme matches the equivalent Riemannian baselines, empirically illustrating the reparametrization invariance. SPD Matrix Optimization: The first two plots on the left show the performance of our full-matrix update scheme for learning SPD matrices. Our update scheme matches the baselines, as our scheme is RGD… view at source ↗

**Figure 4.** Experiments demonstrate the efficiency of our update schemes for low-precision NN training. The plots show the performance of our Kronecker-based scheme for training vision transformers with half precision. All models are trained for 210 epochs, including 10 epochs for warmup. For SOAP and our method, we update their preconditioners every two iterations. SOAP performs much slower than the other methods because it ha… view at source ↗

Figures from the paper (8 more)

**Figure 5.** Figure 5: Comparison between the original full-matrix update scheme (e.g., full-matrix RMSprop when γ = 1) and our update scheme when the exponential map is truncated. We can see that our update scheme is a decomposition-free version of the adaptive method. Here, we can use the …

**Figure 6.** Figure 6: Comparison between the Kronecker-based update scheme (Lin et al., 2024) with p = 1 and our update scheme when the exponential map is truncated. We define Q(C) := G(S (K) ) −1GT , Q(K) := GT (S (C) ) −1G, and W(l) = (B (l) ) T Q(l)B (l) , where S (l) := B (l)Diag(d (l)…

**Figure 7.** Figure 7: Empirical validation of our full-matrix update scheme on estimating preconditioning matrix S ∈ R 100×100. The first two figures on the left show that our update scheme converges to a fixed-point solution as fast as the default update scheme in S and the Cholesky-based …

**Figure 8.** Figure 8: Empirical validation of our structured update scheme on estimating a preconditioner S using a Kronecker product S (C) ⊗ S (K) , where S (C) ∈ R 9×9 and S (K) ∈ R 11×11. The first two figures on the left show that our update scheme gives a structural approximation of a …

**Figure 9.** Figure 9: Experiments demonstrate the efficacy of our update schemes for gradient-free optimization problems defined at Appx. C. These problems represent diverse optimization settings, including multimodality and narrow curvature. We can see that our method performs similarly to…

**Figure 10.** Figure 10: Comparison of our Kronecker-based update schemes with p = 2 (see the bottom box of [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

**Figure 11.** Figure 11: Comparison of the methods on the ImageWoof-10 dataset in terms of iteration efficiency and wall-clock time. These plots demonstrate the efficiency of our update schemes for low-precision NN training compared to state-of-the-art methods such as Muon and SOAP. D. Additi…

**Figure 12.** Figure 12: Comparison of the methods on the ImageNet-25 dataset in terms of iteration efficiency and wall-clock time. These plots demonstrate the efficiency of our update schemes for low-precision NN training compared to state-of-the-art methods such as Muon and SOAP. allow d to…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

72 extracted references · 46 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Optimization algorithms on matrix manifolds

Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization algorithms on matrix manifolds. Princeton University Press, 2009

work page 2009
[3]

Efficient full-matrix adaptive regularization

Agarwal, N., Bullins, B., Chen, X., Hazan, E., Singh, K., Zhang, C., and Zhang, Y. Efficient full-matrix adaptive regularization. In International Conference on Machine Learning, pp.\ 102--110. PMLR, 2019

work page 2019
[4]

Learning rate grafting: Transferability of optimizer tuning

Agarwal, N., Anil, R., Hazan, E., Koren, T., and Zhang, C. Learning rate grafting: Transferability of optimizer tuning. 2021

work page 2021
[5]

Natural gradient works efficiently in learning

Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

1998
[6]

Information geometry and its applications, volume 194

Amari, S.-i. Information geometry and its applications, volume 194. Springer, 2016

work page 2016
[7]

Scalable second order optimization for deep learning

Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

arXiv 2002
[8]

and Newhouse, L

Bernstein, J. and Newhouse, L. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024

arXiv 2024

Show all 72 references

[9]

Better plain vit baselines for imagenet-1k

Beyer, L., Zhai, X., and Kolesnikov, A. Better plain vit baselines for imagenet-1k. arXiv preprint arXiv:2205.01580, 2022

2022 arXiv
[10]

Manopt, a matlab toolbox for optimization on manifolds

Boumal, N., Mishra, B., Absil, P.-A., and Sepulchre, R. Manopt, a matlab toolbox for optimization on manifolds. The Journal of Machine Learning Research, 15 0 (1): 0 1455--1459, 2014

2014
[11]

S., Sra, S., and Tropp, J

Brickell, J., Dhillon, I. S., Sra, S., and Tropp, J. A. The metric nearness problem. SIAM Journal on Matrix Analysis and Applications, 30 0 (1): 0 375--396, 2008

2008
[12]

Closing the generalization gap of adaptive gradient methods in training deep neural networks

Chen, J., Zhou, D., Tang, Y., Yang, Z., Cao, Y., and Gu, Q. Closing the generalization gap of adaptive gradient methods in training deep neural networks. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, ...

2021
[13]

Symbolic discovery of optimization algorithms

Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675, 2023

2023 arXiv
[14]

Z., Huang, J., Reich, S., and Stuart, A

Chen, Y., Huang, D. Z., Huang, J., Reich, S., and Stuart, A. M. Efficient, multimodal, and derivative-free bayesian inference with fisher--rao gradient flows. Inverse Problems, 40 0 (12): 0 125001, 2024

2024
[15]

J., Nado, Z., Lee, J., Maddison, C

Choi, D., Shallue, C. J., Nado, Z., Lee, J., Maddison, C. J., and Dahl, G. E. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446, 2019

1910 arXiv
[16]

and Mehta, H

Cutkosky, A. and Mehta, H. Momentum improves normalized sgd. In International conference on machine learning, pp.\ 2260--2268. PMLR, 2020

2020
[17]

Adaptive subgradient methods for online learning and stochastic optimization

Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12 0 (7), 2011

2011
[18]

Proposal of distance-weighted exponential natural evolution strategies

Fukushima, N., Nagata, Y., Kobayashi, S., and Ono, I. Proposal of distance-weighted exponential natural evolution strategies. In 2011 IEEE Congress of Evolutionary Computation (CEC), pp.\ 164--171. IEEE, 2011

2011
[19]

Exponential natural evolution strategies

Glasmachers, T., Schaul, T., Yi, S., Wierstra, D., and Schmidhuber, J. Exponential natural evolution strategies. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp.\ 393--400, 2010

2010
[20]

Natural gradient variational bayes without fisher matrix analytic calculation and its inversion

Godichon, Baggioni, A., Nguyen, D., and Tran, M.-N. Natural gradient variational bayes without fisher matrix analytic calculation and its inversion. Journal of the American Statistical Association, pp.\ 1--12, 2024

2024
[21]

Shampoo: Preconditioned Stochastic Tensor Optimization

Gupta, V., Koren, T., and Singer, Y. Shampoo: Preconditioned Stochastic Tensor Optimization . In Proceedings of the 35th International Conference on Machine Learning, pp.\ 1842--1850, 2018

2018
[22]

K., and Gao, J

Han, A., Mishra, B., Jawanpuria, P. K., and Gao, J. On riemannian optimization over positive definite matrices with the bures-wasserstein geometry. Advances in Neural Information Processing Systems, 34: 0 8940--8953, 2021

2021
[23]

Ivey, T. A. and Landsberg, J. M. Cartan for beginners: differential geometry via moving frames and exterior differential systems, volume 61. American Mathematical Society Providence, RI, 2003

2003
[24]

NanoGPT (124M) quality in 8.2 minutes

Jordan, K. NanoGPT (124M) quality in 8.2 minutes . https://github.com/KellerJordan/modded-nanogpt, 2024

2024
[25]

Khan, M. E. and Rue, H. The bayesian learning rule. Journal of Machine Learning Research, 24 0 (281): 0 1--46, 2023

2023
[26]

E., Nielsen, D., Tangkaratt, V., Lin, W., Gal, Y., and Srivastava, A

Khan, M. E., Nielsen, D., Tangkaratt, V., Lin, W., Gal, Y., and Srivastava, A. Fast and scalable B ayesian deep learning by weight-perturbation in A dam . In Proceedings of the 35th International Conference on Machine Learning, pp.\ 2611--2620, 2018

2018
[27]

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015

2015
[28]

Momentum stiefel optimizer, with applications to suitably-orthogonal attention, and optimal transport

Kong, L., Wang, Y., and Tao, M. Momentum stiefel optimizer, with applications to suitably-orthogonal attention, and optimal transport. arXiv preprint arXiv:2205.14173, 2022

2022 arXiv
[29]

Krishnan, S., Xiao, Y., and Saurous, R. A. Neumann optimizer: A practical optimization algorithm for deep neural networks. arXiv preprint arXiv:1712.03298, 2017

2017 arXiv
[30]

Lee, J. M. Introduction to Riemannian manifolds, volume 2. Springer, 2018

2018
[31]

Efficient riemannian optimization on the stiefel manifold via the cayley transform

Li, J., Fuxin, L., and Todorovic, S. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint arXiv:2002.01113, 2020

2002 arXiv
[32]

Preconditioned stochastic gradient descent

Li, X.-L. Preconditioned stochastic gradient descent. IEEE transactions on neural networks and learning systems, 29 0 (5): 0 1454--1466, 2017

2017
[33]

E., and Schmidt, M

Lin, W., Khan, M. E., and Schmidt, M. Fast and simple natural-gradient variational inference with mixture of exponential-family approximations. In International Conference on Machine Learning, pp.\ 3992--4002, 2019

2019
[34]

Lin, W., Schmidt, M., and Khan, M. E. Handling the positive-definite constraint in the bayesian learning rule. In International Conference on Machine Learning, pp.\ 6116--6126. PMLR, 2020

2020
[35]

M., and Schmidt, M

Lin, W., Nielsen, F., Emtiyaz, K. M., and Schmidt, M. Tractable structured natural-gradient descent using local parameterizations. In International Conference on Machine Learning, pp.\ 6680--6691. PMLR, 2021

2021
[36]

E., and Schmidt, M

Lin, W., Duruisseaux, V., Leok, M., Nielsen, F., Khan, M. E., and Schmidt, M. Simplifying momentum-based positive-definite submanifold optimization with applications to deep learning. In International Conference on Machine Learning, pp.\ 21026--21050. PMLR, 2023

2023
[37]

E., and Makhzani, A

Lin, W., Dangel, F., Eschenhagen, R., Bae, J., Turner, R. E., and Makhzani, A. Can we remove the square-root in adaptive gradient methods? a second-order perspective. In International Conference on Machine Learning, 2024

2024
[38]

Sophia: A scalable stochastic second-order optimizer for language model pre-training

Liu, H., Li, Z., Hall, D., Liang, P., and Ma, T. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342, 2023

2023 arXiv
[39]

M., Paull, L., Xiong, L., Song, L., and Weller, A

Liu, W., Lin, R., Liu, Z., Rehg, J. M., Paull, L., Xiong, L., Song, L., and Weller, A. Orthogonal over-parameterized training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 7251--7260, 2021

2021
[40]

Optimizing millions of hyperparameters by implicit differentiation

Lorraine, J., Vicol, P., and Duvenaud, D. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, pp.\ 1540--1552. PMLR, 2020

2020
[41]

and Hutter, F

Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

2017 arXiv
[42]

Lecture Notes: Mathematical Modelling of DNA

Maddocks, J. Lecture Notes: Mathematical Modelling of DNA . https://lcvmwww.epfl.ch/teaching/modelling_dna/index.php?dir=exercises&file=corr03.pdf, 2021. Accessed: 2024/09/25

2021
[43]

Manasse, F. K. and Misner, C. W. Fermi normal coordinates and some basic concepts in differential geometry. Journal of mathematical physics, 4 0 (6): 0 735--745, 1963

1963
[44]

and Grosse, R

Martens, J. and Grosse, R. Optimizing neural networks with K ronecker-factored approximate curvature . In International Conference on Machine Learning, pp.\ 2408--2417, 2015

2015
[45]

Mixed precision training

Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed precision training. ICLR, 2018

2018
[46]

and Archambeau, C

Opper, M. and Archambeau, C. The variational Gaussian approximation revisited . Neural computation, 21 0 (3): 0 786--792, 2009

2009
[47]

Osawa, K., Swaroop, S., Khan, M. E. E., Jain, A., Eschenhagen, R., Turner, R. E., and Yokota, R. Practical deep learning with Bayesian principles . In Advances in neural information processing systems, pp.\ 4287--4299, 2019

2019
[48]

B., Pedersen, M

Petersen, K. B., Pedersen, M. S., et al. The matrix cookbook. Technical University of Denmark, 7 0 (15): 0 510, 2008

2008
[49]

Controlling text-to-image diffusion by orthogonal finetuning

Qiu, Z., Liu, W., Feng, H., Xue, Y., Feng, Y., Liu, Z., Zhang, D., Weller, A., and Sch \"o lkopf, B. Controlling text-to-image diffusion by orthogonal finetuning. Advances in Neural Information Processing Systems, 36: 0 79320--79362, 2023

2023
[50]

and Goldfarb, D

Ren, Y. and Goldfarb, D. Tensor normal training for deep learning models. Advances in Neural Information Processing Systems, 34: 0 26040--26052, 2021

2021
[51]

Topmoumoute online natural gradient algorithm

Roux, N., Manzagol, P.-A., and Bengio, Y. Topmoumoute online natural gradient algorithm. Advances in neural information processing systems, 20, 2007

2007
[52]

Hiera: A hierarchical vision transformer without the bells-and-whistles

Ryali, C., Hu, Y.-T., Bolya, D., Wei, C., Fan, H., Huang, P.-Y., Aggarwal, V., Chowdhury, A., Poursaeed, O., Hoffman, J., et al. Hiera: A hierarchical vision transformer without the bells-and-whistles. In International Conference on Machine Learning, pp.\ 29441--29454. PMLR, 2023

2023
[53]

Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models

Salimbeni, H., Eleftheriadis, S., and Hensman, J. Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models . International Conference on Artificial Intelligence and Statistics (AISTATS), 2018

2018
[54]

M., Bazan, C., Yokota, R., Gurevych, I., Cremers, D., Khan, M

Shen, Y., Daheim, N., Cong, B., Nickl, P., Marconi, G. M., Bazan, C., Yokota, R., Gurevych, I., Cremers, D., Khan, M. E., et al. Variational learning is effective for large deep networks. arXiv preprint arXiv:2402.17641, 2024

2024 arXiv
[55]

M., Lee, T.-H., Iwasaki, S., Gallego-Posada, J., Li, Z., Rangadurai, K., Mudigere, D., and Rabbat, M

Shi, H.-J. M., Lee, T.-H., Iwasaki, S., Gallego-Posada, J., Li, Z., Rangadurai, K., Mudigere, D., and Rabbat, M. A distributed data-parallel pytorch implementation of the distributed shampoo optimizer for training neural networks at-scale. arXiv preprint arXiv:2309.06497, 2023

2023 arXiv
[56]

Tagare, H. D. Notes on optimization on stiefel manifolds. Yale University, New Haven, 2011

2011
[57]

Tan, L. S. Analytic natural gradient updates for cholesky factor in gaussian variational approximation. arXiv preprint arXiv:2109.00375, 2022

2022 arXiv
[58]

and Hinton, G

Tieleman, T. and Hinton, G. Rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera, 2012

2012
[59]

H., and Nguyen, D

Tran, M.-N., Nguyen, D. H., and Nguyen, D. Variational bayes on manifolds. Statistics and Computing, 31: 0 1--17, 2021

2021
[60]

Van Loan, C. F. and Pitsianis, N. Approximation with Kronecker products. Springer, 1993

1993
[61]

Invariance properties of the natural gradient in overparametrised systems

van Oostrum, J., M \"u ller, J., and Ay, N. Invariance properties of the natural gradient in overparametrised systems. Information geometry, 6 0 (1): 0 51--67, 2023

2023
[62]

Soap: Improving and stabilizing shampoo using adam

Vyas, N., Morwani, D., Zhao, R., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

2024 arXiv
[63]

Natural evolution strategies

Wierstra, D., Schaul, T., Peters, J., and Schmidhuber, J. Natural evolution strategies. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp.\ 3381--3387. IEEE, 2008

2008
[64]

Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8 0 (3-4): 0 229--256, 1992

1992
[65]

J., Chun, S., Choe, J., and Yoo, Y

Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 6023--6032, 2019

2019
[66]

Noisy natural gradient as variational inference

Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In International Conference on Machine Learning, pp.\ 5847--5856, 2018

2018
[67]

mixup: Beyond empirical risk minimization

Zhang, H. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017

2017 arXiv
[68]

P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S

Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020

2020
[69]

Why transformers need adam: A hessian perspective

Zhang, Y., Chen, C., Ding, T., Li, Z., Sun, R., and Luo, Z.-Q. Why transformers need adam: A hessian perspective. arXiv preprint arXiv:2402.16788, 2024

2024 arXiv
[70]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
[71]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
[72]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Optimization algorithms on matrix manifolds

Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization algorithms on matrix manifolds. Princeton University Press, 2009

work page 2009

[3] [3]

Efficient full-matrix adaptive regularization

Agarwal, N., Bullins, B., Chen, X., Hazan, E., Singh, K., Zhang, C., and Zhang, Y. Efficient full-matrix adaptive regularization. In International Conference on Machine Learning, pp.\ 102--110. PMLR, 2019

work page 2019

[4] [4]

Learning rate grafting: Transferability of optimizer tuning

Agarwal, N., Anil, R., Hazan, E., Koren, T., and Zhang, C. Learning rate grafting: Transferability of optimizer tuning. 2021

work page 2021

[5] [5]

Natural gradient works efficiently in learning

Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

1998

[6] [6]

Information geometry and its applications, volume 194

Amari, S.-i. Information geometry and its applications, volume 194. Springer, 2016

work page 2016

[7] [7]

Scalable second order optimization for deep learning

Anil, R., Gupta, V., Koren, T., Regan, K., and Singer, Y. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

arXiv 2002

[8] [8]

and Newhouse, L

Bernstein, J. and Newhouse, L. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024

arXiv 2024