Pith. sign in

REVIEW 4 major objections 7 minor 66 references

Adam-style variance adaptation carries over to Muon-style spectral descent, and the resulting optimizer, DeVA_S8, reaches a target NanoGPT perplexity with ~6.6% fewer tokens than Muon.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:48 UTC pith:TEXG43US

load-bearing objection Fresh decomposition and a genuinely new spectral update rule, but Algorithm 2 omits the rotation back to parameter space, so the experiments may not be testing the method the theory derives. the 4 major comments →

arxiv 2602.06880 v2 pith:TEXG43US submitted 2026-02-06 cs.LG

Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization

Zitao Song , Cedar Site Bai , Zhe Zhang , Brian Bullins , David F. Gleich This is my paper
classification cs.LG MSC 68T0790C3090C5365K05
keywords adaptive optimizationvariance adaptationspectral descentmatrix sign functionKronecker-factored preconditioningsignal-to-noise ratioMuonNanoGPT
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that every preconditioned gradient update can be split into a variance-adaptation factor times a scale-invariant direction, and that this single split turns the puzzle of "Adam for matrices" into a coordinate-wise formula. The vector case of the split reproduces Adam; the matrix case yields DeVA_S8, which scales each spectral direction of Muon's matrix-sign update by a signal-to-noise ratio built from products of singular values of the rotated gradient. Empirically, DeVA_S8 reaches the target NanoGPT validation perplexity with ~6.6% fewer tokens than Muon (and ~4.3% fewer than SOAP), at about 54% higher wall-clock time per token that narrows as batch size grows, and it also leads early ImageNet training. The paper also proves that the variance-adaptation term reduces a weighted blockwise-smoothness error in the convergence bound, giving a theoretical reason adaptive spectral descent should beat plain Muon — provided the curvature eigenbases stay locally stable and the adaptation weights stay at or below 1.

Core claim

The paper's core move is an identity: Δx = −E[H]^{−1/2}g refactors into −(E[H]^{−1/2}H^{1/2})·(H^{−1/2}g), splitting every preconditioned update into a curvature-ratio factor and a scale-invariant direction. For vectors with H = diag(gg^T), the direction is sign(g) and the factor is the gradient signal-to-noise ratio, recovering Adam. For matrices with Kronecker curvature H = L^{1/2}⊗R^{1/2} (L = GG^T, R = G^TG), the direction is msign(G), which replaces each singular value of the gradient by +1, and the factor becomes a spectral adaptation matrix. Theorem 3.1 shows that under 'local stability' the update simplifies to ΔX = −Q_L(E^{−1/2}⊙msign(Q_L^T G Q_R))Q_R^T with E_ij = E[σ_iσ_j]/(σ_iσ_j

What carries the argument

The load-bearing object is the decoupling identity Δx = −(E[H]^{−1/2}H^{1/2})·(H^{−1/2}g), which presents every adaptive gradient update as a variance-adaptation factor times a scale-invariant steepest-descent direction (sign under ℓ∞; the matrix sign msign under the Schatten-∞ norm). Around it, the paper builds the Kronecker curvature model H = L^{1/2}⊗R^{1/2} on the covariance factors L = GG^T, R = G^TG, and projects into their eigenbasis Q_L, Q_R (refreshed periodically, with a single power-iteration plus QR approximation). Theorem 3.1 is the compression step: it turns the nm×nm diagonal curvature ratio into an n×m Hadamard product whose entries are the singular-value-product SNR E[σ_iσ_j

Load-bearing premise

The load-bearing premise is 'local stability': the derivation of DeVA_S8's update assumes the eigenbases of the accumulated covariance factors L_t and R_t stay fixed over the EMA window, so the expectation acts only on the eigenvalues (Eq. 22 of the proof). When the bases drift in real training, the coordinate-wise update is no longer the exact optimal update and Γ_t becomes a heuristic; a second, separate premise is that the smoothness-reduction theorems need adaptation weig

What would settle it

Two concrete checks settle the claim. (1) During an actual training run, measure the subspace drift of Q_L and Q_R between consecutive eigendecomposition refreshes; if the drift is non-negligible over the EMA window, the factorization E[L^{1/2}⊗R^{1/2}] = (Q_L⊗Q_R)E[K^{1/2}](Q_L^T⊗Q_R^T) fails and Theorem 3.1's update is not the exact optimal preconditioner. (2) Log the entries of Γ_t during training; if they systematically exceed 1, the premise γ̂² ≤ 1 behind the smoothness-reduction theorems is violated exactly in the regime where the method is used, so the theorems would not explain the obs

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • DeVA_S8 reaches the target NanoGPT validation perplexity with ~6.6% fewer tokens than Muon (final perplexity 3.271 vs 3.305 in this run), and the per-token overhead shrinks as gradient accumulation grows, so the token savings translate into wall-clock savings at scale.
  • Adam, Muon, Shampoo, SOAP, and SPlus all appear as special cases of one preconditioned update; DeVA_S8 is the instance that keeps Adam's variance adaptation while using the Kronecker eigenbasis, providing a principled alternative to bolting Adam stepsizes onto Muon ad hoc.
  • The convergence theorems show the variance-adaptation weights scale the blockwise-smoothness error inside the bound; with weights ≤ 1 the error is actively reduced relative to non-adaptive spectral descent, and the resulting stationarity complexity matches prior Muon analysis (Corollary 4.16).
  • The paper establishes that the right quantity to accumulate for matrix adaptation is the covariance of singular values of the rotated gradient, E[σ_iσ_j], and that a memory-efficient O(n+m) variant (EMA of row/column norm outer products) loses little (best validation loss 3.271 → 3.309).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The decoupling identity driving DeVA is norm-agnostic, so the same variance/scale split could produce 'adaptive' versions of other scale-invariant base updates — ℓ_p sign updates, operator-norm (Schatten) steps, or norm-constrained LMO methods — by multiplying the base direction by (E[quadratic form]/quadratic form)^{−1/2} in the appropriate basis. The paper only instantiates ℓ∞ and Schatten-∞.
  • Because the smoothness-reduction theorems require adaptation weights ≤ 1 while SNR weights exceed 1 whenever signal dominates noise, a clamping variant (Γ_t ≤ 1 element-wise) would satisfy the theory's premise and simultaneously give a causal test of whether the theoretical mechanism drives the observed speedup.
  • The 'local stability' assumption carrying Theorem 3.1 is testable in situ: record Q_L and Q_R at successive refreshes during a real run and measure their subspace drift. Small drift confirms the update is the true preconditioned step; large drift means the practical Γ_t is an effective heuristic whose explanation lies elsewhere.
  • The efficient second moment estimates E[σ_iσ_j] by an EMA of the outer product of row and column norms (rank-1 per step); tracking the full n×m matrix of singular-value products would reveal how much of DeVA_S8's gain comes from correlations between different singular-value pairs — a natural ablation the paper leaves open.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces DeVA, a framework that rewrites the AdaGrad-style preconditioned update E[H]^{-1/2}g as a product of a 'decoupled variance adaptation' term E[H]^{-1/2}H^{1/2} and a 'scale-invariant' update H^{-1/2}g (Eq. 10). In the vector case this recovers Adam/SignSGD; in the matrix case, with H = L^{1/2}⊗R^{1/2}, it yields DeVA_S8, an adaptive spectral descent method. Theorem 3.1 derives a coordinate-wise update in the eigenbasis of the Kronecker factors, with a spectral adaptation matrix rE (Eq. 14). Theorems 4.12 and 4.15 give convergence bounds, under blockwise smoothness, for the vector and matrix variants. Experiments on NanoGPT, ViT-L/16, and CIFAR-10 report that DeVA_S8 outperforms Muon and SOAP, using about 6.6% fewer tokens to reach a target perplexity. The paper also provides an implementation link and several appendices with proofs and ablations.

Significance. The core decomposition in Eq. (10) is elegant and has the potential to unify vector adaptive methods (Adam) and spectral matrix optimizers (Muon/SOAP). The convergence theorems are stated for arbitrary positive weight sequences and are not circular: they hold for any γ_t/Γ_t, and the algebra of Theorem 3.1 (conditional on Eq. 22) is clean. This generality is a strength. However, the manuscript in its current form has a decisive theory-algorithm mismatch: Algorithm 2 does not rotate the preconditioned update back to the original parameter basis, so the experiments do not implement the update whose optimality is claimed in Theorem 3.1. In addition, the 'local stability' assumption in Eq. (22) is asserted rather than validated, and the claimed smoothness improvement requires Γ≤1, which is neither proven nor measured for the proposed SNR-style weights. These issues are load-bearing for the central claims.

major comments (4)
  1. [§3.3, Algorithm 2] Algorithm 2 computes G'_t = Q_L^T G_t Q_R, forms M_t as an EMA of G'_t, computes Γ_t in that rotated basis, and then updates X_{t+1} = X_t - η_t D_t with D_t = Γ_t ⊙ msign(M_t) ⊙ 0.2√max(n,m). Theorem 3.1, Eq. (14), states the derived update as ΔX* = -Q_L( rE^{-1/2} ⊙ msign(Q_L^T G Q_R) ) Q_R^T. The algorithm never applies Q_L on the left and Q_R^T on the right. Since G_t is the gradient with respect to X_t in the original parameter space, omitting the inverse rotation means the update is applied in the spectral basis while X_t lives in the original basis. For non-identity Q_L, Q_R this is not the derived preconditioned update. Consequently the empirical results in §5 cannot be attributed to the method whose optimality is claimed in Theorem 3.1. The fix is local (add the rotation), but without it the theory-algorithm link is broken.
  2. [Appendix C.1, Eq. (22)] The proof of Theorem 3.1 substitutes E[L^{1/2}⊗R^{1/2}] = (Q_L⊗Q_R)E[K^{1/2}](Q_L^T⊗Q_R^T), calling this 'local stability.' This treats the eigenbases Q_L, Q_R, which are functions of the random gradient through the EMA-accumulated L_t, R_t, as constant over the averaging window. The assumption is asserted inside the proof and is never derived from conditions on the gradient process or validated with measurements. When the eigenbases drift, Eq. (14) is not the exact solution of Eq. (10), and the practical Γ_t is a heuristic rather than a derived preconditioner. Please either prove the assumption under explicit conditions, provide empirical evidence (e.g., subspace distances between successive eigenbases), or explicitly downgrade Theorem 3.1 to a heuristic derivation.
  3. [§4.1–4.2, Theorems 4.12/4.15] The paper's central narrative — that the variance adaptation term 'reduces the blockwise smoothness' — relies on γ̂_i²≤1 (page 9) and Γ_ij≤1 (page 10). For the algorithms actually run, γ_t=(v_t/(m_t²+ε))^{-1/2}=|m_t|/√(v_t+ε) and Γ_t=(V_t/(r_t c_t^T+ε))^{-1/2}; these quantities exceed 1 whenever the signal dominates the noise, which is typical in early training and for large-gradient coordinates. The paper does not prove a bound on these weights and provides no empirical measurement of max_t γ_t or max_t Γ_t. The theorems themselves are valid for arbitrary positive weights, but the claimed improvement of the smoothness term by the specific DeVA weights is unsupported without the inequality or a measurement.
  4. [§3.2–3.3, Eq. (13)–(15) and Algorithm 2 lines 12–14] Theorem 3.1's spectral adaptation matrix rE uses E[σ_iσ_j], where σ_i, σ_j are the singular values of the current gradient G_t. Algorithm 2 instead computes r_t and c_t from M_t, the EMA of the rotated gradient, and then forms V_t = EMA(r_t c_t^T). This is a reasonable heuristic but a second substitution beyond the theorem; the convergence bounds in §4 do not cover this particular estimator. The manuscript should state explicitly that Algorithm 2 is an approximation of Eq. (14), not its exact implementation, or extend the derivation to cover the EMA-based estimator.
minor comments (7)
  1. [Algorithm 1, line 6] The update reads 'v_t = β_2 s_{t-1} + (1-β_2) m_t²', but s_{t-1} is undefined; it should be v_{t-1}.
  2. [Appendix D.2, final display] The final displayed inequality uses √d ||σ²_m||_{Γ̂²,F} and (d/2)||√L_m||²_{Γ̂,F}, whereas the theorem statement and proof body use r = min(n,m). Replace d with r.
  3. [Appendix B] The verbal description of H_hom and H_het appears reversed relative to the displayed matrices: H_hom has blocks [1,2,3], [99,100,101], [4998,4999,5000], i.e., similar magnitudes within each block, while H_het has blocks [1,99,4998], etc., i.e., dissimilar magnitudes within each block. The text says the opposite.
  4. [Algorithms 3 and 4, line 14] The decay factor in the second-moment updates is written as 'β' without the subscript 2; it should be β_2 for consistency with Algorithm 2.
  5. [Table 1, DeVA_S8 row] The preconditioner expression is missing a closing parenthesis / exponent: as typeset, 'pErpGG T q1{2 b pGT Gqs1{2q´1{2' is ambiguous.
  6. [§3.1] 'single-to-noise ratio' should be 'signal-to-noise ratio.'
  7. [§4.2] The statement 'for any Γ_ij≤1, we will have 1 ≤ √L_m,ij Γ_ij ≤ √L_m,ij' assumes L_m,ij≥1; this should be stated explicitly or the lower bound removed.

Circularity Check

0 steps flagged

No significant circularity found; the derivation is self-contained.

full rationale

The derivation chain is self-contained. Equation (10) is an algebraic identity: (E[H]^{-1/2}H^{1/2}) · (H^{-1/2}g) = E[H]^{-1/2}g, so the 'decoupled variance adaptation' is a factorization of the preconditioned update rather than a fitted quantity. Theorem 3.1 is a basis rotation of that identity under the explicitly stated local-stability assumption (Eq. 22); it is not an estimate obtained from data and then repackaged as a prediction. The convergence bounds in Theorems 4.12 and 4.15 are general inequalities valid for arbitrary γ_t and Γ_t; the claimed 'improvement' is explicitly conditional on γ̂²≤1 or Γ̂≤1 and is not derived from the specific spectral estimator, so the bounds do not reduce to the algorithm's own fit. The paper credits external prior work for the SNR interpretation (Orvieto & Gower) and the eigendecomposition trick (SOAP), and its minor self-citations (Gleich et al. 2023, Luo et al. 2025) appear only in related-work surveys and are not load-bearing. Separately, Algorithm 2 omits the Q_L D_t Q_R^T rotation back to parameter space required by Theorem 3.1, and the local-stability assumption is unverified; these are correctness or assumption concerns, not circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 2 invented entities

The central algorithmic contribution rests on several premises the reader does not pay for upstream: (i) the blockwise-smoothness geometry (Assumptions 4.8/4.13) that all convergence theorems require; (ii) the local-stability of eigenbases (Eq. 22, inside Thm 3.1's proof) — the load-bearing premise of the spectral derivation, asserted but unmeasured; (iii) the unproven condition γ̂²≤1/Γ̂≤1 under which variance adaptation is claimed to reduce the smoothness error terms. These are not fitted parameters in the p-hacking sense; the hand-chosen constants (β₁/β₂/β₃, freq, ε, 0.2√max(n,m), uniform η=0.001) are standard hyperparameters, but since the theorems treat γ_t/Γ_t as free inputs, the specific adaptation's benefit is conditional rather than derived.

free parameters (4)
  • EMA/momentum coefficients β₁, β₂, β₃ = β₁: 0.9–0.95; β₂: 0.95–0.999; β₃: 0.95 (per table)
    Chosen by hand per optimizer and table. The adaptation weights γ_t/Γ_t inherit their meaning from these choices, but the theoretical bounds treat γ_t/Γ_t as given inputs and do not pin them.
  • eigendecomposition frequency (freq) = 10 (NanoGPT/ImageNet), 5 (CIFAR-10)
    Hand-chosen; the ablation (Fig. 6) shows robustness from 10 to 1000 steps, so not load-bearing, but it controls the staleness that the local-stability assumption relies on.
  • RMS alignment constant 0.2·√max(n,m) = 0.2·√max(n,m)
    Imported from Bai et al. (2025); scales the msign update to Adam's RMS and affects the learning-rate-transfer claims (Fig. 4).
  • Uniform learning rate η = 0.001 for all optimizers in main LLM runs = 0.001
    Standard modded-nanogpt recipe choice; the claimed gains are conditional on this LR being favorable to DeVA_S8 (LR sweeps are provided only for CIFAR-10).
axioms (6)
  • domain assumption Blockwise L_v-smoothness (Assumption 4.8) and elementwise smoothness matrix L_m (Assumption 4.13)
    Both convergence theorems (4.12, 4.15) are conditional on a Hessian geometry that is cited to prior empirical block-diagonal observations (Ghorbani et al. 2019; Zhang et al. 2024b) but assumed for the target functions.
  • standard math Unbiased stochastic gradients with bounded coordinate-wise variance (Assumption 4.10)
    Standard stochastic-optimization assumption; the variance bounds σ²_i are precisely what the adaptation weights γ_t/Γ_t are intended to track.
  • ad hoc to paper Local stability of eigenbases: E[L^{1/2}⊗R^{1/2}] = (Q_L⊗Q_R)E[K^{1/2}](Q_L^T⊗Q_R^T)
    Introduced inside Theorem 3.1's proof (Eq. 22) so the expectation passes through to the eigenvalues; generally false when gradient-covariance eigenbases drift over the EMA window, and never validated. Load-bearing for the coordinate-wise spectral update (Eq. 14).
  • ad hoc to paper Adaptation weights bounded above by 1 (γ̂²_i ≤ 1, Γ̂_ij ≤ 1)
    The claimed smoothness-error reduction over non-adaptive SignSGD/Muon requires these; the paper asserts them only in prose ('in the case of γ̂²_i ≤ 1', 'for any Γ_ij ≤ 1') and the SNR weights can exceed 1.
  • standard math Shampoo preconditioner identity: (L^{1/2}⊗R^{1/2})^{-1/2}vec(G) = vec(UV^T) (Eq. 8)
    Known algebraic identity from prior spectral-descent literature (Carlson et al. 2015a; Jordan et al. 2024b), correctly invoked.
  • standard math Matrix norm inequalities in Theorem 4.15's proof (‖A⊙B‖_F ≤ ‖A‖_F‖B‖_F; nuclear ≤ √r·Frobenius)
    Verified: A⊙B is a submatrix of A⊗B, so the Frobenius bound holds; nuclear-norm rank bound is standard.
invented entities (2)
  • Spectral adaptation matrix rE (entries E[σ_iσ_j]/σ_iσ_j) and its practical estimator V_t ⊘ (r_t c_t^T) independent evidence
    purpose: Reweights each (i,j) spectral coordinate of the rotated momentum before the inverse rotation — the mechanism claimed to reduce blockwise smoothness (Thm 4.15).
    Fully specified update rule (Algorithm 2) that any practitioner can test; its empirical benefit is the paper's falsifiable claim. No separate physical handle.
  • 'Decoupled Variance Adaptation' operator (E[H]^{-1/2}H^{1/2}) no independent evidence
    purpose: Conceptual factor in Eq. (10); interpretive division of the preconditioned update into variance adaptation × scale-invariant sign.
    Algebraic identity — carries no content beyond the original preconditioned update; its value is heuristic (SNR reading).

pith-pipeline@v1.3.0-alltime-deepseek · 26284 in / 40975 out tokens · 357257 ms · 2026-08-03T03:48:45.012386+00:00 · methodology

0 comments
read the original abstract

Adaptive methods like Adam have become the $\textit{de facto}$ standard for large-scale vector and Euclidean optimization due to their coordinate-wise adaptation with a second-order nature. More recently, matrix-based spectral optimizers like Muon (Jordan et al., 2024b) show the power of treating weight matrices as matrices rather than long vectors. Linking these is hard because many natural generalizations are not feasible to implement, and we also cannot simply move the Adam adaptation to the matrix spectrum. To address this, we reformulate the AdaGrad update and decompose it into a variance adaptation term and a scale-invariant term. This decoupling produces $\textbf{DeVA}$ ($\textbf{De}$coupled $\textbf{V}$ariance $\textbf{A}$daptation), a framework that bridges between vector-based variance adaptation and matrix spectral optimization, enabling a seamless transition from Adam to adaptive spectral descent. Extensive experiments across language modeling and image classification demonstrate that DeVA consistently outperforms state-of-the-art methods such as Muon and SOAP (Vyas et al., 2024), reducing token usage by around 6.6\%. Theoretically, we show that the variance adaptation term effectively improves the blockwise smoothness, facilitating faster convergence. Our implementation is available at https://github.com/Tsedao/Decoupled-Variance-Adaptation

Figures

Figures reproduced from arXiv: 2602.06880 by Brian Bullins, Cedar Site Bai, David F. Gleich, Zhe Zhang, Zitao Song.

Figure 1
Figure 1. Figure 1: NanoGPT pretraining on FineWeb at a uniform 0.001 learning rate. Compared to non-adaptive Muon, the adaptive method SOAP and our method DeVAS8 achieve the target validation perplexity using 4.3% and 6.6% fewer to￾kens, respectively. A fundamental acceleration strategy in modern optimization involves adapting stochastic gradi￾ent descent to second-order curvature (Amari, 1998). This adaptation enables accel… view at source ↗
Figure 2
Figure 2. Figure 2: Trace Quadratic Function Optimization (median and 25%/75% quantiles over 100 seeds). (a)–(b): Training performance for various optimizers on 9-dimensional trace quadratic problems with homogeneous vs. heterogeneous Hessians (details see Appendix Section B). DeVAS8 significantly outperforms Muon in the heterogeneous setting. (c)–(d): Dynamic of weighted dual norm }H}1,Γ (see Definition 4.1) for adaptive met… view at source ↗
Figure 4
Figure 4. Figure 4: ResNet-20 validation accuracy on CIFAR-10 after 40 epochs. Optimal learning rates cluster near 0.005 for vector methods (left) and 0.01 for matrix methods (right), where DeVAℓ8 and SOAP achieve peak per￾formance, respectively. GPT. We evaluate our methods using the modded-nanogpt codebase (Jordan et al., 2024a), which incorporates modern archi￾tectural enhancements: rotary embeddings, RMSNorm, a linear dec… view at source ↗
Figure 5
Figure 5. Figure 5: Batch size sensitivity on NanoGPT (274M). The generalization performance of [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Sensitivity to eigen-decomposition frequency on NanoGPT (274M). [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Training and Validation Performance of different DeVA [PITH_FULL_IMAGE:figures/full_fig_p029_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 19 linked inside Pith

  1. [1]

    Natural gradient works efficiently in learning

    Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

    1998

  2. [2]

    Asgo: Adaptive structured gradient optimization

    An, K., Liu, Y., Pan, R., Ren, Y., Ma, S., Goldfarb, D., and Zhang, T. Asgo: Adaptive structured gradient optimization. arXiv preprint arXiv:2503.20762, 2025

    arXiv 2025

  3. [3]

    Kimi k2: Open agentic intelligence

    Bai, Y., Bao, Y., Chen, G., Chen, J., Chen, N., Chen, R., Chen, Y., Chen, Y., Chen, Y., et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    and Newhouse, L

    Bernstein, J. and Newhouse, L. Modular duality in deep learning. arXiv preprint arXiv:2410.21265, 2024 a

    Pith/arXiv arXiv 2024

  5. [5]

    and Newhouse, L

    Bernstein, J. and Newhouse, L. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024 b

    Pith/arXiv arXiv 2024

  6. [6]

    signsgd: Compressed optimisation for non-convex problems

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signsgd: Compressed optimisation for non-convex problems. In International conference on machine learning, pp.\ 560--569. PMLR, 2018

    2018

  7. [7]

    and Vandenberghe, L

    Boyd, S. and Vandenberghe, L. Convex Optimization. Cambridge University Press, March 2004. ISBN 9780511804441. doi:10.1017/cbo9780511804441. URL http://dx.doi.org/10.1017/CBO9780511804441

    doi:10.1017/cbo9780511804441 2004

  8. [8]

    H., Hansen, S

    Byrd, R. H., Hansen, S. L., Nocedal, J., and Singer, Y. A stochastic quasi-newton method for large-scale optimization. SIAM Journal on Optimization, 26 0 (2): 0 1008--1031, 2016

    2016

  9. [9]

    Candès, E. J. and Recht, B. Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 9 0 (6): 0 717–772, April 2009. ISSN 1615-3383. doi:10.1007/s10208-009-9045-5. URL http://dx.doi.org/10.1007/s10208-009-9045-5

    doi:10.1007/s10208-009-9045-5 2009

  10. [10]

    Stochastic spectral descent for restricted boltzmann machines

    Carlson, D., Cevher, V., and Carin, L. Stochastic spectral descent for restricted boltzmann machines. In Artificial intelligence and statistics, pp.\ 111--119. PMLR, 2015 a

    2015

  11. [11]

    E., Collins, E., Hsieh, Y.-P., Carin, L., and Cevher, V

    Carlson, D. E., Collins, E., Hsieh, Y.-P., Carin, L., and Cevher, V. Preconditioned spectral descent for deep learning. Advances in neural information processing systems, 28, 2015 b

    2015

  12. [12]

    Lion secretly solves constrained optimization: As lyapunov predicts

    Chen, L., Liu, B., Liang, K., and Liu, Q. Lion secretly solves constrained optimization: As lyapunov predicts. arXiv preprint arXiv:2310.05898, 2023 a

    Pith/arXiv arXiv 2023

  13. [13]

    Muon optimizes under spectral norm constraints

    Chen, L., Li, J., and Liu, Q. Muon optimizes under spectral norm constraints. arXiv preprint arXiv:2506.15054, 2025

    arXiv 2025

  14. [14]

    Symbolic discovery of optimization algorithms

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36: 0 49205--49233, 2023 b

    2023

  15. [15]

    and Drusvyatskiy, D

    Davis, D. and Drusvyatskiy, D. When do spectral gradient updates help in deep learning? arXiv preprint arXiv:2512.04299, 2025

    arXiv 2025

  16. [16]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  17. [17]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  18. [18]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12 0 (7), 2011

    2011

  20. [20]

    Promise: Preconditioned stochastic optimization methods by incorporating scalable curvature estimates

    Frangella, Z., Rathore, P., Zhao, S., and Udell, M. Promise: Preconditioned stochastic optimization methods by incorporating scalable curvature estimates. Journal of Machine Learning Research, 25 0 (346): 0 1--57, 2024 a

    2024

  21. [21]

    Sketchysgd: reliable stochastic optimization via randomized curvature estimates

    Frangella, Z., Rathore, P., Zhao, S., and Udell, M. Sketchysgd: reliable stochastic optimization via randomized curvature estimates. SIAM Journal on Mathematics of Data Science, 6 0 (4): 0 1173--1204, 2024 b

    2024

  22. [22]

    What really matters in matrix-whitening optimizers? arXiv preprint arXiv:2510.25000, 2025 a

    Frans, K., Abbeel, P., and Levine, S. What really matters in matrix-whitening optimizers? arXiv preprint arXiv:2510.25000, 2025 a

    arXiv 2025

  23. [23]

    A stable whitening optimizer for efficient neural network training

    Frans, K., Levine, S., and Abbeel, P. A stable whitening optimizer for efficient neural network training. arXiv preprint arXiv:2506.07254, 2025 b

    arXiv 2025

  24. [24]

    S., and Valiant, G

    Garg, S., Tsipras, D., Liang, P. S., and Valiant, G. What can transformers learn in-context? a case study of simple function classes. Advances in neural information processing systems, 35: 0 30583--30598, 2022

    2022

  25. [25]

    and Lan, G

    Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23 0 (4): 0 2341--2368, 2013

    2013

  26. [26]

    An investigation into neural net optimization via hessian eigenvalue density

    Ghorbani, B., Krishnan, S., and Xiao, Y. An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pp.\ 2232--2241. PMLR, 2019

    2019

  27. [27]

    Goemans, M. X. and Williamson, D. P. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM, 42 0 (6): 0 1115–1145, November 1995. ISSN 1557-735X. doi:10.1145/227683.227684. URL http://dx.doi.org/10.1145/227683.227684

    arXiv 1995

  28. [28]

    Solving ridge regression using sketched preconditioned svrg

    Gonen, A., Orabona, F., and Shalev-Shwartz, S. Solving ridge regression using sketched preconditioned svrg. In International conference on machine learning, pp.\ 1397--1405. PMLR, 2016

    2016

  29. [29]

    Shampoo: Preconditioned stochastic tensor optimization

    Gupta, V., Koren, T., and Singer, Y. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp.\ 1842--1850. PMLR, 2018

    2018

  30. [30]

    Huang, Y., Seshadhri, C., and Gleich, D. F. Theoretical bounds on the network community profile from low-rank semi-definite programming. In Proceedings of the 40th International Conference on Machine Learning, ICML 2023, pp.\ 13976--13992, 2023

    2023

  31. [31]

    modded-nanogpt: Speedrunning the nanogpt baseline, 2024 a

    Jordan, K., Bernstein, J., Rappazzo, B., @fernbear.bsky.social, Vlado, B., Jiacheng, Y., Cesista, F., Koszarsky, B., and @Grad62304977. modded-nanogpt: Speedrunning the nanogpt baseline, 2024 a . URL https://github.com/KellerJordan/modded-nanogpt/tree/ca964e982191830eebbd155e185937077511a8aa

    2024

  32. [32]

    Muon: An optimizer for hidden layers in neural networks, 2024 b

    Jordan, K., Jin, Y., Boza, V., You, J., Cesista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks, 2024 b . URL https://kellerjordan.github.io/posts/muon/

    2024

  33. [33]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  34. [34]

    C., and Platt, J

    Kulis, B., Surendran, A. C., and Platt, J. C. Fast low-rank semidefinite programming for embedding and clustering. In Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research, pp.\ 235--242. PMLR, 2007. URL https://proceedings.mlr.press/v2/kulis07a.html

    2007

  35. [35]

    Scalable optimization in the modular norm

    Large, T., Liu, Y., Huh, M., Bahng, H., Isola, P., and Bernstein, J. Scalable optimization in the modular norm. Advances in Neural Information Processing Systems, 37: 0 73501--73548, 2024

    2024

  36. [36]

    Normuon: Making muon more efficient and scalable

    Li, Z., Liu, L., Liang, C., Chen, W., and Zhao, T. Normuon: Making muon more efficient and scalable. arXiv preprint arXiv:2510.05491, 2025

    arXiv 2025

  37. [37]

    Deepseek-v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Muon is scalable for llm training

    Liu, J., Su, J., Yao, X., Jiang, Z., Lai, G., Du, Y., Qin, Y., Xu, W., Lu, E., Yan, J., et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025 a

    Pith/arXiv arXiv 2025

  39. [39]

    Cosmos: A hybrid adaptive optimizer for memory-efficient training of llms

    Liu, L., Xu, Z., Zhang, Z., Kang, H., Li, Z., Liang, C., Chen, W., and Zhao, T. Cosmos: A hybrid adaptive optimizer for memory-efficient training of llms. arXiv preprint arXiv:2502.17410, 2025 b

    arXiv 2025

  40. [40]

    S., Li, B., Drineas, P., Zhang, R., and Bullins, B

    Luo, X., Bai, C. S., Li, B., Drineas, P., Zhang, R., and Bullins, B. Stacey: Promoting stochastic steepest descent via accelerated lp-smooth nonconvex optimization. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=TaqwI9qF5Q

    2025

  41. [41]

    Preconditioning benefits of spectral orthogonalization in muon

    Ma, J., Huang, Y., Chi, Y., and Chen, Y. Preconditioning benefits of spectral orthogonalization in muon. arXiv preprint arXiv:2601.13474, 2026

    arXiv 2026

  42. [42]

    New insights and perspectives on the natural gradient method

    Martens, J. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  43. [43]

    and Grosse, R

    Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp.\ 2408--2417. PMLR, 2015

    2015

  44. [44]

    A new perspective on shampoo's preconditioner

    Morwani, D., Shapira, I., Vyas, N., Malach, E., Kakade, S., and Janson, L. A new perspective on shampoo's preconditioner. arXiv preprint arXiv:2406.17748, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    Nesterov, Y. et al. Lectures on convex optimization, volume 137. Springer, 2018

    2018

  46. [46]

    and Mahoney, M

    Orecchia, L. and Mahoney, M. W. Implementing regularization implicitly via approximate eigenvector computation. In Getoor, L. and Scheffer, T. (eds.), Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp.\ 121--128, New York, NY, USA, 2011. ACM. URL http://www.icml-2011.org/papers/120_icmlpaper.pdf

    2011

  47. [47]

    and Gower, R

    Orvieto, A. and Gower, R. In search of adam's secret sauce. arXiv preprint arXiv:2505.21829, 2025

    arXiv 2025

  48. [48]

    A., Von Werra, L., Wolf, T., et al

    Penedo, G., Kydl \' c ek, H., Lozhkov, A., Mitchell, M., Raffel, C. A., Von Werra, L., Wolf, T., et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 0 30811--30849, 2024

    2024

  49. [49]

    Training deep learning models with norm-constrained lmos

    Pethick, T., Xie, W., Antonakopoulos, K., Zhu, Z., Silveti-Falls, A., and Cevher, V. Training deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Recht, B., Fazel, M., and Parrilo, P. A. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52 0 (3): 0 471–501, January 2010. ISSN 1095-7200. doi:10.1137/070697835. URL http://dx.doi.org/10.1137/070697835

    doi:10.1137/070697835 2010

  51. [51]

    and Mahoney, M

    Roosta-Khorasani, F. and Mahoney, M. W. Sub-sampled newton methods. Mathematical Programming, 174 0 (1): 0 293--326, 2019

    2019

  52. [52]

    and Stern, M

    Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp.\ 4596--4604. PMLR, 2018

    2018

  53. [53]

    On the convergence analysis of muon

    Shen, W., Huang, R., Huang, M., Shen, C., and Zhang, J. On the convergence analysis of muon. arXiv preprint arXiv:2505.23737, 2025

    Pith/arXiv arXiv 2025

  54. [54]

    Adamuon: Adaptive muon optimizer

    Si, C., Zhang, D., and Shen, W. Adamuon: Adaptive muon optimizer. arXiv preprint arXiv:2507.11005, 2025

    arXiv 2025

  55. [55]

    Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude

    Tieleman, T. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4 0 (2): 0 26, 2012

    2012

  56. [56]

    Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent

    Tong, T., Ma, C., and Chi, Y. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research, 22 0 (150): 0 1--63, 2021

    2021

  57. [57]

    Preconditioned norms: A unified framework for steepest descent, quasi-newton and adaptive methods

    Veprikov, A., Bolatov, A., Horv \'a th, S., Beznosikov, A., Tak \'a c , M., and Hanzely, S. Preconditioned norms: A unified framework for steepest descent, quasi-newton and adaptive methods. arXiv preprint arXiv:2510.10777, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    Soap: Improving and stabilizing shampoo using adam

    Vyas, N., Morwani, D., Zhao, R., Kwun, M., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    and Saul, L

    Weinberger, K. and Saul, L. Unsupervised learning of image manifolds by semidefinite programming. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp.\ 988–995. IEEE, 2004. doi:10.1109/cvpr.2004.1315272. URL http://dx.doi.org/10.1109/CVPR.2004.1315272

    Pith/arXiv arXiv 2004

  60. [60]

    Fantastic pretraining optimizers and where to find them

    Wen, K., Hall, D., Ma, T., and Liang, P. Fantastic pretraining optimizers and where to find them. arXiv preprint arXiv:2509.02046, 2025

    Pith/arXiv arXiv 2025

  61. [61]

    Dissecting hessian: Understanding common structure of hessian in neural networks

    Wu, Y., Zhu, X., Wu, C., Wang, A., and Ge, R. Dissecting hessian: Understanding common structure of hessian in neural networks. arXiv preprint arXiv:2010.04261, 2020

    Pith/arXiv arXiv 2010

  62. [62]

    Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp.\ 581--590. IEEE, 2020

    2020

  63. [63]

    Zhang, G., Fattahi, S., and Zhang, R. Y. Preconditioned gradient descent for overparameterized nonconvex burer--monteiro factorization with global optimality certification. Journal of Machine Learning Research, 24 0 (163): 0 1--55, 2023

    2023

  64. [64]

    Adagrad meets muon: Adaptive stepsizes for orthogonal updates

    Zhang, M., Liu, Y., and Schaeffer, H. Adagrad meets muon: Adaptive stepsizes for orthogonal updates. arXiv preprint arXiv:2509.02981, 2025

    Pith/arXiv arXiv 2025

  65. [65]

    Why transformers need adam: A hessian perspective

    Zhang, Y., Chen, C., Ding, T., Li, Z., Sun, R., and Luo, Z. Why transformers need adam: A hessian perspective. Advances in neural information processing systems, 37: 0 131786--131823, 2024 a

    2024

  66. [66]

    P., Ye, Y., Luo, Z.-Q., and Sun, R

    Zhang, Y., Chen, C., Li, Z., Ding, T., Wu, C., Kingma, D. P., Ye, Y., Luo, Z.-Q., and Sun, R. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793, 2024 b

    Pith/arXiv arXiv 2024