Pith. sign in

REVIEW 3 major objections 5 minor 119 references

Muon does not consistently outperform AdamW on low-rank matrix factorization; its reported gains are sensitive to learning-rate tuning and problem conditioning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 04:17 UTC pith:MRDNH4AO

load-bearing objection Useful, honest empirical check: Muon's edge on simple matrix factorization is problem-dependent and mostly an NMF story, but 'equal tuning' is really 'equal LR tuning' and the paper should say so. the 3 major comments →

arxiv 2607.13246 v2 pith:MRDNH4AO submitted 2026-07-14 cs.LG cs.AI

Reassessing Muon for Matrix Factorization

Ali Parviz , Gal Mishne , Alex Cloninger This is my paper
classification cs.LG cs.AI
keywords Muon optimizerAdamWmatrix factorizationnonnegative matrix factorizationoptimizer hyperparameter tuningcondition numberNewton-Schulz orthogonalizationlow-rank recovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that Muon, an optimizer known for large-language-model training speedups via orthogonalized gradient updates, loses its edge when tested on low-rank matrix factorization, a simple problem with rich spectral structure. Under systematic learning-rate tuning, AdamW and plain gradient descent match or beat Muon on factorization and completion tasks, while Muon retains a clear advantage on nonnegative matrix factorization. The paper argues that this problem-dependence means Muon's large-scale benefits may come from deep-learning-specific properties, and that optimizer comparisons should be evaluated across tuned hyperparameter ranges rather than at a single default configuration.

Core claim

The paper's central claim is that Muon's strong large-scale results do not transfer to controlled low-rank factorization. Across five problem families—symmetric factorization, matrix completion, nonnegative factorization, Gaussian-kernel factorization, and tensor-train factorization—the authors sweep learning rate exhaustively while holding model capacity fixed. Under this tuning, AdamW and plain gradient descent match or beat Muon on symmetric factorization and completion, reaching losses as low as 1e-13 while Muon plateaus around 1e-8 to 1e-5; on NMF, Muon consistently wins, reaching 1e-9 to 1e-6 while AdamW stalls near 1e-3. Rankings at a fixed default learning rate often reverse after tu

What carries the argument

The load-bearing mechanism is Muon's update rule: maintain a momentum buffer, replace it by an approximate polar factor computed with Newton-Schulz iterations, then step in that direction. The polar factor sets every singular value of the update to 1 while preserving singular vectors, making Muon a spectral-norm steepest-descent method. The paper's experimental machinery is the comparison protocol: fixed small matrix problems, log-spaced learning-rate sweeps, and the stable learning-rate range as the meaningful performance metric. An auxiliary analytical result casts AdamW as solving a per-coordinate adaptive trust-region problem and Muon as solving the Orthogonal Procrustes problem, giving

Load-bearing premise

The paper defines 'well-tuned' as sweeping only the learning rate, while Muon's momentum, Newton-Schulz iteration count, and orthogonalization polynomial coefficients are held fixed at values inherited from large-model practice, so those fixed choices could be masking a better Muon configuration on small factorization problems.

What would settle it

Repeat the same factorization benchmark while sweeping Muon's Newton-Schulz iteration count (for example, J = 3, 5, 8, 12) and the orthogonalization polynomial coefficients alongside the learning rate; if some Muon configuration reaches AdamW's 1e-13 loss floor or beats AdamW at condition number 625, the paper's central claim would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Muon's reported advantage over AdamW on large-scale models does not automatically transfer to simple, well-understood optimization problems.
  • Well-tuned AdamW and plain gradient descent match or exceed Muon on symmetric low-rank factorization and matrix completion, so claims of Muon superiority need conditioning on problem structure.
  • Muon retains a consistent advantage on nonnegative matrix factorization, suggesting its orthogonalized updates help when the target has nonnegativity constraints and redundant factor directions.
  • Single-configuration optimizer rankings are unreliable: in nearly all tested settings, the ranking at a default learning rate differs from the ranking after per-optimizer tuning.
  • Ill-conditioning and the shape of the eigenvalue distribution, not merely the condition number, determine which optimizer wins, so benchmark conclusions should report a range of conditioning regimes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: if the same pattern holds in other structured problems, Muon's benefit may be tied to problems with symmetry or redundancy constraints rather than to spectral conditioning alone.
  • Editorial extension: the paper's 'well-tuned' definition sweeps only the learning rate; jointly tuning Muon's Newton-Schulz iteration count and orthogonalization coefficients could shrink the gap on factorization and completion.
  • Editorial extension: the spectral-subspace alignment dynamics suggest a testable prediction that Muon will excel exactly where per-coordinate adaptivity overspecializes to dominant directions, such as highly imbalanced or long-tailed spectra.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies the Muon optimizer on low-rank matrix factorization, matrix completion, nonnegative matrix factorization, Gaussian-kernel factorization, and tensor-train factorization. Its central claim is that Muon does not consistently outperform AdamW in these settings and that its previously reported advantages are problem-dependent: with per-optimizer learning-rate tuning, AdamW and GD match or beat Muon on plain factorization, AdamW and Muon are comparable on completion, and Muon wins on NMF. The authors also analyze sensitivity to spectral shape at fixed condition number and investigate spectral subspace recovery dynamics. They conclude that optimizer rankings should be reported across tuned hyperparameter ranges and conditioning regimes, not at single default configurations.

Significance. If fully established, the paper would be a useful counterweight to LLM-scale claims that Muon dominates AdamW, and it would support the methodological recommendation to compare optimizers over tuned hyperparameter ranges. The paper has genuine strengths: multiple controlled problem settings, explicit caveats about problem-dependence, a detailed spectrum-family analysis, an analytical derivation of the AdamW and Muon updates in Appendix K, and a mechanistic alignment study. However, the headline 'under equal tuning' is stronger than the protocol supports, because the paper tunes only the learning rate while holding Muon-specific hyperparameters at LLM-scale defaults. The central negative claim is therefore not yet fully established; the evidence is consistent with a more modest claim about LR-only tuning with standard Muon hyperparameters.

major comments (3)
  1. [§3.2 and Findings] The protocol tunes only the learning rate: 'holding all other hyperparameters (momentum, and for Muon the Newton–Schulz count J and the orthogonalization coefficients) at standard values.' The Muon-specific values (µ=0.95, J=5, and the quintic coefficients (3.4445,-4.7750,2.0315)) are inherited from LLM-scale practice. These are optimizer hyperparameters, so the conclusion that Muon's advantage disappears 'under equal tuning' is not established by this protocol. The paper should either add a sensitivity analysis for J, the orthogonalization polynomial, and momentum (at least on the factorization and completion settings where Muon loses), or revise the claim to 'when only the learning rate is tuned and Muon uses its standard defaults.' Appendix I already contains a similar acknowledgment for the depth experiment; a parallel caveat is needed for the main claim.
  2. [Appendix K, Eq. (36); §3.2] Appendix K derives that, for the same learning rate η, AdamW's update has Frobenius norm roughly √N times larger than Muon's (Eq. (36)); with N=100 this factor is about 10. Yet the learning-rate grid is identical for all optimizers (25 points in [1e-4, 5e-1] for factorization/completion, 20 points in [1e-4, 3.2] for NMF). If Muon's optimal learning rate lies above the grid, the reported 'tuned' Muon loss is not its true tuned loss. The paper does not report the location of each optimizer's best learning rate relative to the grid boundaries. Please add this information, and either extend the grid or rescale the Muon learning rate by roughly √N to rule out an asymmetric search artifact.
  3. [Table 1; §4] Several 'tuned winner' conclusions are based on 3-seed geometric means with differences of only a factor of 2–3 (e.g., completion κ=1: Muon 1.4×10^-16 vs AdamW 4.8×10^-16; completion κ=5: 5.7×10^-15 vs 5.5×10^-15). These are within the reported log-std bands and cannot support a ranking. The large gaps on factorization and NMF are robust, but for the near-tie rows the paper should report per-seed values or otherwise state a criterion for declaring a winner. This matters because the paper's 'tuned ranking varies across problems' narrative depends on many such comparisons.
minor comments (5)
  1. [§2] Typographical errors: 'We an analyze the exact softplus' and 'Moreover, We analyze' should be cleaned up.
  2. [Appendix K.2] The text cites 'Loshchilov & Hutter (2017)' but this reference is missing from the reference list. Please add it.
  3. [Figure 6 and Eq. (15)] The alignment matrix M_ij uses 'learned singular vectors u_i' and 'true eigenvectors e_j', but the relation between u_i and the factor matrices is not defined in the main text. Please clarify the notation and ordering used to compute M.
  4. [Table 1] The row labels for the completion and NMF blocks are visually confusing ('Completion (κ)', 'κ=14.6×10^-16', etc.). Align the condition number/rank values with the loss values, e.g., by using separate columns.
  5. [General] No code or data repository is linked. For an empirical benchmark paper, a reproducibility statement with code and seeds would substantially strengthen the contribution.

Circularity Check

0 steps flagged

No circular derivation; all central claims are empirical comparisons with acknowledged protocol limitations.

full rationale

The paper does not derive any prediction from a fitted parameter or from a self-citation chain. Its central claims (Muon does not consistently beat AdamW on factorization/completion, but wins on NMF) are empirical findings from learning-rate sweeps, summarized in Table 1 and Figures 1-2. Appendix K derives the AdamW and Muon updates from their defining proxy problems (Propositions 1 and 2, Eqs. 19-36); these derivations do not assume the comparative conclusion. The only possible concern is Section 3.2's protocol, which tunes only the learning rate while holding momentum and Muon's Newton-Schulz J/coefficients 'at standard values'; that is a fairness/coverage limitation that could affect the strength of the 'equal tuning' wording, but it is not a circular reduction: no fitted value is renamed as a prediction and no equation equals the target result by construction. Appendix I also transparently states two limitations of the TT-depth comparison. Self-citations (Parviz & Yoshida 2025; Jha et al. 2026; Sghaier et al. 2026) appear only in related-work passages and are not load-bearing for any central result. Score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper introduces no fitted model parameters or invented theoretical entities. Its central claim depends on experimental controls rather than on a fitted derivation; the free parameters listed are hand-chosen hyperparameters inherited from prior Muon practice that are held fixed during tuning.

free parameters (3)
  • Muon orthogonalization coefficients (a,b,c) = 3.4445, -4.7750, 2.0315
    Chosen by Jordan (2024) and held fixed for all Muon runs (Appendix A). The paper's central comparison depends on this particular approximate polar map; different coefficients could change rankings.
  • Newton-Schulz iteration count J = 5
    Set to the standard value in Section 3.2; increasing J makes Muon closer to the idealized polar update and could alter the conclusions.
  • Muon momentum mu = 0.95
    Nesterov momentum held fixed in all Muon runs; momentum is a hyperparameter with material effect on convergence, yet only the learning rate is swept.
axioms (3)
  • domain assumption Low-rank matrix factorization is a valid controlled testbed for spectral optimizers whose advantages were reported in deep learning.
    The paper motivates MF as a way to isolate Muon's spectral behavior (Section 1, Appendix H); if MF does not capture the mechanism behind LLM-scale gains, the negative results may not transfer.
  • ad hoc to paper Sweeping only the learning rate, while holding all other hyperparameters fixed, constitutes fair per-optimizer tuning.
    Section 3.2 fixes momentum and Muon's Newton-Schulz settings; this is a load-bearing assumption for the phrase 'under equal tuning'.
  • domain assumption A fixed 3000-iteration budget with a patience-based decay schedule is sufficient for all optimizers to reach their attainable floors.
    Section 3.2 describes the budget and schedule; if some methods need substantially more iterations, the ranking could be an artifact of the budget rather than of the optimizer.

pith-pipeline@v1.3.0-alltime-deepseek · 21310 in / 14274 out tokens · 144818 ms · 2026-08-04T04:17:39.934765+00:00 · methodology

0 comments
read the original abstract

Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.

Figures

Figures reproduced from arXiv: 2607.13246 by Alex Cloninger, Ali Parviz, Gal Mishne.

Figure 1
Figure 1. Figure 1: Low-rank factorization, conditioning sweep (target rank r=15, matched search rank k=15, dimension d=100). Tuned final loss vs. learning rate for each condition number κ, from well-conditioned (κ=1) to strongly ill-conditioned (κ=625) (geometric mean over 3 seeds, ±log-std band). The target’s 15 singular values are spaced linearly from κ down to 1, so κ is exactly the condition number. When tuned, AdamW and… view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Non-negative factorization. Final loss (y-axis, log scale; geometric mean over 3 seeds, shaded ±one-log-standard-deviation band) versus learning rate (x-axis, log scale). Panels (a–c) use the uniform spectrum and (d–f) the decayed spectrum, at factor ranks r = 10, 50, 100 respectively. In both spectra, tuned Muon is the only method to fit the target across ranks; AdamW trails by several orders and GD/SignG… view at source ↗
Figure 3
Figure 3. Figure 3: AdamW outperforms Muon variants across spectrum shapes (matrix factorization, target m×n = 150×150, rank r=5). Tuned final loss for each of the seven spectral profiles at fixed κ = 104 (mean over 3 seeds). Both Muon and Muon without Nesterov-style momentum exhibit degraded relative performance on most spectral shapes, whereas AdamW remains effective despite the extreme ill-conditioning. 0 20 40 60 80 0 20 … view at source ↗
Figure 4
Figure 4. Figure 4: Spectral subspace recovery (Gaussian-kernel target K ∈ R 100×100, full rank k=100, tuned LR, 12 inits). Alignment Mij = |⟨ui , ej ⟩| (Eq. 16) between learned singular vectors and true eigenvectors of K. (a) AdamW and (b) Muon both recover the top eigen￾vectors, diverging in the near-degenerate mid-band. (c) The difference isolates it: AdamW holds a tighter ordered diagonal (blue), Muon disperses onto neigh… view at source ↗
Figure 4
Figure 4. Figure 4: Tensor-Train Noiseless regime (true rank r ⋆ = 4). Final loss versus learning rate for each optimizer, across search ranks r ∈ {2, 4, 8, 20} spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max. • Matrix completion (conditioning). A rank-4 symmetric PSD target with the same conditioning sweep, observed on a unif… view at source ↗
Figure 5
Figure 5. Figure 5: Tensor-Train Noiseless regime (true rank r ⋆ = 4). Final loss versus learning rate for each optimizer, across search ranks r ∈ {2, 4, 8, 20} spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max. 3.2. Evaluation protocol We evaluate recovery with the normalized mean-squared error (NMSE), the squared recon￾structi… view at source ↗
Figure 5
Figure 5. Figure 5: Gaussian-kernel NMF at a principled bottleneck. Non-negative factoriza￾tion of a fixed Gaussian RBF kernel K ∈ R 100×100 at search rank r=24 (the smallest rank capturing 95% of K’s spectral energy). On this smooth, well-behaved landscape Muon’s orthogonalized updates give no advantage: AdamW reaches the same global floor as Muon. families share the same endpoints (smin, smax) = (10−3 , 10) and hence the sa… view at source ↗
Figure 6
Figure 6. Figure 6: Spectral subspace recovery: Muon vs. AdamW. Absolute alignment Mij = |⟨ui , ej ⟩| (Eq. 16) between learned singular vectors ui and true eigenvectors ej of K, every 500 steps (AdamW top, Muon bottom). A bright, correctly-ordered diagonal indicates faithful recovery; off-diagonal mass reflects mixing. AdamW sharpens its diagonal steadily, while Muon stays near-random for ∼1500 steps, then snaps into a more d… view at source ↗
Figure 6
Figure 6. Figure 6: Spectral subspace recovery: Muon vs. AdamW. Absolute alignment Mij = |⟨ui , ej ⟩| (Eq. 15) between learned singular vectors ui and true eigenvectors ej of K, every 500 steps (AdamW top, Muon bottom). A bright, correctly-ordered diagonal indicates faithful recovery; off-diagonal mass reflects mixing. AdamW sharpens its diagonal steadily, while Muon stays near-random for ∼1500 steps, then snaps into a more d… view at source ↗
Figure 7
Figure 7. Figure 7: Matrix completion, conditioning sweep. Each panel plots the final re￾construction loss (y-axis, log scale; geometric mean over 3 seeds with a shaded ±one-log￾standard-deviation band) against the learning rate (x-axis, log scale), for condition number κ. Muon and AdamW are indistinguishable in the well-conditioned regime; all methods degrade and converge toward one another as κ grows. 1e-04 4e-04 2e-03 7e-0… view at source ↗
Figure 7
Figure 7. Figure 7: Matrix completion, conditioning sweep. Each panel plots the final re￾construction loss (y-axis, log scale; geometric mean over 3 seeds with a shaded ±one-log￾standard-deviation band) against the learning rate (x-axis, log scale), for condition number κ. Muon and AdamW are indistinguishable in the well-conditioned regime; all methods degrade and converge toward one another as κ grows. such spectral updates … view at source ↗
Figure 8
Figure 8. Figure 8: Matrix completion, search-rank sweep (true rank 4, κ=5). Over￾parameterization (rank 100) widens the band of effective learning rates and lets every method except GD reach machine precision. Implicit bias and generalization. A second line of work studies the implicit bias induced by Muon. Fan et al. (2025) show that in linear classification, idealized Muon converges to a solution that maximizes margin with… view at source ↗
Figure 8
Figure 8. Figure 8: Matrix completion, search-rank sweep (true rank 4, κ=5). Over￾parameterization (rank 100) widens the band of effective learning rates and lets every method except GD reach machine precision. 10 4 10 3 10 2 10 1 10 0 10 1 10 5 10 4 10 3 10 2 10 1 10 0 final loss k = 5 (under-param.) 10 4 10 3 10 2 10 1 10 0 10 1 10 5 10 4 10 3 10 2 10 1 10 0 k = 15 (under-param.) 10 4 10 3 10 2 10 1 10 0 10 1 10 6 10 5 10 4… view at source ↗
Figure 9
Figure 9. Figure 9: Tensor-Train Noisy regime (true rank r ⋆ = 30, additive observation noise). Final loss versus learning rate for each optimizer, across search ranks r ∈ {5, 15, 30, 60} spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max; the loss floors at the noise level rather than at zero. been drawn between Muon and second-… view at source ↗
Figure 9
Figure 9. Figure 9: Tensor-Train Noisy regime (true rank r ⋆ = 30, additive observation noise). Final loss versus learning rate for each optimizer, across search ranks r ∈ {5, 15, 30, 60} spanning under-, exactly- (r = r ⋆ , bold panel), and over-parameterized settings. Solid lines are the median over 3 seeds; shaded bands the min–max; the loss floors at the noise level rather than at zero. rather than focusing on dominant di… view at source ↗
Figure 10
Figure 10. Figure 10: Gaussian-kernel NMF at a principled bottleneck. Non-negative factoriza￾tion of a fixed Gaussian RBF kernel K ∈ R 100×100 at search rank r=24 (the smallest rank capturing 95% of K’s spectral energy). Tuned final loss vs. learning rate (geometric mean over 3 seeds, ±log-std band; patience-based decay schedule). On this smooth, well-behaved landscape Muon’s orthogonalized updates give no advantage: AdamW rea… view at source ↗
Figure 10
Figure 10. Figure 10: Best-tuned reconstruction loss versus tensor-train depth. Each curve shows the final MSE of one optimizer at its best learning rate (minimum over the grid) for a given depth, averaged over three seeds; shaded bands span the per-seed min–max. Left: clean, low-rank target. Right: noisy, higher-rank target. Search bond rank is fixed to the over-parameterized regime (R=12 clean, R=30 noisy). Lower is better; … view at source ↗
Figure 11
Figure 11. Figure 11: Best-tuned reconstruction loss versus tensor-train depth. Each curve shows the final MSE of one optimizer at its best learning rate (minimum over the grid) for a given depth, averaged over three seeds; shaded bands span the per-seed min–max. Left: clean, low-rank target. Right: noisy, higher-rank target. Search bond rank is fixed to the over-parameterized regime (R=12 clean, R=30 noisy). Lower is better; … view at source ↗
Figure 11
Figure 11. Figure 11: Learning-rate stability across activation and capacity. Final MSE (log) vs. learning rate (log) for four optimizers fitting the Gaussian kernel as Kˆ = ϕ(XW)V ⊤; mean over three seeds, bands show min–max, divergent runs capped at 105 . The activation ϕ is exact softplus or its second-order Taylor surrogate, and R is the search rank: (a) softplus, R=10; (b) Taylor, R=10; (c) softplus, R=100; (d) Taylor, R=… view at source ↗
Figure 12
Figure 12. Figure 12: Learning-rate stability across activation and capacity. Final MSE (log) vs. learning rate (log) for four optimizers fitting the Gaussian kernel as Kˆ = ϕ(XW)V ⊤; mean over three seeds, bands show min–max, divergent runs capped at 105 . The activation ϕ is exact softplus or its second-order Taylor surrogate, and R is the search rank: (a) softplus, R=10; (b) Taylor, R=10; (c) softplus, R=100; (d) Taylor, R=… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

119 extracted references · 23 linked inside Pith

  1. [1]

    2024 , note =

    Keller Jordan , title =. 2024 , note =

    2024

  2. [2]

    CoRR , volume =

    Jingyuan Liu and Jianlin Su and Xingcheng Yao and Zhejun Jiang and Guokun Lai and Yulun Du and Yidao Qin and Weixin Xu and Enzhe Lu and Junjie Yan and Yanru Chen and Huabin Zheng and Yibo Liu and Shaowei Liu and Bohong Yin and Weiran He and Han Zhu and Yuzhi Wang and Jianzhou Wang and Mengnan Dong and Zheng Zhang and Yongsheng Kang and Hao Zhang and Xinra...

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.16982 2025

  3. [3]

    Improved Convergence Rates of

    Shuntaro Nagashima and Hideaki Iiduka , year=. Improved Convergence Rates of. 2601.19400 , archivePrefix=

    arXiv

  4. [4]

    Training Deep Learning Models with Norm-Constrained LMOs , booktitle =

    Thomas Pethick and Wanyun Xie and Kimon Antonakopoulos and Zhenyu Zhu and Antonio Silveti. Training Deep Learning Models with Norm-Constrained LMOs , booktitle =. 2025 , url =

    2025

  5. [5]

    2022 , eprint=

    Orthogonalising gradients to speed up neural network optimisation , author=. 2022 , eprint=

    2022

  6. [6]

    Proceedings of the 35th International Conference on Machine Learning,

    Jeremy Bernstein and Yu. Proceedings of the 35th International Conference on Machine Learning,. 2018 , url =

    2018

  7. [7]

    Preconditioning Benefits of Spectral Orthogonalization in

    Jianhao Ma and Yu Huang and Yuejie Chi and Yuxin Chen , year=. Preconditioning Benefits of Spectral Orthogonalization in. 2601.13474 , archivePrefix=

    arXiv

  8. [8]

    arXiv preprint arXiv:2604.04356 , year=

    REAM: Merging improves pruning of experts in LLMs , author=. arXiv preprint arXiv:2604.04356 , year=

    Pith/arXiv arXiv

  9. [9]

    New Perspectives in Graph Machine Learning , year=

    Nonlinear Laplacians Improve Signed-Directed Graph Learning , author=. New Perspectives in Graph Machine Learning , year=

  10. [10]

    International conference on machine learning , pages=

    On the optimization of deep networks: Implicit acceleration by overparameterization , author=. International conference on machine learning , pages=. 2018 , organization=

    2018

  11. [11]

    arXiv preprint arXiv:1711.00811 , year=

    Expressive power of recurrent neural networks , author=. arXiv preprint arXiv:1711.00811 , year=

    Pith/arXiv arXiv

  12. [12]

    arXiv preprint arXiv:2605.19943 , year=

    Probabilistic Tiny Recursive Model , author=. arXiv preprint arXiv:2605.19943 , year=

    Pith/arXiv arXiv

  13. [13]

    Conference on learning theory , pages=

    On the expressive power of deep learning: A tensor analysis , author=. Conference on learning theory , pages=. 2016 , organization=

    2016

  14. [14]

    Advances in neural information processing systems , volume=

    Tensorizing neural networks , author=. Advances in neural information processing systems , volume=

  15. [15]

    A Note on the Convergence of

    Jiaxiang Li and Mingyi Hong , year=. A Note on the Convergence of. 2502.02900 , archivePrefix=

    Pith/arXiv arXiv

  16. [16]

    Gluon: Making

    Artem Riabinin and Egor Shulgin and Kaja Gruntkowska and Peter Richt. Gluon: Making. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.13416 , eprinttype =. 2505.13416 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.13416 2025

  17. [17]

    CoRR , volume =

    Wei Shen and Ruichuan Huang and Minhui Huang and Cong Shen and Jiawei Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.23737 , eprinttype =. 2505.23737 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.23737 2025

  18. [18]

    CoRR , volume =

    Lizhang Chen and Jonathan Li and Qiang Liu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.15054 , eprinttype =. 2506.15054 , timestamp =

    doi:10.48550/arxiv.2506.15054 2025

  19. [19]

    Convergence Bound and Critical Batch Size of

    Naoki Sato and Hiroki Naganuma and Hideaki Iiduka , year=. Convergence Bound and Critical Batch Size of. 2507.01598 , archivePrefix=

    Pith/arXiv arXiv

  20. [20]

    Journal of Machine Learning Research , year =

    Vardan Papyan , title =. Journal of Machine Learning Research , year =

  21. [21]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    An Investigation into Neural Net Optimization via Hessian Eigenvalue Density , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

    2019

  22. [22]

    Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond , journal =

    Levent Sagun and L. Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond , journal =. 2017 , url=

    2017

  23. [23]

    Levent Sagun and Utku Evci and V. Ugur G. Empirical Analysis of the Hessian of Over-Parametrized Neural Networks , booktitle =. 2018 , url =

    2018

  24. [24]

    Investigating the Overlooked Hessian Structure: From

    Qian-Yuan Tang and Yufei Gu and Yunfeng Cai and Mingming Sun and Ping Li and zhou Xun and Zeke Xie , booktitle=. Investigating the Overlooked Hessian Structure: From. 2025 , url=

    2025

  25. [25]

    2022 , isbn =

    Measure Theory, Probability, and Stochastic Processes , series =. 2022 , isbn =. doi:10.1007/978-3-031-14205-5 , url =

    doi:10.1007/978-3-031-14205-5 2022

  26. [26]

    Polloreno and Karl Stratos and Philip Monk and Adarsh Chaluvaraju and Andrew Hojel and Andrew Ma and Anil Thomas and Ashish Tanwer and Darsh J

    Ishaan Shah and Anthony M. Polloreno and Karl Stratos and Philip Monk and Adarsh Chaluvaraju and Andrew Hojel and Andrew Ma and Anil Thomas and Ashish Tanwer and Darsh J. Shah and Khoi Nguyen and Kurt Smith and Michael Callahan and Michael Pust and Mohit Parmar and Peter Rushton and Platon Mazarakis and Ritvik Kapila and Saurabh Srivastava and Somanshu Si...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.02222 2025

  27. [27]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Only tails matter: Average-Case Universality and Robustness in the Convex Regime , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , volume =

    2022

  28. [28]

    2019 , eprint=

    Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization , author=. 2019 , eprint=

    2019

  29. [29]

    2019 , url=

    Gradient Descent Happens in a Tiny Subspace , author=. 2019 , url=

    2019

  30. [30]

    Minhak Song and Kwangjun Ahn and Chulhee Yun , booktitle=. Does. 2025 , url=

    2025

  31. [31]

    Proceedings of the 37th International Conference on Machine Learning , pages =

    Acceleration through spectral density estimation , author =. Proceedings of the 37th International Conference on Machine Learning , pages =. 2020 , volume =

    2020

  32. [32]

    CoRR , volume =

    Dmitry Kovalev , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2503.12645 , eprinttype =. 2503.12645 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.12645 2025

  33. [33]

    Beyond the Ideal: Analyzing the Inexact

    Egor Shulgin and Sultan AlRashed and Francesco Orabona and Peter Richt. Beyond the Ideal: Analyzing the Inexact. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.19933 , eprinttype =. 2510.19933 , timestamp =

    doi:10.48550/arxiv.2510.19933 2025

  34. [34]

    Error Feedback for

    Kaja Gruntkowska and Alexander Gaponov and Zhirayr Tovmasyan and Peter Richt. Error Feedback for. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.00643 , eprinttype =. 2510.00643 , timestamp =

    doi:10.48550/arxiv.2510.00643 2025

  35. [35]

    Convergence of

    Kim, Gyu Yeol and Oh,. Convergence of. The Fourteenth International Conference on Learning Representations , year=

  36. [36]

    CoRR , volume =

    Damek Davis and Dmitriy Drusvyatskiy , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2512.04299 , eprinttype =. 2512.04299 , timestamp =

    doi:10.48550/arxiv.2512.04299 2025

  37. [37]

    CoRR , volume =

    Weijie Su , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2511.00674 , eprinttype =. 2511.00674 , timestamp =

    doi:10.48550/arxiv.2511.00674 2025

  38. [38]

    CoRR , volume =

    Noah Amsel and David Persson and Christopher Musco and Robert Gower , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.16932 , eprinttype =. 2505.16932 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.16932 2025

  39. [39]

    2025 , eprint=

    Factorization-free Orthogonal Projection onto the Positive Semidefinite Cone with Composite Polynomial Filtering , author=. 2025 , eprint=

    2025

  40. [40]

    2024 , url =

    Keller Jordan and Jeremy Bernstein and Brendan Rappazzo and @fernbear.bsky.social and Boza Vlado and You Jiacheng and Franz Cesista and Braden Koszarsky and @Grad62304977 , title =. 2024 , url =

    2024

  41. [41]

    Rakhuba , title =

    Ekaterina Grishina and Matvey Smirnov and Maxim V. Rakhuba , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2506.10935 , eprinttype =. 2506.10935 , timestamp =

    doi:10.48550/arxiv.2506.10935 2025

  42. [42]

    2025 , month =

    Franz Louis Cesista and You Jiacheng and Keller Jordan , title =. 2025 , month =

    2025

  43. [43]

    PolarGrad:

    Tim Tsz. PolarGrad:. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2505.21799 , eprinttype =. 2505.21799 , timestamp =

    doi:10.48550/arxiv.2505.21799 2025

  44. [44]

    2025 , url =

    Boumal, Nicolas and Gonon, Antoine , title =. 2025 , url =

    2025

  45. [45]

    Advances in neural information processing systems , volume=

    Implicit regularization in deep matrix factorization , author=. Advances in neural information processing systems , volume=

  46. [46]

    Advances in neural information processing systems , volume=

    Implicit regularization in matrix factorization , author=. Advances in neural information processing systems , volume=

  47. [47]

    2025 , eprint=

    Dion: Distributed Orthonormalized Updates , author=. 2025 , eprint=

    2025

  48. [48]

    CoRR , volume =

    Chuan He and Zhanwang Deng and Zhaosong Lu , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.11983 , eprinttype =. 2509.11983 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.11983 2025

  49. [49]

    CoRR , volume =

    Feihu Huang and Yuning Luo and Songcan Chen , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2509.14562 , eprinttype =. 2509.14562 , timestamp =

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.14562 2025

  50. [50]

    CoRR , volume =

    Ahmed Khaled and Kaan Ozkara and Tao Yu and Mingyi Hong and Youngsuk Park , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.16981 , eprinttype =. 2510.16981 , timestamp =

    doi:10.48550/arxiv.2510.16981 2025

  51. [51]

    CoRR , volume =

    Rui Pan and Yang Luo and Yuxing Liu and Yang You and Tong Zhang , title =. CoRR , volume =. 2025 , url =. doi:10.48550/ARXIV.2510.17802 , eprinttype =. 2510.17802 , timestamp =

    doi:10.48550/arxiv.2510.17802 2025

  52. [52]

    Why transformers need

    Zhang, Yushun and Chen, Congliang and Ding, Tian and Li, Ziniu and Sun, Ruoyu and Luo, Zhiquan , journal=. Why transformers need

  53. [53]

    Provable acceleration of

    Xu, Zhenghao and Wang, Yuqing and Zhao, Tuo and Ward, Rachel and Tao, Molei , booktitle=. Provable acceleration of

  54. [54]

    Conference On Learning Theory , pages=

    Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations , author=. Conference On Learning Theory , pages=. 2018 , organization=

    2018

  55. [55]

    arXiv preprint arXiv:2409.20325 , year=

    Old optimizer, new norm: An anthology , author=. arXiv preprint arXiv:2409.20325 , year=

    Pith/arXiv arXiv

  56. [56]

    Advances in neural information processing systems , volume=

    Preconditioned spectral descent for deep learning , author=. Advances in neural information processing systems , volume=

  57. [57]

    Training deep learning models with norm-constrained

    Pethick, Thomas and Xie, Wanyun and Antonakopoulos, Kimon and Zhu, Zhenyu and Silveti-Falls, Antonio and Cevher, Volkan , journal=. Training deep learning models with norm-constrained

  58. [58]

    BIT Numerical Mathematics , volume=

    Perturbation bounds in connection with singular value decomposition , author=. BIT Numerical Mathematics , volume=. 1972 , publisher=

    1972

  59. [59]

    Proceedings of the International Congress of Mathematicians 2010 , pages=

    Non-asymptotic theory of random matrices: extreme singular values , author=. Proceedings of the International Congress of Mathematicians 2010 , pages=. 2010 , organization=

    2010

  60. [60]

    Numerical Linear Algebra with Applications , volume=

    Some remarks on the perturbation of polar decompositions for rectangular matrices , author=. Numerical Linear Algebra with Applications , volume=. 2006 , publisher=

    2006

  61. [61]

    International Conference on Machine Learning , pages=

    Shampoo: Preconditioned stochastic tensor optimization , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    2018

  62. [62]

    2018 , organization=

    Bernstein, Jeremy and Wang, Yu-Xiang and Azizzadenesheli, Kamyar and Anandkumar, Animashree , booktitle=. 2018 , organization=

    2018

  63. [63]

    Stat , volume=

    On the non-asymptotic and sharp lower tail bounds of random variables , author=. Stat , volume=. 2020 , publisher=

    2020

  64. [64]

    SIAM journal on Matrix Analysis and Applications , volume=

    The geometry of algorithms with orthogonality constraints , author=. SIAM journal on Matrix Analysis and Applications , volume=. 1998 , publisher=

    1998

  65. [65]

    2013 , publisher=

    Matrix computations , author=. 2013 , publisher=

    2013

  66. [66]

    SIAM Journal on Matrix Analysis and Applications , volume=

    New perturbation bounds for the unitary polar factor , author=. SIAM Journal on Matrix Analysis and Applications , volume=. 1995 , publisher=

    1995

  67. [67]

    SIAM journal on matrix analysis and applications , volume=

    Perturbation bounds of unitary and subunitary polar factors , author=. SIAM journal on matrix analysis and applications , volume=. 2002 , publisher=

    2002

  68. [68]

    arXiv preprint arXiv:2410.21265 , year=

    Modular duality in deep learning , author=. arXiv preprint arXiv:2410.21265 , year=

    Pith/arXiv arXiv

  69. [69]

    IEEE Journal of Selected Topics in Signal Processing , volume=

    Stochastic spectral descent for discrete graphical models , author=. IEEE Journal of Selected Topics in Signal Processing , volume=. 2015 , publisher=

    2015

  70. [70]

    Transformers are

    Katharopoulos, Angelos and Vyas, Apoorv and Pappas, Nikolaos and Fleuret, Fran. Transformers are. International conference on machine learning , pages=. 2020 , organization=

    2020

  71. [71]

    2008 , publisher=

    Functions of matrices: theory and computation , author=. 2008 , publisher=

    2008

  72. [72]

    Stochastic spectral descent for restricted

    Carlson, David and Cevher, Volkan and Carin, Lawrence , booktitle=. Stochastic spectral descent for restricted. 2015 , organization=

    2015

  73. [73]

    arXiv preprint arXiv:2202.07052 , year=

    Orthogonalising gradients to speed up neural network optimisation , author=. arXiv preprint arXiv:2202.07052 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2504.16041 , year=

    Muon Optimizer Accelerates Grokking , author=. arXiv preprint arXiv:2504.16041 , year=

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:2310.01769 , year=

    How over-parameterization slows down gradient descent in matrix sensing: The curses of symmetry and initialization , author=. arXiv preprint arXiv:2310.01769 , year=

    Pith/arXiv arXiv

  76. [76]

    The Polar Express: Optimal Matrix Sign Methods and Their Application to the

    Amsel, Noah and Persson, David and Musco, Christopher and Gower, Robert , journal=. The Polar Express: Optimal Matrix Sign Methods and Their Application to the

  77. [77]

    Practical efficiency of

    Shah, Ishaan and Polloreno, Anthony M and Stratos, Karl and Monk, Philip and Chaluvaraju, Adarsh and Hojel, Andrew and Ma, Andrew and Thomas, Anil and Tanwer, Ashish and Shah, Darsh J , journal=. Practical efficiency of

  78. [78]

    Understanding

    Maes, Lucas and Zhang, Tianyue H and Jolicoeur-Martineau, Alexia and Mitliagkas, Ioannis and Scieur, Damien and Lacoste-Julien, Simon and Guille-Escuret, Charles , journal=. Understanding

  79. [79]

    Towards quantifying the preconditioning effect of

    Das, Rudrajit and Agarwal, Naman and Sanghavi, Sujay and Dhillon, Inderjit S , journal=. Towards quantifying the preconditioning effect of

  80. [80]

    arXiv preprint arXiv:2502.01763 , year=

    On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning , author=. arXiv preprint arXiv:2502.01763 , year=

    Pith/arXiv arXiv

Showing first 80 references.