Pith. sign in

REVIEW 3 major objections 5 minor 6 cited by

AOL preconditioning lets the Muon optimizer drop one Newton-Schulz iteration without loss of accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 18:33 UTC pith:2Z6G4CWZ

load-bearing objection Plausible trick, genuine new combination, but the central attribution is untested and the abstract oversells the gains. the 3 major comments →

arxiv 2512.04632 v2 pith:2Z6G4CWZ submitted 2025-12-04 cs.AI

Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates

Thibaut Boissin (IRIT-MISFIT) , Thomas Massena (DTIPG - SNCF , IRIT-MISFIT) , Franck Mamalet , Mathieu Serrurier (IRIT-MISFIT) This is my paper
classification cs.AI
keywords Muon optimizerNewton-Schulz iterationpolar factoralmost-orthogonal preconditioningorthogonalizationtraining speedupsteepest descentgradient preconditioning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that replacing the usual Frobenius-norm normalization with an almost-orthogonal column rescaling before Newton-Schulz iterations makes the iterative orthogonalization converge fast enough that Muon can run four iterations instead of the usual five while keeping equal polar accuracy and final model quality. If true, this is a drop-in change: no hyperparameter retuning, roughly 20 percent lower orthogonalization cost, and consistent end-to-end runtime gains—around 3 percent in tightly tuned speed-run benchmarks and more at medium scales where the orthogonalization overhead is visible. The paper also argues that the bias introduced by preconditioning is benign: the resulting update remains a strict descent direction, so training cannot be pushed uphill by the approximation. Why this matters: the main practical barrier to orthogonality-based optimizers is the cost of the orthogonalization step, and any reduction widens the range of models where they are affordable.

Core claim

The paper's central claim is that the initial normalization step shared by Muon and Muon+—dividing the gradient by its Frobenius norm—is a poor starting point for Newton-Schulz, because it preserves the condition number and typically sends the spectral norm far below 1. Turbo-Muon replaces it with AOL preconditioning: scale the columns of the gradient by the inverse square roots of the row sums of the Gram matrix X^T X. This yields an initial matrix much closer to orthogonal, especially for large matrices, and the Gram matrix computed for the preconditioner can be cached and reused as the first product of the first Newton-Schulz iteration, so the extra cost is nearly negligible. The improved

What carries the argument

The central object is the almost-orthogonal (AOL) rescaling: a diagonal matrix S whose entries are the inverse square roots of the row sums of the Gram matrix A0 = X^T X, applied to the gradient columns before Newton-Schulz iterations. Its work is to bound the column-space coherence—via Gershgorin-style row-sum bounds on the Gram matrix—so that the initial matrix is closer to orthogonal and better conditioned than Frobenius-normalized input. The computational trick is that A0, once computed for the preconditioner, is reused as the first matrix product inside the first Newton-Schulz step, reducing the preconditioner's overhead to a cheap elementwise rescaling and making the entire scheme near

Load-bearing premise

The load-bearing premise is that a same-budget baseline—Muon or Muon+ run with four Newton-Schulz iterations under the same truncated coefficient schedule—would be meaningfully worse in polar error and final loss; the paper's comparisons always give baselines one extra iteration, so the attribution of the speedup to AOL preconditioning depends on this unshown comparison.

What would settle it

Run Muon+ with exactly four Newton-Schulz iterations using the same truncated coefficient schedule as Turbo-Muon, and compare polar error on random and heavy-tailed matrices plus end-to-end validation loss on the paper's GPT and CIFAR-10 speed-runs; if the four-iteration baseline matches Turbo-Muon's loss and polar error, the central claim that AOL enables dropping the iteration is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Muon-family optimizers can adopt Turbo-Muon as a drop-in replacement, replacing the initial Frobenius normalization with AOL rescaling and reusing the Gram matrix, to gain about a 20 percent reduction in orthogonalization cost with no loss of final loss or accuracy.
  • End-to-end training time should improve by a few percent in regimes where orthogonalization is a visible fraction of each step (medium batch sizes, non-sharded training), with the largest gains on large matrices where Newton-Schulz iterations dominate.
  • The polar-error improvement grows with matrix size, so the method becomes more beneficial as layers widen, directly targeting the regime where exact SVD is impractical and Newton-Schulz is most expensive.
  • The descent-direction guarantee means the preconditioned update cannot diverge from the true gradient direction; thus the usual stability arguments for Muon carry over to Turbo-Muon even when many Newton-Schulz iterations are used.
  • Practitioners who need higher orthogonalization accuracy can keep five iterations and obtain lower polar error than the usual five-iteration baselines, or trade the saved iteration for other computational savings during training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Our inference: because the benefit comes from a better-conditioned starting point, the same AOL rescaling likely transfers to other iterative orthogonalization schemes—higher-order Newton-Schulz variants, Chebyshev-type schedules, and non-square matrices—where a cheap, reuse-friendly preconditioner could similarly accelerate convergence.
  • Our inference: the appendix shows that polynomial coefficients tuned for Frobenius-normalized Newton-Schulz are suboptimal when combined with AOL. Retuning the coefficients for the preconditioned start could plausibly remove a second iteration or push polar error further down, a natural and testable follow-up that the paper leaves open.
  • Our inference: the Gershgorin-coherence interpretation suggests AOL rescaling actively shrinks updates for feature directions with high redundancy; this hints at a robustness property against feature collapse that goes beyond runtime, though the paper only gestures at it and does not claim it as a demonstrated effect.
  • Our inference: the descent-direction theorem is proven for the exact polar factor of the rescaled gradient; with a truncated four-iteration Newton-Schulz approximation, the guarantee transfers only approximately, so the practical safety of the method ultimately rests on how close the four-iteration approximation is to the exact polar factor in realistic training regimes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Turbo-Muon, a modification of the Newton–Schulz orthogonalization step used in the Muon optimizer. The key idea is to replace Frobenius-norm pre-scaling with an Almost-Orthogonal Layer (AOL) column rescaling, which the authors argue reduces the initial polar error and allows one of the usual five Newton–Schulz iterations to be removed. The paper reports synthetic polar-error experiments, heavy-tailed robustness checks, CIFAR-10 and NanoGPT training runs, and a theoretical steepest-descent interpretation of the resulting update. It claims a parameter-free drop-in replacement that is faster than Muon/Muon+ while matching final model quality.

Significance. If the central attribution were established, the contribution would be practically valuable: a 20% reduction in orthogonalization cost, no hyperparameter tuning, public implementations, and consistent small end-to-end runtime gains. The steepest-descent-in-a-rescaled-norm result and the strict-descent-direction lemma are useful theoretical additions, and the paper correctly avoids fitted parameters or self-derived constants. However, as submitted, the central claim that AOL specifically enables removing one Newton–Schulz iteration is not isolated from other implementation changes, and several numerical statements in the abstract outrun the reported tables.

major comments (3)
  1. [§4.2, Fig. 3a/4b, Table 3] The central claim that AOL preconditioning enables removing one Newton–Schulz iteration is not isolated. Every head-to-head comparison gives Turbo-Muon k iterations while baselines get k+1. Since Turbo-Muon also differs by a fused Triton kernel and by the coefficient-truncation schedule (App. D), the speedup could be due to fewer matrix multiplications or kernel implementation rather than AOL. A necessary control is a 4-iteration Muon+ run with the same truncated coefficients and the same kernel as Turbo-Muon but with Frobenius normalization. If that baseline matches Turbo-Muon's final loss and polar error, the claimed enabling role of AOL is not supported. This is directly load-bearing for the abstract and conclusions.
  2. [Table 3 / Abstract] The abstract claims 5–10% end-to-end improvement and 'matching reference performance'. The reported end-to-end reductions are 273.75→266.65 s (2.6%) in Table 3 and 2.66→2.64 s (0.75%) in Table 1. The 8–10% figure in Fig. 4a is a step-time simulation, not end-to-end training time. Moreover, Table 3 shows 3.2791 (Turbo-Muon, 4it) vs 3.2774 (Muon+ PE, 5it) with stds ≈0.0013–0.0014 over 10 runs; this difference is statistically significant (t≈2.8), so the phrase 'matching reference performance' overstates the result. Claims should be aligned with the measured effect sizes and significance.
  3. [Fig. 3a, §6] The polar-error evidence is suggestive but does not establish that training metrics are insensitive to the polar-error gap at four iterations. Fig. 3a compares Turbo-Muon at 4 iterations with baselines at 5; a same-iteration polar-error plot is needed. Additionally, the bias analysis in §6 and Table 2 reports εbias≈0.1, which is not negligible relative to εapprox, yet final accuracy is unchanged; the paper does not explain why a large polar bias has no effect on optimization in that experiment. A brief analysis or reference would make the training-insensitivity claim credible.
minor comments (5)
  1. [Algorithm 2, line 4] A1 = s^T A0 s is dimensionally inconsistent for a matrix A1; the correct update is A1 = S A0 S with S = diag(s), or componentwise A1_ij = s_i A0_ij s_j.
  2. [Appendix B] Figures 6, 7, and 8 say they reproduce 'Fig. 1' and 'Fig. 3a' of the main paper, but the preconditioning comparison appears to be Fig. 2; please verify the cross-references.
  3. [Abstract and Conclusion] The numerical claims are inconsistent across versions: the abstract mentions '~3%' training-time reduction and '5–10%' improvement, while Table 3 shows 2.6% and Table 1 shows 0.75%; the conclusion says 8–10% step-time speedups. Please reconcile these numbers so the reader can tell which quantity is being reported.
  4. [Table 2 / §6] There are typos such as 'high regimes' and 'hight regimes'; also clarify whether εbias and εapprox are normalized as in §3. The large εbias value deserves a one-sentence interpretation.
  5. [Eq. (6), §4.1] The notation |X^T X| should be defined as the elementwise absolute value, and the diagonal inverse square root should be stated explicitly; this will avoid confusion with matrix absolute value or determinant.

Circularity Check

0 steps flagged

No significant circularity: AOL preconditioning is parameter-free and externally cited, and the claimed speedup is an empirical combination not reduced to the paper's own inputs.

full rationale

The derivation chain is not circular. The AOL preconditioner is parameter-free and imported from Prach and Lampert [28], with the paper's polar-error comparisons (Figs. 2, 3a, App. B/C) being external numerical measurements rather than fitted values. The bias decomposition in Sec. 6 is a direct triangle inequality, and the steepest-descent result (Prop. 1, App. A) is an algebraic reparametrization of the cited Proposition 5 of Bernstein and Newhouse [5], supported by an independent positivity lemma (Lemma 1). No load-bearing premise rests on a self-citation: Muon [16], Dion [1], AOL [28], and Bernstein-Newhouse [5] are all by other authors and independently established. The main weakness is experimental: every head-to-head training comparison gives Turbo-Muon one fewer Newton-Schulz iteration than the Muon/Muon+ baseline, with no same-iteration, same-kernel control that isolates AOL from the removed iteration and the fused Triton kernel. That is a missing control and a correctness risk, not a reduction of the claimed speedup to the paper's own definitions or fitted parameters. There are no fitted inputs renamed as predictions, no imported uniqueness theorem from the authors' prior work, and no ansatz smuggled in via self-citation. Score 0.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The method introduces no fitted parameters; it imports AOL scaling (Eq. 6) and Newton-Schulz coefficients from prior work. The main load-bearing assumptions are the convergence condition for NS, the spectral-norm bound of AOL scaling, and the cited steepest-descent characterization of polar-factor updates.

axioms (5)
  • standard math Newton-Schulz iteration converges to the polar factor when the input spectral norm is <= 1.
    Used to justify the NS scheme after AOL scaling (Sec. 3, Eq. 1).
  • domain assumption AOL row-sum scaling keeps the spectral norm of the scaled matrix <= 1.
    Paper asserts Eq. (6) enforces ||X0||2 <= 1; a proof via PSD/Gershgorin-type bound is not given in the paper (Sec. 4.1).
  • standard math Steepest descent in the spectral norm equals the polar factor (Proposition 5 of Bernstein and Newhouse).
    Basis of Proposition 1 in Appendix A; imported from cited reference [5].
  • domain assumption Gradient distributions are heavy-tailed with tail index around 1.0-1.8, motivating Levy-distribution stress tests.
    Motivates the protocol in Appendix B; not load-bearing for the end-to-end runtime claim.
  • domain assumption The AOL scaling vector s has strictly positive entries.
    Lemma 1 requires S positive definite; this fails if a gradient column is identically zero (Sec. 6, App. A.1).

pith-pipeline@v1.3.0-alltime-deepseek · 16308 in / 14587 out tokens · 134943 ms · 2026-08-03T18:33:17.602415+00:00 · methodology

0 comments
read the original abstract

Orthogonality-based optimizers, such as Muon, have recently shown strong performance across large-scale training and community-driven efficiency challenges. However, these methods rely on a costly gradient orthogonalization step. Even efficient iterative approximations such as Newton-Schulz remain expensive, typically requiring dozens of matrix multiplications to converge. We introduce a pre-conditioning procedure that improves the initialization of the Newton--Schulz iterations while incurring negligible overhead. Furthermore, our pre-conditioning reduces the initial polar error and enables the removal of one Newton-Schulz iteration (out of the five iterations usually used in practice). The resulting implementation significantly reduces Muon's overhead. At the end-to-end training level, we observe consistent runtime improvements across speed-run and standard benchmarks, including $\sim$3% reductions in training time on multiple fast training benchmarks, while matching reference performance on both language and vision tasks. Crucially, these improvements require no hyperparameter tuning and can be adopted as a simple drop-in replacement. Beyond empirical gains, we provide theoretical insight into the geometry of the update and its potential robustness against feature collapse. Our code is publicly available on github, in optax and huggingface kernels.

Figures

Figures reproduced from arXiv: 2512.04632 by Franck Mamalet, IRIT-MISFIT), Mathieu Serrurier (IRIT-MISFIT), Thibaut Boissin (IRIT-MISFIT), Thomas Massena (DTIPG - SNCF.

Figure 1
Figure 1. Figure 1: Practical implementations of orthogonalization face a [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Turning preconditioning into runtime: Applying AOL before the algorithm improves its convergence speed (a). This can be used to remove an iteration while achieving a similar polar error. Removing one iteration out of 5 improves the runtime of the algorithm (b), making optimizers like Muon more scalable to large matrices. tion3 and compared three standard implementations listed in Sec. 2. We then compute th… view at source ↗
Figure 4
Figure 4. Figure 4: Turbo-Muon can make realistic training faster without impact on final loss. Fig. 4a shows that it can achieve non-negligible speedups on medium-scale training, with runtime improvements nearing 10% of the total step runtime. Fig. 4b shows that our approach does not induce perceptible loss degradation when an iteration is removed. of total training computational cost, which is usually domi￾nated by forward … view at source ↗
Figure 5
Figure 5. Figure 5: Understanding the nature of the remaining polar error. We decompose the polar error of the Turbo-Muon algorithm as an approximation error that depends on the number of NS iterations, along with a bias error that is introduced by AOL pre-conditioning. Results are measured on 100 random normal matrices. PolarFactor(AOL(X)) = Qaol: εbias(AOL, X) = ∥Q − Qaol∥F √ n This can be interpreted as the irreducible err… view at source ↗
Figure 6
Figure 6. Figure 6: We reproduced the Fig [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: We reproduced the Fig. 3a using heavy-tailed distributions. Despite an overall degradation in the resulting polar error, Turbo-Muon outperforms existing approaches by a significant margin. (a) Levy distribution: α = 1.0, β = 0 (b) Levy distribution: α = 1.5, β = 0 (c) Levy distribution: α = 2.0, β = 0 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Reproduction of Fig [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Reproduction of Fig [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Coefficients from [2] are suboptimal when combined with AOL-preconditioning. We recomputed the optimal coefficient for each number of iterations, using default parameters (l = 10−3 , cushion = 0.024, and safety factor = 2 × 10−2 ). For both Muon+ and Turbo-Muon , we report the performance using Polar Express (PE) and compare it with the original. On heavy-tailed distributions, these coefficients improve t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Why Muon Outperforms Adam: A Curvature Perspective

    cs.LG 2026-06 conditional novelty 7.0

    Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.

  2. Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

    math.OC 2026-05 conditional novelty 7.0

    Proposes equivariant optimizers matched to the symmetry groups of embeddings, SwiGLU projections and MoE routers, with experiments showing consistent gains over AdamW on language model pre-training.

  3. Muse: Representation Geometry of Muon Beyond Normalized Momentum

    cs.LG 2026-07 conditional novelty 6.0

    The matrix shape given to Muon's orthonormalization step is a genuine optimizer axis: square-ish reshapes match native Muon, skinnier reshapes interpolate toward normalized SGD with momentum.

  4. Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

    math.OC 2026-05 unverdicted novelty 6.0

    Proposes equivariant optimizer updates matched to layer symmetries for embeddings, SwiGLU MLPs, and MoE routers, with reported gains in validation loss and training stability on several language model architectures.

  5. MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization

    cs.LG 2026-05 conditional novelty 6.0

    A 4-bit quantization scheme for the Muon optimizer — combining pre-quantization normalization, singular-structure-aligned decomposition, and µ-law companding — recovers most full-precision training quality with up to ...

  6. MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization

    cs.LG 2026-05 unverdicted novelty 5.0

    MuonQ achieves stable 4-bit quantization of Muon optimizer states via pre-quantization normalization, singular component decomposition with power iteration, and μ-law companding, matching full-precision loss and accur...

Reference graph

Works this paper leans on

41 extracted references · 8 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Dion: Distributed orthonormalized updates

    Kwangjun Ahn, Byron Xu, Natalie Abreu, Ying Fan, Gagik Magakyan, Pratyusha Sharma, Zheng Zhan, and John Lang- ford. Dion: Distributed orthonormalized updates. arXiv preprint arXiv:2504.05295, 2025. 2, 3, 5, 6, 15

    arXiv 2025

  2. [2]

    The polar express: Optimal matrix sign methods and their application to the muon algorithm

    Noah Amsel, David Persson, Christopher Musco, and Robert M Gower. The polar express: Optimal matrix sign methods and their application to the muon algorithm. arXiv preprint arXiv:2505.16932, 2025. 3, 7, 8, 15, 16

    Pith/arXiv arXiv 2025

  3. [3]

    Sorting out lipschitz function approximation

    Cem Anil, James Lucas, and Roger Grosse. Sorting out lipschitz function approximation. InInternational Conference on Machine Learning, pages 291–301. PMLR, 2019. 2

    2019

  4. [4]

    Deriving muon, 2025, 2025

    Jeremy Bernstein. Deriving muon, 2025, 2025. 1, 2

    2025

  5. [5]

    Old optimizer, new norm: An anthology

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. InOPT 2024: Optimization for Machine Learning, 2024. 1, 2, 8, 11

    2024

  6. [6]

    Modular duality in deep learning

    Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning. In Proceedings of the 42nd International Conference on Machine Learning, pages 3920–3930. PMLR,

  7. [7]

    An iterative algorithm for computing the best estimate of an orthogonal matrix

    ˚Ake Bj ¨orck and Clazett Bowie. An iterative algorithm for computing the best estimate of an orthogonal matrix. SIAM Journal on Numerical Analysis, 8(2):358–364, 1971. 1, 2, 3, 7

    1971

  8. [8]

    Sur quelques Propri´et´es des D´eterminants Gauches, page 332–336

    Arthur Cayley. Sur quelques Propri´et´es des D´eterminants Gauches, page 332–336. Cambridge University Press, 1846. 2

  9. [9]

    Layer sharding for large-scale training with muon, 2025, 2025

    Essential-AI. Layer sharding for large-scale training with muon, 2025, 2025. 3, 6

    2025

  10. [10]

    Cesista Franz Louis, Jiacheng You, and Jordan. Keller. Squeezing 1-2% efficiency gains out of muon by optimiz- ing the newton-schulz coefficients, 2025, 2025. 1, 2, 3, 15

    2025

  11. [11]

    ¨uber die abgrenzung der eigen- werte einer matrix

    Semen Aronovich Gershgorin. ¨uber die abgrenzung der eigen- werte einer matrix. Izvestiya Rossiiskoi Akademii Nauk. Seriya Matematicheskaya, 6:749–754, 1931. 13

    1931

  12. [12]

    Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials, 2025

    Ekaterina Grishina, Matvey Smirnov, and Maxim Rakhuba. Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials, 2025. arXiv:2506.10935 [math]. 3, 15, 16

    arXiv 2025

  13. [13]

    A recipe for improved certifiable robustness

    Kai Hu, Klas Leino, Zifan Wang, and Matt Fredrikson. A recipe for improved certifiable robustness. In International Conference on Learning Representations, 2023. 2 4https://www.deel.ai/

    2023

  14. [14]

    cifar10-airbench

    Keller Jordan. cifar10-airbench. https://github.com/ KellerJordan/cifar10-airbench , 2024. GitHub repository. 7

    2024

  15. [15]

    Jordan Keller, Bernstein Jeremy, Rappazzo Brendan, Vlado Boza, Jiacheng You, Cesista Franz, and Koszarsky. Braden. modded-nanogpt: Speedrunning the nanogpt baseline, 2024,

    2024

  16. [16]

    Muon: An optimizer for hidden layers in neural networks, 2024, 2024

    Jordan Keller, Jin Yuchen, Boza Vlado, You Jiacheng, Cesista Franz, Newhouse Laker, and Bernstein Jeremy. Muon: An optimizer for hidden layers in neural networks, 2024, 2024. 1, 2, 3, 5

    2024

  17. [17]

    Yutian, Chen

    Kimi-AI, Bai Yifan, Bao Yiping, Chen Guanduo, Chen Jiahao, Chen Ningxin, Chen Ruijue, Chen Yanru, Chen Yuankun, and et al. Yutian, Chen. Kimi k2: Open agentic intelligence,

  18. [18]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 1

    2015

  19. [19]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7

    2009

  20. [20]

    Heavy-tailed class imbalance and why adam outperforms gradient descent on language mod- els

    Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent on language mod- els. Advances in Neural Information Processing Systems, 37: 30106–30148, 2024. 5, 14

    2024

  21. [21]

    Th´eorie analytique des probabilit´es

    Pierre Simon LaPlace. Th´eorie analytique des probabilit´es. Courcier, 1820. 2

  22. [22]

    Scalable optimization in the modular norm

    Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. Advances in Neural Information Processing Systems, 37:73501–73548, 2024. 1

    2024

  23. [23]

    Flash-muon: An efficient implementation of muon optimizer, 2025

    Tianyang Lin. Flash-muon: An efficient implementation of muon optimizer, 2025. 5

    2025

  24. [24]

    Muon is scalable for llm training

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025. 1

    Pith/arXiv arXiv 2025

  25. [25]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. 1

    2017

  26. [26]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl´ıˇcek, Anton Lozhkov, Mar- garet Mitchell, Colin A Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024. 6

    2024

  27. [27]

    Train- ing deep learning models with norm-constrained lmos

    Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, and V olkan Cevher. Train- ing deep learning models with norm-constrained lmos. arXiv preprint arXiv:2502.07529, 2025. 1

    Pith/arXiv arXiv 2025

  28. [28]

    Almost-orthogonal layers for efficient general-purpose lipschitz networks

    Bernd Prach and Christoph H Lampert. Almost-orthogonal layers for efficient general-purpose lipschitz networks. In European Conference on Computer Vision, pages 350–365. Springer, 2022. 2, 4

    2022

  29. [29]

    Language models are unsuper- vised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsuper- vised multitask learners. OpenAI blog, 1(8):9, 2019. 6 9

    2019

  30. [30]

    Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs), 2025

    Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Pe- ter Richt´arik. Gluon: Making Muon & Scion Great Again! (Bridging Theory and Practice of LMO-based Optimizers for LLMs), 2025. arXiv:2505.13416 [cs]. 2

    Pith/arXiv arXiv 2025

  31. [31]

    Prac- tical efficiency of muon for pretraining

    Ishaan Shah, Anthony M Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, et al. Prac- tical efficiency of muon for pretraining. arXiv preprint arXiv:2505.02222, 2025. 1, 6

    Pith/arXiv arXiv 2025

  32. [32]

    Beyond the ideal: Analyzing the inexact muon update

    Egor Shulgin, Sultan AlRashed, Francesco Orabona, and Pe- ter Richt´arik. Beyond the ideal: Analyzing the inexact muon update. arXiv preprint arXiv:2510.19933, 2025. 1

    arXiv 2025

  33. [33]

    AdaMuon: Adap- tive Muon Optimizer, 2025

    Chongjie Si, Debing Zhang, and Wei Shen. AdaMuon: Adap- tive Muon Optimizer, 2025. arXiv:2507.11005 [cs] version:

    arXiv 2025

  34. [34]

    A tail-index analysis of stochastic gradient noise in deep neural networks

    Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning, pages 5827–5837. PMLR, 2019. 5, 13, 14

    2019

  35. [35]

    Skew orthogonal convolu- tions

    Sahil Singla and Soheil Feizi. Skew orthogonal convolu- tions. In International Conference on Machine Learning, pages 9756–9766. PMLR, 2021. 2

    2021

  36. [36]

    Tri- ton: an intermediate language and compiler for tiled neu- ral network computations

    Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Tri- ton: an intermediate language and compiler for tiled neu- ral network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshopon Machine Learning and Programming Languages, pages 10–19, 2019. 3

    2019

  37. [37]

    High-dimensional probability, 2009

    Roman Vershynin. High-dimensional probability, 2009. 4

    2009

  38. [38]

    Fan- tastic Pretraining Optimizers and Where to Find Them, 2025

    Kaiyue Wen, David Hall, Tengyu Ma, and Percy Liang. Fan- tastic Pretraining Optimizers and Where to Find Them, 2025. arXiv:2509.02046 [cs]. 1, 2, 6

    Pith/arXiv arXiv 2025

  39. [39]

    L ´evy distribution - Wikipedia, the free encyclope- dia

    Wikipedia. L ´evy distribution - Wikipedia, the free encyclope- dia. http://en.wikipedia.org/w/index.php? title = L % C3 % A9vy % 20distribution & oldid = 1322680363, 2025. 14

    2025

  40. [40]

    Glm-4.5: Agentic, reasoning, and coding (arc) foundation models

    Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.06471, 2025. 1 10 A. About the convergence of Turbo-Muon In this section, we build upon the work of Bernstein and Newhouse [5], as they show ...

    Pith/arXiv arXiv 2025

  41. [2025]

    arXiv preprint arXiv:2507.20534, 2025

    Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025. 1, 2

    Pith/arXiv arXiv 2025