Pith. sign in

REVIEW 3 major objections 6 minor 11 references

The Active Ingredient in Muon's Grokking

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that orthogonalization of the momentum buffer, not spectral scaling, is the active ingredient behind Muon's grokking speedup on modular arithmetic.

desk verdict A clean ablation showing orthogonalization, not spectral scaling, drives Muon's grokking speedup—but the 'spectral scaling is inert' half is asserted from underpowered nulls, so worth a careful referee. read the letter →

arxiv 2607.20512 v1 pith:GOTXSE5P submitted 2026-07-06 cs.LG cs.AI

Yufeng Wang This is my paper · ORCID
classification cs.LGcs.AI
keywords grokkingMuonoptimizerorthogonalizationNewton-Schulziterationspectralscalingmodulararithmeticoptimizationdynamicsablationstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to pin down why the Muon optimizer reaches the grokking threshold faster than AdamW on modular arithmetic. Through a 2×2 ablation that independently toggles momentum orthogonalization and spectral scaling, it claims the speedup comes almost entirely from the Newton–Schulz orthogonalization step: an orthogonalize-only variant statistically matches full Muon, while spectral scaling alone is no faster than AdamW and is less reliable. The paper also finds that orthogonalizing optimizers settle into a lower-norm, Fourier-spread solution rather than merely moving the embedding less, and that a single Newton–Schulz iteration reaches the threshold sooner but makes the grokked solution fragile, so the canonical five iterations are the robust default. If true, it would mean Muon's advantage on this task family can be captured by a simpler update and the spectral scaling term can be dropped for free.

What carries the argument

The central object is the Newton–Schulz iteration applied to the momentum buffer: B_t → O_t = NS_ns(B_t), which polar-factorizes the buffer by driving its singular values toward one (the orthogonalization step), alongside a separate per-matrix RMS-and-dimension scale factor (the spectral scaling knob). The paper uses a 2×2 ablation design that toggles each mechanism independently, plus the 'stable-grok' metric (validation accuracy at least 0.95 sustained to the end of training) to separate real generalization improvements from threshold oscillation.

What would settle it

Run a high-powered replication (say 50+ seeds) of the primary contrasts M0 vs M1 and M2 vs M3 on modular addition at lr=1e-3; if M0 stably beats M1, or M2 stably beats M3, with a consistent effect size, then the claim that orthogonalization is the sole active ingredient and spectral scaling is inert is falsified.

Watch

Extended reading notes

Core claim

The central claim is that orthogonalization of the momentum buffer is the active ingredient in Muon's grokking speedup, and spectral scaling is inert. The paper supports this by showing that orthogonalize-only (M1) matches full Muon (M0) and beats AdamW, while spectral-only (M2) is no faster than AdamW and is erratic, with the verdict holding across learning rates and across modular addition, subtraction, and multiplication. It further claims that orthogonalizing optimizers reach generalization at roughly 3× lower spectral norm and find a Fourier-uniform solution, with an embedding-movement control ruling out the 'moves the embedding less' explanation. A speed–stability frontier then shows t

Load-bearing premise

The paper treats p-values around 0.2–0.3 from 5–8 seed runs as evidence that spectral scaling has no effect, so if spectral scaling has a real but modest effect, the central 'inert' claim and the active-ingredient decomposition would be overstated.

Editorial extensions

If this is right

  • An orthogonalize-only variant (no spectral scaling) is sufficient to match full Muon on modular arithmetic, meaning the spectral scale term can be removed without measured cost.
  • Under a stability-aware metric, Muon still beats AdamW across learning rates, but the first-crossing rank can invert between metrics, so grokking-speed claims should be reported under both.
  • Reducing Newton–Schulz iterations from five to one speeds first touch of the 0.95 threshold by about 17–20% at lr=1e-3 but makes the grokked solution fragile, so iteration count trades speed against stability.
  • The speed–stability frontier implies the canonical five Newton–Schulz iterations are a robust operating point across learning rates.
  • On modular subtraction, dropping spectral scaling clearly helps, suggesting the spectral term can hurt on some tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the active-ingredient claim scales beyond this one-layer transformer and modular arithmetic, it would suggest research on adaptive optimizers for grokking should focus on the geometry of momentum orthogonalization rather than spectral-norm regularization.
  • The methodological point about metric inversion likely generalizes: any 'faster to generalize' claim about an optimizer should be rechecked under a sustained-generalization metric, not just first crossing.
  • The fragility of ns=1 predicts that under-orthogonalized momentum may be a general cause of post-grok collapse, a connection the paper itself hints at but does not test beyond its hypothesis.
  • A direct testable extension is to measure the FLOP and wall-clock cost of Newton–Schulz on larger models; the 'drop spectral scaling for free' claim may depend on the scalar factor being negligible, which could change at scale.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports a multi-seed ablation of the Muon optimizer on modular-arithmetic grokking, aiming to isolate which component of Muon (orthogonalization via Newton–Schulz vs. spectral scaling) causes its faster grokking relative to AdamW. The authors construct a 2×2 ablation (orthogonalization on/off, spectral scale on/off), a Newton–Schulz iteration sweep, a learning-rate sweep, robustness checks on subtraction/multiplication, and a short mechanistic analysis of the grokked embedding spectrum. Their headline claims are: (i) orthogonalization, not spectral scaling, is the active ingredient; (ii) orthogonalizing optimizers reach generalization at lower spectral norm with a more uniform Fourier spectrum; (iii) reducing Newton–Schulz iterations from five to one speeds first crossing but makes the grokked solution fragile; and (iv) spectral scaling can be dropped at no measured cost. All results are reported under both first-crossing and stable-grok metrics, and the full code and analysis harness are released.

Significance. If the claims hold, this is a useful and reproducible decomposition of Muon’s grokking advantage: it narrows the explanation to orthogonalization, connects it to a lower-norm solution, and supplies a practical speed–stability trade-off. The paper’s strengths include a 258-run multi-seed sweep, Holm-corrected primary contrasts, a deterministic bit-for-bit reproducible harness with released code, and a dual-metric approach that shows how metric choice can invert rankings. However, the inertness claim for spectral scaling is the weaker leg: it rests on failures to reject null differences without equivalence margins, and several robustness statements rely on marginal or uncorrected p-values. The central orthogonalization-necessity contrast (ortho-only vs. AdamW, p=0.010; Lean1 vs. Muon, p=0.005) is reasonably supported, but the abstract/conclusion overstates the evidence for ‘no measured cost’ of dropping spectral scaling.

major comments (3)
  1. [§3.1, §5.1] The claim that spectral scaling is inert/removable is inferred from non-significant differences without an equivalence margin. With n=5 in §3.1, M0 vs. M1 (p=0.21) gives a 95% CI for M1−M0 of roughly [−57, 281] steps, so ortho-only could be ~15% slower than full Muon; M2 vs. M3 (p=0.25) has a CI width of hundreds of steps on an AdamW mean of 2372, so ‘spectral-only is no faster than AdamW’ is not established. In §5.1, Lean5 vs. Muon (p=0.27, n=8) excludes a >5% slowdown but still lacks a pre-specified margin. Please add an equivalence test (e.g., two one-sided tests with a justified margin) or soften the abstract/conclusion claim to ‘no significant difference detected’.
  2. [§3.2, §5.4] Some robustness claims are based on marginal or uncorrected p-values. In §3.2, the ‘verdict holds across learning rates’ relies on p=0.052 and p=0.053 for ortho-only vs. spectral-only at lr=1e-3 and 3e-3 (4 seeds); these are directional at best. In §5.4, ‘Lean5 beats Muon, p=0.021’ on subtraction is one of many pairwise comparisons across operations; the manuscript does not state whether this survives Holm correction, despite the stated rule that secondary contrasts not surviving correction are exploratory. Report adjusted p-values/CIs and label these as exploratory if not corrected.
  3. [§5.2 / Appendix A] The claim that Lean1 is ‘fast and stable’ at lr=3e-4 rests on the equality of first-crossing and stable-grok in one 4-seed cell, with no test against Muon’s stable-grok (4875 vs. 5481). More generally, the frontier comparison across rates uses p=0.32, 0.043, 0.001 without multiple-comparison correction across the three rates and variants. Please provide adjusted significance or explicit exploratory status for these comparisons.
minor comments (6)
  1. [Abstract] Typo: ‘Moun’s behavior’ should be ‘Muon’s behavior’.
  2. [§2, §3.3] The notation ‘ns’ is overloaded: it denotes both the Newton–Schulz iteration count and is used as a label in the sweep. Consider using, e.g., n_NS for the count.
  3. [§4] The term ‘Fourier-IPR’ is used before being defined. Please define the inverse participation ratio and how the averaged-power IPR is computed in the main text, not only in the appendix.
  4. [§3.1, §5.1, §5.4] The tables are not numbered. Numbering them would make the text references (e.g., ‘Table 3, period-5 row’ style comparisons) easier and more precise.
  5. [§2] The word ‘pre-registered’ appears without a registration link or timestamp. If a preregistration exists, include it; otherwise use ‘pre-specified’.
  6. [§5.3] The destabilization hypothesis in §5.3 is explicitly labeled a hypothesis, which is fine, but consider adding one sentence on how it could be tested (e.g., measuring distance to the grokking manifold under ns=1).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central ablation and mechanistic analyses are measured comparisons, not definitions or fitted predictions; the flagged statistical issue is a power/equivalence concern, not circularity.

full rationale

The paper's central claim is an empirical ablation. M0-M3 are defined by independently toggling two optimizer components (Newton-Schulz orthogonalization and spectral scale), and the comparisons are measured over seeds: "Orthogonalize-only (M1) matches full Muon (M0 vs. M1: p=0.21)" is a comparison of two distinct configurations, not a quantity fitted from that comparison. No parameter is fitted to the target metric, no 'prediction' is derived from the data that defines it, and no result is asserted by construction from its own equations. The ns-sweep consistency checks (ns=0 matches M2, ns=5 reproduces M0) are checks of determinism and implementation, not circular reductions. The mechanistic section controls for embedding movement by measuring relative embedding change independently, ruling out a trivial explanation rather than assuming it. Citations to prior work (Tveit et al., Jordan et al., Nanda et al.) provide background, the optimizer definition, and the reference circuit; none is a self-citation and none is load-bearing as an unverified external theorem. The reviewer-flagged issue that null contrasts (p=0.21, p=0.25, p=0.27) are treated as evidence of equivalence is a statistical power/equivalence-testing limitation, not a circularity: it concerns whether a conclusion is overstated, not whether the derivation reduces to its inputs. Under the stated rules, this is a self-contained empirical study with no significant circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No fitted constants are used to manufacture the claims; the only hand-chosen quantities are metric thresholds and experimental design choices. The paper introduces no new physical or architectural entities; 'Lean-Muon' and 'Lean5/Lean1' are just optimizer configurations. The main unstated burden is the equivalence-from-null inference rule and the coarse spectral proxy.

free parameters (2)
  • grok threshold (validation accuracy) = 0.95
    Hand-chosen, pre-registered threshold. All first-crossing and stable-grok times are defined by this number; rankings could shift at a different threshold.
  • stable-grok definition = sustained >= 0.95 for remainder of training
    A hand-chosen metric definition. The paper's methodological claim that grokking-speed rankings can invert depends on this definition, and on the fixed training horizon.
assumptions (5)
  • domain assumption Validation accuracy 0.95, sustained to the end of training, defines grokking.
    All speed comparisons run through this threshold and horizon; the paper itself shows the metric can invert rankings, so the choice is substantive.
  • domain assumption Grokked modular addition uses a Fourier-multiplication circuit (Nanda et al., 2023), so Fourier spectrum is a meaningful reference.
    Used in §4 to interpret uniform vs concentrated Fourier spectra as different solution types; if the circuit story differs at this scale, the mechanistic reading weakens.
  • domain assumption Averaged-power Fourier IPR is a coarse but usable proxy for per-neuron solution structure.
    The paper itself calls it a 'coarse proxy' in §4 and §6; it underpins the lower-norm/uniform-spectrum mechanism claim.
  • ad hoc to paper Non-significant p-values can be read as evidence of equivalence ('matches', 'no measured cost').
    Used implicitly in §3.1 (M0 vs M1 p=0.21; M2 vs M3 p=0.25) and §5.1 (Lean5 vs Muon p=0.27). This is an unstated inference rule, and it is underpowered at 5–8 seeds.
  • domain assumption Applying AdamW to embeddings/unembeddings and the optimizer under test only to hidden matrices isolates the optimizer effect.
    Stated in §2 as standard Muon practice; if the interaction between embedding optimizers and the hidden-matrix optimizer matters, the ablation's interpretation changes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Active Ingredient in Muon's Grokking." pith.science (2026). https://pith.science/paper/GOTXSE5P

@misc{pith2026260720512,
  author       = {Pith},
  title        = {Pith review of: The Active Ingredient in Muon's Grokking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GOTXSE5P}},
  note         = {Machine review of arXiv:2607.20512}
}
read the original abstract

The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW. Prior work attributes this to "spectral-norm constraints plus orthogonalized momentum" but does not isolate which mechanism matters. To better understand Moun's behavior, we run multi-seed and multi-learning-rate sweeps to decompose and stress-test the effect. First, an ablation shows the speedup comes from orthogonalization (the Newton-Schulz iteration): orthogonalize-only matches full Muon, whereas spectral-only is no faster than AdamW and is unreliable, and this verdict holds across learning rates. Second, a mechanistic analysis finds that orthogonalizing optimizers reach generalization at roughly 3x lower spectral norm and, controlling for how much the embedding actually moves, settle into a lower-norm solution rather than simply perturbing the embedding less. Third, reducing the Newton-Schulz iteration count from five to one accelerates reaching the threshold but makes the grokked solution fragile, prone to transient collapse, with fragility that grows with learning rate; a single iteration is fast and stable only at small learning rate, while the canonical five iterations are the learning-rate-robust choice. We also show spectral scaling can be dropped at no measured cost. A methodological thread runs throughout: under a stability-aware metric, "faster" claims about grokking optimizers can invert, so we report both first-crossing and sustained-grok times. To support reproducibility, we release our full training and analysis code at https://github.com/louiswang524/muon-grokking-frontier

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 5 linked inside Pith

  1. [1]

    ICLR MATH-AI Workshop , year=

    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets , author=. ICLR MATH-AI Workshop , year=

  2. [2]

    International Conference on Learning Representations (ICLR) , year=

    Progress Measures for Grokking via Mechanistic Interpretability , author=. International Conference on Learning Representations (ICLR) , year=

  3. [3]

    2024 , howpublished=

    Muon: An optimizer using Newton--Schulz orthogonalization of momentum , author=. 2024 , howpublished=

    2024

  4. [4]

    arXiv preprint arXiv:2405.20233 , year=

    Grokfast: Accelerated Grokking by Amplifying Slow Gradients , author=. arXiv preprint arXiv:2405.20233 , year=

    arXiv

  5. [5]

    arXiv preprint arXiv:2504.16041 , year=

    Muon Optimizer Accelerates Grokking , author=. arXiv preprint arXiv:2504.16041 , year=

    arXiv

  6. [6]

    International Conference on Learning Representations (ICLR) , year=

    Decoupled Weight Decay Regularization , author=. International Conference on Learning Representations (ICLR) , year=

  7. [7]

    International Conference on Learning Representations (ICLR) , year=

    Omnigrok: Grokking Beyond Algorithmic Data , author=. International Conference on Learning Representations (ICLR) , year=

  8. [8]

    arXiv preprint arXiv:2309.02390 , year=

    Explaining Grokking Through Circuit Efficiency , author=. arXiv preprint arXiv:2309.02390 , year=

    arXiv

Show all 11 references
  1. [9]

    arXiv preprint arXiv:2206.04817 , year=

    The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon , author=. arXiv preprint arXiv:2206.04817 , year=

    arXiv

  2. [10]

    arXiv preprint arXiv:2409.20325 , year=

    Old Optimizer, New Norm: An Anthology , author=. arXiv preprint arXiv:2409.20325 , year=

    arXiv

  3. [11]

    International Conference on Learning Representations (ICLR) , year=

    Spectral Normalization for Generative Adversarial Networks , author=. International Conference on Learning Representations (ICLR) , year=

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.