Pith. sign in

REVIEW 5 minor 17 references

In a fully readable 12K-parameter model, grokking is a conditional, fragile phase transition whose unit of evidence is a multi-seed rate under a pinned numerical environment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 08:45 UTC pith:7R5CC6H5

load-bearing objection Solid multi-seed evidence that grokking at 12K is coverage-gated, weight-decay-sensitive, and numerically fragile; the methodological standard is the real payload.

arxiv 2607.05104 v1 pith:7R5CC6H5 submitted 2026-07-06 cs.LG cs.AI

Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters

Yoshiyuki Ootani This is my paper
classification cs.LG cs.AI
keywords grokkingmulti-seed ratecoverage thresholdoutput cardinalitynumerical knife-edgeweight decaymodular arithmetictractable transformer
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most work studies grokking in models too large to read completely and treats a single training run as evidence. This paper instead uses a publicly released ~11,856-parameter Llama-style transformer on modular arithmetic, small enough that its weights, attention, and full input-output map can be enumerated, and reports grokking as a multi-seed rate inside a fixed numerical environment. In that regime grokking is gated by how much of the input domain is covered in training; the coverage threshold tracks the number of possible answers (the modulus) more than the algebraic structure of the task, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay produces the known inverted-U in grok rate, confirming that the rate measurement itself responds to a real intervention. Two different floating-point perturbations (CPU thread count, which only changes reduction order, and CPU versus GPU) each flip a minority of same-seed outcomes without shifting the aggregate rate. Task decomposition helps mainly by turning sparse coverage into dense sub-task coverage rather than by adding supervision. Multi-seed control under a fixed environment overturns three clean single-run stories that appeared in the authors’ own data. The practical claim is therefore methodological: for phase transitions like grokking the unit of evidence must be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.

Core claim

In this fully-tractable 12K-parameter regime, grokking is not an intrinsic property of a task but a conditional, fragile phase transition: it is gated by training-set coverage whose threshold tracks output cardinality more than composition structure, is promoted then destroyed by weight decay, and sits on a numerical knife-edge where pure changes of floating-point reduction order or device flip a minority of same-seed outcomes without detectable aggregate bias. Consequently the only reliable unit of evidence is a multi-seed rate measured inside a fixed numerical environment and, where possible, checked against a complete reading of weights and input-output map.

What carries the argument

The multi-seed, fixed-environment grok-rate (fraction of seeds whose best held-out accuracy exceeds a fixed threshold, typically 0.70, under pinned thread count and device). It converts seed- and numerics-sensitive binary outcomes into a controlled rate that can be tested for aggregate bias (exact McNemar, Newcombe intervals) and that responds to known interventions such as weight decay.

Load-bearing premise

That the coverage-cardinality pattern, numerical knife-edge, and multi-seed protocol observed at 12K parameters remain informative for the much larger models and non-modular tasks where most grokking claims are made.

What would settle it

A dense multi-seed coverage sweep on a larger-width modular model (or a non-modular algorithmic task) that either (a) shows no modulus-tracking coverage threshold once domain size and structure are controlled, or (b) shows that floating-point reduction-order or device changes no longer produce same-seed flips once averaging over many more parameters is present.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper studies grokking in a fully tractable ~11.9K-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic tasks. Using multi-seed rates (typically 10–300 seeds) under a pinned numerical environment rather than single runs, it shows that grokking is a conditional, fragile phase transition: gated by training-set coverage whose threshold tracks output cardinality (modulus M) more than task structure (holding across a 10 imes domain-size change), modulated by weight decay into the Omnigrok inverted-U, and sensitive to floating-point reduction order and device (CPU threads or CPU vs GPU) which flip a minority of same-seed outcomes without shifting the aggregate rate. Decomposition into specialists helps primarily by converting sparse coverage into dense sub-domain coverage (isolated via a scratchpad control). Mechanistically, generalizing solutions exhibit more periodic output maps (flagged as partly definitional) while embeddings do not form the textbook Fourier circle. Methodologically, multi-seed paired controls overturn three single-run narratives as seed confounds, so the unit of evidence must be a multi-seed rate under fixed numerics, checked against direct model readout where possible.

Significance. If the results hold, the work supplies a clean, fully-legible testbed that makes multi-seed rates, seed-paired McNemar/Newcombe statistics, and an Omnigrok positive control standard for claims about phase-transition phenomena such as grokking. The coverage–cardinality regularity, numerical knife-edge (sub-ULP flips without aggregate bias), and decomposition-as-data-efficiency result are concrete and falsifiable inside the claimed regime; the public checkpoint, scripts, and per-seed records further raise the bar for reproducibility. Even though transfer to larger models is left open, the methodological prescription (report rates, pin threads/device, pair by seed, distrust single-run stories) is scale-independent and timely for any sharp transition measured near threshold.

minor comments (5)
  1. Table 1 still reports 3–5-seed cells that the text correctly flags as provisional and later supersedes with 10-seed grids (Table 2, Fig. 3). Consider moving the provisional numbers to an appendix or adding an explicit “superseded” note in the caption so a casual reader does not treat them as primary evidence.
  2. Figure 1 caption and surrounding text describe a “brief shaded train−held-out gap”; the shading itself is not visible in the rendered figure description. A short note that the gap is at most a few tenths of an epoch would help readers who cannot inspect the raw plot.
  3. Section 4.4: the Pearson r values for logit-Fourier concentration are reported with Fisher-z CIs, yet the class imbalance (39:3 and 5:37) is acknowledged only later. Leading with the group-mean separation and Spearman ρ (already present) would make the “partly definitional” caveat even clearer on first reading.
  4. Reproducibility statement lists Python/PyTorch/Transformers versions and the exact GPU; adding the precise MKL/OpenBLAS version string used for the CPU thread experiments would complete the numerical-environment pin.
  5. A few long sentences in the Introduction and Discussion (e.g., the paragraph beginning “This combination matters…”) could be split for readability without changing content.

Circularity Check

1 steps flagged

Only minor self-definitional consistency check on output periodicity (explicitly flagged by authors as partly definitional); central empirical claims are independent multi-seed measurements with external positive control.

specific steps
  1. self definitional [Section 4.4 (Mechanism: generalization as periodicity)]
    "More fundamentally, because the read-out is the periodicity of the true-answer logit over a grid generated by a periodic rule, a model that answers correctly is periodic on that grid almost by construction; the correlation is thus partly definitional, and we report it as a consistency check on the periodicity framing rather than as an independent causal probe."

    The claimed positive correlate (logit Fourier concentration tracks held-out accuracy) is nearly tautological: any model that correctly implements a modular (hence periodic) map will, by construction, produce a periodic true-answer logit surface on the (a,b) grid. The authors correctly flag and demote it; it does not underwrite the paper’s central rate or fragility claims.

full rationale

The paper is an empirical multi-seed study of grokking rates under controlled interventions (coverage, weight decay, floating-point environment, decomposition) on a fully enumerable 12K-parameter model. Its load-bearing results are measured rates (10–300 seeds), paired McNemar/Newcombe tests, an Omnigrok inverted-U reproduction used strictly as external positive control (Liu et al. 2023), a scratchpad arm isolating coverage from supervision, and threshold re-tabulations. No parameters are fitted then re-labeled as predictions; no uniqueness theorems or self-citation chains force the conclusions; the three single-run narratives are refuted by the multi-seed protocol itself. The sole near-circular element is the logit-Fourier concentration vs. accuracy correlation in §4.4, which the authors themselves label “partly definitional” and demote to a consistency check rather than an independent causal claim; the independent mechanistic result is the negative (no textbook Fourier embedding circle). This is honest self-containment inside the claimed fully-tractable regime; transfer is left open. Score 2 reflects only that flagged minor step.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claims rest on standard floating-point non-associativity, the public Glimmer-1-Base architecture, the conventional definition of grokking via a held-out accuracy threshold, and the modular-arithmetic task family. No new physical entities are postulated; free parameters are experimental knobs (threshold, weight-decay values, seed counts) whose sensitivity is checked. Domain assumptions about single-token answers and CPU BLAS reduction trees are stated and used only inside the claimed regime.

free parameters (3)
  • grok threshold τ = 0.70 (primary)
    Conventionally set to 0.70; re-tabulated at 0.60 and 0.80 to confirm qualitative invariance of inverted-U, flip counts, and decomposition gap.
  • weight-decay values in Omnigrok sweep = 0 / 0.01 / 0.1 / 1.0
    Hand-chosen grid {0, 0.01, 0.1, 1.0} that produces the inverted-U; used as positive control rather than fitted to a continuous model.
  • coverage fractions and data budgets
    Discrete experimental points (5–40 %, n=160/2000/3000) chosen to straddle the transition; not free parameters of a fitted theory.
axioms (4)
  • standard math Floating-point addition is non-associative, so reduction order (thread count) or device kernels can change results at the level of rounding while leaving the mathematical program unchanged.
    Invoked in §4.3 with citation to Goldberg 1991; verified by bit-identical weights at 4 vs 16 threads.
  • domain assumption A run ‘groks’ if best held-out accuracy ≥ τ, with τ far above chance 1/M.
    Definitional convention stated in §3; sensitivity checked across three thresholds.
  • domain assumption At hidden size 16 and M≤10 the model and tasks remain fully enumerable (weights, attention, full I/O map).
    Enabling premise of the entire tractable-regime design (§1, §3).
  • domain assumption The released Glimmer-1-Base checkpoint (11 856 parameters, pretrained on 500 K FineWeb-Edu tokens) is a valid substrate that starts at chance on the modular tasks.
    Public model used throughout; no claim that it is a useful language model.

pith-pipeline@v1.1.0-grok45 · 19757 in / 3061 out tokens · 26053 ms · 2026-07-11T08:45:33.151491+00:00 · methodology

0 comments
read the original abstract

Grokking -- the delayed onset of generalization long after a network has fit its training set - -is usually studied in models too large to read completely and reported from single training runs. We instead study a publicly released ~11,856-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic, small enough to enumerate its weights, attention, and full input-output map, and we measure grokking as a multi-seed rate rather than a single outcome. In this fully-tractable regime grokking is a conditional, fragile phase transition. It is gated by training-set coverage, whose threshold tracks output cardinality (the modulus) more than task structure, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay reproduces the Omnigrok inverted-U at 12K parameters, a positive control on the rate measurement. Grokking also sits on a numerical knife-edge: two perturbations of the floating-point environment -- CPU thread count (reduction order) and CPU-versus-GPU execution -- each flip a minority of same-seed outcomes without a detectable shift in the aggregate rate. Decomposition into sub-task specialists helps chiefly by making coverage cheap rather than by adding supervision. Methodologically, multi-seed control under a fixed numerical environment overturns three dramatic single-run narratives in our own data, each a seed confound. The unit of evidence for grokking must therefore be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.

Figures

Figures reproduced from arXiv: 2607.05104 by Yoshiyuki Ootani.

Figure 1
Figure 1. Figure 1: Grokking dynamics for (a+b) mod 10 at weight decay 0.1 (all ten seeds grok). Left: answer-token accuracy for a representative seed (bold) over the other nine seeds’ held-out curves (grey); held-out accuracy holds near chance, then transitions sharply at a seed-dependent epoch (≈320–760). Right: train and held-out cross-entropy, plateaued at the chance value ln M ≈ 2.30 before collapsing. The brief shaded t… view at source ↗
Figure 2
Figure 2. Figure 2: Grok-rate versus training coverage for four composite tasks. Lower-cardinality moduli ( [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cardinality governs the coverage threshold across a ten-fold change in domain size. Left: on [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Grok-rate and mean best held-out accuracy versus weight decay for [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Paired per-seed best accuracy under two numerical perturbations: reduction order (1 versus 4 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Held-out accuracy versus logit-Fourier concentration (output periodicity) for 42 add-specialists [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Left: grok-rate of monolith versus pipeline at a matched 160-example budget; the pipeline groks [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 17 linked inside Pith

  1. [1]

    arXiv:2207.08799. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent. Ac- counting for variance in machine learning benc...

    Pith/arXiv arXiv

  2. [2]

    Bilal Chughtai, Lawrence Chan, and Neel Nanda

    arXiv:2103.03098. Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. InInternational Conference on Machine Learning (ICML),

    Pith/arXiv arXiv

  3. [3]

    CompactAI

    arXiv:2302.03025. CompactAI. Glimmer-1: An 11.9K-parameter Llama-style transformer,

    Pith/arXiv arXiv

  4. [4]

    co/Glint-Research

    URLhttps://huggingface. co/Glint-Research. Darshil Doshi, Tianyu He, Aritra Das, and Andrey Gromov. Grokking modular polynomials.arXiv preprint arXiv:2406.03495,

    Pith/arXiv arXiv

  5. [5]

    Grokking modular arithmetic.arXiv preprint arXiv:2301.02679,

    Andrey Gromov. Grokking modular arithmetic.arXiv preprint arXiv:2301.02679,

    Pith/arXiv arXiv

  6. [6]

    1https://github.com/otanl/grokking-microscope, archived athttps://doi.org/10.5281/zenodo.21221306 13 Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J

    arXiv:2310.06110. 1https://github.com/otanl/grokking-microscope, archived athttps://doi.org/10.5281/zenodo.21221306 13 Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv doi:10.5281/zenodo.21221306

  7. [7]

    Ziming Liu, Eric J

    arXiv:2205.10343. Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data. InInter- national Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  8. [8]

    William Merrill, Nikolaos Tsilivis, and Aman Shukla

    arXiv:2210.01117. William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two circuits: Grokking as competition of sparse and dense subnetworks.arXiv preprint arXiv:2303.11873,

    Pith/arXiv arXiv

  9. [9]

    Pascal Jr

    arXiv:2301.05217. Pascal Jr. Tikeng Notsawo, Hattie Zhou, Mohammad Pezeshki, Irina Rish, and Guillaume Dumas. Predicting grokking long before it happens: A look into the loss landscape of models which grok.arXiv preprint arXiv:2306.13253,

    Pith/arXiv arXiv

  10. [10]

    torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision.arXiv preprint arXiv:2109.08203,

    David Picard. torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision.arXiv preprint arXiv:2109.08203,

    Pith/arXiv arXiv

  11. [11]

    Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177,

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets.arXiv preprint arXiv:2201.02177,

    Pith/arXiv arXiv

  12. [12]

    Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo

    arXiv:2501.04697. Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  13. [13]

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean

    arXiv:2304.15004. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  14. [14]

    Cecelia Summers and Michael J

    arXiv:1701.06538. Cecelia Summers and Michael J. Dinneen. Nondeterminism and instability in neural network optimization. InProceedings of the 38th International Conference on Machine Learning (ICML),

    Pith/arXiv arXiv

  15. [15]

    The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon.arXiv preprint arXiv:2206.04817,

    Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua Susskind. The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon.arXiv preprint arXiv:2206.04817,

    Pith/arXiv arXiv

  16. [16]

    Explaining grokking through circuit efficiency.arXiv preprint arXiv:2309.02390,

    Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency.arXiv preprint arXiv:2309.02390,

    Pith/arXiv arXiv

  17. [17]

    arXiv:2306.17844. 14

    Pith/arXiv arXiv