Structure-Specific Representational Priors Causally Control the Grokking Delay

Gunner Levi Howe

The grokking delay is the time needed to form the right task-structured features, not labels or weight norms.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 10:08 UTC pith:RXBWQD4P

load-bearing objection Clean causal contingency on structure priors for grokking, but the matching claim and full methods are unverified from the abstract alone. the 3 major comments →

arxiv 2607.04333 v3 pith:RXBWQD4P submitted 2026-07-05 cs.LG

Structure-Specific Representational Priors Causally Control the Grokking Delay

Gunner Levi Howe This is my paper

classification cs.LG

keywords grokkingrepresentational priorsmodular additionsupervised contrastive lossfeature formationweight normgeneralization delayone-layer transformer

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Grokking is the delayed jump from memorization to generalization long after a model has fit its training set. This paper asks whether that delay is specifically the time required to assemble the right internal structure for the task, and tests the idea by injecting three kinds of representational prior into a one-layer transformer learning modular addition. A supervised-contrastive loss is used whose positive pairs encode either the true modular-addition structure, a coherent but wrong sibling structure (modular subtraction), or a purely random partition—all matched for loss form, strength, class sizes, and geometry. Generalization occurs cleanly according to prior content (true structure succeeds most often, sibling next, random never), while a weight-norm-matched control that merely replays the same norm trajectory under ordinary cross-entropy never generalizes. Structure formation, measured by probes, precedes and predicts the generalization jump in every successful run. The result implies that the delay is causal and feature-level: once the right representation is present, generalization follows; when only memorizable structure is supplied, it never does.

Core claim

The grokking delay is, causally, the time to form the right representational structure—decided at the level of features, not labels. Whether generalization occurs follows a clean gradation by the content of an injected prior: true modular-addition structure yields generalization in 22/30 runs, a coherent sibling structure in 14/15, and a random partition in 0/20, while a weight-norm-matched control generalizes in 0/15.

What carries the argument

A supervised-contrastive loss whose positives encode one of three partitions—(a+b) mod p, (a−b) mod p, or a random partition—while holding loss form, strength, class sizes, and geometry fixed, thereby isolating structural content as the sole experimental factor.

Load-bearing premise

That the three contrastive losses differ only in the intended structural content of their positive pairs and that every other factor (loss form, strength, class sizes, geometry) is fully matched, so outcome differences can be attributed solely to that content.

What would settle it

Re-run the same one-layer modular-addition setup with the three matched contrastive priors and the weight-norm-matched control; if the random prior or the norm-matched control begin to generalize at rates comparable to the true prior, or if probes show structure formation no longer preceding generalization, the causal claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Injecting the true task structure as a representational prior can make generalization occur and can accelerate it (up to 2.75×), though the acceleration is dose-dependent and bimodal.
A coherent but incorrect sibling structure still permits generalization far more often than a random partition, showing that usable periodic features matter even when the combination rule is wrong.
Weight-norm trajectory alone is not the mediator: replaying the identical norm path under plain cross-entropy yields zero generalization.
Clamping weight norm during ordinary training, without any contrastive prior, produces a reliable standalone accelerator (median 8.6×, up to 22×) whose residual stalls largely vanish when pooled across mitigations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the delay is truly feature-formation time, then any intervention that forces the correct features earlier—curriculum ordering, architectural inductive bias, or synthetic pre-training on the same structure—should shrink or eliminate the grokking plateau on modular arithmetic and related algorithmic tasks.
The sibling-structure result suggests a graded notion of “almost-right” features: periodic structure shared by addition and subtraction is already enough for later generalization, so intermediate probes for shared versus task-specific features could predict which wrong priors still help.
Because norm clamping alone accelerates without injecting structure, the two factors (feature content and weight-norm side-effects) are separable; future work could map the interaction surface by jointly varying prior strength and clamped norm.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Clean causal contingency on structure priors for grokking, but the matching claim and full methods are unverified from the abstract alone.

read the letter

The one thing to know is the reported contingency: true modular-addition structure via contrastive prior generalizes in 22/30 runs, sibling structure 14/15, random 0/20 (Fisher p=1.3e-7), with a weight-norm-matched plain-CE control at 0/15. If the matching holds, this converts the usual observational story about the grokking delay into a feature-level causal claim and yields a predicted standalone accelerator (norm clamping, median 8.6×).

What is actually new is the content-matched design. Structure-agnostic interventions already exist; here they vary only the structural content of the positives (true (a+b) mod p, sibling (a-b) mod p, random partition) while asserting identical loss form, strength, class sizes and geometry. The sibling arm is a useful control—same periodic features, wrong combination—and probes that structure formation precedes generalization add a mechanistic check. Closing the loop by predicting the clamp from the observed norm side-effect of the true prior is clean experimental practice.

Soft spots are real but match an abstract-only read. The load-bearing assumption is that the three priors differ solely in content. Residual differences in effective hardness, positive-pair density or contrastive curvature would turn the random arm into a harder auxiliary task rather than a pure content control; the abstract asserts matching but we cannot verify it. Acceleration is dose-dependent and bimodal; residual-stall significance appears only when the two mitigations are pooled. Free parameters (contrastive strength, clamp level, architecture) remain. No trajectories, code or matching diagnostics are visible here, so confidence stays low.

This is for people already working on grokking and small transformers. A serious referee should see the full methods, matching checks and per-seed plots. Send it to review rather than desk-reject; the contingency table is strong enough and the design idea is worth stress-testing. If the matching holds up, the result is useful for the subfield.

Referee Report

3 major / 4 minor

Summary. The manuscript claims that the grokking delay is causally the time required to form the right task-structured representations (features, not labels). In a one-layer transformer on modular addition, the authors inject supervised-contrastive priors whose positives encode (i) true structure (a+b) mod p, (ii) a coherent sibling (a-b) mod p, or (iii) a random partition, asserting identical loss form, strength, class sizes, and geometry. Generalization occurrence grades by prior content (true 22/30, sibling 14/15, random 0/20; Fisher p=1.3e-7). A weight-norm-matched replay onto plain cross-entropy generalizes 0/15, ruling out norm as mediator of occurrence. Probes show structure formation precedes and predicts generalization. Only the true prior accelerates grokking (up to 2.75x, dose-dependent and bimodal); the authors then predict and confirm that norm clamping alone is a reliable accelerator (median 8.6x, up to 22x), with residual stalls vanishing when pooled across mitigations.

Significance. If the matching and causal attribution hold, this is a substantial contribution: it converts an observational story about grokking into an interventional one at the level of representational content, and it yields a simple, falsifiable, practical accelerator (norm clamping) derived from a side-effect of the true prior. Strengths visible in the abstract include the three-way content contrast under claimed matched geometry, the norm-trajectory control, probe precedence, Fisher contingency evidence, and the closed-loop prediction that clamping should accelerate. These design elements, if fully documented, would raise the standard for causal claims about grokking.

major comments (3)

[Abstract (matching claim; contingency 22/30 vs 14/15 vs 0/20)] The central causal claim (delay = time to form the right features) rests on the assertion that the three contrastive priors differ only in structural content once loss form, strength, class sizes, and geometry are matched. That matching is load-bearing and only stated, not demonstrated, in the available text. Residual differences in positive-pair density, effective number of classes, hardness, or contrastive curvature (especially for the random partition) would reframe the random condition as a harder auxiliary task rather than a pure content control, collapsing the feature-level interpretation. The full manuscript must supply explicit matching diagnostics (class-size and pair-count tables, loss-landscape or gradient-geometry comparisons, and any hardness controls).
[Abstract (dose-dependent/bimodal acceleration; norm-clamping prediction)] Acceleration is reported as dose-dependent and bimodal for the true prior only, and the subsequent norm-clamping prediction is gated by a weight-norm side-effect of that prior. Without a clear operational definition of dose, a characterization of the bimodality, and a quantitative link from the side-effect to the clamp levels, the claim that 'only the true structure accelerates' and the mechanistic derivation of clamping remain incompletely supported. These details are needed to make the predictive confirmation load-bearing rather than post-hoc.
[Abstract (pooled residual-stall test)] The residual-stall result is significant only when pooled over two mitigations at both strengths (0/40 vs 6/20, p=7.7e-4), not per method. Pooling is acceptable as secondary evidence but should not be presented as confirming each mitigation independently; the manuscript should report per-method rates and power, and temper the claim accordingly so that the occurrence gradation (the primary causal result) is not over-extended.

minor comments (4)

[Abstract] Report exact definitions of 'generalization' (e.g., accuracy threshold and evaluation set) and of a 'run' (seed protocol, early-stopping or epoch budget) so the contingency fractions are reproducible from the text alone.
[Abstract (methods claim)] State the supervised-contrastive temperature, batch construction, and how positives/negatives are sampled for modular vs random partitions; these affect geometry even when class sizes are matched.
[Abstract (contingency table)] Clarify whether the sibling prior's 14/15 rate used a different n or selection rule than the true prior's 22/30; unequal denominators invite questions about optional stopping or condition-specific budgets.
[Abstract (probes)] When full text is available, include probe architecture, training stage, and the quantitative 'precedes and predicts' criterion (e.g., lag, correlation, or causal mediation stats).

Circularity Check

0 steps flagged

No significant circularity: interventional priors and a derived norm-clamping prediction are empirical tests, not definitional reductions.

full rationale

The abstract presents a causal intervention design: supervised-contrastive priors whose positives encode true modular addition, sibling modular subtraction, or a random partition are injected under claimed-matched loss form, strength, class sizes, and geometry, with a separate weight-norm-matched replay control on plain cross-entropy. Outcome gradation (true 22/30, sibling 14/15, random 0/20; control 0/15) and probe timing are empirical results of those interventions, not quantities forced by fitting the same target or by redefining the delay in terms of the prior. The subsequent claim that norm clamping accelerates grokking is framed as a mechanistic prediction from an observed side-effect of the true prior (acceleration gated by weight-norm trajectory), then tested as a standalone intervention (median 8.6× speedup). That prediction is not the fitted parameter renamed as a result, nor is any equation in the abstract equivalent by construction to its inputs. No self-citation chain, uniqueness theorem, or ansatz-smuggling appears in the available text. Residual concerns about whether the three contrastive conditions are fully matched on hardness or gradient geometry are validity/confound issues, not circularity. With only the abstract available, the derivation chain as stated is self-contained and interventional; score 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

Abstract-only; free parameters and domain assumptions are those necessarily implied by the described setup. No new physical entities are postulated. The central causal claim rests on the matched-prior design and on modular addition in a one-layer transformer as a valid testbed for feature-level structure formation.

free parameters (3)

contrastive_loss_strength
Strength of the supervised-contrastive prior relative to cross-entropy; described as matched across conditions but its absolute value is a free hyperparameter that gates both generalization and the dose-dependent acceleration.
weight_norm_clamp_level
The level at which weight norm is held; acceleration grows monotonically as the norm is held lower, so the clamp value is a free control parameter of the reported speedup.
architecture_and_training_hyperparameters
Embedding size, learning rate, p (modulus), batch size, and related choices for the one-layer transformer are not fixed by theory and must be set by the experimenters.

axioms (3)

domain assumption Modular addition on a one-layer transformer is a valid testbed for the causal role of representational structure in grokking.
The entire causal claim is demonstrated only in this standard algorithmic setting; transfer beyond it is unstated.
ad hoc to paper Positives of the supervised-contrastive loss that encode (a+b) mod p, (a−b) mod p, or a random partition differ only in structural content once loss form, strength, class sizes, and geometry are matched.
This matching is the load-bearing design claim that converts outcome differences into a feature-level causal conclusion.
domain assumption Linear probes of internal activations correctly measure formation of the task structure that mediates generalization.
Probe precedence and prediction of generalization are used as mechanistic evidence; probe validity is assumed.

pith-pipeline@v1.1.0-grok45 · 6275 in / 2739 out tokens · 36629 ms · 2026-07-15T10:08:03.676255+00:00 · methodology

0 comments

read the original abstract

Grokking -- generalization long after training-set interpolation -- has been accelerated by structure-agnostic interventions (gradient filtering, weight-norm clamping, geometric penalties). Whether the delay specifically measures the time to form task-structured representations has remained observational. We test it causally by injecting representational priors of varying content into a one-layer transformer learning modular addition, via a supervised-contrastive loss whose positives encode (i) the task's true structure ($(a+b) \bmod p$), (ii) a coherent-but-wrong sibling ($(a-b) \bmod p$), or (iii) a random partition -- all with identical loss form, strength, class sizes, and geometry. Whether generalization occurs follows a clean gradation: true 22/30 runs, sibling (same periodic features, wrong combination) 14/15, random (only memorizable) 0/20 (Fisher $p=1.3\times10^{-7}$). A weight-norm-matched control replaying the norm trajectory onto plain cross-entropy generalizes 0/15, ruling out the norm as mediator. Probes show structure formation precedes and predicts generalization in all runs. Only the true structure also accelerates grokking (up to $2.75\times$), but this is dose-dependent and bimodal. We then confirm the mechanism by prediction: because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator with a median $8.6\times$ speedup (up to $22\times$ on the fastest seeds, under 1000 epochs), growing monotonically as the norm is held lower; the residual stalls also vanish, though significant only pooled over the two mitigations run at both strengths ($0/40$ vs $6/20$, $p=7.7\times10^{-4}$), not per method. The grokking delay is, causally, the time to form the right representational structure -- decided at the level of features, not labels.