{"id":"f8eb43f0-750f-4819-8138-8e31a561226c","arxiv_id":"2412.10898","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper reproduces the grokking phenomenon in a Transformer on modular arithmetic, finds no grokking in MLPs and LSTMs under the tested settings, and attributes the difference to missing weight decay.","lead":"This paper re-runs grokking experiments on modular addition with Transformers, LSTMs, and MLPs, and shows how data fraction, architecture, and optimizer affect the sudden jump from memorization to generalization. It reproduces earlier results that weight decay helps grokking, and reviews two proposed mechanisms, so it is most useful as a replication and tutorial study.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MLP/LSTM non-grokking result is confounded by optimizer: Transformer uses AdamW weight decay 1 while MLP/LSTM use plain Adam, and the paper never tests the large-init, small-weight-decay regime it cites.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: the model comparison conflates architecture with optimizer and initialization choices. The paper's central empirical contribution—that MLP and LSTM do not grok under these conditions—is presented as a model effect, but the Transformer uses AdamW while MLP/LSTM use plain Adam, and the paper never varies initialization scale. The authors themselves cite the Omnigrok regime in Section 5 and concede in Section 6 that the observed absence may be due to not using weight decay. This does not make the paper fraudulent or its individual experiments wrong, but it means the architecture-comparison conclusion is not established. The reader's CONDITIONAL verdict already captures this, so I do not recommend changing the verdict; rather, the concrete test above would settle whether the concern actually lands. If the test shows grokking in MLP/LSTM under the Omnigrok regime, the paper's model-variation claim would need to be substantially revised.","tokens_in":6679,"tokens_out":4473,"duration_ms":40228,"concrete_test":"Re-run the MLP and LSTM experiments from Section 4.2 on (x+y) mod 97 using the Omnigrok-style recipe: AdamW with weight decay 1, initialization scale multiplied by roughly 10-100, and training data fractions α in {30%, 45%, 60%}, tracking validation accuracy over at least 100k steps. Also run the Transformer under plain Adam with the same initialization scale as a control. If either MLP or LSTM exhibits a delayed, sharp validation accuracy jump in this regime, the paper's model-variation conclusion is undercut.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2's comparison is presented as a model/architecture effect, but it does not control for optimizer or initialization. The Transformer is trained with AdamW and weight decay 1 (Section 3), while the MLP and LSTM are trained with plain Adam and no weight decay. The cited Omnigrok work [6] and Lyu et al. [8] identify large initialization plus small weight decay as sufficient to induce grokking in MLPs, a regime the paper never tests. The paper itself concedes this in Section 6: 'we understand this might be attributed to not employing weight decay techniques.' Thus the observed absence of grokking in MLP/LSTM is exactly what the confounded setup predicts, and it cannot support the claim that architecture, rather than optimization/initialization, is the determining factor. Without this control, the statement that 'the MLP model does not exhibit the grokking phenomenon in this context' is at best conditional on settings that prior work has shown to be the wrong regime for MLPs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports experiments on grokking for the modular addition task with p=97, varying the training data fraction, the model architecture (Transformer, MLP, LSTM), and the optimizer/regularization scheme. It claims that the Transformer exhibits grokking most clearly at intermediate data fractions around 50%, that MLP and LSTM models do not exhibit grokking in this context, and that AdamW weight decay improves generalization the most among the interventions tested. It also summarizes two existing mechanistic accounts of grokking: structured representations in a 'Goldilocks zone' and implicit biases from large initialization and small weight decay.","tokens_in":6979,"tokens_out":5477,"duration_ms":48598,"significance":"If the empirical claims were fully supported, the paper would provide a useful replication and extension of Power et al.'s grokking experiments and would highlight interactions among architecture, data fraction, and optimizer. The paper deserves credit for reproducing the original OpenAI setup, providing a code link for the MLP experiments, and explicitly acknowledging in its conclusion that the absence of grokking in MLP/LSTM may be due to the lack of weight decay. However, the central comparative claim about architecture is currently under-supported because the models are compared under different optimizers and initializations, and no seed statistics are reported. The mechanism section is a literature review rather than a new mechanistic investigation, so the paper's value is primarily as a conditional empirical observation.","major_comments":[{"comment":"The architecture comparison is confounded by optimizer and initialization. The Transformer is trained with AdamW and weight decay 1 (Section 3), while the MLP and LSTM are trained with plain Adam and no weight decay. The paper itself cites [6] and [8] showing that large initialization with small weight decay can induce grokking in MLPs, and Section 6 concedes that the absence 'might be attributed to not employing weight decay techniques.' Figures 3 and 4 therefore do not establish that MLP and LSTM fail to grok as a property of architecture; they establish a property of the combined hyperparameter configuration. Please rerun the MLP and LSTM in the large-initialization/small-weight-decay regime, or explicitly limit the claim to the tested configuration and remove the implication that architecture is the determining factor.","section":"Section 4.2 / Section 6"},{"comment":"The data-fraction scan omits the claimed 50% point. The text states that grokking is 'most pronounced when alpha is around 50%,' but the experiments use α = 5%, 30%, 45%, 60%, 75%, and 90%; α = 50% is never run. The observed gap at 45% and 60% is consistent with a peak near 50%, but it does not directly measure that point. Either add the α = 50% experiment or rephrase the claim as an interpolation from neighboring fractions.","section":"Section 4.1, Figures 1-2, Figure 7"},{"comment":"No error bars, confidence intervals, or multiple-seed trials are reported for any accuracy curve or for the steps-until-generalization values in Figure 7. Grokking transition times are known to be sensitive to seed and initialization, and a single trajectory is insufficient to support quantitative statements such as 'it takes around 6000 epoch for generalization' (Section 4.1). Please report statistics over at least a few seeds, or explicitly label these as single illustrative runs and avoid quantitative claims based on a single trajectory.","section":"Section 4.1, Figure 7"},{"comment":"The claim that 'weight decay, i.e. AdamW, improves generalization the most' is not supported by a defined comparison. Figure 5 presents eight panels with different optimizers, learning rates, and batch sizes, but the text gives no metric (e.g., final validation accuracy, area under the curve, or time to a threshold) by which AdamW is judged best, and all panels appear to be single runs. In addition, the caption labels panel (a) as '3x baseline LR' while the text bullet list starts with 3e-4, which is 0.3x baseline; please align the caption and text and specify the comparison protocol.","section":"Section 4.3, Figure 5"}],"minor_comments":[{"comment":"The definition of α would be clearer if it explicitly stated that the total data size is p^2 for the modular addition task, since the text currently refers to p^2 only indirectly.","section":"Section 2.1"},{"comment":"The LSTM is described as having 'hidden size of 20' in Section 3 but 'number of hidden layers to 20' in Section 4.2; please make the terminology and notation consistent.","section":"Section 3 / Section 4.2"},{"comment":"There is a typo: 'updats' should be 'updates'.","section":"Section 4.3"},{"comment":"The mechanism section is a literature review rather than a new mechanistic investigation; this is acceptable as background, but the framing should state this explicitly so readers do not expect a novel mechanism.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"This manuscript has the character of a course project report rather than a full research article: the empirical portion is a replication/extension of Power et al., and the mechanism section is a review. If the journal publishes replication studies or explicitly conditional empirical results, the paper is within scope once the confound in the model comparison is addressed. The main risk is that the headline claim about MLP/LSTM not grokking will be cited without the hyperparameter caveat, so the revision should make the conditional nature of that claim prominent and ideally add the missing control experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this reads like a solid undergrad project, not a research paper. The authors reproduce the data-fraction dependence for grokking on modular addition with a Transformer, and they run a wide-ish optimization sweep. That part is fine and honestly reported. What is not fine is the model comparison. The Transformer is trained with AdamW and weight decay 1; the MLP and LSTM are trained with plain Adam and no weight decay. The paper itself concedes in the conclusion that the absence of grokking in MLP/LSTM \"might be attributed to not employing weight decay techniques.\" That concession is fatal to the Section 4.2 claim that these models do not exhibit grokking in this context. The stress-test note is right: the setup predicts exactly what they observed, and the cited Omnigrok/Lyu work gives a regime (large init, small weight decay) they never tested. So the negative result is a statement about hyperparameters, not architectures.\n\nWhat is good: the exploration of data fractions at least qualitatively tracks Power et al.; the optimizer comparison in Fig 5 (even without seeds) shows the expected large benefit of weight decay; the mechanism discussion is a fair, brief summary of Liu et al. and Lyu et al. The writing is clear and the acknowledgments of their own limitations are candid. But the experimental gaps are real: no error bars, no multiple seeds, some internal inconsistencies (they say the MLP uses parameters identical to the Transformer's feed-forward network, but the Transformer's FFN is a one-layer MLP of width 512 while the MLP is described as 2 layers with hidden 512), and the claim of reproducing the 'alpha around 50%' result is weakened by their scan skipping 50% (45% and 60% shown). These are the kind of gaps you'd expect in a course report.\n\nI would not send this to a serious referee. It reproduces known results, and the only attempted novel contribution is methodologically confounded. The right venue for this, if any, is a course archive or a blog post. It does show the authors engaged seriously with the literature and with the replication, so as a pedagogical artifact it has some value.","headline":"A careful student replication of Power et al.'s grokking result, whose only new-looking claim (MLP/LSTM do not grok) is confounded by the fact that the Transformer got weight decay and the others did not.","tokens_in":7410,"tokens_out":1839,"would_cite":false,"duration_ms":15675,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that grokking in modular addition is conditional: the Transformer shows the delayed validation-accuracy jump at intermediate training-data fractions, while MLP and LSTM do not under the tested hyperparameters, and AdamW…","keywords":["grokking","modular addition","generalization","Transformer","MLP","LSTM","AdamW","weight decay"],"falsifier":"Train the same MLP on the same modular addition task with large initialization and small weight decay; if its validation accuracy shows the delayed sharp jump after training accuracy reaches 100%, the negative MLP result is a hyperparameter effect rather than an architectural one. This experiment would settle the paper's central model comparison.","tokens_in":6446,"feed_emoji":"🧮","tokens_out":8392,"duration_ms":74250,"temperature":0.7,"pith_summary":"This paper reports a broad set of experiments on when grokking appears in modular addition, the task where a network first memorizes its training inputs and later, after much more training, suddenly generalizes. The central finding is that this delayed jump in validation accuracy is strongest at intermediate training-data fractions: with modulus 97, fractions near 45–60% produce the classic long gap between perfect training accuracy and near-chance validation accuracy before generalization. The authors also find that, under the hyperparameter settings they chose, the Transformer exhibits the jump while the MLP and LSTM models do not, and that the AdamW form of weight decay gives the largest generalization improvement within a fixed budget. These results map the conditions that make grokking visible and show how sensitive the phenomenon is to architecture and optimizer choice. The paper is an experimental study that does not claim a new mechanism, and it explicitly notes that other results have produced grokking in MLPs with different initialization and weight-decay settings.","feed_headline":"Grokking appears only in Transformer in modular-additions test","feed_subtitle":"Delayed generalization jump peaks near 50% training data, and AdamW weight decay gives the largest boost.","key_machinery":"The load-bearing object is the delayed generalization transition itself: the interval during which training loss is already near zero but validation accuracy remains at chance, ending in a sharp rise. The paper generates this object with the modular addition task, a training-data fraction $\\alpha$ that controls how many of the $p^2$ pairs the model sees, and three model families. The Transformer is a two-layer decoder-only model with four attention heads and ReLU feed-forward blocks; the MLP and LSTM are matched to comparable size but trained with their own optimizers. The experimental argument compares the shape of the training and validation accuracy curves across $\\alpha$, across architectures, and across optimizer variants, treating the gap between the two curves as the signature of grokking.","core_discovery":"The central claim is that grokking is a real but configuration-dependent transition in the modular addition problem $(x,y)\\mapsto (x+y)\\bmod 97$ viewed as a classification task over the $97^2$ possible input pairs. In the Transformer, training accuracy reaches 100% within a few hundred steps while validation accuracy stays near zero for thousands of steps, then jumps sharply to perfect generalization; this behavior is most pronounced when roughly 30–60% of the input pairs are used for training. At very low fractions the model never generalizes, and at high fractions validation accuracy tracks training almost immediately. The same delayed jump is not seen in the MLP or LSTM under the hyperparameters tested, where training and validation accuracy rise together. In the optimizer comparison, AdamW with weight decay 1 produces the strongest improvement in validation accuracy within the fixed training budget, while full-batch Adam without weight decay still generalizes at higher data fractions, and poor learning-rate or minibatch choices sharply limit generalization.","pith_inferences":["Editorial inference: the absence of grokking in the MLP and LSTM is likely an artifact of the chosen optimization regime rather than an architectural fact, because the paper itself cites a setting with large initialization and small weight decay that produces grokking in MLPs.","Editorial inference: a direct test of that possibility would be to train the same MLP on the same modular addition task with large initialization and small weight decay; a delayed validation jump there would identify the optimizer configuration as the sufficient condition for grokking.","Editorial inference: the strong effect of weight decay suggests a follow-up sweep of weight-decay values at a fixed data fraction, which could reveal a sharp regularization threshold analogous to the sharp threshold in $\\alpha$.","Editorial inference: if grokking is primarily an optimization-trajectory effect, then the structured-representation and implicit-bias explanations are not competing but complementary, and progress measures built on representation geometry should be tested in the large-initialization regime."],"forward_implications":["For modular addition with $p=97$, the classic grokking gap appears near training-data fractions of 45–60%, with generalization arriving only after roughly 6,000 additional steps beyond perfect training accuracy.","Under the hyperparameters chosen in the paper, the Transformer, MLP, and LSTM separate into two behaviors: the Transformer shows the delayed validation jump, while MLP and LSTM show accuracy curves that rise together.","AdamW with weight decay 1 is the strongest single intervention for improving generalization within the training budget, but weight decay is not strictly required at higher data fractions, where full-batch Adam also generalizes.","The encoding of the arithmetic problem changes the difficulty: the more complex dictionary-style encoding from the original setup requires a larger training-data fraction before the Transformer can generalize than the direct classification encoding.","Grokking results are sensitive to optimizer configuration: suboptimal learning rates or minibatch choices can suppress the generalization transition within the fixed budget, so negative grokking results need to be tied to the exact training settings."],"supporting_citations":[{"why":"Supplies the original grokking setup, the modular-arithmetic task, and the data-fraction curves this paper reproduces.","marker":"[11]"},{"why":"Provides the result that large initialization and small weight decay can induce grokking in MLPs, the main qualification to the paper's negative MLP finding.","marker":"[6]"},{"why":"Gives the structured-representation and 'Goldilocks zone' framework that the paper reviews as one candidate mechanism.","marker":"[5]"},{"why":"Gives the early-vs-late phase implicit-bias theory that the paper reviews as the second candidate mechanism.","marker":"[8]"},{"why":"Introduces AdamW and weight-decay decoupling, the optimizer intervention the paper finds most effective.","marker":"[7]"},{"why":"Describes the slingshot mechanism, the reason the paper chooses full-batch training to smooth dynamics.","marker":"[13]"},{"why":"Defines the Transformer architecture used as the grokking-positive model in the comparisons.","marker":"[15]"}],"fun_headline_variants":["Grokking seen only in Transformer on modular addition","Delayed generalization jump peaks in Transformer, not MLP or LSTM","AdamW weight decay boosts grokking in modular addition","Grokking appears near 50% training data in Transformer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that the different optimizer and initialization settings chosen for each model are the fair common ground, so the conclusion that MLPs and LSTMs do not grok is a conclusion about those settings rather than about the architectures themselves.","fun_headline_variants_meta":{"raw":{"variants":["Grokking seen only in Transformer on modular addition","Delayed generalization jump peaks in Transformer, not MLP or LSTM","AdamW weight decay boosts grokking in modular addition","Grokking appears near 50% training data in Transformer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1301,"prompt_tokens":835,"completion_tokens":466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":396}},"tokens_in":451,"tokens_out":466,"duration_ms":4010,"temperature":1.0,"reasoning_tokens":396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:29:59.318950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same MLP on the same modular addition task with large initialization and small weight decay; if its validation accuracy shows the delayed sharp jump after training accuracy reaches 100%, the negative MLP result is a hyperparameter effect rather than an architectural one. This experiment would settle the paper's central model comparison.","supporting_citations":[{"cited_title":"Michaud, Max Tegmark, and Mike Williams","cited_arxiv_id":null,"evidence_quote":"Gives the structured-representation and 'Goldilocks zone' framework that the paper reviews as one candidate mechanism."},{"cited_title":"The slingshot mechanism: An empirical study of adaptive optimizers and the grokking phenomenon, 2022","cited_arxiv_id":null,"evidence_quote":"Describes the slingshot mechanism, the reason the paper chooses full-batch training to smooth dynamics."},{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Defines the Transformer architecture used as the grokking-positive model in the comparisons."}],"review_version":1}