REVIEW 3 major objections 5 minor 1 cited by
What Can Grokking Teach Us About Learning Under Nonstationarity?
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The effective learning rate is the lever that lets a network overwrite old features.
desk verdict Genuinely useful recipe and a clean delayed-grokking result, but the paper's own LayerNorm experiment undercuts the single-knob mechanism story the abstract sells. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the effective learning rate, a scale-invariant measure of learning-rate magnitude: for scale-invariant networks it equals the learning rate divided by a power of the parameter norm (squared for gradient descent, first power for adaptive optimizers). Re-warming is implemented as a modification of Normalize-and-Project: after each optimizer step the weights are rescaled to their initial norm, and the optimizer's learning rate is raised on a fixed schedule or triggered by a CUSUM changepoint detector on the loss. This lets the effective learning rate be read directly from the schedule and makes feature learning controllable throughout training. Two metrics, change in normalized feature covariance and change in ReLU activation patterns, measure whether the network is actually learning features. A short Gaussian-gradient derivation shows that the probability a ReLU unit flips or an embedding rotates depends on the effective learning rate, which is the mechanism the method exploits.
What would settle it
Train a network on task A with fixed parameter norms and an annealed learning rate, switch to task B, and raise the effective learning rate: if activation-pattern and feature-covariance metrics do not jump while the generalization gap on task B closes, the causal claim fails. Equivalently, find any trained network in which the effective learning rate is still high but primacy bias persists; that would show the effective learning rate is not the only gate on feature overwriting.
Extended reading notes
Core claim
On the paper's own terms: networks generalize only when they are in a feature-learning regime, in which hidden representations change substantially; memorization and primacy bias are both signs that the network is stuck in a lazy, kernel-like regime where updates barely alter the representation. The effective learning rate controls which regime the network is in: as parameter norms grow or the optimizer anneals, it decays and feature learning stops. The paper's central discovery is that the effective learning rate can be re-warmed late in training, by projecting parameters to a fixed norm and raising the learning rate on a fixed cycle or when a changepoint detector like CUSUM flags a distribution shift, so that the network overwrites stale or memorizing features with generalizing ones. This converts grokking from a curiosity into a controllable transition: grokking can be induced at arbitrary points in training, warm-started networks recover the test accuracy of networks trained on the full dataset, and high-update-to-data DQN and Rainbow agents improve when learning rate cycles are run early and then annealed. The paper concludes that a single scalar control, the effective learning rate, determines whether a network can overwrite old features.
Load-bearing premise
The load-bearing assumption is that the effective learning rate keeps governing feature-learning dynamics long after training has left initialization, so that raising it late can still force the network to overwrite old features.
Editorial extensions
If this is right
- Grokking, primacy bias, and at least part of plasticity loss reduce to one failure: the effective learning rate has decayed too far for the network to change its features.
- Grokking can be induced on demand: even after hundreds of thousands of low-learning-rate steps, a sufficiently large effective-learning-rate increase triggers feature learning and closes the test gap.
- Warm-starting generalization gaps can be closed regardless of the initial data fraction, as long as parameter norms stay fixed and the re-warmed effective learning rate is large enough.
- In high-update-to-data reinforcement learning, early learning-rate cycling followed by annealing beats both constant low learning rates and parameter resets, though high-learning-rate phases must be transient to avoid instability.
- Feature learning alone is not sufficient: the loss landscape must also permit convergence to a generalizing solution, which is why scale decay on normalization layers is needed in some architectures.
Reading between the lines
- If one mechanism governs all three phenomena, then plasticity-loss treatments that re-initialize or perturb parameters could be replaced by a cheaper schedule intervention that re-warms the effective learning rate; this is a testable prediction the paper does not run.
- This account suggests a concrete diagnostic for continual learning: track the effective learning rate and activation-pattern change across a task switch; a generalization gap appearing while the effective learning rate remains high would be evidence against the mechanism.
- A likely side effect of effective-learning-rate re-warming in continual learning is accelerated forgetting of earlier tasks, since overwriting is exactly what the method induces; the paper's experiments measure final-task generalization, not retention, so the stability trade-off is left open.
- Because scale-invariance is what makes the projection exact, architectures with substantial non-scale-invariant components, such as biases or unnormalized layers, may need per-module norm control before effective-learning-rate re-warming transfers; the paper's LayerNorm experiment already hints at this boundary.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper conjectures that the feature-learning dynamics that drive grokking are the same dynamics needed to overwrite previously learned features in nonstationary learning, and that both are controlled by the effective learning rate (ELR), defined as the ratio between update and parameter norms. It proposes ELR re-warming, implemented as a modification of the Normalize-and-Project (NaP) method: keep parameter norms fixed by projection and periodically reset the optimizer learning rate upward, either on a fixed schedule or triggered by CUSUM changepoint detection. The method is evaluated in three settings: grokking on modular arithmetic, warm-starting image classification on CIFAR-10, and Atari RL with DQN and Rainbow. In all three settings, the paper reports improved generalization relative to constant-learning-rate baselines. The authors also propose feature-learning metrics based on changes in normalized feature covariance and activation patterns, and provide a toy analysis in Appendix C relating the ELR to rotations and activation flips in a single-hidden-layer ReLU network.
Significance. If the central claim holds, the paper offers a unifying account of grokking, primacy bias, and loss of plasticity, with a remarkably simple practical intervention: a non-monotone learning-rate schedule under norm control. The empirical evidence is consistent across diverse domains, and the paper is honest in labeling the framework as a conjecture. The main strengths are the breadth of the evaluation, the use of explicit feature-learning metrics rather than test accuracy alone, and the clear statement of the proposed mechanism. However, the paper's own LayerNorm experiment shows that ELR is not a standalone scalar control over feature learning, and the theoretical appendix is explicitly restricted to near-initialization dynamics. These issues do not invalidate the practical method, but they do mean the central mechanism claim as stated in the abstract and conclusion needs substantial re-scoping.
major comments (3)
- [Section 4.1, Figure 2 RHS] The LayerNorm experiment is an internal counterexample to the claim that ELR re-warming alone induces feature learning and generalization. With NaP projection holding the ELR high, the network completely fails to generalize; grokking appears only after scale decay is added to the LayerNorm parameters. Algorithm 1 in Section 3.2 does not include scale decay, and the abstract and Section 7 attribute the results to ELR re-warming. The practical recipe is therefore a combination of at least two interventions: ELR re-warming and either scale decay or a parameterization in which the ELR is sufficient. Please reframe the central claim as 'norm-controlled ELR re-warming under a compatible parameterization,' and either add scale decay to Algorithm 1 or explicitly state that the algorithm requires a scale-invariant or LayerNorm-free parameterization.
- [Section 6.1-6.2, Figure 7] The high-UTD RL experiments do not directly test ELR re-warming because they are run without NaP projection. Section 6.1 explicitly states that the DQN agent is trained 'without weight decay or projection,' and the Rainbow high-UTD results in Section 6.2 appear to follow the same protocol. In that setting, raw learning-rate cycling does not control the ELR: parameter norms can grow over training and offset the increase in the raw learning rate, so the effective learning rate may not be re-warmed at all. Please either apply NaP in the high-UTD experiments or report ELR traces over training and reframe the RL section as evidence for learning-rate cycling rather than ELR re-warming.
- [Appendix C, text after Eq. (5)] The theoretical support for the mechanism is stated to hold 'when the network is close to initialization,' while the method is applied after hundreds of thousands of optimizer steps. The Gaussian-gradient model also assumes updates are approximately orthogonal to parameters and that row norms stay near one, assumptions that are unlikely to hold late in training. The derivation therefore cannot support the conclusion in Section 7 that re-warming the ELR 'can rapidly induce feature-learning, and thus generalization' in the late-training regime where the method is actually used. Please either extend the analysis to later training or explicitly restrict the theoretical claim to the near-initialization regime and present the late-training behavior as an empirically supported conjecture throughout the paper, including the conclusion.
minor comments (5)
- [Section 4.2] The sentence 'the optimization process much satisfy two criteria' appears to be missing the word 'must'; please correct this typo.
- [Section 4.1] The phrase 'a technique which we refer to as which we refer to by the term scale decay' contains a duplicated clause and should be rewritten as 'a technique we refer to as scale decay.'
- [Appendix C, paragraph after Eq. (5)] The statement 'the expected norm for each row satisfies E[||g_i||] = sigma_g^2' is dimensionally inconsistent; it should be E[||g_i||^2] = sigma_g^2, i.e., 'expected squared norm.' Please correct this and adjust the surrounding probability statements accordingly.
- [Algorithm 1] The function ResetLR(St+1) is not defined; please specify whether it resets only the step size or also clears the optimizer moment estimates, since this choice affects the practical behavior of the method.
- [General] The paper uses inconsistent capitalization 'Cusum' and 'CUSUM' for the changepoint detector; please standardize on one form.
Circularity Check
Partial circularity: ELR is defined as the quantity governing network-function dynamics, and the paper's feature-learning metrics measure those same dynamics, so the 'ELR induces feature-learning' claim is partly definitional; the practical grokking, warm-start, and RL results remain external and non-circular.
-
self definitional
[Section 2.3 (Eq. 1), Section 3.1 (Eqs. 3-4), and Section 4.1]
"the effective learning rate ˜η to replicate the learning dynamics on θ with unit-norm parameters ˜θ is given by ˜η(θ) = η/∥θ∥^2 ... Such a rescaling is exact for scale-invariant functions ... based on our previous arguments, we predict that a large effective learning rate will induce feature-learning dynamics"
The feature-learning metrics in Eqs. (3)-(4) are defined as changes in normalized layer outputs fℓ(X) and in activation patterns, i.e., observables of the network function. Eq. (1) defines η̃ precisely as the learning rate that reproduces the network-function dynamics after rescaling: the trajectory of f under (θ, η) is identical to the trajectory under (θ/∥θ∥, η̃) for scale-invariant f. Under the NaP projection, parameter norms are pinned, so η̃ is just η times a constant, and the claim that a larger η̃ 'induces feature-learning' (larger Δ_C, Δ_A) follows from the definition of η̃ together with the fact that a larger normalized step moves the network function farther.
full rationale
The paper's central practical claims are anchored by external benchmarks: test accuracy in modular-arithmetic grokking, CIFAR-10 warm-start generalization, and Atari returns in DQN/Rainbow. Those results do not reduce to the definition of the effective learning rate, so the paper is not globally circular. The main definitional component is in the mechanism story: ELR is introduced as the rescaled learning rate that exactly replicates network-function dynamics for scale-invariant functions (Eq. 1), while feature learning is measured by changes in normalized network features (Eqs. 3-4). For a scale-invariant network with NaP projection, asserting that ELR governs the rate of feature change is therefore close to a tautology, and Appendix C merely recomputes this dependence under a Gaussian-gradient model. This raises the circularity score above the 0-2 range. However, the LayerNorm counterexample in Section 4.1 (where high ELR with projection fails to grok without scale decay) is a correctness and scope limitation rather than a circularity, and the self-citations to Lyle et al. (2024a) for NaP are not load-bearing in a circular way: NaP is prior published work used as a tool, and its effectiveness is judged here against external generalization benchmarks. Overall, the practical contributions stand independently, but the mechanistic claim that ELR 'induces feature-learning dynamics' is partially self-definitional, justifying a moderate score of 4.
Assumptions & free parameters
free parameters (5)
- Re-warming learning rate magnitude =
0.01 vs 0.001 (grokking); 6.25e-5 / 1e-6 cyclic bounds (Atari); 1x to 10x initial LR (warm-starting)
- Cycle length and annealing point =
1000-step warmup (grokking); 4M-step cycles with stop at 8, 20, or 40M frames (RL); 5 cycles over 140 epochs…
- Scale decay coefficient =
1.0 and 10.0 (sweep, Figure 2 RHS)
- CUSUM detector parameters =
not reported
- Gradient noise scale sigma_g =
not estimated
assumptions (5)
- domain assumption The network is scale-invariant or approximately so: f(theta) = f(alpha*theta), making the effective learning rate the governing quantity (Eq. 1)
- ad hoc to paper Gradients are independent Gaussian perturbations with g[i,j] ~ N(0, sigma_g^2/d), and updates are approximately orthogonal to parameters
- ad hoc to paper The conjecture that the feature-learning dynamics of grokking are the same dynamics needed to overwrite previously learned features in continual learning
- domain assumption Activation-pattern change (Eq. 4) and normalized feature-covariance change (Eq. 3) measure the feature learning that causes generalization
- standard math Lazy-to-rich (NTK to feature-learning) transition is the correct description of grokking in finite networks
Cite this review
Pith. "Pith review of What Can Grokking Teach Us About Learning Under Nonstationarity?." pith.science (2026). https://pith.science/paper/5HSEEXCW
@misc{pith2026250720057,
author = {Pith},
title = {Pith review of: What Can Grokking Teach Us About Learning Under Nonstationarity?},
year = {2026},
howpublished = {\url{https://pith.science/paper/5HSEEXCW}},
note = {Machine review of arXiv:2507.20057}
}
read the original abstract
In continual learning problems, it is often necessary to overwrite components of a neural network's learned representation in response to changes in the data stream; however, neural networks often exhibit \primacy bias, whereby early training data hinders the network's ability to generalize on later tasks. While feature-learning dynamics of nonstationary learning problems are not well studied, the emergence of feature-learning dynamics is known to drive the phenomenon of grokking, wherein neural networks initially memorize their training data and only later exhibit perfect generalization. This work conjectures that the same feature-learning dynamics which facilitate generalization in grokking also underlie the ability to overwrite previous learned features as well, and methods which accelerate grokking by facilitating feature-learning dynamics are promising candidates for addressing primacy bias in non-stationary learning problems. We then propose a straightforward method to induce feature-learning dynamics as needed throughout training by increasing the effective learning rate, i.e. the ratio between parameter and update norms. We show that this approach both facilitates feature-learning and improves generalization in a variety of settings, including grokking, warm-starting neural network training, and reinforcement learning tasks.
Figures
Figures from the paper (10 more)
Forward citations
Cited by 1 Pith paper
-
The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
Capability formation in small transformers is claimed to obey a driven-nucleation rate law J = Nνσ(c)e^{−βK} − D, read forward as emergence, backward as plasticity loss, and completed as circuit control.
Reference graph
Works this paper leans on
-
[1]
The architecture consists of an input embedding layer using absolute positional encoding, followed by a single attention block, followed by a two- layer MLP, followed by a decoding layer. LayerNorm layers, when incorporated, are applied to the attention inputs and outputs, and the MLP outputs. We omit bias terms in the network and use RMSNorm by default. ...
work page 2015
-
[2]
So the likelihood of a perturbation changing the activity of a particular unit scales as arctan( 1 η ), assuming our original (not scale-invariant) setup. Incorporating 2https://math.stackexchange.com/questions/4433691/what-is-pxy0-mid-x0-given-that-x-y-two-different-normal 21 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 scale-in...
-
[4]
Unveiling grokking: Analyzing feature learning dynamics during training
Javier Sanguino Bautiste, Gregor Bachmann, Bobby He, Lorenzo Noci, and Thomas Hofmann. Unveiling grokking: Analyzing feature learning dynamics during training. In High-dimensional Learning Dynamics 2024: The Emer- gence of Structure and Reasoning,
work page 2024
-
[7]
Unifying grokking and double descent
Xander Davies, Lauro Langosco, and David Krueger. Unifying grokking and double descent. arXiv preprint arXiv:2303.06173,
-
[8]
Continual backprop: Stochastic gradient descent with persistent randomness
Shibhansh Dohare, A Rupam Mahmood, and Richard S Sutton. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.06325,
-
[12]
Learning continually by spectral regularization
Alex Lewandowski, Michał Bortkiewicz, Saurabh Kumar, Andr ´as Gy¨orgy, Dale Schuurmans, Mateusz Ostaszewski, and Marlos C Machado. Learning continually by spectral regularization. In 13th International Conference on Learning Representations, ICLR 2025,
work page 2025
-
[14]
An exponential learning rate schedule for deep learning
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In 8th International Confer- ence on Learning Representations, ICLR 2020,
work page 2020
-
[15]
Normalization and effective learning rates in reinforcement learning
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. 12 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 Clare Lyle, Zeyu ...
arXiv 2025
Show all 30 references
-
[16]
Progress measures for grokking via mechanistic interpretability
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217,
-
[19]
Cyclical learning rates for training neural networks
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464–472. IEEE,
2017
-
[20]
On the infinite width limit of neural networks with a standard parameterization
Jascha Sohl-Dickstein, Roman Novak, Samuel S Schoenholz, and Jaehoon Lee. On the infinite width limit of neural networks with a standard parameterization. arXiv preprint arXiv:2001.07301,
2001 arXiv
-
[21]
Susskind
Vimal Thilak, Etai Littwin, Shuangfei Zhai, Omid Saremi, Roni Paiss, and Joshua M. Susskind. The slingshot mech- anism: An empirical study of adaptive optimizers and the\emph{Grokking Phenomenon}. In Has it Trained Yet? NeurIPS 2022 Workshop,
2022
-
[22]
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil
URL https://openreview.net/forum?id=lY1e0PNkSJ. Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep rein- forcement learning and the deadly triad. arXiv preprint arXiv:1812.02648,
-
[23]
Explaining grokking through circuit efficiency
Vikrant Varma, Rohin Shah, Zachary Kenton, J´anos Kram´ar, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390,
-
[24]
Small-scale proxies for large-scale transformer training instabilities
Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309.14322,
-
[25]
Wide feedforward or recurrent neural networks of any architecture are gaussian processes
13 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32,
2025
-
[27]
14 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 Parameter Value num layers 2 num heads 4 query/key/value dimension 32 feed-forward hidden size 512 dropout rate 0 relative position embeddings False absolute position length 5 LayerNorm position before a...
2025
-
[29]
is the ReLU function (when it is applied to a vector, it is done coordinatewise). We assume a simplified optimization setting with inputs X∈ Rn×m, wheren denotes the number of data points and m the input dimension, so that each row ofX is of the form xk for some data point xk∈...
2024
-
[2013]
A study on the plasticity of neural networks
Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042,
-
[2015]
Early stopping in deep networks: Double descent and how to eliminate it
11 Published at 4th Conference on Lifelong Learning Agents (CoLLAs), 2025 Reinhard Heckel and Fatih Furkan Yilmaz. Early stopping in deep networks: Double descent and how to eliminate it. In International Conference on Learning Representations,
2025
-
[2016]
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra
URL https://proceedings.neurips.cc/paper/2016/file/ 148510031349642de5ca0c544f31b2ef-Paper.pdf. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177,
2016 arXiv
-
[2017]
Second-order regression models exhibit progressive sharp- ening to the edge of stability
Atish Agarwala, Fabian Pedregosa, and Jeffrey Pennington. Second-order regression models exhibit progressive sharp- ening to the edge of stability. arXiv preprint arXiv:2210.04860,
-
[2018]
Maintaining plasticity via regenerative regularization
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity via regenerative regularization. arXiv preprint arXiv:2308.11958,
-
[2019]
Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466,
-
[2020]
Implicit gradient regularization
David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009.11162,
2009 arXiv
-
[2021]
Gradient descent on neural networks typically occurs at the edge of stability
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065,
-
[2022]
On the interplay between stepsize tuning and progressive sharpening
Vincent Roulet, Atish Agarwala, and Fabian Pedregosa. On the interplay between stepsize tuning and progressive sharpening. In OPT 2023: Optimization for Machine Learning,
2023
-
[2023]
Critical learning periods in deep neural networks
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv:1711.08856,
-
[2024]
Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado
URL https://proceedings.mlr.press/v235/lee24d.html. Alex Lewandowski, Haruto Tanaka, Dale Schuurmans, and Marlos C Machado. Curvature explains loss of plasticity. arXiv preprint arXiv:2312.00246,
-
[2025]
The large learning rate phase of deep learning: the catapult mechanism
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218,
2003 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.