{"id":"a839ee6c-8d8e-41c1-a54b-107e9ee59bf6","arxiv_id":"2507.20057","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Periodically increasing the effective learning rate while constraining parameter norms induces feature-learning dynamics and mitigates primacy bias in grokking, warm-starting, and reinforcement learning.","lead":"The paper connects two puzzles in neural network training, grokking (sudden generalization after memorization) and primacy bias (early data blocking later learning), and proposes that the same feature-rewriting dynamics drive both. It shows that periodically raising the effective learning rate, while keeping weights normalized, improves generalization on modular arithmetic, CIFAR-10 warm-starting, and Atari games.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own LayerNorm experiment shows ELR is not a standalone scalar control: with NaP projection and high ELR, grokking fails unless scale decay is added, so the claimed mechanism is parameterization-dependent.","rationale":"The reader's verdict and weakest assumption point in the same direction, but I want to sharpen the emphasis. The most load-bearing problem is not primarily the near-initialization caveat of Appendix C, which the authors state themselves; it is that the paper's own Figure 2 (RHS) provides an internal counterexample to the strong 'one scalar control' version of the claim. In a LayerNorm transformer, high ELR maintained by projection does not induce grokking; scale decay of the LayerNorm parameters is required. This means the actual control variable is not just eta/||W|| but also the scale of the normalization parameters, i.e., the parameterization. The near-init derivation cannot rescue this, because it is derived under scale invariance and so cannot predict the LayerNorm failure. The practical method may still be useful, and the authors explicitly include scale decay, so the conditional verdict stands; but the abstract and conclusion framing overstates the role of ELR alone. My proposed test settles whether there is any ELR value that makes the LayerNorm network grok without scale decay; if there is not, the paper should be revised to present ELR re-warming plus scale control as the method, and the mechanism as a two-knob account. This is consistent with the reader's CONDITIONAL verdict, so I mark no change.","tokens_in":22112,"tokens_out":9232,"duration_ms":98032,"concrete_test":"Run the Figure 2 RHS grokking setup (transformer with LayerNorm, NaP projection, fixed ELR) and sweep ELR over at least two orders of magnitude, e.g., eta/||W|| in {1e-4, 1e-3, 1e-2, 1e-1, 1}, with no scale decay, measuring test accuracy and the activation-pattern metric (Eq. 4). If no ELR value produces grokking or a sustained increase in activation-pattern change, then ELR alone does not control feature learning in this architecture, and the headline claim must be restricted to scale-decayed or otherwise normalized parameterizations. As a complementary control, hold ELR fixed and vary only the LayerNorm scale-decay coefficient; if generalization appears or disappears with scale decay at constant ELR, the missing control is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 7) is that re-warming the effective learning rate (ELR) can rapidly induce feature-learning and generalization. For this to be true as stated, ELR must be a well-defined control over feature-learning dynamics. But Section 2 defines ELR exactly only for scale-invariant functions, and Algorithm 1 explicitly takes a scale-invariant network as input. The grokking experiment in Section 4.1 (Figure 2, RHS) is therefore a direct stress test: the authors add LayerNorm to the same transformer, apply weight projection so that the ELR is held at a high value, and observe that the network completely fails to generalize; grokking only appears after an additional mechanism, scale decay on the LayerNorm parameters, is introduced. This is an internal counterexample to the claim that one scalar control (ELR) determines whether old features can be overwritten: in the LayerNorm parameterization, the effective sensitivity of features depends on the scale of the LayerNorm parameters, not only on eta/||W||. The theoretical support in Appendix C models only the scale-invariant case and is stated to hold near initialization, so it cannot explain this failure. Consequently, the practical recipe is a combination of at least two interventions, ELR re-warming plus scale decay or a suitable parameterization, and the 'single mechanism' account is incomplete. The paper acknowledges this in Section 4.1, but the abstract and conclusion still attribute the result to ELR re-warming, which weakens the central conjecture as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper conjectures that the feature-learning dynamics that drive grokking are the same dynamics needed to overwrite previously learned features in nonstationary learning, and that both are controlled by the effective learning rate (ELR), defined as the ratio between update and parameter norms. It proposes ELR re-warming, implemented as a modification of the Normalize-and-Project (NaP) method: keep parameter norms fixed by projection and periodically reset the optimizer learning rate upward, either on a fixed schedule or triggered by CUSUM changepoint detection. The method is evaluated in three settings: grokking on modular arithmetic, warm-starting image classification on CIFAR-10, and Atari RL with DQN and Rainbow. In all three settings, the paper reports improved generalization relative to constant-learning-rate baselines. The authors also propose feature-learning metrics based on changes in normalized feature covariance and activation patterns, and provide a toy analysis in Appendix C relating the ELR to rotations and activation flips in a single-hidden-layer ReLU network.","tokens_in":22266,"tokens_out":5918,"duration_ms":60293,"significance":"If the central claim holds, the paper offers a unifying account of grokking, primacy bias, and loss of plasticity, with a remarkably simple practical intervention: a non-monotone learning-rate schedule under norm control. The empirical evidence is consistent across diverse domains, and the paper is honest in labeling the framework as a conjecture. The main strengths are the breadth of the evaluation, the use of explicit feature-learning metrics rather than test accuracy alone, and the clear statement of the proposed mechanism. However, the paper's own LayerNorm experiment shows that ELR is not a standalone scalar control over feature learning, and the theoretical appendix is explicitly restricted to near-initialization dynamics. These issues do not invalidate the practical method, but they do mean the central mechanism claim as stated in the abstract and conclusion needs substantial re-scoping.","major_comments":[{"comment":"The LayerNorm experiment is an internal counterexample to the claim that ELR re-warming alone induces feature learning and generalization. With NaP projection holding the ELR high, the network completely fails to generalize; grokking appears only after scale decay is added to the LayerNorm parameters. Algorithm 1 in Section 3.2 does not include scale decay, and the abstract and Section 7 attribute the results to ELR re-warming. The practical recipe is therefore a combination of at least two interventions: ELR re-warming and either scale decay or a parameterization in which the ELR is sufficient. Please reframe the central claim as 'norm-controlled ELR re-warming under a compatible parameterization,' and either add scale decay to Algorithm 1 or explicitly state that the algorithm requires a scale-invariant or LayerNorm-free parameterization.","section":"Section 4.1, Figure 2 RHS"},{"comment":"The high-UTD RL experiments do not directly test ELR re-warming because they are run without NaP projection. Section 6.1 explicitly states that the DQN agent is trained 'without weight decay or projection,' and the Rainbow high-UTD results in Section 6.2 appear to follow the same protocol. In that setting, raw learning-rate cycling does not control the ELR: parameter norms can grow over training and offset the increase in the raw learning rate, so the effective learning rate may not be re-warmed at all. Please either apply NaP in the high-UTD experiments or report ELR traces over training and reframe the RL section as evidence for learning-rate cycling rather than ELR re-warming.","section":"Section 6.1-6.2, Figure 7"},{"comment":"The theoretical support for the mechanism is stated to hold 'when the network is close to initialization,' while the method is applied after hundreds of thousands of optimizer steps. The Gaussian-gradient model also assumes updates are approximately orthogonal to parameters and that row norms stay near one, assumptions that are unlikely to hold late in training. The derivation therefore cannot support the conclusion in Section 7 that re-warming the ELR 'can rapidly induce feature-learning, and thus generalization' in the late-training regime where the method is actually used. Please either extend the analysis to later training or explicitly restrict the theoretical claim to the near-initialization regime and present the late-training behavior as an empirically supported conjecture throughout the paper, including the conclusion.","section":"Appendix C, text after Eq. (5)"}],"minor_comments":[{"comment":"The sentence 'the optimization process much satisfy two criteria' appears to be missing the word 'must'; please correct this typo.","section":"Section 4.2"},{"comment":"The phrase 'a technique which we refer to as which we refer to by the term scale decay' contains a duplicated clause and should be rewritten as 'a technique we refer to as scale decay.'","section":"Section 4.1"},{"comment":"The statement 'the expected norm for each row satisfies E[||g_i||] = sigma_g^2' is dimensionally inconsistent; it should be E[||g_i||^2] = sigma_g^2, i.e., 'expected squared norm.' Please correct this and adjust the surrounding probability statements accordingly.","section":"Appendix C, paragraph after Eq. (5)"},{"comment":"The function ResetLR(St+1) is not defined; please specify whether it resets only the step size or also clears the optimizer moment estimates, since this choice affects the practical behavior of the method.","section":"Algorithm 1"},{"comment":"The paper uses inconsistent capitalization 'Cusum' and 'CUSUM' for the changepoint detector; please standardize on one form.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a strong empirical paper with a clear and useful message, and I see no grounds for rejection. The main risk is that the title, abstract, and conclusion overclaim relative to the parameterization dependence revealed by the paper's own LayerNorm experiment and the lack of norm control in the high-UTD RL section. These concerns are addressable within the manuscript's scope by re-scoping the claims and, ideally, adding the missing ablations or ELR traces. No issues with novelty or citation patterns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the practical recipe seems to work in their experiments: pin parameter norms with NaP, push the learning rate back up periodically, and you close the warm-start generalization gap and induce grokking at arbitrary late points. Figure 3 is a genuinely clean new result. Second, the unifying story is shakier than the abstract implies. The paper's own LayerNorm experiment (Figure 2, RHS) shows ELR is not a standalone control: with projection holding ELR high, the network fails to grok until scale decay is added. That is an internal counterexample to \"one scalar determines whether features can be overwritten.\" The authors acknowledge it in Section 4.1 but the abstract and conclusion still attribute the results to ELR re-warming alone.\n\nThe paper earns credit for honesty: the ELR-feature-learning link is explicitly a conjecture, and the empirical support is broad—grokking, warm-started CIFAR-10, and Atari. The delayed-grokking experiment matters because it rules out the trivial reading that the effect only lives near initialization. The weakest piece is Appendix C: the Gaussian-gradient toy model is stated to hold near initialization, exactly the regime the method is applied away from. That is a real gap, not fatal, because the conjecture framing carries it.\n\nThe stress-test note is right, and I do not think it is manufactured. The recipe is a package—norm projection, scale decay or a suitable parameterization, and re-warming—not one knob. The headline claim should say that.\n\nMinor but real reporting gaps: no code shipped, no seed spread in the grokking figures, a hand-picked 20-game subset for the headline RL gains, and the early-stopping point for the cyclic schedule is admitted to be arbitrary (B.3). The feature-learning metrics (Eqs. 3-4) scale with update magnitude almost by construction, so the mechanism verification is mildly circular; the test-accuracy results are external benchmarks and anchor the practical claim.\n\nWho this is for: anyone working on continual learning, plasticity, or grokking. It is a useful organizing frame even if the single-mechanism story turns out to be parameterization-dependent. It deserves a serious referee: the top-line results are worth verifying and the conjecture is productive. I would send it to review, asking the authors to reframe the abstract to match the LayerNorm result and to ship code.","headline":"Genuinely useful recipe and a clean delayed-grokking result, but the paper's own LayerNorm experiment undercuts the single-knob mechanism story the abstract sells.","tokens_in":22958,"tokens_out":4474,"would_cite":true,"duration_ms":39990,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The effective learning rate is the lever that lets a network overwrite old features.","keywords":["grokking","primacy bias","effective learning rate","feature learning","continual learning","plasticity loss","learning rate cycling","warm-starting"],"falsifier":"Train a network on task A with fixed parameter norms and an annealed learning rate, switch to task B, and raise the effective learning rate: if activation-pattern and feature-covariance metrics do not jump while the generalization gap on task B closes, the causal claim fails. Equivalently, find any trained network in which the effective learning rate is still high but primacy bias persists; that would show the effective learning rate is not the only gate on feature overwriting.","tokens_in":1971,"feed_emoji":"🔁","tokens_out":5970,"duration_ms":101296,"temperature":0.7,"pith_summary":"Neural networks that have memorized their training data can suddenly generalize, a phenomenon known as grokking, when their training dynamics shift from refining a fixed representation to changing the representation itself. This paper argues that the same shift explains why networks trained on early data resist learning new tasks: the effective learning rate, the ratio of update size to parameter size, has decayed to the point where the network can no longer overwrite its old features. The proposed fix is to keep parameter norms fixed by projection and periodically reset the optimizer's learning rate upward, which re-induces feature learning on demand. Doing so accelerates grokking in modular arithmetic, closes the generalization gap in warm-started image classification, and improves high-update-to-data Atari agents. A sympathetic reader would take away that primacy bias, plasticity loss, and grokking may be one phenomenon governed by one scalar.","feed_headline":"Raising the effective learning rate re-triggers feature learning","feed_subtitle":"Keeping parameter norms fixed and resetting the optimizer LR makes networks overwrite stale features and generalize.","key_machinery":"The central object is the effective learning rate, a scale-invariant measure of learning-rate magnitude: for scale-invariant networks it equals the learning rate divided by a power of the parameter norm (squared for gradient descent, first power for adaptive optimizers). Re-warming is implemented as a modification of Normalize-and-Project: after each optimizer step the weights are rescaled to their initial norm, and the optimizer's learning rate is raised on a fixed schedule or triggered by a CUSUM changepoint detector on the loss. This lets the effective learning rate be read directly from the schedule and makes feature learning controllable throughout training. Two metrics, change in normalized feature covariance and change in ReLU activation patterns, measure whether the network is actually learning features. A short Gaussian-gradient derivation shows that the probability a ReLU unit flips or an embedding rotates depends on the effective learning rate, which is the mechanism the method exploits.","core_discovery":"On the paper's own terms: networks generalize only when they are in a feature-learning regime, in which hidden representations change substantially; memorization and primacy bias are both signs that the network is stuck in a lazy, kernel-like regime where updates barely alter the representation. The effective learning rate controls which regime the network is in: as parameter norms grow or the optimizer anneals, it decays and feature learning stops. The paper's central discovery is that the effective learning rate can be re-warmed late in training, by projecting parameters to a fixed norm and raising the learning rate on a fixed cycle or when a changepoint detector like CUSUM flags a distribution shift, so that the network overwrites stale or memorizing features with generalizing ones. This converts grokking from a curiosity into a controllable transition: grokking can be induced at arbitrary points in training, warm-started networks recover the test accuracy of networks trained on the full dataset, and high-update-to-data DQN and Rainbow agents improve when learning rate cycles are run early and then annealed. The paper concludes that a single scalar control, the effective learning rate, determines whether a network can overwrite old features.","pith_inferences":["If one mechanism governs all three phenomena, then plasticity-loss treatments that re-initialize or perturb parameters could be replaced by a cheaper schedule intervention that re-warms the effective learning rate; this is a testable prediction the paper does not run.","This account suggests a concrete diagnostic for continual learning: track the effective learning rate and activation-pattern change across a task switch; a generalization gap appearing while the effective learning rate remains high would be evidence against the mechanism.","A likely side effect of effective-learning-rate re-warming in continual learning is accelerated forgetting of earlier tasks, since overwriting is exactly what the method induces; the paper's experiments measure final-task generalization, not retention, so the stability trade-off is left open.","Because scale-invariance is what makes the projection exact, architectures with substantial non-scale-invariant components, such as biases or unnormalized layers, may need per-module norm control before effective-learning-rate re-warming transfers; the paper's LayerNorm experiment already hints at this boundary."],"forward_implications":["Grokking, primacy bias, and at least part of plasticity loss reduce to one failure: the effective learning rate has decayed too far for the network to change its features.","Grokking can be induced on demand: even after hundreds of thousands of low-learning-rate steps, a sufficiently large effective-learning-rate increase triggers feature learning and closes the test gap.","Warm-starting generalization gaps can be closed regardless of the initial data fraction, as long as parameter norms stay fixed and the re-warmed effective learning rate is large enough.","In high-update-to-data reinforcement learning, early learning-rate cycling followed by annealing beats both constant low learning rates and parameter resets, though high-learning-rate phases must be transient to avoid instability.","Feature learning alone is not sufficient: the loss landscape must also permit convergence to a generalizing solution, which is why scale decay on normalization layers is needed in some architectures."],"supporting_citations":[{"why":"Supplies the Normalize-and-Project method and the definition of effective learning rate that the re-warming algorithm modifies.","marker":"Lyle et al. 2024a"},{"why":"Introduced grokking on algorithmic datasets, the phenomenon the paper uses as the testbed for feature-learning dynamics.","marker":"Power et al. 2022"},{"why":"Defined the warm-starting generalization gap and the Shrink-and-Perturb baseline that effective-learning-rate re-warming is compared against.","marker":"Ash & Adams 2020"},{"why":"Introduced primacy bias in deep reinforcement learning, the nonstationary-learning phenomenon the paper aims to explain.","marker":"Nikishin et al. 2022"},{"why":"Supplies the CUSUM changepoint detector used to trigger adaptive learning-rate resets.","marker":"Page 1954"},{"why":"Defines the neural tangent kernel / lazy regime whose contrast with rich feature learning frames the paper's analysis.","marker":"Jacot et al. 2018"},{"why":"Supplies the feature-learning definition, a nonvanishing change in feature covariance, that the paper's metrics operationalize.","marker":"Yang et al. 2022"},{"why":"Provides the transformer architecture and training setup for the modular arithmetic grokking experiments.","marker":"Varma et al. 2023"},{"why":"Establishes that parameter-norm growth reduces effective step size, the link between norm growth and loss of plasticity.","marker":"Lyle et al. 2024b"},{"why":"Describes grokking as a transition from lazy to rich training dynamics and provides the gradient form the appendix's Gaussian approximation uses.","marker":"Xu et al. 2024"}],"fun_headline_variants":["Rewarm effective LR to overwrite stale features","Effective LR controls when networks can generalize","Feature learning triggered by re-warming LR","Grokking key: effective learning rate re-warm","Overcoming primacy bias with LR re-warming"],"cache_read_input_tokens":24832,"weakest_assumption_plain":"The load-bearing assumption is that the effective learning rate keeps governing feature-learning dynamics long after training has left initialization, so that raising it late can still force the network to overwrite old features.","fun_headline_variants_meta":{"raw":{"variants":["Rewarm effective LR to overwrite stale features","Effective LR controls when networks can generalize","Feature learning triggered by re-warming LR","Grokking key: effective learning rate re-warm","Overcoming primacy bias with LR re-warming"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000879,"raw_usage":{"total_tokens":3816,"prompt_tokens":979,"completion_tokens":2837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":2766}},"tokens_in":595,"tokens_out":2837,"duration_ms":18401,"temperature":1.0,"reasoning_tokens":2766,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:52:09.477728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a network on task A with fixed parameter norms and an annealed learning rate, switch to task B, and raise the effective learning rate: if activation-pattern and feature-covariance metrics do not jump while the generalization gap on task B closes, the causal claim fails. Equivalently, find any trained network in which the effective learning rate is still high but primacy bias persists; that would show the effective learning rate is not the only gate on feature overwriting.","supporting_citations":[],"review_version":1}