Pith. sign in

REVIEW 4 major objections 5 minor 29 references

Two transformers with identical 100% test accuracy part ways under noise: the high-entropy one keeps roughly 95% of the original task, while the conventionally trained one falls to about 75%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:39 UTC pith:VGTJBZDP

load-bearing objection Large robustness gap between WLMD and AdamW in grokking, but the missing loss-matched control leaves the entropy explanation unproven. the 4 major comments →

arxiv 2607.29503 v1 pith:VGTJBZDP submitted 2026-07-31 cs.LG

The Grokked Illusion: True Equilibrium Mitigates Catastrophic Forgetting

Xiaotian Zhang , Lai Shun Chan , Yue Shang , Entao Yang , Ge Zhang This is my paper
classification cs.LG MSC 68T0782C32
keywords grokkingcatastrophic forgettingBoltzmann entropyeffective rankrobustnessmodular arithmetichigh-entropy advantagecontinual learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a model that generalizes perfectly is also robust when forced to memorize new, meaningless data. In a controlled modular-arithmetic task, it compares transformers trained two ways: conventional gradient-based optimization and high-entropy sampling from the Boltzmann entropy landscape. Despite identical 100% test accuracy on the original task, the conventionally trained model's accuracy drops to about 75% after memorizing random labels, while the high-entropy model keeps about 95%. The paper calls this gap the 'grokked illusion' and attributes it to the high-entropy model's broader effective rank in attention and MLP layers, which buffers the disruption of new learning. If right, the result separates generalization from memory robustness and points to entropy-based training as a target for continual learning.

Core claim

The central discovery is that two neural networks with the same perfect test accuracy on a task can have sharply different resilience to later parameter updates. In grokking on modular arithmetic, a transformer trained by gradient descent (AdamW) memorizes injected random-label noise and, in doing so, loses about 25 points of original-task test accuracy; a transformer sampled from a high-entropy equilibrium state of the Boltzmann entropy landscape, with the same 100% test accuracy, loses only about 5 points. The paper names this divergence the 'grokked illusion': from the outside both models have 'grokked' the task, but their internal representations are not equally robust. Singular value de

What carries the argument

The paper's load-bearing objects are (1) the Boltzmann entropy of a neural network state, defined as the log-volume of parameter space occupied by configurations with a given training loss and test accuracy, and (2) the effective rank of a weight matrix, ER = exp(−Σ σ̃ᵢ log σ̃ᵢ) with normalized singular values σ̃ᵢ, which quantifies how many significant independent directions the layer uses. The experiment constructs two matched models: one sampled from the high-entropy equilibrium of the entropy landscape via molecular dynamics, and one produced by standard gradient-descent optimization, both with weight norm fixed and both at 100% test accuracy. It then forces each to memorize random or str

Load-bearing premise

The paper attributes the robustness gap to Boltzmann entropy, but the high-entropy model was optimized to a higher training loss (ln L ≈ −1) than the AdamW model (ln L ≈ −2), so the two are not matched on optimization depth; if less-complete optimization is the true cause of retention, the entropy explanation fails.

What would settle it

Train a conventional AdamW model until its training loss reaches ln L ≈ −1 (the equilibrium model's loss) and run the same random-noise injection; if it also retains ~95% test accuracy on the original task, the entropy explanation is confounded. Alternatively, take the AdamW model and artificially raise its weight matrices' effective rank (e.g., by spectral regularization) without changing its entropy landscape, and check whether its retention improves.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the high-entropy robustness advantage is real, evaluation of trained models should include interference or adaptation tests, not just static test accuracy.
  • Continual learning could be reframed as moving models into high-entropy regions of parameter space rather than adding regularization to standard optimizers.
  • The effective-rank buffer suggests that spectral diversity of weight matrices is a measurable indicator of a model's resistance to forgetting.
  • Since the robustness gap narrows when the new data is structurally similar to the original task, the effect is specific to interference from unrelated information, providing a taxonomy of forgetting.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to vary effective rank directly—for instance by spectral regularization or low-rank factorization—while keeping entropy roughly fixed, to see whether the buffer effect is causal or merely correlated.
  • The confounded comparison (higher training loss in the equilibrium state) leaves open the possibility that less complete optimization, not entropy, confers robustness; matching optimization depth would settle this.
  • If the entropy/robustness link scales to large language models, post-training and fine-tuning pipelines might select high-entropy checkpoints to preserve pretrained capabilities during instruction tuning.
  • The finding suggests a new diagnostic: the cosine displacement during noise injection is larger for the high-entropy model, yet its performance holds—so robustness is not about staying close to the original parameters but about sitting in a wide basin.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper asks whether high-entropy (WLMD-equilibrium) transformers are more robust to catastrophic forgetting than AdamW-trained transformers that achieve identical perfect test accuracy. Using modular arithmetic (x+y mod 67) as a controlled setting, the authors compare a single WLMD-equilibrium state selected at ln(L_train)≈−1 with an AdamW-goldilocks state trained to ln(L_train)≈−2. After expanding the vocabulary and fine-tuning on a mixed dataset containing the original training data plus 500 noise samples of three types (random, x^2+y mod 37, x+y mod 37), they report that AdamW-goldilocks original-task test accuracy drops to about 75%, while WLMD retains about 95%. They attribute this robustness gap to the higher Boltzmann entropy of the WLMD state and to higher effective rank in attention and MLP layers, which they interpret as a representational buffer against parameter displacement.

Significance. If the robustness advantage of high-entropy states is confirmed, it would meaningfully extend the high-entropy advantage from generalization to robustness and motivate entropy-guided strategies for continual learning. The experimental setting is well chosen: the modular arithmetic task is controlled, both models are constrained to the same weight norm, and both reach 100% test accuracy before noise injection. The noise-injection protocol is a clear, falsifiable stress test. However, the central comparison is currently confounded by a training-loss mismatch, and the reliance on a single WLMD checkpoint undermines the generality of the claim. The effective-rank analysis, while suggestive, is correlational and would need interventions to support a causal interpretation.

major comments (4)
  1. [Methodology / Experimental Results] The WLMD-equilibrium state is explicitly selected at ln(L_train)≈−1 (Training Protocols), whereas the AdamW-goldilocks model is trained to ln(L_train)≈−2 (same section). Yet the Results section states that the two achieve 'similar training loss' (first paragraph of Experimental Results). This is an internal contradiction and a load-bearing confound: a model with higher training loss is less fully optimized and may have more parameter slack to absorb noise without disrupting the original computation. An AdamW control trained to the same loss (e.g., via early stopping or adjusted learning rate) is required to isolate the effect of entropy from optimization depth. Without such a control, the robustness advantage cannot be attributed to Boltzmann entropy.
  2. [Methodology / Table 1] Only a single WLMD checkpoint is used. Pre-injection effective-rank values in Table 1 are reported as point values with no uncertainty, while post-injection values are averaged over 10 seeds. It is therefore unclear whether the observed robustness (and the very high ER values, e.g., WQ=96.8 vs 2.2) is typical of equilibrium states or an artifact of one selected configuration. The paper should sample multiple independent WLMD states, or report the variance across WLMD runs, to demonstrate that the phenomenon is a property of the equilibrium ensemble rather than of a particular draw.
  3. [Experimental Results / Figure 1] Ten seeds are used for noise injection, but the final original-task test accuracies are only described qualitatively ('about 75%' and 'approximately 95%'). No exact means, standard deviations, or significance tests are reported. Given the modest effect size in the structured-noise conditions and the overlap suggested by the trajectory plots, the paper should report mean±std and perform a two-sample t-test or bootstrap to establish that the robustness gap is statistically significant. Additionally, the y-axis in Fig. 1(b)–(d) extends above 1.0, which is misleading for a bounded accuracy metric; clip or explain the averaging overshoot.
  4. [Singular Value Analysis / Table 1] The effective-rank analysis is correlational and based on the same single pre-injection checkpoint. The paper concludes that 'richer feature representations can serve as a buffer against catastrophic forgetting' (Abstract) and states in Conclusions that the authors 'revealed' this mechanism, but the SVD evidence cannot distinguish whether high ER is a cause or a correlate. The Future Directions section correctly acknowledges the need for causal intervention; the main text should either temper the causal language or include an intervention experiment (e.g., rank-constrained fine-tuning or explicitly increased ER) to support the claim.
minor comments (5)
  1. [Methodology] There are formatting errors from LaTeX, e.g., 'withinthelowtraininglossrangeofinterestconverges correctly' should have spaces. Please proofread.
  2. [Figure 1] The shaded regions and mean trajectories would benefit from a legend that explicitly distinguishes 'AdamW' (blue) and 'WLMD' (red) in subplots (b)–(d); currently the legend is only in (a).
  3. [Table 1] Pre-injection ER columns should be marked as single-checkpoint values (e.g., with a footnote) to avoid appearing as mean±std across seeds.
  4. [Abstract / Introduction] The term 'grokked illusion' is introduced but the word 'grokked' may confuse readers; define it at first use as 'a model that appears to have fully learned (grokked) the task yet remains fragile to subsequent interference.'
  5. [Experimental Results] The paper should report the exact final test accuracies for all three noise types, not just qualitative descriptions, to support the claim of a monotonic trend with structural similarity.

Circularity Check

0 steps flagged

No circularity: the robustness comparison is measured, not derived from the cited entropy theory; the training-loss mismatch is a confound but not a circular step.

full rationale

I walked the derivation chain and found no circular step. The paper's central claim is an empirical comparison: WLMD-equilibrium NNs (sampled at ln(L_train) ≈ −1) and AdamW-goldilocks NNs (trained to ln(L_train) ≈ −2) are both taken to 100% original-task test accuracy, then subjected to noise injection, and retention is measured. The robustness numbers are measured outcomes, not formal consequences of the cited entropy landscape. The effective-rank analysis is likewise a post-hoc measurement. No parameter is fitted to the retention outcome and then renamed as a prediction, and no uniqueness theorem is imported to force the choice of explanation. The self-citations (Yang et al. 2026b; Zhang et al. 2026) are used to justify the WLMD sampling method and the claim that equilibrium states can reach perfect test accuracy, but the present paper independently selects and evaluates its checkpoints, so the new robustness observation does not reduce to those citations. The one substantive issue is a control mismatch, not circularity: Methodology states the WLMD state has ln(L_train) ≈ −1 and the AdamW state has ln(L_train) ≈ −2, while Results asserts both achieve 'similar training loss under identical weight norm constraints' — an internal contradiction. This confounds the entropy attribution (a less-optimized WLMD state may simply have more slack), but it does not make the retention result equivalent to the inputs by construction. I therefore score 2 for minor, non-load-bearing self-citation rather than for circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim is empirical; no new parameters are fit to the outcome. The main free parameters are experimental hyperparameters chosen from prior work. The key domain assumptions are the validity of the entropy landscape and WLMD sampling from earlier papers by the same group.

free parameters (4)
  • Weight norm ||w|| = 30
    Chosen based on prior 'Goldilocks zone' (Liu et al. 2022) to ensure both models achieve perfect test accuracy; not fitted to the robustness result.
  • Entropy sampling range = ln(L) ∈ [-20, 1]
    Used in WLMD to constrain sampling; chosen from prior work to converge correctly.
  • Fine-tuning learning rate = 1e-4
    Hand-chosen; not fitted to the claim.
  • Memorization threshold = >99.8% train accuracy
    Criterion to define 'fully memorized'; arbitrary but consistent across models.
axioms (4)
  • domain assumption Boltzmann entropy of NN state is well-defined and computable via WLMD
    Adopted from Yang et al. 2026b; the paper relies on this to classify high-entropy states.
  • ad hoc to paper The selected WLMD state is a maximum-entropy state for its loss
    The paper selects one state and calls it equilibrium, but does not prove it is at the entropy peak.
  • domain assumption The high-entropy advantage in generalization holds
    Used to motivate the hypothesis; not re-derived here.
  • standard math Effective rank measures representation richness
    Standard definition from Roy & Vetterli; used to connect entropy to mechanism.

pith-pipeline@v1.3.0-daily-deepseek · 178 in / 10142 out tokens · 132434 ms · 2026-08-03T05:39:09.767487+00:00 · methodology

0 comments
read the original abstract

While neural networks are typically evaluated by their training and test performance, these metrics do not reveal how robust a learned representation is. Recent studies have shown that solutions occupying larger volumes in parameter space, as quantified by Boltzmann entropy, often exhibit superior generalizability compared to those reached by conventional optimization, a phenomenon known as the high entropy advantage. Here we ask whether this advantage persists beyond generalization. Specifically, we investigate models' robustness, the ability to retain the learned knowledge when the model is subsequently trained to acquire new information. Using grokking in modular arithmetic as a controlled setting, we design a noise injection experiment to evaluate the robustness difference between AdamW-trained transformers and high-entropy model sampled from Wang-Landau Molecular Dynamics with identical saturated performance. By forcing both models to fully remember new data with random labels, we find that AdamW-trained models suffer from catastrophic forgetting, with original task test accuracy dropping from 100% to below 75%, whereas the high-entropy models maintain approximately 95% test accuracy. We term this hidden fragility behind apparent generalization the "grokked illusion." Through singular value decomposition of the neural network weights, we discover that high-entropy neural networks possess significantly higher effective rank in attention and MLP layers both before and after noise injection, indicating richer feature representations can serve as a buffer against catastrophic forgetting. Our findings demonstrate that perfect generalization does not imply equal robustness, offering a new perspective on what makes a trained model robust to interference.

Figures

Figures reproduced from arXiv: 2607.29503 by Entao Yang, Ge Zhang, Lai Shun Chan, Xiaotian Zhang, Yue Shang.

Figure 1
Figure 1. Figure 1: Training dynamics and robustness trajectories. (a) Noise training accuracy versus epoch (log scale) for AdamW [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 4 linked inside Pith

  1. [1]

    International Conference on Learning Representations , year=

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author=. International Conference on Learning Representations , year=

  2. [2]

    The Thirteenth International Conference on Learning Representations (ICLR) , year =

    Theory on Mixture-of-Experts in Continual Learning , author =. The Thirteenth International Conference on Learning Representations (ICLR) , year =

  3. [3]

    arXiv:2201.02177

    Grokking: Generalization beyond overfitting on small algorithmic datasets , author=. arXiv:2201.02177

    Pith/arXiv arXiv

  4. [4]

    International Conference on Machine Learning , pages=

    An Investigation into Neural Net Optimization via Hessian Eigenvalue Density , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  5. [5]

    Physical Review Letters , volume =

    Efficient, Multiple-Range Random Walk Algorithm to Calculate the Density of States , author =. Physical Review Letters , volume =. 2001 , month =. doi:10.1103/PhysRevLett.86.2050 , url =

    doi:10.1103/physrevlett.86.2050 2001

  6. [6]

    The Eleventh International Conference on Learning Representations (ICLR) , year =

    Progress measures for grokking via mechanistic interpretability , author =. The Eleventh International Conference on Learning Representations (ICLR) , year =

  7. [7]

    npj Artificial Intelligence , volume=

    High-entropy advantage in neural networks' generalizability , author=. npj Artificial Intelligence , volume=. 2026 , publisher=

    2026

  8. [8]

    Advances in Neural Information Processing Systems , volume=

    Is Grokking a Computational Glass Relaxation? , author=. Advances in Neural Information Processing Systems , volume=

  9. [9]

    The Eleventh International Conference on Learning Representations (ICLR) , year =

    Omnigrok: Grokking Beyond Algorithmic Data , author =. The Eleventh International Conference on Learning Representations (ICLR) , year =

  10. [10]

    2007 15th European signal processing conference , pages=

    The effective rank: A measure of effective dimensionality , author=. 2007 15th European signal processing conference , pages=. 2007 , organization=

    2007

  11. [11]

    Advances in Neural Information Processing Systems , volume =

    Towards Understanding Grokking: An Effective Theory of Representation Learning , author =. Advances in Neural Information Processing Systems , volume =

  12. [12]

    3rd International Conference on Learning Representations (ICLR) , year =

    Explaining and Harnessing Adversarial Examples , author =. 3rd International Conference on Learning Representations (ICLR) , year =

  13. [13]

    Psychology of learning and motivation , volume=

    Catastrophic interference in connectionist networks: The sequential learning problem , author=. Psychology of learning and motivation , volume=. 1989 , publisher=

    1989

  14. [14]

    Neural computation , volume=

    Flat minima , author=. Neural computation , volume=. 1997 , publisher=

    1997

  15. [15]

    International Conference on Machine Learning , pages=

    Sharp minima can generalize for deep nets , author=. International Conference on Machine Learning , pages=. 2017 , organization=

    2017

  16. [16]

    5th International Conference on Learning Representations (ICLR) , year =

    Understanding deep learning requires rethinking generalization , author =. 5th International Conference on Learning Representations (ICLR) , year =

  17. [17]

    Journal of Statistical Mechanics: Theory and Experiment , volume=

    Entropy-sgd: Biasing gradient descent into wide valleys , author=. Journal of Statistical Mechanics: Theory and Experiment , volume=. 2019 , publisher=

    2019

  18. [18]

    Annual review of condensed matter physics , volume=

    Statistical mechanics of deep learning , author=. Annual review of condensed matter physics , volume=. 2020 , publisher=

    2020

  19. [19]

    arXiv:2103.10427

    The low-rank simplicity bias in deep networks , author=. arXiv:2103.10427

    Pith/arXiv arXiv

  20. [20]

    arXiv:1901.10371

    On the effect of low-rank weights on adversarial robustness of neural networks , author=. arXiv:1901.10371

    Pith/arXiv arXiv 1901

  21. [21]

    Proceedings of the national academy of sciences , volume=

    Overcoming catastrophic forgetting in neural networks , author=. Proceedings of the national academy of sciences , volume=. 2017 , publisher=

    2017

  22. [22]

    International conference on machine learning , pages=

    Continual learning through synaptic intelligence , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  23. [23]

    arXiv:2603.02224

    Subspace geometry governs catastrophic forgetting in low-rank adaptation , author=. arXiv:2603.02224

    arXiv

  24. [24]

    and Peng, Hao , booktitle =

    Du, Yufeng and Tian, Minyang and Ronanki, Srikanth and Rongali, Subendhu and Bodapati, Sravan Babu and Galstyan, Aram and Wells, Azton and Schwartz, Roy and Huerta, Eliu A. and Peng, Hao , booktitle =. Context Length Alone Hurts

  25. [25]

    The Fourteenth International Conference on Learning Representations (ICLR) , year =

    Learning on the Job: An Experience-Driven Self-Evolving Agent for Long-Horizon Tasks , author =. The Fourteenth International Conference on Learning Representations (ICLR) , year =

  26. [26]

    arXiv:2508.14704

    Mcp-universe: Benchmarking large language models with real-world model context protocol servers , author=. arXiv:2508.14704

    Pith/arXiv arXiv

  27. [27]

    Proceedings of the 43rd International Conference on Machine Learning (ICML) , year =

    UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon Scenarios , author =. Proceedings of the 43rd International Conference on Machine Learning (ICML) , year =

  28. [28]

    Proceedings of the 43rd International Conference on Machine Learning (ICML) , year =

    LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning , author =. Proceedings of the 43rd International Conference on Machine Learning (ICML) , year =

  29. [29]

    FutureX: An Advanced Live Benchmark for

    Zeng, Zhiyuan and Liu, Jiashuo and Chen, Siyuan and He, Tianci and Liao, Yali and Tian, Yixiao and Wang, Jinpeng and Wang, Zaiyuan and Yang, Yang and Yin, Lingyue and Yin, Mingren and Zhenwei, Zhu and Cai, Tianle and Chen, Xinjie and Chen, Zehui and Chen, Jiecao and Du, Yantao and Gao, Xiang and Guo, Jiacheng and Hu, Liang and Jiao, Jianpeng and Li, Xiang...

    2026