REVIEW 1 cited by
Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper studies emergent phenomena in neural networks by focusing on grokking where models suddenly generalize after delayed memorization. To understand this phase transition, we utilize higher-order mutual information to analyze the collective behavior (synergy) and shared properties (redundancy) between neurons during training. We identify distinct phases before grokking allowing us to anticipate when it occurs. We attribute grokking to an emergent phase transition caused by the synergistic interactions between neurons as a whole. We show that weight decay and weight initialization can enhance the emergent phase.
Forward citations
Cited by 1 Pith paper
-
Emergent Generalization by Representation Learning in Artificial Neural Networks
An explicit low-dimensional bottleneck is necessary for OOD generalisation in reservoir networks, and the non-monotonic rise of causal emergence in the latent code predicts generalisation both in silico and in mouse CA1.
Discussion (0). Continue with ORCID to comment.