Pith. sign in

REVIEW 3 cited by

Unifying Grokking and Double Descent

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.06173 v1 pith:DM4YMEDJ submitted 2023-03-10 cs.LG cs.AI

Xander Davies , Lauro Langosco , David Krueger This is my paper · ORCID
classification cs.LGcs.AI
keywords grokkingdescentdoubleframeworklearningemphgeneralizationperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

A principled understanding of generalization in deep learning may require unifying disparate observations under a single conceptual framework. Previous work has studied \emph{grokking}, a training dynamic in which a sustained period of near-perfect training performance and near-chance test performance is eventually followed by generalization, as well as the superficially similar \emph{double descent}. These topics have so far been studied in isolation. We hypothesize that grokking and double descent can be understood as instances of the same learning dynamics within a framework of pattern learning speeds. We propose that this framework also applies when varying model capacity instead of optimization steps, and provide the first demonstration of model-wise grokking.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decomposing Prediction Mechanisms for In-Context Recall

    cs.LG 2025-07 conditional novelty 7.0 of 10

    In a toy in-context recall task, label-based task initiation and observation-based continuation are distinct mechanisms with separate emergence times, and the same first-token versus second-token gap appears in an OLM...

  2. A Spin Glass Characterization of Neural Networks

    cond-mat.dis-nn 2025-08 unverdicted novelty 6.0 of 10

    A Hopfield-type spin glass constructed from a feedforward network yields replica overlap statistics that serve as a per-instance descriptor of the network's generalization, capacity, and robustness.

  3. Mechanistic Insights into Grokking from the Embedding Layer

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Trainable embeddings in a simple MLP cause delayed generalization (grokking) on modular arithmetic, and a higher embedding learning rate plus balanced sampling accelerates it.

Pith tools