pith. sign in

arxiv: 2404.17563 · v3 · pith:I6ECRPKTnew · submitted 2024-04-26 · 💻 cs.LG · cond-mat.dis-nn· stat.ML

An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem

Yoonsoo Nam , Nayara Fonseca , Seok Hyeong Lee , Chris Mingard , Ard A. Louis This is my paper
classification 💻 cs.LG cond-mat.dis-nnstat.ML
keywords modelsizeemergencetrainingdatatimeabilityincreases
0
0 comments X
read the original abstract

Deep learning models can exhibit what appears to be a sudden ability to solve a new problem as training time, training data, or model size increases, a phenomenon known as emergence. In this paper, we present a framework where each new ability (a skill) is represented as a basis function. We solve a simple multi-linear model in this skill-basis, finding analytic expressions for the emergence of new skills, as well as for scaling laws of the loss with training time, data size, model size, and optimal compute. We compare our detailed calculations to direct simulations of a two-layer neural network trained on multitask sparse parity, where the tasks in the dataset are distributed according to a power-law. Our simple model captures, using a single fit parameter, the sigmoidal emergence of multiple new skills as training time, data size or model size increases in the neural network.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent

    stat.ML 2026-05 unverdicted novelty 7.0

    In the linear-width regime, the second GD step yields a spiked random matrix whose number of outliers is floor(alpha2 / (1/2 - alpha1)), and batch reuse enables learning directions with information exponent greater th...

  2. Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent

    stat.ML 2026-05 unverdicted novelty 7.0

    Two steps of gradient descent on first-layer weights in linear-width two-layer networks produce a spiked random matrix with floor(alpha2/(1/2-alpha1)) outliers, each a learned direction, and batch reuse allows capturi...

  3. Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics

    cs.LG 2026-05 conditional novelty 6.0

    Weight decay controls distinct learning regimes in grokking transformers on modular arithmetic, tracked by new cheap attention-based diagnostics with empirical critical value and exponent fits.