Pith. sign in

REVIEW 7 cited by

Broken Neural Scaling Laws

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2210.14891 v17 pith:SF5W7II5 submitted 2022-10-26 cs.LG cs.AI

Ethan Caballero , Kshitij Gupta , Irina Rish , David Krueger This is my paper · ORCID
classification cs.LGcs.AI
keywords scalinglearningfunctionalbehaviorneuralbrokenformtasks
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present a smoothly broken power law functional form (that we refer to as a Broken Neural Scaling Law (BNSL)) that accurately models & extrapolates the scaling behaviors of deep neural networks (i.e. how the evaluation metric of interest varies as amount of compute used for training (or inference), number of model parameters, training dataset size, model input size, number of training steps, or upstream performance varies) for various architectures & for each of various tasks within a large & diverse set of upstream & downstream tasks, in zero-shot, prompted, & finetuned settings. This set includes large-scale vision, language, audio, video, diffusion, generative modeling, multimodal learning, contrastive learning, AI alignment, AI capabilities, robotics, out-of-distribution (OOD) generalization, continual learning, transfer learning, uncertainty estimation / calibration, OOD detection, adversarial robustness, distillation, sparsity, retrieval, quantization, pruning, fairness, molecules, computer programming/coding, math word problems, "emergent phase transitions", arithmetic, supervised learning, unsupervised/self-supervised learning, & reinforcement learning (single agent & multi-agent). When compared to other functional forms for neural scaling, this functional form yields extrapolations of scaling behavior that are considerably more accurate on this set. Moreover, this functional form accurately models & extrapolates scaling behavior that other functional forms are incapable of expressing such as the nonmonotonic transitions present in the scaling behavior of phenomena such as double descent & the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic. Lastly, we use this functional form to glean insights about the limit of the predictability of scaling behavior. Code is available at https://github.com/ethancaballero/broken_neural_scaling_laws

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 15 citations worldwide. Full citation record

  1. Reliability Scaling Laws for Quantized Large Language Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Reliability of quantized LLMs peaks nonlinearly at 4-bit precision under fixed total model bits, while accuracy scales monotonically, and quantization can improve robustness to natural perturbations.

  2. Information-Theoretic Limits of Reliability and Scaling in Language Models

    cs.CL 2026-05 conditional novelty 6.0 of 10

    A theoretical framework derives a reliability ceiling and a max-form Chinchilla-type scaling law for LLMs from task entropy and dependency spectra.

  3. A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.

  4. Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Loss deceleration, a piecewise-linear break in log-log loss curves, is attributed to zero-sum learning where per-example gradients oppose one another, and scaling helps by mitigating it.

  5. Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A Prior-data Fitted Network with a scaling-law-specific prior gives better point and uncertainty predictions for neural scaling law extrapolation than MCMC, BNSL, and LC-PFN baselines.

  6. Position: Stop Reactively Patching Your Model Every Time and Start Proactive Test-Driven AI Development

    cs.LG 2026-07 conditional novelty 5.0 of 10

    In a stylized model, a proactive flywheel that fixes whole groups of related scenarios needs Θ(K log K) update rounds versus Θ(M log M) for reactive patching.

  7. X-Factor: Quality Is a Dataset-Intrinsic Property

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Across 2,500 class-balanced MNIST subsets and 10 model architectures, test-error Z-scores correlate strongly across models (mean R2=0.82 excluding GNB), supporting dataset quality as an intrinsic property.

Pith tools