REVIEW 8 cited by
A Constructive Prediction of the Generalization Error Across Scales
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data.
Forward citations
Cited by 8 Pith papers
-
In situ fine-tuning of in silico trained Optical Neural Networks
GIFT computes a gradient-informed direction based on how the training loss gradient changes with the assumed noise level and line-searches along it in situ, improving accuracy under noise misspecification.
-
A Theory of Inference Compute Scaling: Reasoning through Directed Stochastic Skill Search
A skill-graph random-walk model gives closed-form accuracy-versus-compute formulas for four reasoning strategies and connects them to training scaling.
-
Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
A Prior-data Fitted Network with a scaling-law-specific prior gives better point and uncertainty predictions for neural scaling law extrapolation than MCMC, BNSL, and LC-PFN baselines.
-
Scaling Pre-training to One Hundred Billion Data for Vision Language Models
Scaling VLM pretraining from 10B to 100B image-text pairs yields saturation on standard benchmarks but large gains on cultural diversity, low-resource language retrieval, and subgroup disparity.
-
Lightweight Dataset Pruning without Full Training via Example Difficulty and Prediction Uncertainty
A lightweight score combining prediction mean and variance, plus ratio-adaptive Beta sampling, prunes datasets early in training and reaches 60% ImageNet accuracy at 90% pruning.
-
Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification
Classification accuracy on inertial HAR and SLR tasks follows a consistent logarithmic growth with training-set size, enabling a MAPD-based stability-point metric that often saturates far below traditional heuristics.
-
Unifying Learning Dynamics and Generalization in Transformers Scaling Law
Claims a two-stage transformer scaling law (exponential then C^{-1/6}) with matching bounds, but the lower bounds are missing, the exponent is inconsistent (-1/7 vs -1/6), and the law is an artifact of hand-set M = Θ(...
-
Beyond Scaling Curves: Internal Dynamics of Neural Networks Through the NTK Lens
Using NTK trace and effective rank, this paper shows that model and data scaling improve test loss at similar rates but drive internal dynamics in opposite directions, and estimates a feature-learning width limit well...
Discussion (0). Continue with ORCID to comment.