REVIEW 4 cited by
On Regularization via Early Stopping for Least Squares Regression
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A fundamental problem in machine learning is understanding the effect of early stopping on the parameters obtained and the generalization capabilities of the model. Even for linear models, the effect is not fully understood for arbitrary learning rates and data. In this paper, we analyze the dynamics of discrete full batch gradient descent for linear regression. With minimal distributional assumptions, we characterize the trajectory of the parameters and the expected excess risk. Using this characterization, we show that when training with any learning rate schedule and finite time horizon, the early stopped solution is equivalent to the minimum norm solution for a generalized ridge regression problem. We also prove that early stopping is beneficial for generic data with arbitrary spectrum and for a wide variety of learning rate schedules. We provide an estimate for the optimal stopping time and empirically demonstrate the accuracy of our estimate.
Forward citations
Cited by 4 Pith papers
-
Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent
Early-stopped negative-shifted gradient descent beats every stable negative-ridge endpoint and every positive shrinker in gapped high-dimensional linear models, by polynomial risk factors.
-
Learning Curves of Stochastic Gradient Descent in Kernel Regression
Single-pass SGD with exponentially decaying steps is claimed to reach minimax-optimal excess risk in high-dimensional kernel regression for well-specified problems, with averaging handling misspecified problems.
-
Models of Heavy-Tailed Mechanistic Universality
A new random matrix model with one structure parameter explains heavy-tailed spectra in trained networks, and yields scaling laws, optimizer-tail behavior, and a description of the five-plus-one phases of training.
-
Design Considerations in Offline Preference-based RL
A unified theory of offline RLHF shows that loss curvature and data coverage control suboptimality, explaining why squared-loss IPO is more stable than logistic-loss DPO.
Discussion (0). Continue with ORCID to comment.