REVIEW 2 cited by
A Simplified Analysis of SGD for Linear Regression with Weight Averaging
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Theoretically understanding stochastic gradient descent (SGD) in overparameterized models has led to the development of several optimization algorithms that are widely used in practice today. Recent work by~\citet{zou2021benign} provides sharp rates for SGD optimization in linear regression using constant learning rate, both with and without tail iterate averaging, based on a bias-variance decomposition of the risk. In our work, we provide a simplified analysis recovering the same bias and variance bounds provided in~\citep{zou2021benign} based on simple linear algebra tools, bypassing the requirement to manipulate operators on positive semi-definite (PSD) matrices. We believe our work makes the analysis of SGD on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling, leading to improvements in the training of realistic models.
Forward citations
Cited by 2 Pith papers
-
Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
When a cosine schedule would halve the learning rate, Seesaw cuts it by √2 and doubles the batch, matching loss curves with ~36% fewer serial steps.
-
On the Effectiveness of the z-Transform Method in Quadratic Optimization
Using z-transforms and Tauberian theory, this paper derives sharp asymptotic rates, such as 1/k^omega for gradient descent and 1/k^{min(2omega, omega+1)} for Nesterov acceleration, for quadratic optimization in infini...
Discussion (0). Continue with ORCID to comment.