Pith. sign in

REVIEW 5 cited by

The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.18965 v2 pith:OUZ7MZ3Y submitted 2025-01-31 cs.LG math.OCstat.ML

Fabian Schaipp , Alexander Hägele , Adrien Taylor , Umut Simsekli , Francis Bach This is my paper · ORCID
classification cs.LGmath.OCstat.ML
keywords learning-ratetrainingboundoptimizationtheoryconvexcooldownlarge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedule with linear cooldown; in particular, the practical benefit of cooldown is reflected in the bound due to the absence of logarithmic terms. Further, we show that this surprisingly close match between optimization theory and practice can be exploited for learning-rate tuning: we achieve noticeable improvements for training 124M and 210M Llama-type models by (i) extending the schedule for continued training with optimal learning-rate, and (ii) transferring the optimal learning-rate across schedules.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Defense of the Quadratic Model

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Local Taylor-expanded quadratic models reproduce a 150M-parameter LLM's validation loss for up to 10% of training late in the run, and LLM pretraining operates within a factor of 2 of a stochastic or deterministic edg...

  2. Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

    cs.LG 2025-07 conditional novelty 7.0 of 10

    Compute-optimally trained networks of different sizes show loss curves that collapse onto one universal curve after normalization; with learning rate decay, the collapse is tighter than seed-to-seed noise, providing a...

  3. AGNFormer I: Reconstruction of AGN spectra using a probabilistic transformer model

    astro-ph.GA 2026-07 conditional novelty 6.0 of 10

    An uncertainty-aware transformer reconstructs masked AGN broad lines and spectral halves with 4-16% flux errors and beats eleven purpose-built Lyα-reconstruction algorithms on a blind benchmark.

  4. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused ...

  5. Why Do We Need Warm-up? A Theoretical Perspective

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.

Pith tools