Pith. sign in

REVIEW 16 cited by

Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.05192 v3 pith:IFVTDPZT submitted 2024-10-07 cs.LG cs.CLstat.ML

Kaiyue Wen , Zhiyuan Li , Jason Wang , David Hall , Percy Liang , Tengyu Ma This is my paper
classification cs.LGcs.CLstat.ML
keywords learningphaserateriverbranchcomputeduringloss
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contrast, the Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can in principle continue indefinitely without a pre-specified compute budget. Then, given any compute budget, one can branch out from the main branch at a proper time with a rapidly decaying learning rate to produce a strong model. Empirically, WSD generates a non-traditional loss curve: the loss remains elevated during the stable phase but sharply declines during the decay phase. Towards explaining this phenomenon, we conjecture that pretraining loss exhibits a river valley landscape, which resembles a deep valley with a river at its bottom. Under this assumption, we show that during the stable phase, the iterate undergoes large oscillations due to the high learning rate, yet it progresses swiftly along the river. During the decay phase, the rapidly dropping learning rate minimizes the iterate's oscillations, moving it closer to the river and revealing true optimization progress. Therefore, the sustained high learning rate phase and fast decaying phase are responsible for progress in the river and the mountain directions respectively, and are both critical. Our analysis predicts phenomenons consistent with empirical observations and shows that this landscape can emerge from pretraining on a simple bi-gram dataset. Inspired by the theory, we introduce WSD-S, a variant of WSD that reuses previous checkpoints' decay phases and keeps only one main branch, where we resume from a decayed checkpoint. WSD-S empirically outperforms WSD and Cyclic-Cosine in obtaining multiple language model checkpoints across various compute budgets in a single run for parameters scaling from 0.1B to 1.2B.

Discussion (0). Sign in to comment.

Forward citations

Cited by 16 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Universal One-third Time Scaling in Learning Peaked Distributions

    cs.LG 2026-02 conditional novelty 7.0 of 10

    Softmax + cross-entropy on peaked targets yields loss ∼ t^{−1/3}, giving an architecture-driven explanation for power-law LLM training time without power-law data.

  2. Scaling Latent Reasoning via Looped Language Models

    cs.CL 2025-10 unverdicted novelty 7.0 of 10

    Looped language models with latent iterative computation and entropy-regularized depth allocation achieve performance matching up to 12B standard LLMs through superior knowledge manipulation.

  3. World Modeling with Probabilistic Structure Integration

    cs.CV 2025-09 conditional novelty 7.0 of 10

    A single probabilistic video model extracts optical flow, depth, and segments via counterfactual prompts, then integrates those structures as new token types to improve its own video predictions.

  4. Theoretical Analysis on how Learning Rate Warmup Accelerates Convergence

    cs.LG 2025-09 conditional novelty 7.0 of 10

    Under a new smoothness assumption linking local curvature to the loss gap, increasing learning rates provably accelerate GD and SGD convergence, with up to Theta(T) speedup in special cases.

  5. Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks

    cs.LG 2025-07 conditional novelty 7.0 of 10

    Compute-optimally trained networks of different sizes show loss curves that collapse onto one universal curve after normalization; with learning rate decay, the collapse is tighter than seed-to-seed noise, providing a...

  6. AGNFormer I: Reconstruction of AGN spectra using a probabilistic transformer model

    astro-ph.GA 2026-07 conditional novelty 6.0 of 10

    An uncertainty-aware transformer reconstructs masked AGN broad lines and spectral halves with 4-16% flux errors and beats eleven purpose-built Lyα-reconstruction algorithms on a blind benchmark.

  7. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused ...

  8. NITP: Next Implicit Token Prediction for LLM Pre-training

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    NITP augments standard next-token prediction with implicit semantic prediction in representation space using shallow-layer self-supervision, reporting consistent downstream gains on 0.5B-9B models including 5.7% on MM...

  9. ByteSpan: Information-Driven Subword Tokenisation

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A tokeniser that groups predictable bytes using a byte-level LM's surprisal or entropy achieves higher morphological alignment than BPE without loss of compression.

  10. BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A continued-pretrained ModernBERT encoder for biomedical and clinical text claims SOTA on several clinical NLP tasks, with caveats about data overlap between pretraining and evaluation.

  11. Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Loss deceleration, a piecewise-linear break in log-log loss curves, is attributed to zero-sum learning where per-example gradients oppose one another, and scaling helps by mitigating it.

  12. Recipes for Pre-training LLMs with MXFP8

    cs.LG 2025-05 conditional novelty 6.0 of 10

    MXFP8-E4M3 with round-up scale computation matches BF16 pre-training accuracy for LLMs up to 8B parameters, enabling 2x faster training on Blackwell.

  13. Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Weight decay on scale-invariant weights creates a norm-dependent sharpness boundary; crossing it predicts loss spikes in normalized networks.

  14. Why Do We Need Warm-up? A Theoretical Perspective

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.

  15. WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Checkpoint merging during constant-LR training can replace LR decay and yields improved LLM benchmark scores over Warmup-Stable-Decay.

  16. Mpemba Effect in Large-Language Model Training Dynamics: A Minimal Analysis of the Valley-River model

    cs.AI 2025-07 reject novelty 5.0 of 10

    The authors derive conditions under which a higher plateau learning rate accelerates LLM training convergence via the Mpemba effect, but the central derivative formula is incorrect.

Pith tools