Pith. sign in

REVIEW 3 major objections 8 minor 1 cited by

Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The cooldown shape of the Warmup-Stable-Decay learning-rate scheduler sets up a bias-variance trade-off, and the sqrt and lowered-linear-0.7 shapes sit at its optimum.

desk verdict Solid empirical paper on WSD cooldown shape selection, with a beta2 claim that needs a replication pass before being taken at face value. read the letter →

arxiv 2508.01483 v1 pith:XGIFGUCB submitted 2025-08-02 cs.LG cs.AI

Aleksandr Dremov , Alexander Hägele , Atli Kosson , Martin Jaggi This is my paper · ORCID
classification cs.LGcs.AI
keywords learningrateschedulingwarmup-stable-decaycooldownshapebias-variancetrade-offAdamWbetatuningrivervalleylosslandscapevalidationperplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper isolates the cooldown phase of the Warmup-Stable-Decay (WSD) learning-rate scheduler — the short final annealing that produces the sharp loss drop in transformer training — and asks what makes one cooldown better than another. It argues that cooldown shapes create a bias-variance trade-off: aggressive shapes explore the loss surface widely and reach better solutions on average but with large run-to-run spread, while conservative shapes descend steadily into the current basin, giving consistent but worse final models. Measuring both terms against longer-trained reference models, the paper finds that the sqrt shape and the 'lowered linear' shape starting at 0.7 of the stable learning rate sit at the minimum of bias plus variance, which is why they beat cosine, square, and mirror-cosine cooldowns. It also reports that raising AdamW's $\beta_2$ during cooldown consistently improves validation perplexity by an amount comparable to choosing the right cooldown shape. If these results hold, WSD users should tune the cooldown shape and $\beta_2$ as seriously as any other hyperparameter.

What carries the argument

The load-bearing object is the bias-variance decomposition of Equation (1), which writes a cooldown shape's expected validation loss as $$\mathbb{E}[L(m_i) - L(m^*)] = \underbrace{L(\mathbb{E}[m_i]) - L(m^*)}_{\text{bias}} + \underbrace{\mathbb{E}[L(m_i)] - L(\mathbb{E}[m_i])}_{\text{variance}},$$ where $m_i$ is the model produced by data permutation $i$ and $m^*$ is the average of reference models trained much longer with the sqrt shape. Because both terms are measurable, every cooldown shape receives a point in a bias-variance plane, and the paper's recommendation — sqrt and lowered linear 0.7 — is the point nearest the bias+variance minimum line. The 'lowered linear' family, linear schedules that start at a fraction 0.1–1.0 of the stable learning rate and decay to zero, is the probe family that traces the trade-off curve.

What would settle it

Retrain the reference set with the square cooldown shape instead of sqrt, keeping the same 59,400 steps and data permutations, and re-plot Figure 6: if sqrt and lowered linear 0.7 no longer sit closest to the bias+variance minimum, the optimal-shape claim depends on the chosen reference rather than being a property of the shapes themselves. A second check is to rerun the $\beta_2$ sweep at a larger model scale and see whether validation perplexity still improves monotonically as $\beta_2$ approaches 1.

Watch

Extended reading notes

Core claim

The paper's central claim is that the cooldown stage of WSD training is governed by a bias-variance trade-off, and that the cooldown shape selects where on that trade-off the final model lands. For each shape, the authors train several models from the same pre-cooldown checkpoint on different data orderings and decompose the expected validation loss into a bias term — the gap between the weight-space average of those models and a reference model trained much longer on the same data — and a variance term — the gap between the average loss of individual runs and the loss of the averaged model. Empirically, shapes fall along a curve from high-variance, low-bias (mirror cosine, square) to low-variance, high-bias (lowered linear 0.1, cosine), and the sqrt shape and lowered linear 0.7 sit at the minimum of bias plus variance; the paper states there is nothing special about sqrt other than occupying this trade-off point. A second claim is that AdamW's $\beta_2$ during cooldown matters as much as shape selection: larger $\beta_2$ values consistently improve validation perplexity, and the optimal shape is unchanged across reasonable $\beta$ choices. Loss-landscape plots along the global optimization direction and the Adam-step direction show a river valley at cooldown start that narrows into a basin by the end, offered as empirical support for the river-valley view of WSD.

Load-bearing premise

The load-bearing premise is that each cooldown shape's bias can be measured against a single reference — the sqrt-cooldown model trained for 59,400 steps on the same data — because a different reference shape, duration, or data permutation would shift the bias values and could move which shapes sit at the bias-plus-variance minimum; the paper also notes in Section 4.2 that its recency-bias shift statistic is computed over the cooldown portion only, without a firm justification.

Editorial extensions

If this is right

  • Practitioners using WSD should adopt the sqrt or lowered-linear-0.7 cooldown, since cosine, square, and mirror-cosine shapes land measurably worse in final validation perplexity.
  • Raising AdamW's $\beta_2$ during cooldown is a nearly free performance gain of the same size as shape selection, and the optimal cooldown shape stays fixed across reasonable $\beta_2$ choices.
  • Cooldown-stage hyperparameters deserve the same tuning attention as stable-phase choices, because a short cooldown moves final perplexity as much as conventional hyperparameter changes do.
  • When the final model comes from averaging multiple runs, the ranking inverts: high-variance, low-bias shapes like mirror cosine average better than the trade-off-optimal shapes, although one longer run using the same compute beats all averages.
  • Cooldown shape controls recency bias: post-cooldown models show a U-shaped bias over recent training data instead of the pre-cooldown model's strong recency bias, so shape choice changes which data the model remembers best.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A scaling prediction follows that the paper does not test: as models grow and runs become single-shot, variance from data ordering should shrink relative to bias, which would push the optimal cooldown shape toward the high-variance end of the trade-off.
  • The $\beta_2$ result implies the optimizer's EMA memory is too short during cooldown, so an untested improvement would be annealing $\beta_2$ toward 1 over the cooldown itself, mirroring the learning-rate decay.
  • The river-valley visualization suggests a directly checkable mechanism: the angle between single-batch gradients and the global pre-cooldown-to-final direction should be near orthogonal at cooldown start and grow aligned as cooldown progresses, with the alignment curve depending on cooldown shape.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. The paper studies the cooldown (final decay) phase of the Warmup-Stable-Decay (WSD) learning rate scheduler for transformer language models. Using a 210M-parameter Llama-like decoder trained on SlimPajama, the authors compare cooldown LR shapes (linear, cosine, mirror cosine, square, sqrt, and lowered-linear variants) and show that shapes differ substantially in final validation perplexity. They introduce a bias-variance decomposition (Eq. 1) of the loss relative to a longer-trained reference model and report that sqrt and lowered-linear 0.7 achieve the best bias-variance balance, which corresponds to the lowest average validation loss across data permutations. The paper also analyzes recency bias toward recent training data, batch-size effects during cooldown, the influence of AdamW beta parameters, and model averaging, and it visualizes the loss landscape during cooldown to support the 'river valley' hypothesis. The shape-ranking claim is reproduced for a 60M-parameter model and on the fineweb-edu dataset in Appendix D.

Significance. The empirical shape-ranking result is the strongest contribution: it is directly measured, is reproduced at a smaller scale and on a second dataset, and yields actionable guidance (prefer sqrt or lowered-linear-0.7 cooldowns). The model-averaging experiment in Section 5 usefully shows that high-variance, low-bias shapes perform best when weights are averaged, and that a longer single run beats all averages. The bias-variance framework is transparently an algebraic identity, so its interpretive value is limited, but the underlying measurements are solid and the paper is honest about several exploratory choices (e.g., Section 4.2). The landscape visualizations provide qualitative support for the river-valley view. The beta_2 recommendation, however, is the least-supported load-bearing claim and requires additional replication and confound control before it can be stated as a general practical guideline.

major comments (3)
  1. [Section 6.2 / Abstract] The abstract and Section 9 state that 'consistent improvements with higher values of beta_2' are observed, but Section 6.2 reports a single sweep without the permutation-level replication and error bars that Section 6.1 provides for the batch-size study. Moreover, the beta_2 manipulation is entangled with the effective learning-rate trajectory: with beta_2 close to 1, the second-moment estimate is nearly frozen, so the AdamW step size changes over cooldown in a way that mimics a different cooldown shape. The text itself acknowledges (Section 6.2) that at small p 'the optimizer state is almost not updated' and 'requires higher learning rates.' Please provide per-permutation replicates (or at least standard errors) for the beta_2 sweep and, ideally, a control that keeps the effective step-size schedule matched across beta_2 values; if such controls are not possible, the unconditional abstract claim should be qualified to the studied setting only.
  2. [Section 10 / Appendix D] Section 10 and the conclusion say the main findings are validated across model sizes and datasets, citing Appendix D. However, Appendix D reproduces only the bias-variance plot for shape ranking (in loss- and weight-space); it does not reproduce the beta_2 hyperparameter result, the recency-bias shift-deviation analysis, or the batch-size findings. Please qualify the validation claim to refer specifically to the shape-ranking result so that readers do not infer that all headline recommendations were cross-checked at 60M scale and on fineweb-edu.
  3. [Section 4.1, Eq. (1)] The bias-variance framework is presented as explaining why some cooldown shapes perform better, but Eq. 1 makes it an algebraic identity: bias + variance = E_i[L(m_i)] - L(m*). Consequently, the claim that the optimal shape 'minimizes bias+variance' is equivalent to the claim that it has the lowest average validation loss across data permutations, which is precisely the measured ranking. The paper should state this equivalence explicitly and frame Eqs. 1-4 as a descriptive decomposition rather than an independent explanation. It should also note that the bias coordinate is relative to the chosen reference model and that any reference with a fixed loss L(m*) shifts all bias coordinates by the same constant, so the relative ranking is reference-invariant; stating this explicitly would preempt the natural concern that the reference choice drives the conclusions.
minor comments (8)
  1. [Equations and notation] Several equations and inline symbols (e.g., in Eq. 1 and Section 4.1) appear garbled from the text conversion, making it hard to parse the definitions of m_i, m*, and the expectations; please ensure all mathematical symbols render correctly in the final version.
  2. [Section 4.2, Eq. (2)] The decision to compute shift only over cooldown batches is acknowledged to lack firm justification; please add a short sensitivity discussion or a supplementary plot showing how the shift-deviation ranking changes if the window is extended, since the correlation results in Appendix E depend on this choice.
  3. [Figure 11] The beta_2 sweeps in Figure 11 are shown without error bars or a statement of the number of data permutations; Section 6.1 explicitly reports error bars for 5 permutations for the batch-size study, and the same reporting standard should be applied here.
  4. [Figure 11 caption] The caption contains a typo: 'miror cosine' should be 'mirror cosine.'
  5. [Section 6.1] The learning-rate scaling factors for larger batch sizes (Appendix H) were selected as the best among several tried values; please state this selection procedure in the main text and discuss how the tuning of LR per batch size affects the batch-size conclusions.
  6. [Appendix D] The statements that the reproduced plots 'agree with our earlier discussion' are vague; consider overlaying the 210M results or reporting a correlation/rank comparison between the 210M and 60M / fineweb-edu bias-variance plots.
  7. [Section 7 and Figure 30] The loss-landscape visualizations appear to be from a single trajectory; please state how many runs were examined and whether the qualitative river-valley pattern was consistent across data permutations or seeds.
  8. [Figure 9] The dashed horizontal line for the 52.8k sqrt run is not labeled in the legend; please clarify in the caption or legend that the dashed line corresponds to the longer single run.

Circularity Check

1 steps flagged · score 6.0 of 10

The bias-variance optimality criterion is a tautological restatement of the measured average validation loss; the empirical shape and beta_2 comparisons are otherwise independent experiments.

  1. self definitional [Section 4.1, Eq. (1) and 'Plot interpretation' paragraph]
    "With this notation, we can formulate the bias-variance decomposition of model performance in relation to a better model m_*: E_i[L(m_i) - L(m_*)] = E_i[L(m_i)] - L(m_*) = (L(E_i[m_i]) - L(m_*)) + (E_i[L(m_i)] - L(E_i[m_i])). ... In this setting, the optimal cooldown shape is one that achieves the minimum of bias + variance. We note that the lowered linear shape with parameter 0.7 and the sqrt shape occupy an optimal position, achieving a balance between bias and variance."

    Equation (1) defines bias = L(E_i[m_i]) - L(m_*) and variance = E_i[L(m_i)] - L(E_i[m_i]), so bias + variance = E_i[L(m_i)] - L(m_*). Since the reference loss L(m_*) is a single constant shared by all cooldown shapes, minimizing bias + variance is exactly minimizing the average validation loss E_i[L(m_i)], which the paper measures directly in Figure 4. The 'explanation' that sqrt and lowered-linear-0.7 occupy the optimal trade-off therefore reduces by construction to the observation that they have the lowest average final perplexity; the framework does not predict the ranking independently of the loss values used to define it.

full rationale

The central bias-variance framework is tautological: Eq. (1) telescopes to average loss minus a reference constant, so the optimal bias+variance criterion is equivalent to the lowest measured validation loss. This makes the headline explanation circular, although the underlying measurements (shape ranking, beta_2 sweep, loss-landscape plots) are genuine experiments and the practical advice is not forced by the identity. Changing the reference model would only shift all bias coordinates by the same constant, so the reference choice is not the source of circularity. The paper also cites prior work by overlapping authors (Hagele et al. 2024) for WSD background and the 20% cooldown fraction, but those citations are not load-bearing for the new derivation. The beta_2 claim is empirical and not circular, though its lack of permutation-level replication is a correctness risk, not a circularity risk. Overall: one 'prediction' reduces by construction; score 6.

Assumptions & free parameters 4 free parameters · 7 assumptions · 0 invented entities

The central claim rests on the measured identity in Eq. 1, the hand-selected reference model (longer sqrt cooldown), and the domain assumption that validation perplexity and weight-space averaging capture model quality. The freely chosen elements are the cooldown length, the reference duration and shape, per-batch-size learning-rate scaling factors fitted on validation performance, and the post hoc data window for the shift metric. No new physical or formal entities are postulated; the 'river valley' picture comes from Wen et al. (2024) and the bias-variance framework is a re-description of a standard identity.

free parameters (4)
  • Learning-rate scaling factors per batch size = 1.06 (batch 200), 1.50 (400), 2.12 (800), 3.00 (1600), 3.35 (2000)
    Section 6.1 and App. H: chosen by trying several LR values per batch size and selecting the best on validation perplexity; the batch-size conclusions in Figures 10a and 10b use these fitted LRs.
  • Cooldown length fraction = 20% of training
    Adopted from Hagele et al. (2024), not re-derived; all shape and hyperparameter comparisons keep this fixed.
  • Reference model training duration = 59,400 steps (6B tokens)
    Section 4.1: the reference defines the bias axis; the choice of duration and the sqrt cooldown for the reference are hand-selected.
  • Shift metric data window in Eq. 2 = cooldown batches only (i with P < i <= N)
    Section 4.2: chosen post hoc because it 'leads to an unskewed shift-deviation plot and high correlation with the previously observed results'; the paper states no firm justification.
assumptions (7)
  • domain assumption Validation perplexity on the held-out split is an adequate measure of final model quality.
    All conclusions use validation perplexity; Section 10 (Future Work) notes downstream-task relevance is not studied.
  • domain assumption Weight-space averaging of models trained on different data permutations is a meaningful representative point (implicit linear mode connectivity).
    Eq. 1 defines variance via L(E[m_i]); this requires interpreting the weight-space average as a sensible model.
  • ad hoc to paper The longer sqrt-cooldown reference run is the correct 'better solution' toward which bias should be measured.
    Section 4.1: the entire bias axis is anchored to this reference; a different reference could re-rank shapes.
  • domain assumption Nine data permutations adequately estimate the expectation over data orderings.
    Section 4.1: E_i is estimated from N=9 runs with no confidence intervals on the bias-variance points.
  • domain assumption The token half-life parameterization (beta_tilde = beta^p) is a valid way to transfer AdamW statistics across batch sizes.
    Section 6.1: follows Chiley et al. (2019), Busbridge et al. (2023), Pagliardini et al. (2024); the batch-size conclusions depend on this parameterization.
  • domain assumption Cooldown behavior at 60M and 210M parameters transfers to larger models.
    Section 10: 'our analysis is limited to moderate model sizes'; no large-scale validation is provided.
  • standard math The bias-variance decomposition in Eq. 1 is an algebraic identity; its empirical content is the measured position of each shape on the curve.
    Bias plus variance equals E[L(m_i)] minus L(m_ref) by construction, so 'min bias+variance' equals 'min average loss' for a fixed reference.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler." pith.science (2026). https://pith.science/paper/XGIFGUCB

@misc{pith2026250801483,
  author       = {Pith},
  title        = {Pith review of: Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGIFGUCB}},
  note         = {Machine review of arXiv:2508.01483}
}
abstract

Learning rate scheduling is essential in transformer training, where the final annealing plays a crucial role in getting the best performance. However, the mechanisms behind this cooldown phase, with its characteristic drop in loss, remain poorly understood. To address this, we provide a comprehensive analysis focusing solely on the cooldown phase in the Warmup-Stable-Decay (WSD) learning rate scheduler. Our analysis reveals that different cooldown shapes reveal a fundamental bias-variance trade-off in the resulting models, with shapes that balance exploration and exploitation consistently outperforming alternatives. Similarly, we find substantial performance variations $\unicode{x2013}$ comparable to those from cooldown shape selection $\unicode{x2013}$ when tuning AdamW hyperparameters. Notably, we observe consistent improvements with higher values of $\beta_2$ during cooldown. From a loss landscape perspective, we provide visualizations of the landscape during cooldown, supporting the river valley loss perspective empirically. These findings offer practical recommendations for configuring the WSD scheduler in transformer training, emphasizing the importance of optimizing the cooldown phase alongside traditional hyperparameter tuning.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Why Do We Need Warm-up? A Theoretical Perspective

    cs.LG 2025-10 conditional novelty 5.0 of 10

    Under the proposed (H0,H1)-smoothness condition, gradient descent with a warm-up-style adaptive step-size provably converges faster than with any fixed step-size.

Reference graph

Works this paper leans on

25 extracted references · 17 canonical work pages · cited by 1 Pith paper

  1. [1]

    The 210M parameter model is used in the majority of the experiments

    Model Size d����� n������ ffw dim head dim n����� 60M 512 10 1536 64 8 210M 768 24 2048 64 12 Table 1: Model configurations with varying parameter counts. The 210M parameter model is used in the majority of the experiments. The 60M parameter model is used for experiments in App. D. B Used Cooldown Shapes Formulas The formulas for cooldown shapes used in t...

    work page 2025

  2. [2]

    All runs are trained with bfloat16 automatic mixed precision

    = (0 .9, 0.95), ε = 10 � 8, a decoupled weight decay of 0.1 (Loshchilov & Hutter, 2019), and gradient clipping at1.0. All runs are trained with bfloat16 automatic mixed precision. We generally apply a short warmup of 300 steps. The batch size is set to 200, corresponding to 0.1 million tokens for a sequence length of

    work page 2019

  3. [5]

    URL�������������������������������� . Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929,

    arXiv 2010

  4. [7]

    Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule

    Nikhil Iyer, V. Thejas, Nipun Kwatra, Ramachandran Ramjee, and Muthian Sivathanu. Wide-minima density hypothesis and the explore-exploit learning rate schedule. CoRR, abs/2003.03977,

    work page Pith review arXiv 2003

  5. [8]

    Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training

    URL �������������������������������� . 14 Published in Transactions on Machine Learning Research (08/2025) Paul Janson, Vaibhav Singh, Paria Mehrbod, Adam Ibrahim, Irina Rish, Eugene Belilovsky, and Benjamin Thérien. Beyond cosine decay: On the effectiveness of infinite learning rate schedule for continual pre- training. arXiv preprint arXiv:2503.02844,

    work page Pith review arXiv 2025

  6. [10]

    15 Published in Transactions on Machine Learning Research (08/2025) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67,

    work page 2025

  7. [11]

    doi: 10.18653/v1/2024.acl-long.293

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.293. URL������������������������������ ������������� . Fabian Schaipp, Alexander Hägele, Adrien Taylor, Umut Simsekli, and Francis Bach. The surprising agree- ment between convex optimization theory and learning-rate scheduling for large model training,

    doi:10.18653/v1/2024.acl-long.293 2024

  8. [12]

    Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin

    URL�������������������������������� . Yikang Shen, Zhen Guo, Tianle Cai, and Zengyi Qin. Jetmoe: Reaching llama2 performance with 0.1m dollars. Apr 2024a. URL��������������������������������� . Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D. Cox, and Rameswar Panda. Power scheduler: A bat...

    arXiv

Show all 25 references
  1. [13]

    Shreyas Subramanian, Vignesh Ganapathiraman, and Corey D Barrett

    URL�������������������������������� . Shreyas Subramanian, Vignesh Ganapathiraman, and Corey D Barrett. Hop, skip, jump to convergence: Dy- namicsoflearningratetransitionsforimprovedtrainingoflargelanguagemodels. InYaserAl-Onaizan, Mo- hit Bansal, and Yun-Nung Chen (eds.),Find...

    2024

  2. [14]

    doi: 10.18653/v1/2024.findings-emnlp.954

    Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.954. URL ��������������������������������������������� ����. Akiyoshi Tomihari and Issei Sato. Understanding linear probing then fine-tuning language models from NTK perspective. In The Thirty-eigh...

    2024 doi

  3. [15]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al

    URL ������������������������������������������ . Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv p...

    arXiv

  4. [16]

    16 Published in Transactions on Machine Learning Research (08/2025) Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S

    URL������������������ �������������� . 16 Published in Transactions on Machine Learning Research (08/2025) Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornbli...

    2025

  5. [17]

    17 Published in Transactions on Machine Learning Research (08/2025) A Experiments Context The experimental setup mirrors that of Hägele et al

    URL������������������ �������������� . 17 Published in Transactions on Machine Learning Research (08/2025) A Experiments Context The experimental setup mirrors that of Hägele et al. (2024). Specifically, we employ a decoder-only transformer model resembling Llama3 (Meta AI, 20...

    2024

  6. [19]

    From the plots, it is evident that the general structure and relative positions of different shapes agree with the previous experiments. 4 5 6 Bias loss contribution,[10� � ] 0.25 0.50 0.75 1.00 1.25 Variance loss contribution [10� � ] ��� ��� ��� ��� ��� 3.75 4.00 4.25 4.50 B...

    2025

  7. [20]

    Presumably, variations between plots should become less noticeable with an increase in the number of experiments with different data permutations

    We see that while some variations are observed, the general structure of the plot remains the same. Presumably, variations between plots should become less noticeable with an increase in the number of experiments with different data permutations. 300 325 350 Bias, model weight...

    2025

  8. [21]

    While the result is similar to the basis space used in the main text, we believe that coordinates choice is less interpretable and therefore we are not using it as the main version. 21 Published in Transactions on Machine Learning Research (08/2025) D Bias Variance Plot Reprod...

    2025

  9. [24]

    The correlation with bias is not as strong, but the general structure remains the same, though skewed. 23 Published in Transactions on Machine Learning Research (08/2025) 80 85 90 95 100 Batch index, % � 6 � 4 � 2 Perplexity improvement, �(��) � � (��) 0 25 50 75 100 Batch ind...

    2025

  10. [25]

    effective learning rate

    It is clear that low-deviation shapes such aslowered linear 0.1lie close to zero on the right plot, indicating more uniform data point bias. However, on the left plot, they lie high, indicating low improvement over the pre-cooldown model (shift). Conversely, high-deviation sha...

    2024

  11. [26]

    during the cooldown stage at intervals of 1000 steps across all transformer hidden layers. The linear layers are trained for 2,000 steps (1 million tokens) using the AdamW optimizer on a subset of the training data, then evaluated on 1,000 evaluation batches from a held-out po...

    2025

  12. [2017]

    AtliKosson, BettinaMessmer, andMartinJaggi

    URL ������ ������������������������� . AtliKosson, BettinaMessmer, andMartinJaggi. Rotationalequilibrium: Howweightdecaybalanceslearning across neural networks. InForty-first International Conference on Machine Learning, 2024a. URL������ ������������������������������������ . ...

    arXiv

  13. [2019]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al

    arXiv:1905.05894. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113,

    1905 arXiv

  14. [2020]

    Understanding emergent abilities of language models from the loss perspective

    Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. Rong Ge, Sham M. Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: A near optimal, geometrically decaying learning rate pr...

    1904 arXiv

  15. [2023]

    Team Cohere, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Al- thammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, Neeral Beladia,...

    2025

  16. [2024]

    Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness

    URL�������������������������������� . Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to zero: Why linearly decaying the learning rate to zero works best for llms.arXiv preprint arXiv:2502.15938,

  17. [2025]

    Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open-source language models with longtermism.arXiv preprint arXiv:2401.02954,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.