Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Merging checkpoints can replace the learning-rate decay phase in LLM pre-training.

desk verdict WSM gives LLM pretrainers a genuinely simpler schedule—constant LR plus checkpoint averaging—but the headline gain over WSD rests on single-run, best-of-many evaluations and a few numeric inconsistencies. read the letter →

arxiv 2507.17634 v2 pith:IHQ3APXU submitted 2025-07-23 cs.CL cs.LG

Changxin Tian , Jiapeng Wang , Qian Zhao , Kunlong Chen , Jia Liu , Ziqi Liu , Jiaxin Mao , Wayne Xin Zhao
show 2 more authors
Zhiqiang Zhang Jun Zhou
This is my paper
classification cs.CLcs.LG
keywords learningrateschedulingcheckpointmergingmodelaveragingLLMpre-trainingWarmup-Stable-Decaydecay-freetrainingmixture-of-experts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Warmup-Stable and Merge (WSM), a learning-rate schedule for large language model pre-training that eliminates the decay phase entirely. The authors try to establish that the optimization benefits normally obtained from gradually lowering the learning rate can be reproduced by averaging saved checkpoints from a constant-learning-rate run. They derive a theorem that converts any monotonically decreasing decay curve (cosine, linear, inverse square root) into a set of checkpoint merge weights, and they argue this makes the theoretical connection between decay and merging exact under idealized gradient updates. On a 16.3B-parameter mixture-of-experts model annealed on 400B tokens, WSM outperforms the Warmup-Stable-Decay (WSD) baseline on most benchmarks, with reported improvements of +3.5% on MATH, +2.9% on HumanEval, and +5.5% on MMLU-Pro. If the claim holds, practitioners can keep training at a constant learning rate indefinitely and obtain annealed models on demand by merging, without pre-specifying a total step count or decay function.

What carries the argument

The central object is the checkpoint-merge-as-decay identity: for an ordered list of checkpoints, merging with weights $\{c_j\}$ applies effective coefficients $\{w_i\}$ to the gradient updates after the base checkpoint, where $w_i = \sum_{j=i}^k c_j$. Theorem 3.1 inverts this map: from any bounded monotone decay sequence $\{w_i\}$ it constructs the checkpoint weights $c_k = w_k$, $c_j = w_j - w_{j+1}$, $c_0 = 1 - w_1$. This identity is what lets the paper replace a live decay phase with an offline or sliding-window merge of saved checkpoints; the shape of the decay curve becomes a choice of merge weights rather than a change to the optimizer or learning rate.

What would settle it

Train two identical models from the same checkpoint on the same data for the same number of tokens, one with a true cosine or linear decay schedule and one with constant learning rate plus checkpoints merged using the Theorem 3.1 weights, then compare their benchmark scores and loss trajectories across several random seeds and data orders; if the merged model does not track the decayed model, the claimed equivalence fails in practice. A sharper version compares SGD, where the update is literally the gradient times the learning rate, with AdamW: if the mapping holds, merge weights derived from a decay curve should reproduce the decayed model under both optimizers.

Watch

Extended reading notes

Core claim

The central claim is that the optimization benefits of learning-rate decay can be decoupled from the live training process and instead achieved by checkpoint merging. Concretely, the paper shows that a weighted average of checkpoints, $\hat\theta_{n+k} = \sum_{j=0}^k c_j \theta_{n+j}$, can be rewritten as $\theta_n - \sum_{i=1}^k w_i g_{n+i-1}$, where $w_i = \sum_{j=i}^k c_j$ are effective weights applied to the gradient updates $g$. Given a desired monotonically non-increasing decay sequence $1 \ge w_1 \ge \cdots \ge w_k \ge 0$, the unique checkpoint weights are $c_k = w_k$, $c_j = w_j - w_{j+1}$, and $c_0 = 1 - w_1$. Therefore cosine, linear, inverse-square-root, or any monotone decay can be emulated post hoc by averaging checkpoints with these weights, leaving the live training at a constant learning rate. The paper reports that this decay-free schedule consistently outperforms WSD, and identifies the merge duration, rather than checkpoint interval or number of checkpoints, as the dominant factor.

Load-bearing premise

The mapping from merge weights to a decay schedule assumes the gradient updates between checkpoints are independent and that optimizer states such as Adam's moments can be ignored, which is only approximately true in real large-scale training.

Editorial extensions

If this is right

  • Training runs can be extended indefinitely at a constant learning rate; an annealed model is available at any point by merging the latest $n$ checkpoints, removing the need to pre-specify $T_{\max}$ or a decay start step.
  • Any monotone decay curve, including cosine, linear, and inverse square root, can be instantiated as a weighted average of checkpoints, so decay-strategy selection becomes a post-hoc offline search over merge weights.
  • The merge window size (duration) is the primary lever; the paper finds it matters more than how often checkpoints are saved or how many are merged, so practitioners should tune duration first.
  • The gains transfer to supervised fine-tuning: models produced by WSM retain an advantage after 5 epochs of SFT, indicating the benefit is not confined to the pre-training loss.
  • The method is optimizer-agnostic in principle, so the same pipeline should work with SGD, AdamW, or other optimizers without touching the training loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested implication is that the theoretical equivalence is exact only when gradient updates between checkpoints are independent and optimizer states are ignored; with AdamW, the effective per-step update depends on moment estimates, so the mapping from merge weights to a decay schedule is approximate and may degrade for optimizers with strong state.
  • The merge-duration finding suggests a cheap way to probe post-anneal model quality during training: instead of launching expensive decay runs at each milestone, a merged checkpoint covering the last 100B tokens could serve as a high-fidelity estimate of what a true decay run would produce.
  • The paper's hybrid experiments indicate that decay and merging are substitute pathways to the same objective rather than complementary ones; a stronger test would be whether any combination of partial decay and partial merge ever beats the better of the two alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Warmup-Stable and Merge (WSM), a learning-rate schedule that replaces the decay phase of Warmup-Stable-Decay (WSD) with constant-LR training followed by weighted averaging of recent checkpoints. It derives a formal mapping between checkpoint merge weights and effective gradient decay weights (Theorem 3.1), claims this allows emulation of cosine, linear, and inverse-square-root decay as principled model averaging, and reports experiments on a 16.3B-parameter MoE model (Ling-mini) continued for 400B tokens from a 10.2T-token checkpoint. The central claim is that WSM outperforms WSD by about 1.3 average points on the reported benchmarks, with gains that persist after supervised fine-tuning, and that merge duration is the most important design factor.

Significance. If the central claim holds, WSM would be a practically valuable simplification of LLM pre-training schedules: it removes the need to pre-specify decay functions and total steps, enables continuous training, and offers post-hoc exploration of annealing strategies from stored checkpoints. The paper's theoretical contribution is modest—Theorem 3.1 is an algebraic identity under a strong independence assumption—but the empirical exploration of merge duration, granularity, and merge algorithm is useful and reasonably extensive for a 16.3B MoE at 400B additional tokens. The work also connects to a growing literature on schedule-free and weight-averaging methods. However, the current evidence for the headline gain is weakened by the best-of-many evaluation protocol, missing error bars, and several internal inconsistencies, so the empirical claims are not yet established at a level that would support the paper's strong wording.

major comments (5)
  1. [Section 4.2, Table 1] The comparison between WSM and WSD is asymmetric. WSD is a single deterministic run evaluated at its final checkpoint, whereas WSM results are 'reported based on the checkpoint with the highest average benchmark score' among multiple merge windows (Merge 8, 12, 16, 20) and across the entire 500B-token trajectory. This is a best-of-many versus one comparison, which can inflate the reported gains even if the underlying schedule is no better than WSD. To support the claim that WSM outperforms WSD, the paper should either fix the merge window a priori and compare the corresponding merged checkpoints at matched token budgets, or report the mean and standard deviation (or median and range) over merge windows. The current +1.3 average improvement is not a controlled estimate of the schedule's effect.
  2. [Section 4.2 and Appendix E, Table 7] The abstract reports improvements of +3.5% on MATH, +2.9% on HumanEval, and +5.5% on MMLU-Pro, but these do not match Table 7's numbers (e.g., HumanEval +2.3% for mean and +1.5% for 1-sqrt relative to WSD; MMLU-Pro +4.4% for mean). More importantly, Table 7 shows several benchmark decreases for WSM (e.g., HumanEval_fim -1.2 points, HumanEval_cn -0.6, MBPP -0.0, MBPP_plus -0.3), contradicting the statement in Section 4.2 of 'consistent improvements across all benchmark categories.' The paper should report the complete distribution of per-benchmark deltas, clarify which numbers correspond to which merging algorithm, and temper the 'consistent' language accordingly.
  3. [Section 3.1, Eqs. (2)-(4) and Theorem 3.1] The theoretical connection is derived under the assumption that 'the updates between checkpoints at different time steps are independent and ignore optimizer states.' With AdamW, the effective update for each parameter depends on optimizer moments that persist across checkpoints, so the mapping from merge weights to a decay schedule is approximate and not guaranteed to hold in the experiments. Since the main empirical results (mean merging) do not depend on Theorem 3.1, this is not fatal, but the paper should not claim the framework is a 'principled' or exact emulation of decay without testing the approximation (e.g., by comparing the weight trajectories of a true decay run and a WSM run with matched effective learning rates). As written, the theorem is a definitional identity under a strong simplifying assumption, and its role should be described as interpretive rather than predictive.
  4. [Section 4.1 and Figure 4] The experimental setup states that after branching from the 10.2T checkpoint, training continues for an additional 400B tokens with checkpoints saved every 25B tokens, yielding at most 16 checkpoints. However, Figure 4 plots results for 'Merge 20' in the mean merging panel, and the text refers to merge windows of size 20. Unless checkpoints from before the branch (i.e., from the constant-LR 10.2T pretraining) are included in the merge, a window of 20 checkpoints would require 500B tokens of post-branch training, which is inconsistent with the stated budget. Please clarify the actual checkpoint budgets and whether any pre-branch checkpoints enter the merge windows, as this affects the interpretation of 'merge duration.'
  5. [Section 4.3.3, Table 4] The 'Language Modeling' column in Table 4 contains values (79.95, 80.00, 80.29, 79.83, 64.98) that are implausible relative to the WSD baseline's Language Modeling score of 67.78 in Table 1 and to the other columns in the same table. This appears to be a data error, possibly a misplaced column or a copy-paste mistake. Please verify the table and correct the values, and ensure that the overall averages are computed from the corrected entries.
minor comments (6)
  1. [Abstract] The phrase 'significant improvements' is not supported by any statistical significance testing; consider replacing it with 'reported improvements' or adding error bars and significance tests.
  2. [Section 2] The cosine schedule formula is missing the lr_peak factor in the second line (it should be lr_peak times the cosine term). Also, the piecewise definitions would benefit from explicit variable domains.
  3. [Section 3.2 and Algorithm 1] The term 'merge duration' is used throughout but never formally defined. Define it as the token span covered by the merged checkpoints, e.g., (n-1)*T_cpt, and use this consistently.
  4. [Section 4.3.2, Table 3] The claim that 'the 1-sqrt merge approach shows slight advantages over Mean' is based on a single run with no error bars. Temper the language and note the absence of variance estimates.
  5. [Figure 3] The gray region is mentioned in the text but its meaning in the figure is not explained in the caption or body. Please add a clear legend or caption explanation.
  6. [Section 4.4 and Figure 6] The t-SNE visualization is qualitative and does not directly support the quantitative claims. Please either provide a more rigorous analysis (e.g., distance-to-merged-solution measurements) or explicitly label this as an illustrative observation.

Circularity Check

0 steps flagged · score 0.0 of 10

No load-bearing circularity: the Section 3.1 merge-decay link is an explicit algebraic identity, and the central empirical claims rest on independent benchmark comparisons.

full rationale

The paper's theoretical derivation (Section 3.1) is an algebraic equivalence, not a circular prediction. Equations (2)-(4) express each checkpoint as the initial checkpoint minus accumulated gradient updates, and then define effective gradient weights as tail sums of the checkpoint merge weights: w_i = sum_{j=i}^k c_j. Theorem 3.1 (Eq. 5) simply inverts this linear relation, giving c_k = w_k and c_j = w_j - w_{j+1}. This is a construction from a stated premise to a stated conclusion, with no empirical quantity being fitted and then re-predicted; it asserts only that a given merge-weight sequence corresponds to a given gradient-weight sequence under the paper's explicitly stated independence assumptions. The central claim that checkpoint merging can replace LR decay is tested empirically in Section 4 on external benchmarks (MATH, HumanEval, MMLU-Pro, etc.), so it is not forced by the algebra. The choice of mean averaging and window sizes is explored experimentally rather than derived from the theorem. The cited prior work (Li et al. 2025, Hu et al. 2024, etc.) is external or concurrent and does not supply an unverified uniqueness premise that the paper's conclusion depends on. The reported practice of selecting the checkpoint with the highest average benchmark score ('Results are reported based on the checkpoint with the highest average benchmark score', Table 1) is a potential evaluation-fairness or selection concern, but it is not a circular derivation: the WSM numbers are not constructed from the WSD numbers, and the schedule comparison is an empirical matter rather than an identity. No step in the paper reduces a claimed prediction to its own input by definition or by self-citation.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the design choices for the learning rate, checkpointing interval, merge window size, and the target decay curve, plus the strong assumption that checkpoint merging can substitute for LR decay. No new physical entities are introduced. The algebra in the proof is elementary, so the main burden falls on the empirical demonstration and the transferability of the simplifying assumptions.

free parameters (6)
  • Peak learning rate = 4.78e-4 (Section 4.1) or 3.74e-4 (Appendix A)
    Set via scaling-law experiments; the reported values are inconsistent, and the LR affects all downstream results.
  • Checkpoint interval Tcpt = 25B tokens (default); 5B, 10B, 20B, 40B, 80B in granularity study
    A design choice that determines merge granularity and storage cost.
  • Merge window size n = Varied: 2, 4, 8, 12, 16, 20 checkpoints
    The duration of the merge window is claimed to be the most critical factor; the optimal value is chosen empirically.
  • Target decay curve shape and parameters = 1-sqrt, cosine, linear, EMA; exact curve parameters not fully specified
    The merge weights are computed from a chosen decay function; the curve shape and any floor or scaling are design choices.
  • Annealing data switch point Tswitch = 10.2T tokens (start of the 400B-token branch)
    When the model switches to the high-quality annealing dataset; a manual scheduling choice.
  • SFT training epochs = 5
    Used for the post-training comparison in Table 2; chosen by the authors.
assumptions (6)
  • domain assumption Gradient updates between checkpoints are independent and optimizer states can be ignored.
    Section 3.1 states this assumption explicitly. It makes the checkpoint-merge to LR-decay equivalence exact, but it is an approximation for AdamW.
  • ad hoc to paper Checkpoint merging can decouple the optimization benefits of LR decay from the live training process.
    This is the central hypothesis in Section 3.2, not a consequence of the theory.
  • domain assumption The benchmark average across the chosen suite is an adequate proxy for model quality.
    The headline numbers use the average of a fixed set of benchmarks and select the best checkpoint, which can bias conclusions.
  • standard math The algebraic rearrangement and telescoping sums in the proof are valid.
    Eqs. 8-17 in Appendix B are correct arithmetic under the stated definitions.
  • domain assumption The offline exploration of merge strategies does not bias the final evaluation.
    Section 3.3 claims offline merging does not impact the primary training run, but the checkpoint history is used to select the final evaluation, which is a form of post-hoc selection.
  • domain assumption The 1-sqrt decay is a strong and representative WSD baseline.
    Appendix C establishes this on a smaller Ling-lite model; the main experiments assume it transfers to the 16B model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training." pith.science (2026). https://pith.science/paper/IHQ3APXU

@misc{pith2026250717634,
  author       = {Pith},
  title        = {Pith review of: WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IHQ3APXU}},
  note         = {Machine review of arXiv:2507.17634}
}
read the original abstract

Recent advances in learning rate (LR) scheduling have demonstrated the effectiveness of decay-free approaches that eliminate the traditional decay phase while maintaining competitive performance. Model merging techniques have emerged as particularly promising solutions in this domain. We present Warmup-Stable and Merge (WSM), a general framework that establishes a formal connection between learning rate decay and model merging. WSM provides a unified theoretical foundation for emulating various decay strategies-including cosine decay, linear decay and inverse square root decay-as principled model averaging schemes, while remaining fully compatible with diverse optimization methods. Through extensive experiments, we identify merge duration-the training window for checkpoint aggregation-as the most critical factor influencing model performance, surpassing the importance of both checkpoint interval and merge quantity. Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks, achieving significant improvements of +3.5% on MATH, +2.9% on HumanEval, and +5.5% on MMLU-Pro. The performance advantages extend to supervised fine-tuning scenarios, highlighting WSM's potential for long-term model refinement.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WSqD: A Horizon-Free Learning Rate Schedule for Large Model Training

    cs.LG 2026-07 conditional novelty 6.0 of 10

    WSqD uses a horizon-independent shifted inverse-square-root base plus linear cooldown to get optimal last-iterate rates in convex stochastic optimization and match or beat tuned WSD on LLM pretraining with one reused ...

Reference graph

Works this paper leans on

70 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    e l Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Bj \

    Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Rapha \" e l Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Bj \" o rn Bebensee, Neeral Beladia, Walter Beller - Morales, Alexandre B \' e rard, Andrew Berneshawi, A...

    arXiv doi:10.48550/arxiv.2504.00698 2025

  3. [3]

    GQA: training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee - Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \' o n, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singa...

    doi:10.18653/v1/2023.emnlp-main.298 2023

  4. [4]

    Singular value decomposition for genome-wide expression data processing and modeling

    Orly Alter, Patrick O Brown, and David Botstein. Singular value decomposition for genome-wide expression data processing and modeling. Proceedings of the National Academy of Sciences, 97 0 (18): 0 10101--10106, 2000

    work page 2000

  5. [5]

    Efficient training of language models to fill in the middle

    Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. CoRR, abs/2207.14255, 2022. doi:10.48550/ARXIV.2207.14255. https://doi.org/10.48550/arXiv.2207.14255

    arXiv doi:10.48550/arxiv.2207.14255 2022

  6. [6]

    Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge

    Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR, abs/2102.03315, 2021. https://arxiv.org/abs/2102.03315

    arXiv 2021

  7. [7]

    PIQA: reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in...

    doi:10.1609/aaai.v34i05.6239 2020

  8. [8]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

    arXiv 2021

Show all 70 references
  1. [9]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021. https://arxi...

    arXiv

  2. [10]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek - AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei...

    2024 doi arXiv

  3. [11]

    Optimal linear decay learning rate schedules and further refinements

    Aaron Defazio, Ashok Cutkosky, Harsh Mehta, and Konstantin Mishchenko. Optimal linear decay learning rate schedules and further refinements. arXiv preprint arXiv:2310.07831, 2023

    2023 arXiv

  4. [12]

    The road less scheduled

    Aaron Defazio, Xingyu Yang, Ahmed Khaled, Konstantin Mishchenko, Harsh Mehta, and Ashok Cutkosky. The road less scheduled. In NeurIPS, 2024

    2024

  5. [13]

    Ernie 4.5 technical report, 2025

    Baidu ERNIE-Team. Ernie 4.5 technical report, 2025

    2025

  6. [14]

    A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation

    Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 ....

    2019

  7. [15]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Cruxeval: A benchmark for code reasoning, understanding and execution

    Alex Gu, Baptiste Rozi \` e re, Hugh James Leather, Armando Solar - Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-...

    2024

  9. [17]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview...

    2021

  10. [18]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai - Kit Yeung, editors, Proceedings of the Neural Information Processi...

    2021

  11. [19]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  12. [20]

    Worldsense: Evaluating real-world omnimodal understanding for multimodal llms

    Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. CoRR, abs/2502.04326, 2025. doi:10.48550/ARXIV.2502.04326. https://doi.org/10.48550/arXiv.2502.04326

    2025 doi arXiv

  13. [21]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Z...

    2024 doi arXiv

  14. [22]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Alice Oh, Tristan N...

    2023

  15. [23]

    Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timoth \' e e Lesort, and Irina Rish

    Adam Ibrahim, Benjamin Th \' e rien, Kshitij Gupta, Mats L. Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timoth \' e e Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Trans. Mach. Learn. Res., 2024, 2024. https://o...

    2024

  16. [24]

    Vetrov, and Andrew Gordon Wilson

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artifi...

    2018

  17. [25]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen - Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar - Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on L...

    2025

  18. [26]

    Rethinking learning rate tuning in the era of large language models

    Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, and Yanzhao Wu. Rethinking learning rate tuning in the era of large language models. In 5th IEEE International Conference on Cognitive Machine Intelligence, CogMI 2023, Atlanta, GA, USA, November 1-4, 2023 , pages 112--121. IEE...

    2023

  19. [27]

    Weld, and Luke Zettlemoyer

    Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min - Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  20. [28]

    Stop wasting my time! saving days of imagenet and BERT training with latest weight averaging

    Jean Kaddour. Stop wasting my time! saving days of imagenet and BERT training with latest weight averaging. CoRR, abs/2209.14981, 2022. doi:10.48550/ARXIV.2209.14981. https://doi.org/10.48550/arXiv.2209.14981

    2022 doi arXiv

  21. [29]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  22. [30]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. http://arxiv.org/...

    2015 arXiv

  23. [31]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and...

    2019 doi

  24. [32]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Pro...

    2017 doi

  25. [33]

    CMMLU: measuring massive multitask language understanding in chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: measuring massive multitask language understanding in chinese. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computationa...

    2024 doi

  26. [34]

    Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers

    Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. In Lun - Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of t...

    2024 doi

  27. [35]

    Trainable weight averaging: Efficient training by optimizing historical solutions

    Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable weight averaging: Efficient training by optimizing historical solutions. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net,...

    2023

  28. [36]

    Model merging in pre-training of large language models

    Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, Jin Ma, Xunhao Lai, Deyi Liu, Yao Luo, Xingyan Bin, Hongbin Ren, Mingji Han, Wenhao Hao, Bairen Yi, LingJun Liu, Bole Ma, Xiaoying Jia, Xun Zhou, Siyuan Qiao...

    2025 doi arXiv

  29. [37]

    Every flop counts: Scaling a 300b mixture-of-experts ling llm without premium gpus

    Ling-Team, Binwei Zeng, Chao Huang, Chao Zhang, Changxin Tian, Cong Chen, Dingnan Jin, Feng Yu, Feng Zhu, Feng Yuan, et al. Every flop counts: Scaling a 300b mixture-of-experts ling llm without premium gpus. arXiv preprint arXiv:2503.05139, 2025

    2025 arXiv

  30. [38]

    Checkpoint merging via bayesian optimization in LLM pretraining

    Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. Checkpoint merging via bayesian optimization in LLM pretraining. CoRR, abs/2403.19390, 2024 a . doi:10.48550/ARXIV.2403.19390. https://doi.org/10.48550/arXiv.2403.19390

    2024 doi arXiv

  31. [39]

    Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark

    Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. In Lun - Wei Ku, Andre Martins...

    2024

  32. [40]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edito...

    2023

  33. [41]

    Muon is scalable for llm training, 2025

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang, Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Z...

    2025 arXiv

  34. [42]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  35. [43]

    Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks

    Kaijing Ma, Xeron Du, Yunran Wang, Haoran Zhang, Zhoufutu Wen, Xingwei Qu, Jian Yang, Jiaheng Liu, Minghao Liu, Xiang Yue, Wenhao Huang, and Ge Zhang. Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks. In The Thirteenth International Conference on...

    2025

  36. [44]

    Can a suit of armor conduct electricity? A new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empiri...

    2018 doi

  37. [45]

    Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization

    Qiwei Peng, Yekun Chai, and Xuhong Li. Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization. In Nicoletta Calzolari, Min - Yen Kan, V \' e ronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings...

    2024

  38. [46]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30 0 (4): 0 838--855, 1992

    1992

  39. [47]

    Know what you don't know: Unanswerable questions for squad

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July ...

    2018 doi

  40. [48]

    WARP: on the benefits of weight averaged rewarded policies

    Alexandre Ram \' e , Johan Ferret, Nino Vieillard, Robert Dadashi, L \' e onard Hussenot, Pierre - Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. WARP: on the benefits of weight averaged rewarded policies. CoRR, abs/2406.16768, 2024. doi...

    2024 doi arXiv

  41. [49]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q & a benchmark. CoRR, abs/2311.12022, 2023. doi:10.48550/ARXIV.2311.12022. https://doi.org/10.48550/ar...

    2023 doi arXiv

  42. [50]

    The effective rank: A measure of effective dimensionality

    Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606--610. IEEE, 2007

    2007

  43. [51]

    Winogrande: an adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM , 64 0 (9): 0 99--106, 2021. doi:10.1145/3474381. https://doi.org/10.1145/3474381

    2021 doi

  44. [52]

    Training trajectories, mini-batch losses and the curious role of the learning rate

    Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, and Nolan Miller. Training trajectories, mini-batch losses and the curious role of the learning rate. CoRR, abs/2301.02312, 2023

    2023 arXiv

  45. [53]

    Early weight averaging meets high learning rates for llm pre-training

    Sunny Sanyal, Atula Neerkaje, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi. Early weight averaging meets high learning rates for llm pre-training. arXiv preprint arXiv:2306.03241, 2023

    2023 arXiv

  46. [54]

    Language models are multilingual chain-of-thought reasoners

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Le...

    2023

  47. [55]

    Through the river: Understanding the benefit of schedule-free methods for language model training

    Minhak Song, Beomhan Baek, Kwangjun Ahn, and Chulhee Yun. Through the river: Understanding the benefit of schedule-free methods for language model training. In High-dimensional Learning Dynamics 2025, 2025

    2025

  48. [56]

    Jianlin Su, Murtadha H. M. Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024. doi:10.1016/J.NEUCOM.2023.127063. https://doi.org/10.1016/j.neucom.2023.127063

    2024

  49. [57]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Sch \" a rli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan L. Boyd - Gr...

    2023 doi

  50. [58]

    Mathscale: Scaling instruction tuning for mathematical reasoning

    Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. https://openreview.net...

    2024

  51. [59]

    Enhancing program synthesis with large language models using many-objective grammar-guided genetic programming

    Ning Tao, Anthony Ventresque, Vivek Nallur, and Takfarinas Saber. Enhancing program synthesis with large language models using many-objective grammar-guided genetic programming. Algorithms, 17 0 (7): 0 287, 2024. doi:10.3390/A17070287. https://doi.org/10.3390/a17070287

    2024 doi

  52. [60]

    M. - A - P. Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, Dehua Ma, Yuansheng ...

    2025 doi arXiv

  53. [61]

    Visualizing data using t-SNE

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE . Journal of Machine Learning Research, 9: 0 2579--2605, 2008. http://www.jmlr.org/papers/v9/vandermaaten08a.html

    2008

  54. [62]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024

  55. [63]

    CMATH: can your language model pass chinese elementary school math test? CoRR, abs/2306.16636, 2023

    Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. CMATH: can your language model pass chinese elementary school math test? CoRR, abs/2306.16636, 2023. doi:10.48550/ARXIV.2306.16636. https://doi.org/10.48550/arXiv.2306.16636

    2023 doi arXiv

  56. [64]

    Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective

    Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective. arXiv preprint arXiv:2410.05192, 2024

    2024 arXiv

  57. [65]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy with...

    2022

  58. [66]

    Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Llu \' s M \` a rquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguisti...

    2019 doi

  59. [67]

    Foster, and Sham M

    Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean P. Foster, and Sham M. Kakade. How does critical batch size scale in pre-training? In ICLR . OpenReview.net, 2025

    2025

  60. [68]

    Why gradient clipping accelerates training: A theoretical justification for adaptivity

    Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net...

    2020

  61. [69]

    Evaluating the performance of large language models on GAOKAO benchmark

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on GAOKAO benchmark. CoRR, abs/2305.12474, 2023. doi:10.48550/ARXIV.2305.12474. https://doi.org/10.48550/arXiv.2305.12474

    2023 doi arXiv

  62. [70]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena G \' o mez - Adorno, and Steven Bethard, editors, Findings of the Associa...

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.