Pith. sign in

REVIEW 8 cited by

Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2503.04715 v7 pith:RVO2FDVK submitted 2025-03-06 cs.LG cs.AI

Houyi Li , Wenzhen Zheng , Qiufeng Wang , Hanshan Zhang , Zili Wang , Shijie Xuyang , Yuantao Fan , Zhenyu Ding
show 5 more authors
Haoying Wang Ning Ding Shuigeng Zhou Xiangyu Zhang Daxin Jiang
This is my paper · ORCID
classification cs.LGcs.AI
keywords hyperparametermodeloptimalacrossstepdatascalingsize
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

The impressive capabilities of Large Language Models (LLMs) across diverse tasks are now well established, yet their effective deployment necessitates careful hyperparameter optimization. Although existing methods have explored the influence of hyperparameters on model performance, a principled and generalizable framework across model architectures and data recipes remains absent. In this study, we conduct an unprecedented empirical investigation training over 3,700 LLMs from scratch across 100 trillion tokens, consuming nearly one million NVIDIA H800 GPU hours to establish a universal Scaling Law for hyperparameter optimization in LLM Pre-training, called Step Law. We empirically observe that, under fixed model size ($N$) and dataset size ($D$), the hyperparameter landscape exhibits convexity with a broad optimum, substantially reducing the complexity of hyperparameter search. Building on this insight, we formally define and empirically validate the Step Law: The optimal learning rate follows a power-law relationship with $N$ and $D$, while the optimal batch size is primarily influenced by $D$ and remains largely invariant to $N$.Notably, our estimated optima deviate from the global best performance found via exhaustive search by merely 0.094\% on the test set. To our best known, Step Law is the first that unifies different model shapes and structures, such as Mixture-of-Experts models and dense transformers, as well as establishes optimal hyperparameter scaling laws across diverse data recipes. We contribute a universal, plug-and-play optimal hyperparameter tool for the community, which is expected to advance efficient LLM training at scale. All experimental code, data and checkpoints are publicly available at https://github.com/step-law/steplaw

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.

  2. OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale

    cs.CL 2026-02 conditional novelty 6.0 of 10

    OmniMoE's vector-level atomic experts + Cartesian-product router + expert-centric scheduling gives 50.9% zero-shot accuracy and a 10.9x inference speedup over PEER at matched 1.7B active parameters.

  3. SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A width-progressive training method (RMS-preserving rescaling plus asymmetric optimizer-state reset and LR rewarmup) enables mid-training 2x width expansion with up to 35% compute savings over training from scratch.

  4. Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A newly fitted scaling law with model-size-dependent data exponents predicts LLM loss more accurately than Chinchilla, including at a held-out 25.1B model.

  5. Scaling Law for Quantization-Aware Training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A unified QAT scaling law predicts 4-bit quantization error from model size, training tokens, and group size, showing activation outliers in the FC2 layer are the main W4A4 bottleneck.

  6. LLM-Based Generative Retrieval for Snapchat Content Recommendation

    cs.IR 2026-07 conditional novelty 5.0 of 10

    SnapLGR, a production LLM-based generative retrieval system for Snapchat short video, lifted View Time 0.37% and related engagement metrics in a 7-day A/B test, with offline ablation attributing most of the gain to de...

  7. Convolution for Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    Adding a residual depthwise convolution (kernel 3) to QKV projections before attention raises average downstream accuracy in Qwen3-1.7B/4B by 1.6-3.8 points with negligible parameter cost.

  8. An Overview of Algorithms for Contactless Cardiac Feature Extraction from Radar Signals: Advances and Challenges

    eess.SP 2025-08 unverdicted novelty 4.0 of 10

    The paper claims to be the first review of signal-processing algorithms for radar-based cardiac monitoring, with a new taxonomy and public dataset listings.

Pith tools