Pith. sign in

REVIEW 27 cited by

Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.03466 v2 pith:HIF4C7TW submitted 2022-03-07 cs.LG cond-mat.dis-nncs.NE

Greg Yang , Edward J. Hu , Igor Babuschkin , Szymon Sidor , Xiaodong Liu , David Farhi , Nick Ryder , Jakub Pachocki
show 2 more authors
Weizhu Chen Jianfeng Gao
This is my paper
classification cs.LGcond-mat.dis-nncs.NE
keywords modeltuningparameterscostpretrainingbert-largehyperparametermutransfer
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`.

Discussion (0). Sign in to comment.

Forward citations

Cited by 27 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback

    math.OC 2026-07 conditional novelty 7.0 of 10

    Sign-compressing Muon's update to one bit can make it ascend on linear objectives; error feedback works only on the gradient side, yet the divergent sign-after-the-LMO method wins in experiments.

  2. Bridging Compute- and Data-Optimal Pretraining

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Pretraining loss obeys a single law in which repeated or paraphrased tokens count as η(N, data-per-parameter, expansion-ratio) fresh tokens, with total effective data saturating as derived tokens grow.

  3. Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product

    math.PR 2026-06 unverdicted novelty 7.0 of 10

    In double asymptotic limits, the squared singular value process of non-square matrix products obeys geometric Dyson Brownian motion whose T-transform solves a Burgers equation, producing the free log-normal law via fr...

  4. Deep Delta Learning

    cs.LG 2026-01 unverdicted novelty 7.0 of 10

    Replacing additive residual connections with a gated rank-1 delta update that interpolates identity, projection, and reflection slightly improves language modeling and downstream averages in reported 124M/353M runs.

  5. FM4NPP: A Scaling Foundation Model for Nuclear and Particle Physics

    cs.LG 2025-08 conditional novelty 7.0 of 10

    A 188M-parameter Mamba model pretrained on 11M+ simulated sPHENIX events with a new serialization and neighbor-prediction task beats task-specific baselines on three downstream detector tasks when frozen and paired wi...

  6. Geometry of Neural Reinforcement Learning in Continuous State and Action Spaces

    cs.LG 2025-07 conditional novelty 7.0 of 10

    For wide two-layer linearized neural policies in deterministic continuous RL, the locally attainable states concentrate on a manifold of dimension at most 2da+1, independent of the state dimension.

  7. Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A hybrid NoPE visual diffusion backbone with HeteroP scaling yields ~7× pretraining compute efficiency versus matched full attention and near-stable zero-shot 5s→30s video extrapolation.

  8. Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Splitting weight matrices into a fixed-norm direction and learnable per-row/column magnitudes improves LLM training over AdamW/Muon, removes weight decay and warmup, and transfers the optimal LR across width.

  9. Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Neural scaling occurs because larger models maintain learning on weaker eigenmodes of the eNTK that smaller models cannot access.

  10. SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A width-progressive training method (RMS-preserving rescaling plus asymmetric optimizer-state reset and LR rewarmup) enables mid-training 2x width expansion with up to 35% compute savings over training from scratch.

  11. Scaling depth capacity via zero/one-layer model expansion

    cs.LG 2025-11 conditional novelty 6.0 of 10

    Training GPT2 from a zero/one-layer model and expanding depth at 80% of the schedule reaches fixed-size loss with approximately 5x less compute.

  12. Customizing the Inductive Biases of Softmax Attention using Structured Matrices

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Structured-matrix scoring functions, BTT and MLR, let attention escape the low-rank bottleneck and add a distance-dependent compute bias, improving accuracy for fixed compute on regression, language modeling, and forecasting.

  13. Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Falcon-H1 reports competitive benchmark scores for a 0.5B to 34B family of parallel hybrid attention/Mamba-2 models, claiming 2x to 4x parameter efficiency versus dense transformers.

  14. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

  15. BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A ReLU-routed MoE with chunk-level sparsity training objectives and custom kernels combining activation sparsity with speculative decoding achieves over 70% 8-token chunk sparsity and up to 3.67x end-side speedup.

  16. Simple Convergence Proof of Adam From a Sign-like Descent Perspective

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Adam's O(1/T^1/4) convergence is proven from a sign-like descent perspective, but the dimension-free claim depends on restrictive coordinate-wise assumptions.

  17. Decoupled Relative Learning Rate Schedules

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Per-component relative learning-rate schedules speed up Transformer pretraining by up to 23% for MoE models, and the schedules tuned on a 34M model transfer to models 27x larger.

  18. Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

    cs.LG 2025-06 conditional novelty 6.0 of 10

    Aspect ratio of weight matrices biases heavy-tail spectral metrics; the new FARMS subsampling method removes this bias and improves downstream layer-wise tuning.

  19. Scale Weight Decay and Train Better

    cs.LG 2026-07 conditional novelty 5.0 of 10

    Muon with weight decay scaled by η/η_max reaches the same MoE validation loss ~30% faster than constant-decay Muon while preserving asymptotic stationarity of the unregularized objective.

  20. Index SLM Technical Report

    cs.CL 2026-07 accept novelty 5.0 of 10

    Index-1.9B-Base reaches 64.92 average benchmark score via WSD training with late curated data plus Norm-Head, with open Pure/Boost controls isolating instruction-data inflation.

  21. The Role of Rigor in Artificial Intelligence

    cs.AI 2026-05 conditional novelty 5.0 of 10

    Modern AI's distinctive trajectory is explained by the primacy of operational rigor over conceptual and epistemic rigor across successive paradigms.

  22. Tree-Structured Parzen Estimator Can Solve Black-Box Combinatorial Optimization More Efficiently

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A distance-based categorical kernel and two algorithmic modifications let TPE optimize combinatorial spaces more efficiently than the original TPE on synthetic benchmarks.

  23. SingLoRA: Low Rank Adaptation Using a Single Matrix

    cs.AI 2025-07 conditional novelty 5.0 of 10

    SingLoRA replaces LoRA's two matrices A and B with one matrix A and the symmetric update AA^T, cutting adapter parameters roughly in half while claiming more stable fine-tuning.

  24. MiniCPM4: Ultra-Efficient LLMs on End Devices

    cs.CL 2025-06 conditional novelty 5.0 of 10

    MiniCPM4-8B reportedly matches Qwen3-8B on standard benchmarks while using about 22% of the training tokens, and achieves large long-context speedups on edge devices.

  25. Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs

    cs.LG 2025-07 conditional novelty 4.0 of 10

    High data redundancy and over-training decelerate LLM performance gains, and the authors fit a sub-optimal scaling law with logistic correction terms to predict the slowdown.

  26. NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models

    cs.AI 2025-06 conditional novelty 4.0 of 10

    The authors propose a competition with new scoring metrics to find benchmarks that give clean early-training signals for small language models, and show MMLU-var outperforms MMLU as a baseline.

  27. Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?

    cs.LG 2025-06 reject novelty 3.0 of 10

    LLM4TS_FS achieves the best MSE on four of seven long-term datasets, but the claimed broad advantage of pre-trained large models over small transformers is not consistent across all benchmarks.

Pith tools