Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Gradient Multi-Normalization for Stateless and Scalable LLM Training

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Gradient row-column balancing trains 1B LLaMA faster than Adam

desk verdict Genuinely new stateless optimizer framework, but the 3X speedup headline is inflated by a weak reproduced Adam baseline; the qualitative result survives. read the letter →

arxiv 2502.06742 v1 pith:2EU7VV3S submitted 2025-02-10 cs.LG cs.AI

Meyer Scetbon , Chao Ma , Wenbo Gong , Edward Meeds This is my paper · ORCID
classification cs.LGcs.AI
keywords statelessoptimizergradientnormalizationSinkhornscalingLLMpretrainingmemory-efficienttrainingMulti-NormalizedDescentSinkGDAdam-freeoptimizers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that a stateless optimizer — one that records nothing about past gradients — can match or beat Adam on large language model pretraining. The proposed SinkGD preprocesses each weight-matrix gradient by repeatedly rescaling its rows and columns so all rows have the same length and all columns have the same length, a procedure the paper identifies as the square-root Sinkhorn algorithm. The paper proves this alternating rescaling converges linearly to a fixed point, and it shows that SWAN, a prior stateless optimizer, is a special case of the same multi-normalization framework with a different pair of norms. Empirically, on LLaMA models from 60M to 1.3B parameters trained on C4, SinkGD reaches test perplexity equal to or better than Adam and other memory-efficient baselines while using the memory of plain SGD and achieving a 3x effective throughput speedup over Adam on the largest tested model.

What carries the argument

The load-bearing object is SR-Sinkhorn (Algorithm 3), a parameter-free alternating map that applies row-wise and column-wise $\ell_2$ normalization to a gradient matrix. It is the square-root reparameterization of the classical Sinkhorn algorithm, so it converges linearly (in Hilbert metric) to the unique fixed point where every row has $\ell_2$ norm $\sqrt{n}$, every column has $\ell_2$ norm $\sqrt{m}$, and it does so at $O(mn)$ cost per matrix. Its companion in the framework is the generic MultiNorm alternating-projection scheme (Algorithm 1), for which Theorem 3.6 shows convergence to a fixed point under the assumption that each norm's normalized projection has constant $\ell_2$ norm; SinkGD is the practical instance with two cheap $\ell_2$ norms.

What would settle it

Train the 1.3B LLaMA with SinkGD at $L=0$ iterations (raw gradient rescaled to the same total length, without row/column balancing) using the paper's exact hyperparameters; if test perplexity matches the $L=5$ result, the square-root Sinkhorn preprocessing is not what drives the gains.

Watch

Extended reading notes

Core claim

The paper's core claim is that momentum and second-moment estimates are unnecessary for LLM training if the instantaneous gradient is preprocessed so that every row and every column of each weight-matrix gradient has a prescribed Euclidean length. This multi-normalization is achieved by SR-Sinkhorn (Algorithm 3), which alternates row-wise and column-wise $\ell_2$ normalization; the paper shows this is exactly the square-root Sinkhorn iteration and inherits its linear convergence to a fixed point. By replacing SWAN's spectral-norm and whitening step with column-wise $\ell_2$ normalization, SinkGD reduces preprocessing cost from $O(m^2(m+n))$ to $O(mn)$ while keeping the stateless property. On LLaMA models from 60M to 1.3B trained on C4, SinkGD reaches test perplexity on par with or better than Adam and memory-efficient baselines, at SGD-level memory, with a 3x effective throughput speedup over Adam on the largest tested model.

Load-bearing premise

The load-bearing premise is that rescaling the current gradient so every row and every column has the same length gives the update an Adam-like ability to train transformers, so the optimizer can discard all memory of past gradients.

Editorial extensions

If this is right

  • LLM pretraining can run at SGD-level memory, eliminating the optimizer-state memory that Adam requires.
  • At the largest tested scale (1.3B), reaching Adam's final perplexity takes roughly a third of the wall-clock time, so long training runs shrink proportionally.
  • SWAN is recovered as the two-norm instance (row-wise $\ell_2$ and spectral norm), giving a principled way to design other stateless optimizers by choosing different norm pairs.
  • SinkGD's per-matrix cost is $O(mn)$, compared with SWAN's $O(m^2(m+n))$, so the approach scales to the large weight matrices in modern transformers.
  • The SR-Sinkhorn preprocessing converges linearly to a well-defined fixed point, so the update direction is deterministic and does not depend on random initialization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's pretraining evidence is limited to LLaMA sizes up to 1.3B on C4, with single runs and many baseline numbers taken from other papers; whether the 3x speedup holds at 7B+ scale or on other data distributions remains open.
  • If row/column norm balancing is the true mechanism, the same square-root Sinkhorn map could be applied to other tensors (activations, second-moment estimates, or convolutional weight tensors) rather than only to linear-layer gradients.
  • SinkGD still routes embeddings and output layers through Adam and uses a small learning-rate multiplier ($\alpha=0.05$); a fully stateless variant that removes the Adam components would isolate how much of the gain is due to the new update rule rather than the hybrid schedule.
  • Because the paper finds performance barely changes between $L=1$ and $L=5$ Sinkhorn iterations, the practical bottleneck may be the choice of norms rather than the number of iterations; an adaptive iteration count would test that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces Multi-Normalized Gradient Descent (MNGD), a framework for stateless optimizers that pre-processes each instantaneous gradient by alternating normalized projections with respect to several norms (Algorithm 1). Theorem 3.6 shows that, for two norms with constant-l2-norm projections, the alternating scheme converges to a point that is a fixed point of both projections and is normalized with respect to both norms up to arbitrary precision. SWAN (Ma et al., 2024) is recovered as the L=1 instance with row-wise l2 and spectral-norm constraints. The paper then proposes SinkGD (Algorithm 4), which alternates row-wise and column-wise l2 normalization (Algorithm 3) and is shown to compute exactly the square-root Sinkhorn iterates, giving linear convergence and O(mn) per-step cost. Experiments on LLaMA-style models from 60M to 1.3B parameters trained on C4 report test perplexities at par or better than Adam and several memory-efficient baselines (GaLore, Fira, Apollo, SWAN) with SGD-level memory, and an effective-throughput speedup of 3x over Adam (Table 3).

Significance. The theoretical core is clean and largely self-contained. Algorithm 3 is shown by a simple reparameterization to be exactly the square-root Sinkhorn iteration, so linear convergence is inherited from Franklin-Lorenz; the proof of Theorem 3.6 in Appendix B is coherent; the identification of SWAN as the L=1 instance of MultiNorm with row-l2 and spectral norms is illuminating; and the sqrt(nm) rescaling is derived from the Frobenius norm of Adam's zero-EMA sign-descent limit, making the learning-rate transfer parameter-free rather than fitted. If the empirical claims hold, the contribution is practically significant: an SGD-memory, O(mn)-per-step stateless optimizer that matches or beats Adam on C4/LLaMA up to 1.3B would be genuinely useful for memory-constrained LLM training. The paper is also transparent: it discloses the Adam reproduction failure (Table 1, Remark 1), reports both Adam references, and describes the SWAN-dagger tuning.

major comments (3)
  1. [Abstract; Fig. 1(c); Section 5.2 (Table 3)] The flagship '3x effective throughput' claim is computed against the reproduced Adam baseline and is not robust to the baseline choice. Table 3's effective throughput of 161,769 tokens/s equals raw throughput 57,982 x 2.79, where 2.79 is the 'SinkGD speed up v.s. Adam (reproduced)' at 1.3B in Table 1; however, Remark 1 of Table 1 concedes that the reproduced Adam (16.44 PPL) is worse than the cited Adam (15.56 PPL) by 0.88 PPL. Using the cited-Adam speedup of 2.17x gives an effective throughput of about 125.8K tokens/s, i.e., about 2.4x Adam's 53,047 tokens/s rather than 3x. Because the abstract, Figure 1(c), and Section 5.2 state the 3x number without the baseline caveat, the paper's headline quantitative claim is inflated by reference to the weaker baseline; the authors should quote the speedup against the cited Adam, or report both numbers whenever the speedup appears.
  2. [Section 5.1 (Table 1); Appendix A] The empirical ranking is built on single runs and on a mixture of in-house numbers (Adam-reproduced, SWAN-dagger, SinkGD) with numbers cited from other papers (GaLore, Fira, Apollo, Apollo-mini, SWAN). The authors' own reproduction failure is the key concern: the gap between the reproduced and cited Adam at 1.3B (16.44 vs 15.56, i.e., 0.88 PPL) is larger than several of the cross-method differences used to rank optimizers (e.g., 130M: SinkGD 22.75 vs Fira 22.73; 350M: SinkGD 16.51 vs Apollo 16.85), so setup variance or pipeline mismatch alone could change the ranking. I recommend multiple seeds (at least at 60M-350M, and ideally for the 1.3B SinkGD and Adam runs), in-house execution of the main baselines under the identical verified pipeline, and an explicit statement of which Table 1 rows are in-house versus cited.
  3. [Section 5.2 (Table 2)] The 1B-SinkGD versus 7B-baselines comparison does not isolate the optimizer's contribution. A 1B model trained on 19.5B tokens is close to a balanced token-to-parameter ratio, whereas a 7B model on the same budget is severely undertrained, so the lower PPL of the 1B run is largely a model-capacity and compute-allocation effect rather than an optimizer effect; likewise, the '15 days versus 3.3 days' contrast confounds model size with optimizer speed. The comparison should be reframed as a compute-efficiency illustration, or better, supplemented with a same-size control (e.g., 1B models trained with Adam and Apollo under the identical setup) so that the SinkGD-specific contribution is identifiable.
minor comments (6)
  1. [Section 3.3 (Algorithm 2); Section 4 (Algorithm 4)] Both algorithms end with 'Return x' rather than returning the final iterate of the optimization loop; as written, the return value should presumably be the final theta_T, and the current text is ambiguous.
  2. [Section 4; Conclusion; Appendix B.2] There are several typographical errors that should be corrected: 'SR-Sinhkorn' in Section 4, 'SinGD' in the Conclusion, and 'Thoerem' in the Appendix B.2 header.
  3. [Abstract; Introduction; Table 1] The scale of the models is described inconsistently: the Abstract and Introduction say 'up to 1 billion parameters', while Table 1, Section 5.2, and Figure 1 train and report a 1.3B-parameter model; please unify the terminology.
  4. [Section 5.1] The sentence 'For 60M and 1B, we uses 256 batch with 2 accumulation, and 32 per-device batch size with 2 accumulation and 8xA100s, respectively' is ungrammatical and ambiguous about which configuration applies to the 60M run and which to the 1.3B run; please rewrite.
  5. [Section 4] The linear-convergence guarantee via Franklin-Lorenz is stated for positive matrices, but Algorithm 3 normalizes by row and column l2 norms that are zero for degenerate (exact-zero row or column) gradients; the paper should state the positivity or support assumption on the gradient, or describe the epsilon-regularized variant actually used in the implementation.
  6. [Sections 3.2-3.3] Theorem 3.6 guarantees only that the preprocessing iterates approach a fixed point of the two normalizations, not that this fixed point is a useful descent direction or that the loss converges; the practical case for SinkGD therefore rests entirely on the empirical section. An explicit limitation statement plus one diagnostic on real gradients (e.g., the cosine similarity between the SR-Sinkhorn output and the Adam update direction) would help bridge the theory-practice gap.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: SinkGD's derivation is self-contained, with only a minor, non-load-bearing self-citation to the authors' SWAN work.

full rationale

The derivation chain is self-contained. SinkGD's update (Algorithm 3) is defined by alternating row-wise and column-wise L2 normalizations, and the theoretical claims in Section 4 (exact equivalence to square-root Sinkhorn iterates, linear convergence via Franklin-Lorenz) rest on the external Sinkhorn literature plus the paper's own Theorem 3.6; none of these results assumes the target outcome (good LLM perplexity). The scaling choice in 'On the Importance of the Scaling' is a design heuristic to match the Frobenius norm of an EMA-free Adam update (sqrt(n*m)), not a parameter fitted to the benchmark losses. Hyperparameters (alpha=0.05, global LR) are taken from the authors' prior SWAN work, which is a self-citation (Ma et al. 2024 shares all four authors), but it is used only as motivation, a starting point for hyperparameters, and a baseline; the central claim that SinkGD matches Adam is an independent empirical result measured against Adam, Apollo, and Galore. The re-derivation of SWAN as an instance of MultiNorm is definitional (the norms are chosen so their projections equal the SWAN steps) and is not used as evidence for SinkGD, so it does not make the derivation circular. The main quantitative caveat is the 3X effective-throughput headline: it is computed against the authors' reproduced Adam (PPL 16.44) rather than the cited Adam (PPL 15.56), which the paper explicitly acknowledges in the Table 1 remark; using the cited Adam reduces the speedup to 2.17X and effective throughput to roughly 126K tokens/s. That is a baseline-selection and reproducibility concern, not a circularity of the derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central algorithm introduces no new physical or mathematical entities; SinkGD is a procedure. The main assumptions are the constant-projection-norm condition and the empirical equivalence of instantaneous normalized gradients to Adam's momentum-based updates.

free parameters (3)
  • Learning rate scaling factor α for SinkGD linear modules = 0.05
    Scales the global learning rate for linear projection weights. Set to the same value as SWAN† after a small grid search over {0.01, 0.02, 0.05} for SWAN†, not independently tuned for SinkGD.
  • Global learning rate for SinkGD = 0.02
    Used together with α=0.05 to obtain effective LR 0.001, matching the optimal Adam LR. Selected by 'lazy tuning', shared with SWAN†.
  • Number of SR-Sinkhorn iterations L = 5
    Chosen based on an ablation on 130M showing marginal but consistent improvement versus L=1; used for all main experiments.
assumptions (4)
  • domain assumption Assumption 3.3: normalized projections under g1 and g2 have constant ℓ2-norm c for all inputs.
    Used for Theorem 3.6 and Lemma 3.5. Both the SWAN norms and the SinkGD row/column norms satisfy it, but it is an extra condition not true for general norms.
  • domain assumption The fixed-point of alternating row and column normalization yields an effective descent direction for LLM training.
    No convergence guarantee of the optimizer to a loss minimum is provided; the paper relies on the empirical observation that normalized instantaneous gradients work as well as Adam's preconditioned updates.
  • standard math Sinkhorn's convergence theorem (Franklin & Lorenz 1989) applies to the square-root reparameterization.
    The linear convergence of Algorithm 3 is inherited directly from the Sinkhorn iteration on A = ∇^{⊙2}, which is cited but not re-derived.
  • domain assumption Applying SinkGD to linear layers and Adam to embeddings/norms/output layers is a valid design.
    This hybrid is carried over from GaLore/SWAN and not justified theoretically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gradient Multi-Normalization for Stateless and Scalable LLM Training." pith.science (2026). https://pith.science/paper/2EU7VV3S

@misc{pith2026250206742,
  author       = {Pith},
  title        = {Pith review of: Gradient Multi-Normalization for Stateless and Scalable LLM Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2EU7VV3S}},
  note         = {Machine review of arXiv:2502.06742}
}
read the original abstract

Training large language models (LLMs) typically relies on adaptive optimizers like Adam (Kingma & Ba, 2015) which store additional state information to accelerate convergence but incur significant memory overhead. Recent efforts, such as SWAN (Ma et al., 2024) address this by eliminating the need for optimizer states while achieving performance comparable to Adam via a multi-step preprocessing procedure applied to instantaneous gradients. Motivated by the success of SWAN, we introduce a novel framework for designing stateless optimizers that normalizes stochastic gradients according to multiple norms. To achieve this, we propose a simple alternating scheme to enforce the normalization of gradients w.r.t these norms. We show that our procedure can produce, up to an arbitrary precision, a fixed-point of the problem, and that SWAN is a particular instance of our approach with carefully chosen norms, providing a deeper understanding of its design. However, SWAN's computationally expensive whitening/orthogonalization step limit its practicality for large LMs. Using our principled perspective, we develop of a more efficient, scalable, and practical stateless optimizer. Our algorithm relaxes the properties of SWAN, significantly reducing its computational cost while retaining its memory efficiency, making it applicable to training large-scale models. Experiments on pre-training LLaMA models with up to 1 billion parameters demonstrate a 3X speedup over Adam with significantly reduced memory requirements, outperforming other memory-efficient baselines.

Figures

Figures reproduced from arXiv: 2502.06742 by the authors.

Figure 1
Figure 1. SinkGD performance preview on LLM pretraining. (a) and (b): Comparison of the test perplexities obtained by Adam (Kingma & Ba, 2015; Zhao et al., 2024a), SWAN (Ma et al., 2024), Apollo (Zhu et al., 2024), and our proposed SinkGD (Algorithm 4) on 1B LLaMA pretraining task with C4 dataset. All loss curves of Adam and Apollo-mini are reproduced from the corresponding opensource codes. We also compare with their officia… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Low-rank Momentum Factorization for Memory Efficient Training

    cs.LG 2025-07 reject novelty 6.0 of 10

    MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.

Reference graph

Works this paper leans on

51 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    R., and Hinton, G

    Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv, abs/1607.06450, 2016. URL https://api.semanticscholar.org/CorpusID:8236317

    arXiv 2016

  2. [2]

    Iterative bregman projections for regularized transportation problems

    Benamou, J.-D., Carlier, G., Cuturi, M., Nenna, L., and Peyr \'e , G. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing, 37 0 (2): 0 A1111--A1138, 2015

    work page 2015

  3. [3]

    and Newhouse, L

    Bernstein, J. and Newhouse, L. Old optimizer, new norm: An anthology. arXiv preprint arXiv:2409.20325, 2024

    arXiv 2024

  4. [4]

    signsgd: Compressed optimisation for non-convex problems

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp.\ 560--569. PMLR, 2018

    2018

  5. [5]

    Proximal alternating linearized minimization for nonconvex and nonsmooth problems

    Bolte, J., Sabach, S., and Teboulle, M. Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Mathematical Programming, 146 0 (1): 0 459--494, 2014

    work page 2014

  6. [6]

    Distributed optimization and statistical learning via the alternating direction method of multipliers

    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J., et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning , 3 0 (1): 0 1--122, 2011

    work page 2011

  7. [7]

    Stochastic spectral descent for restricted boltzmann machines

    Carlson, D., Cevher, V., and Carin, L. Stochastic spectral descent for restricted boltzmann machines. In Artificial Intelligence and Statistics, pp.\ 111--119. PMLR, 2015

    work page 2015

  8. [8]

    and Pock, T

    Chambolle, A. and Pock, T. A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of mathematical imaging and vision, 40: 0 120--145, 2011

    work page 2011

Show all 51 references
  1. [10]

    Fira: Can we achieve full-rank training of llms under low-rank constraint?, 2024 b

    Chen, X., Feng, K., Li, C., Lai, X., Yue, X., Yuan, Y., and Wang, G. Fira: Can we achieve full-rank training of llms under low-rank constraint?, 2024 b . URL https://arxiv.org/abs/2410.01623

    2024

  2. [11]

    Symbolic discovery of optimization algorithms

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024 c

    2024

  3. [12]

    and Mehta, H

    Cutkosky, A. and Mehta, H. Momentum improves normalized sgd. In International conference on machine learning, pp.\ 2260--2268. PMLR, 2020

    2020

  4. [13]

    On hilbert’s metric for simplices

    De La Harpe, P. On hilbert’s metric for simplices. Geometric group theory, 1: 0 97--119, 1993

    1993

  5. [14]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al - Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roz...

    2024 arXiv

  6. [15]

    Dykstra, R. L. An algorithm for restricted least squares regression. Journal of the American Statistical Association, 78 0 (384): 0 837--842, 1983

    1983

  7. [16]

    and Lorenz, J

    Franklin, J. and Lorenz, J. On the scaling of multidimensional matrices. Linear Algebra and its applications, 114: 0 717--735, 1989

    1989

  8. [17]

    Eigenvalue-corrected natural gradient based on a new approximation

    Gao, K., Huang, Z.-H., Liu, X., Wang, M., Wang, S., Wang, Z., Xu, D., and Yu, F. Eigenvalue-corrected natural gradient based on a new approximation. Asia-Pacific Journal of Operational Research, 40 0 (01): 0 2340005, 2023

    2023

  9. [18]

    Shampoo: Preconditioned stochastic tensor optimization, 2018

    Gupta, V., Koren, T., and Singer, Y. Shampoo: Preconditioned stochastic tensor optimization, 2018. URL https://arxiv.org/abs/1802.09568

    2018 arXiv

  10. [19]

    Flora: Low-rank adapters are secretly gradient compressors

    Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. ArXiv, abs/2402.03293, 2024. URL https://api.semanticscholar.org/CorpusID:267412117

    2024 arXiv

  11. [20]

    Beyond convexity: Stochastic quasi-convex optimization

    Hazan, E., Levy, K., and Shalev-Shwartz, S. Beyond convexity: Stochastic quasi-convex optimization. Advances in neural information processing systems, 28, 2015

    2015

  12. [21]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  13. [22]

    Iterative normalization: Beyond standardization towards efficient whitening

    Huang, L., Zhou, Y., Zhu, F., Liu, L., and Shao, L. Iterative normalization: Beyond standardization towards efficient whitening. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 4874--4883, 2019

    2019

  14. [23]

    Muon: An optimizer for hidden layers in neural networks, 2024

    Jordan, K., Jin, Y., Boza, V., You, J., Cecista, F., Newhouse, L., and Bernstein, J. Muon: An optimizer for hidden layers in neural networks, 2024. URL https://kellerjordan.github.io/posts/muon/

    2024

  15. [24]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015

    2015

  16. [25]

    A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B

    Korthikanti, V. A., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models. Proceedings of Machine Learning and Systems, 5: 0 341--353, 2023

    2023

  17. [26]

    W., and Schmidt, M

    Kunstner, F., Chen, J., Lavington, J. W., and Schmidt, M. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. arXiv preprint arXiv:2304.13960, 2023

    2023 arXiv

  18. [27]

    Heavy-tailed class imbalance and why adam outperforms gradient descent on language models

    Kunstner, F., Yadav, R., Milligan, A., Schmidt, M., and Bietti, A. Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. arXiv preprint arXiv:2402.19449, 2024

    2024 arXiv

  19. [28]

    Lewis, A. S. and Malick, J. Alternating projections on manifolds. Mathematics of Operations Research, 33 0 (1): 0 216--234, 2008

    2008

  20. [29]

    Towards faster training of global covariance pooling networks by iterative matrix square root normalization

    Li, P., Xie, J., Wang, Q., and Gao, Z. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 947--955, 2018

    2018

  21. [30]

    Relora: High-rank training through low-rank updates

    Lialin, V., Shivagunde, N., Muckatira, S., and Rumshisky, A. Relora: High-rank training through low-rank updates. In International Conference on Learning Representations, 2023. URL https://api.semanticscholar.org/CorpusID:259836974

    2023

  22. [31]

    Swan: Sgd with normalization and whitening enables stateless llm training, 2024

    Ma, C., Gong, W., Scetbon, M., and Meeds, E. Swan: Sgd with normalization and whitening enables stateless llm training, 2024. URL https://arxiv.org/abs/2412.13148

    2024 arXiv

  23. [32]

    Decomposition through formalization in a product space

    Pierra, G. Decomposition through formalization in a product space. Mathematical Programming, 28: 0 96--115, 1984

    1984

  24. [33]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020. URL http://jmlr.org/paper...

    2020

  25. [34]

    Zero: Memory optimizations toward training trillion parameter models

    Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020

    2020

  26. [35]

    Rockafellar, R. T. Conjugate duality and optimization. SIAM, 1974

    1974

  27. [36]

    A relationship between arbitrary positive matrices and doubly stochastic matrices

    Sinkhorn, R. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics, 35 0 (2): 0 876--879, 1964

    1964

  28. [37]

    and Knopp, P

    Sinkhorn, R. and Knopp, P. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21 0 (2): 0 343--348, 1967

    1967

  29. [38]

    Fast differentiable matrix square root and inverse square root

    Song, Y., Sebe, N., and Wang, W. Fast differentiable matrix square root and inverse square root. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (6): 0 7367--7380, 2022

    2022

  30. [39]

    Tibshirani, R. J. Dykstra's algorithm, admm, and coordinate descent: Connections, insights, and extensions. Advances in Neural Information Processing Systems, 30, 2017

    2017

  31. [40]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  32. [41]

    Functional operators: Measures and integrals, volume 1

    Von Neumann, J. Functional operators: Measures and integrals, volume 1. Princeton University Press, 1950

    1950

  33. [42]

    Watson, G. A. Characterization of the subdifferential of some matrix norms. Linear Algebra Appl, 170 0 (1): 0 33--45, 1992

    1992

  34. [44]

    No more adam: Learning rate scaling at initialization is all you need

    Xu, M., Xiang, L., Cai, X., and Wen, H. No more adam: Learning rate scaling at initialization is all you need. arXiv preprint arXiv:2412.11768, 2024 b

    2024 arXiv

  35. [45]

    Large batch training of convolutional networks

    You, Y., Gitman, I., and Ginsburg, B. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888, 2017

    2017 arXiv

  36. [46]

    Large batch optimization for deep learning: Training bert in 76 minutes

    You, Y., Li, J., Reddi, S., Hseu, J., Kumar, S., Bhojanapalli, S., Song, X., Demmel, J., Keutzer, K., and Hsieh, C.-J. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019

    1904 arXiv

  37. [47]

    and Sennrich, R

    Zhang, B. and Sennrich, R. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  38. [48]

    P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S

    Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020

    2020

  39. [49]

    Adam-mini: Use fewer learning rates to gain more

    Zhang, Y., Chen, C., Li, Z., Ding, T., Wu, C., Ye, Y., Luo, Z.-Q., and Sun, R. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793, 2024

    2024 arXiv

  40. [50]

    A., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y

    Zhao, J., Zhang, Z. A., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient llm training by gradient low-rank projection. ArXiv, abs/2403.03507, 2024 a . URL https://api.semanticscholar.org/CorpusID:268253596

    2024 arXiv

  41. [51]

    Deconstructing what makes a good optimizer for language models

    Zhao, R., Morwani, D., Brandfonbrener, D., Vyas, N., and Kakade, S. Deconstructing what makes a good optimizer for language models. arXiv preprint arXiv:2407.07972, 2024 b

    2024 arXiv

  42. [52]

    Z., Wang, Z., and Lee, J

    Zhu, H., Zhang, Z., Cong, W., Liu, X., Park, S., Chandra, V., Long, B., Pan, D. Z., Wang, Z., and Lee, J. Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arXiv:2412.05270, 2024

    2024 arXiv

  43. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.