Pith. sign in

REVIEW 5 major objections 5 minor 45 references

Improving Adaptive Moment Optimization via Preconditioner Diagonalization

T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that rotating gradients into a basis where their covariance is nearly diagonal—via periodic full-rank SVD—makes Adam's diagonal preconditioner markedly more accurate, yielding a 1.8–2x speedup on large language model…

desk verdict Useful SOAP simplification with plausible but unverified speedups; the mechanism argument and convergence proof both have holes. read the letter →

arxiv 2502.07488 v1 pith:QRP64NFL submitted 2025-02-11 cs.LG

Son Nguyen , Bo Liu , Lizhang Chen , Qiang Liu This is my paper · ORCID
classification cs.LG
keywords adaptivemomentestimationpreconditionerdiagonalizationsingularvaluedecompositiongradientprojectionLLaMApretrainingAdamoptimizerAdafactorHamiltoniandescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that Adam's diagonal second-moment preconditioner, which ignores gradient covariance, can be made far more accurate without paying the full cost of second-order optimization. The maneuver is to rotate each gradient into a basis in which its covariance is approximately diagonal—the basis is obtained by an SVD of the gradient, refreshed every T steps—then run the usual adaptive-moment updates in that rotated space and project the result back, since the rotation is an invertible reparameterization of the network. On LLaMA-style pretraining on C4, the paper reports that the resulting optimizers AdaDiag and AdaDiag++ match Adam's validation perplexity with about half the steps, a 1.8–2x sample-efficiency speedup, with consistent convergence gains on ImageNet classification. The same diagonalization can be layered onto memory-efficient optimizers such as Adafactor and Hfac, so the speedup is not purchased with Adam-scale memory. The sympathetic reader cares because the proposal attacks a real weakness of Adam—discarded gradient correlations—while keeping the update simple enough to drop into existing training pipelines.

What carries the argument

The load-bearing object is a periodic full-rank singular-value projection. Every T steps, AdaDiag extracts orthogonal matrices $P_t, Q_t$ from the SVD of the current gradient and uses them to rotate subsequent gradients ($\tilde{G}_t = P_t^\top G_t$ for AdaDiag, $P_t^\top G_t Q_t$ for AdaDiag++) before feeding them into the standard Adam moment accumulators. The identity behind the method is that this rotation diagonalizes the covariance: $C(P^\top G Q) = (Q \otimes P)^\top C(G)(Q \otimes P) = \mathrm{vec}(\Sigma)\mathrm{vec}(\Sigma)^\top$, which is a diagonal matrix because $\Sigma$ is diagonal. Full rank of $P, Q$ makes the reparameterization invertible, so the update can be mapped back to the original parameters; it also distinguishes the method from GaLore's truncated projection, which cannot preserve this diagonal structure.

What would settle it

Run AdaDiag with a random orthogonal matrix drawn once and kept fixed, instead of the periodic SVD basis, on the same LLaMA-60M/C4 setup. The paper's diagonalization rationale predicts this control should lose AdaDiag's speedup and land near Adam, because a generic rotation does not make the gradient covariance diagonal. If the random-rotation control matches AdaDiag's perplexity curve, then the speedup is not due to preconditioner diagonalization and the mechanism needs revision.

Watch

Extended reading notes

Core claim

The central claim is that Adam's diagonal approximation of the second-moment preconditioner leaves a large amount of curvature information on the table, and that information can be recovered by a change of basis rather than by approximating a large matrix. For a gradient matrix $G_\tau$ of shape $m \times n$, the empirical preconditioner is $C(G_\tau) = \mathrm{vec}(G_\tau)\mathrm{vec}(G_\tau)^\top$; Adam keeps only its diagonal. The paper shows that under the SVD $G_\tau = P_\tau \Sigma_\tau Q_\tau^\top$, the fully diagonalized object is $C(\tilde{G}_\tau)$ with $\tilde{G}_\tau = P_\tau^\top G_\tau Q_\tau$, because $C(\tilde{G}_\tau) = (Q_\tau \otimes P_\tau)^\top C(G_\tau)(Q_\tau \otimes P_\tau) = \mathrm{vec}(\Sigma_\tau)\mathrm{vec}(\Sigma_\tau)^\top$, which is nearly diagonal since $\Sigma_\tau$ is diagonal. Since $P_\tau, Q_\tau$ are full-rank orthogonal matrices, the transformation is invertible and is equivalent to reparameterizing the weight as $\tilde{W} = P^\top W Q$, so the rotated-space update can be projected back to the original parameters. The paper's empirical thesis is that with a periodic refresh of $P, Q$ every $T \approx 200$–$500$ steps, this preconditioner diagonalization substantially accelerates Adam: 1.8–2x speedups on LLaMA models, final-perplexity improvements on C4, and convergence gains on ImageNet1k with ResNets and ViTs.

Load-bearing premise

The method's benefit rests on the subspace-stability premise that within each SVD period every gradient is well approximated by the singular bases computed at the period's start, so projected gradients stay approximately diagonal; if gradient subspaces drift faster than the refresh period T, the preconditioner is no longer diagonal in the rotated space and the method is just Adam with an arbitrary orthogonal rotation.

Editorial extensions

If this is right

  • At a fixed GPU budget, a 1.8–2x sample-efficiency speedup on pretraining means reaching a target perplexity in roughly half the wall-clock steps, effectively doubling the amount of data a practitioner can afford to train on.
  • Because the rotation is an invertible reparameterization, the method is a drop-in replacement for Adam: it changes no forward-pass math and returns weights in the original coordinate system.
  • Layering the same diagonalization onto Adafactor and Hfac gives memory-efficient optimizers whose convergence matches or beats Adam, so the speedup does not force a return to Adam's $2mn$ optimizer-state cost.
  • The Hamiltonian-descent convergence argument covers the continuous-time limit; combined with full-rank projection matrices it rules out the degenerate stationary points that truncated low-rank projections (GaLore) can create.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A random-rotation control experiment, not run in the paper, would separate the diagonalization mechanism from the mere benefit of a fixed orthogonal coordinate frame.
  • The T=10k–20k ablation results hint that the optimal refresh period may be much longer than 200–500 on very stable tasks, possibly allowing a one-time rotation; this is an extrapolation, not a paper claim.
  • The stated equivalence with SOAP suggests that the only meaningful difference between the two algorithms is memory cost, so a large-scale comparison of AdaDiag vs SOAP under equal memory budgets would isolate where each method's overhead matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes AdaDiag and AdaDiag++, adaptive optimizers that periodically compute the full SVD of the current minibatch gradient, use the singular vector matrices to rotate the gradient into a new coordinate system, run Adam-style exponential moving averages of the rotated first and second moments, and then project the update back to the original parameter space. The authors argue that this rotation approximately diagonalizes the full-matrix preconditioner of Eq. (3), allowing a diagonal approximation in the rotated space while retaining some off-diagonal gradient correlation information. They report faster convergence than Adam on ImageNet classification with ResNet/ViT and on C4 language modeling with LLaMA-60M/130M/350M, claim a 1.8x-2x speedup in sample efficiency over Adam at matched perplexity, and provide an ODE-based convergence argument via a Hamiltonian descent framework. They also present memory-efficient variants built on Adafactor and Hfac, an ablation over SVD update period T, and a comparison with GaLore for full-rank versus low-rank projection.

Significance. If the empirical speedup claims hold, the proposed method would be practically significant for large-scale pretraining, where even modest sample-efficiency gains matter. The paper also has useful breadth: it tests the method on multiple architectures, integrates with memory-efficient optimizers, and abates the SVD frequency. However, the significance is contingent on the central mechanism being correctly characterized, and on the empirical results being reproducible; the manuscript provides no code, no seeds or error bars, and no direct comparison with SOAP despite admitting that the method is effectively SOAP without accumulations. The theoretical contribution is currently a non-autonomous ODE analysis that does not establish discrete-time convergence under stated assumptions.

major comments (5)
  1. [§3, Eq. (5)-(6) and Figure 1] The central diagonalization claim is not established. For eGτ = Στ with m ≤ n, C(eGτ) = vec(Στ)vec(Στ)^T has nonzero entries σ_i σ_j only in the m×m block corresponding to the nonzero entries of vec(Στ), and the off-diagonal entries σ_i σ_j are of the same order as the diagonal entries σ_i^2 whenever the singular values are comparable; the matrix is therefore not diagonally dominant. The histograms in Figures 1 and 12 count off-diagonal entries and are dominated by the many structural zeros, so they do not measure the relative energy of the off-diagonal block of the EMA preconditioner. The paper should report the ratio of off-diagonal to diagonal Frobenius energy of the actual preconditioner eC_t in Eq. (6), and should compare against a control with random orthogonal projections before attributing the speedup to diagonalization.
  2. [§4 and Appendix B] The convergence argument relies on LaSalle's invariance principle for the non-autonomous ODE (12) with time-varying or piecewise-constant projection matrices P_t and Q_t. The authors do not prove that the ODE trajectories are bounded, that the system is autonomous on any interval, or that the discrete-time Algorithm 1 is a faithful discretization whose limit points satisfy the ODE invariance condition. A formal theorem with explicit assumptions (e.g., coercivity of L, boundedness of trajectories, regularity of P_t and Q_t) is needed; as written, the claim that all trajectories converge to local optima is not supported.
  3. [§5.2 and Appendix C] The central speedup claim rests on single runs without seeds or error bars, and no code is provided. The learning-rate tuning description in Appendix C is ambiguous: the text says the learning rate was tuned over {0.003, 0.001, 0.0003, 0.0001} and the optimal value selected based on validation perplexity, but Table 6 lists one value per model. It is unclear whether the Adam baselines received the same tuning budget and selection procedure, and whether the reported AdaDiag gains are robust to hyperparameter choice.
  4. [§5.3 and §3.3] The manuscript explicitly states in §5.3 that 'we found no significant performance differences relative to our algorithms' for SOAP, and §3.3 describes the proposal as 'effectively equivalent to SOAP without accumulations,' yet no SOAP results or comparison are shown. Because SOAP is a concurrent prior method, the central claim that AdaDiag substantially improves over Adam requires a SOAP baseline to establish what the SVD projection adds beyond the existing eigendecomposition-based approach.
  5. [§5.3, Figure 7] The ablation over T (Figure 7) shows that T=2 and T=10 degrade performance while T=10k and T=20k improve it, but this does not isolate whether the benefit comes from SVD alignment. A control using random orthogonal projection matrices with the same period would test whether the speedup is due to diagonalization of the preconditioner or merely to applying Adam in a rotated coordinate system. Without such a control, the mechanism proposed in §3 remains untested.
minor comments (5)
  1. [Algorithm 1] The algorithm pseudocode is ambiguous: the line 'eGt = P⊤t Gt eGt = P⊤t GtQt' assigns eGt twice, and the two-sided update is not clearly separated from the one-sided update.
  2. [Eq. (8)] Equation (8) reads 'fWt+1 = fWt+1 − ηt ...', which should be 'fWt+1 = fWt − ηt ...'.
  3. [§4] The text says H is 'monotonically non-decreasing' along the ODE trajectory, but the subsequent derivation shows dH/dt ≤ 0; it should say non-increasing.
  4. [Figures 1, 12, and 13] The histograms are truncated and not normalized, which makes visual comparisons of sparsity difficult; reporting the share of off-diagonal energy in log scale or as a ratio would be more informative.
  5. [Table 3] The memory comparison shows that AdaDiag++ uses more than twice the optimizer-state memory of Adam for 350M parameters (3.03G vs. 1.44G), so the claim of 'manageable computational overhead' should be qualified to distinguish compute cost from memory cost.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional support for the diagonalization mechanism; central speedup claim is externally benchmarked and not circular.

  1. self definitional [Section 3, equation following 'Suppose we have Gτ = Pτ Στ Q⊤τ' and Figure 1]
    "Since Στ is a diagonal matrix, we have vec(Στ)vec(Στ)⊤ is almost diagonal (off-diagonal elements are mostly zero). ... eGτ ≜ P⊤τ Gτ Qτ = Στ. This rotation aligns the gradient eGt with coordinate axes and consequently induces a roughly diagonal structure on C(eGτ)."

    The 'almost diagonal' structure of C(eGτ) is presented as evidence supporting the method, but eGτ is defined as the SVD middle factor Στ, so its vectorization has at most min(m,n) nonzero entries by construction. The histograms in Figure 1 therefore display a property directly manufactured by the transformation, not an independent empirical finding. This is a supporting intuition rather than the headline result; the main speedup claim is measured against external baselines (Adam, Shampoo) and does not reduce to this construction.

full rationale

The paper's central empirical claim — AdaDiag achieving roughly 1.8–2x speedup over Adam when pretraining LLaMA models on C4 — is validated against external baselines and is therefore not circular. The only step that is close to definitional is the claim that the projected gradient's covariance matrix becomes 'almost diagonal' because the projected gradient is exactly the diagonal SVD factor at projection iterations; that observation is a mathematical consequence of the chosen transformation, not an independent source of evidence. However, it is not the load-bearing part of the paper's main contribution. The subspace-stability assumption over each period is imported from GaLore (an external citation), not from the authors' own prior work. The convergence analysis constructs its own Hamiltonian/Lyapunov function and carries out the proof in Appendix B; the Hamiltonian-descent framework is attributed to Maddison et al. and the self-citations to Hfac and Online Subspace Descent are contextual rather than load-bearing. No fitted parameter is renamed as a prediction, and the speedup is not forced by any self-citation chain. Score 2 reflects the one definitional supporting observation; the central result retains independent empirical content.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The algorithm introduces no new physical or mathematical entities. The load-bearing assumptions are the subspace-stability premise from GaLore, the continuous-time ODE idealization with fixed projections, and the unproven boundedness needed for LaSalle. The SVD update period and learning rates are hand-tuned free parameters that affect the reported results.

free parameters (2)
  • SVD update period T = 500 (image), 200 (language)
    Chosen per task; Figure 7 shows performance varies strongly with T, with best values around 200-500 and strong degradation at T=2 or 10.
  • Learning rates for LLaMA runs = 0.003 (60M), 0.001 (130M/350M)
    Appendix C reports tuning for AdaDiag over {0.003, 0.001, 0.0003, 0.0001}; it is unclear whether Adam and Shampoo baselines were tuned to the same extent.
assumptions (5)
  • domain assumption Gradient trajectories stay in a persistent low-dimensional subspace: Gτ ≈ Pκ Στ Qκ^T for all τ in a period of length T.
    Introduced in Section 3.1 and borrowed from GaLore; the entire diagonalization argument relies on this empirical subspace stability.
  • domain assumption The discrete optimizer dynamics are well approximated by a continuous-time ODE with fixed projection matrices P_t, Q_t.
    Section 4 and Appendix B define an ODE that omits periodic SVD updates, stochastic gradient noise, and the weight decay term in the actual algorithm.
  • ad hoc to paper LaSalle's invariance principle applies, meaning trajectories are bounded and the dynamics are sufficiently regular despite piecewise-constant projection matrices.
    Appendix B invokes LaSalle without proving boundedness of (W_t, M_t, V_t) or handling discontinuities of P_t, Q_t at SVD update times.
  • standard math The objective L(W; X) is proper, differentiable, and lower bounded.
    Stated at the start of Section 2 as the problem setting for the convergence analysis.
  • domain assumption Gradient distributions are stationary enough for exponential moving averages to approximate expected outer products.
    Section 2.3 says "under the assumption of stationary gradient distribution, we can approximate the expectation by minibatch sampling in conjunction with the exponential moving average technique."

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Adaptive Moment Optimization via Preconditioner Diagonalization." pith.science (2026). https://pith.science/paper/QRP64NFL

@misc{pith2026250207488,
  author       = {Pith},
  title        = {Pith review of: Improving Adaptive Moment Optimization via Preconditioner Diagonalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRP64NFL}},
  note         = {Machine review of arXiv:2502.07488}
}
read the original abstract

Modern adaptive optimization methods, such as Adam and its variants, have emerged as the most widely used tools in deep learning over recent years. These algorithms offer automatic mechanisms for dynamically adjusting the update step based on estimates of gradient statistics. Compared to traditional algorithms like Stochastic Gradient Descent, these adaptive methods are typically more robust to model scale and hyperparameter tuning. However, the gradient statistics employed by these methods often do not leverage sufficient gradient covariance information, leading to suboptimal updates in certain directions of the parameter space and potentially slower convergence. In this work, we keep track of such covariance statistics in the form of a structured preconditioner matrix. Unlike other works, our approach does not apply direct approximations to estimate this matrix. We instead implement an invertible transformation that maps the preconditioner matrix into a new space where it becomes approximately diagonal. This enables a diagonal approximation of the preconditioner matrix in the transformed space, offering several computational advantages. Empirical results show that our approach can substantially enhance the convergence speed of modern adaptive optimizers. Notably, for large language models like LLaMA, we can achieve a speedup of 2x compared to the baseline Adam. Additionally, our method can be integrated with memory-efficient optimizers like Adafactor to manage computational overhead.

Figures

Figures reproduced from arXiv: 2502.07488 by the authors.

Figure 2
Figure 2. Sparsity of one-sided projection. spanning basis. GaLore deployed this idea by periodically applying SVD on the gradients to extract projection matri￾ces. Mathematically, during each period of length T, say [κT,(κ + 1)T], the gradient Gτ can be decomposed as: Gτ ≈ PκΣτQ ⊤ κ ∀τ ∈ [κT,(κ + 1)T] where Pκ, _, Q ⊤ κ ← SVD(GκT ) are kept the same through￾out the period. We can adapt this assumption to our framework. In th… view at source ↗
Figure 3
Figure 3. Illustration of network reparameterization induced [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Top-1 Accuracy of optimizers in pretraining ResNet50, ViT-B/32, and ViT-S/16 from scratch on the ImageNet1k. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Training progression for pre-training LLaMA models on C4 dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: An ablation study on why the full-rank SVD matters. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Performances of AdaDiag on LLaMA-60M and [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Top-1 Accuracy of memory-efficient optimizers in pre-training ResNet50, ViT-B/32, and ViT-S/16 from scratch on [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Training progression of Adam and AdafacDiag for pre-training LLaMA models on C4 dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Training progression of Adam, Hfac, and HfacDiag for pre-training LLaMA models on C4 dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Top-1 Accuracy of optimizers in pre-training (to the end) ResNet50, ViT-B/32, and ViT-S/16 from scratch on the [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Histograms of off-diagonal elements [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Histograms of off-diagonal elements C(Gτ ) (original) and C(Ge τ ) (GaLore), corresponding to the two first layers of ResNet50 trained on ImageNet1k. Compared to the full-rank case, the low-rank GaLore projection (r = min{m, n}/2) does not exhibit a discernible sparsi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 15 canonical work pages

  1. [1]

    Fisher information and natural gradient learning in random deep networks

    Shun-ichi Amari, Ryo Karakida, and Masafumi Oizumi. Fisher information and natural gradient learning in random deep networks. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 694--702. PMLR, 2019

    work page 2019

  2. [2]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020

    arXiv 2002

  3. [3]

    Numerical optimization: theoretical and practical aspects

    Joseph-Fr \'e d \'e ric Bonnans, Jean Charles Gilbert, Claude Lemar \'e chal, and Claudia A Sagastiz \'a bal. Numerical optimization: theoretical and practical aspects. Springer Science & Business Media, 2006

    work page 2006

  4. [4]

    Practical gauss-newton optimisation for deep learning

    Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. In International Conference on Machine Learning, pages 557--565. PMLR, 2017

    work page 2017

  5. [5]

    Lion secretly solves constrained optimization: As lyapunov predicts

    Lizhang Chen, Bo Liu, Kaizhao Liang, and Qiang Liu. Lion secretly solves constrained optimization: As lyapunov predicts. arXiv preprint arXiv:2310.05898, 2023

    arXiv 2023

  6. [6]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems, 36, 2024

    work page 2024

  7. [7]

    Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with a reduced search space. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3008--3017, 2019. URL https://api.semanticscholar.org/CorpusID:208006202

    work page 2020

  8. [8]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12 0 (7), 2011

    2011

Show all 45 references
  1. [9]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126--1135. PMLR, 2017

    2017

  2. [10]

    Practical methods of optimization

    Roger Fletcher. Practical methods of optimization. John Wiley & Sons, 2000

    2000

  3. [11]

    Global convergence of stochastic gradient hamiltonian monte carlo for nonconvex stochastic optimization: Nonasymptotic performance bounds and momentum-based acceleration

    Xuefeng Gao, Mert G \"u rb \"u zbalaban, and Lingjiong Zhu. Global convergence of stochastic gradient hamiltonian monte carlo for nonconvex stochastic optimization: Nonasymptotic performance bounds and momentum-based acceleration. Operations Research, 70 0 (5): 0 2931--2947, 2022

    2022

  4. [12]

    Fast approximate natural gradient descent in a kronecker factored eigenbasis

    Thomas George, C \'e sar Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018

    2018

  5. [13]

    Low-rank gradient approximation for memory-efficient on-device training of deep neural network

    Mary Gooneratne, Khe Chai Sim, Petr Zadrazil, Andreas Kabel, Fran c oise Beaufays, and Giovanni Motta. Low-rank gradient approximation for memory-efficient on-device training of deep neural network. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Sig...

    2020

  6. [14]

    A kronecker-factored approximate fisher matrix for convolution layers

    Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pages 573--582. PMLR, 2016

    2016

  7. [15]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842--1850. PMLR, 2018

    2018

  8. [16]

    Gradient descent happens in a tiny subspace

    Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018

    2018 arXiv

  9. [17]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  10. [18]

    Federated optimization: Distributed machine learning for on-device intelligence

    Jakub Kone c n \`y , H Brendan McMahan, Daniel Ramage, and Peter Richt \'a rik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016

    2016 arXiv

  11. [19]

    Heavy-tailed class imbalance and why adam outperforms gradient descent on language models

    Frederik Kunstner, Robin Yadav, Alan Milligan, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. arXiv preprint arXiv:2402.19449, 2024

    2024 arXiv

  12. [20]

    Federated learning: Challenges, methods, and future directions

    Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37 0 (3): 0 50--60, 2020

    2020

  13. [21]

    Memory-efficient llm training with online subspace descent

    Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu. Memory-efficient llm training with online subspace descent. arXiv preprint arXiv:2408.12857, 2024

    2024 arXiv

  14. [22]

    Sophia: A scalable stochastic second-order optimizer for language model pre-training

    Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342, 2023

    2023 arXiv

  15. [23]

    Rotate your networks: Better weight consolidation and less catastrophic forgetting

    Xialei Liu, Marc Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262--2268. IEEE, 2018

    2018

  16. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  17. [25]

    Hamiltonian descent methods

    Chris J Maddison, Daniel Paulin, Yee Whye Teh, Brendan O'Donoghue, and Arnaud Doucet. Hamiltonian descent methods. arXiv preprint arXiv:1809.05042, 2018

    2018 arXiv

  18. [26]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  19. [27]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pages 2408--2417. PMLR, 2015

    2015

  20. [28]

    H-fac: Memory-efficient optimization with factorized hamiltonian descent

    Son Nguyen, Lizhang Chen, Bo Liu, and Qiang Liu. H-fac: Memory-efficient optimization with factorized hamiltonian descent. arXiv preprint arXiv:2406.09958, 2024

    2024 arXiv

  21. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

    2020

  22. [30]

    On the convergence of adam and beyond

    Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019

    1904 arXiv

  23. [31]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  24. [32]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596--4604. PMLR, 2018

    2018

  25. [33]

    u rb \"u zbalaban, Thanh Huy Nguyen, Ga \

    Umut S im s ekli, Mert G \"u rb \"u zbalaban, Thanh Huy Nguyen, Ga \"e l Richard, and Levent Sagun. On the heavy-tailed theory of stochastic gradient descent for deep neural networks. arXiv preprint arXiv:1912.00018, 2019

    1912 arXiv

  26. [34]

    On the origin of implicit regularization in stochastic gradient descent

    Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176, 2021

    2021 arXiv

  27. [35]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818--2826, 2016

    2016

  28. [36]

    Recent advances in stochastic gradient descent in deep learning

    Yingjie Tian, Yuqi Zhang, and Haibin Zhang. Recent advances in stochastic gradient descent in deep learning. Mathematics, 11 0 (3): 0 682, 2023

    2023

  29. [37]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  30. [38]

    Soap: Improving and stabilizing shampoo using adam

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    2024 arXiv

  31. [39]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  32. [40]

    mixup: Beyond empirical risk minimization

    Hongyi Zhang, Moustapha Ciss \'e , Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv, abs/1710.09412, 2017. URL https://api.semanticscholar.org/CorpusID:3162051

    2017 arXiv

  33. [41]

    Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020

    Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 0 15383--15393, 2020

    2020

  34. [42]

    Adam can converge without any modification on update rules

    Yushun Zhang, Congliang Chen, Naichen Shi, Ruoyu Sun, and Zhi-Quan Luo. Adam can converge without any modification on update rules. Advances in neural information processing systems, 35: 0 28386--28399, 2022

    2022

  35. [43]

    Why transformers need adam: A hessian perspective

    Yushun Zhang, Congliang Chen, Tian Ding, Ziniu Li, Ruoyu Sun, and Zhi-Quan Luo. Why transformers need adam: A hessian perspective. arXiv preprint arXiv:2402.16788, 2024

    2024 arXiv

  36. [44]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024

    2024 arXiv

  37. [45]

    Towards theoretically understanding why sgd generalizes better than adam in deep learning

    Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33: 0 21285--21296, 2020

    2020

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.