Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that choosing a structural family to approximate the per-layer Fisher information matrix, then taking the square-root natural-gradient update, is a unified recipe that reproduces many existing LLM optimizers and yields…

desk verdict A genuinely useful unification of optimizer design with two plausible new optimizers, but the headline speed-up numbers are internally inconsistent and need fixing. read the letter →

arxiv 2502.07752 v2 pith:ACZWA73V submitted 2025-02-11 cs.LG cs.AIstat.ML

Wenbo Gong , Meyer Scetbon , Chao Ma , Edward Meeds This is my paper · ORCID
classification cs.LGcs.AIstat.ML MSC 68T07
keywords FisherinformationmatrixstructuredapproximationLLMoptimizerdesignmemory-efficienttraininglow-rankRACSAlicenaturalgradientdescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to turn LLM optimizer design into a structured approximation problem: pick a family of matrices, approximate the per-layer Fisher information matrix $F = E[\vec g \vec g^T]$ under the Frobenius norm, and derive the corresponding square-root natural-gradient update. It claims that many existing optimizers, including Adam, Shampoo, gradient normalization and whitening, and SOAP, are special cases of this recipe, each with a different structural assumption. From that viewpoint it derives two new memory-efficient optimizers: RACS, which scales gradient rows and columns with two positive diagonal matrices and uses SGD-like memory, and Alice, a low-rank extension of the more general Eigen-Adam optimizer that couples low-rank tracking, subspace switching, and a compensation term. On LLaMA pretraining up to 1B parameters, the paper reports that Alice reaches a given evaluation perplexity more than twice as fast as Adam, and RACS matches or beats stronger memory-efficient baselines at roughly SGD memory. If these claims hold, the framework gives a systematic way to design optimizers with a controlled trade-off between memory, compute, and approximation quality.

What carries the argument

The central object is the structured Fisher information matrix approximation: for a layer's vectorized gradient $\vec g$, define $F = E[\vec g \vec g^T]$ and approximate it by the closest matrix in a structural family $\mathcal{H}$ under the Frobenius norm, then use the square-root inverse $\tilde F^{-1/2}$ as a preconditioner. The family $\mathcal{H}$ is the design variable: diagonal matrices give Adam, Kronecker products give Shampoo, block-diagonal shared-eigenbasis matrices give Eigen-Adam, and an eigenbasis with diagonal scaling gives SOAP. RACS uses the family of positive diagonal matrices $S \otimes Q$ and solves the approximation by a fixed-point iteration equivalent to a power method on $E[G^{\odot 2}]$, producing row-and-column scaling with only $m+n+1$ memory. Alice applies a three-step low-rank extension to Eigen-Adam: low-rank tracking of the projected second moment, subspace switching that mixes the leading eigenbasis with randomly sampled complement basis vectors, and an analytically optimal compensation term that restores the full-rank update direction. The machinery that carries the argument is the equivalence between solving the Frobenius-norm FIM approximation and the resulting preconditioned update.

What would settle it

Run Alice on a task where the gradient covariance eigenspace is measured to rotate substantially within the $K=200$ update block; Proposition 4 predicts the subspace-switching term must then dominate, so disabling switching should degrade perplexity measurably. For RACS, check whether any entry of the empirical $E[G^{\odot 2}]$ is zero or near zero during early training; Proposition 3's fixed-point convergence to positive singular vectors would fail, and the observed speedup over Adam would lack its stated support.

Watch

Extended reading notes

Core claim

The central claim is that the empirical Fisher information matrix $F = E[\vec g \vec g^T]$ for a layer can be approximated by solving $\min_{\tilde F \in \mathcal{H}} \|\tilde F - F\|_F^2$ for a chosen structure $\mathcal{H}$, and that the square-root natural-gradient update with $\tilde F^{-1/2}$ reproduces the update rules of well-known optimizers: the diagonal structure yields Adam, the Kronecker-product structure gives Shampoo's preconditioner as an upper-bound minimizer, and shared- or two-sided eigen-space structures yield Eigen-Adam and SOAP. The paper further claims two concrete design principles: first, choose a structure that generalizes a simple operator while keeping memory low, which leads to RACS, whose two-sided scaling $S \otimes Q$ is solved by a fixed-point iteration that converges to the principal singular vectors of $E[G^{\odot 2}]$; second, for generally structured optimizers, convert them to low-rank form with three components, low-rank tracking, subspace switching, and compensation, which produces Alice. The paper's experimental assertion is that Alice achieves more than $2\times$ faster convergence than Adam in steps and effective throughput on LLaMA pretraining, and RACS gives strong 1B results with SGD-like memory.

Load-bearing premise

The theoretical derivation of both optimizers assumes the gradient statistics are well behaved: entries of the expected squared gradient stay strictly positive for RACS, and gradients keep sharing the tracked eigenbasis within each update block for Alice, so if real training violates those distributional conditions, the FIM-approximation rationale for the speedups weakens even if the optimizers still work empirically.

Editorial extensions

If this is right

  • Adam, Shampoo, gradient normalization and whitening, Eigen-Adam, and SOAP are not separate heuristics but points on one design curve; improving an optimizer can be reinterpreted as changing its structural family.
  • RACS uses $O(m+n)$ memory per matrix layer, so it can replace Adam states in large-model pretraining without storing second-moment matrices.
  • Alice reaches a given LLaMA evaluation perplexity in fewer than half the training steps of Adam at roughly GaLore-level memory, so it converts memory savings into wall-clock speedup rather than only footprint reduction.
  • The low-rank extension framework applies to SOAP as well; deriving Alice's counterpart for two-sided eigenbases is a direct next step, which the paper identifies as future work.
  • Alice's performance degrades only slightly when the last layer is trained without full-rank Adam, indicating that the method does not rely on that common low-rank-optimizer crutch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply the same three-step low-rank recipe to SOAP; if it succeeds, the framework would cover the full two-sided eigenbasis design space at low memory.
  • The FIM viewpoint implies optimizer comparisons could be guided by measuring $\|\tilde F - F\|_F$ for RACS, Alice, and Adam during real training, and checking whether the reported speedups track approximation error.
  • The subspace-switching ablation reveals an exploration-exploitation trade-off in low-rank optimizers: keeping leading eigenvectors stabilizes progress, while mixing in complement basis prevents stagnation; this could be formalized as a bandit-style schedule instead of random sampling.
  • Because RACS's fixed-point iteration is essentially a power method on $E[G^{\odot 2}]$, using randomized or rank-limited estimates of that matrix could interpolate between RACS and full eigen-decomposition methods.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a unified framework for designing LLM optimizers by approximating the layer-wise Fisher information matrix (FIM) under the Frobenius norm with structural constraints. It shows that Adam, Shampoo, normalization/whitening operators, Eigen-Adam, and SOAP can be cast in this framework (in some cases approximately), and it uses the framework to derive two new optimizers: RACS, a row-and-column-scaled SGD with SGD-like memory, and Alice, a low-rank extension of Eigen-Adam with tracking, subspace switching, and compensation. Experiments on LLaMA pretraining at 60M, 130M, 350M, and 1B/1.3B scales report that RACS and Alice outperform Adam and several memory-efficient baselines, with Alice claimed to converge more than 2x faster than Adam.

Significance. If the claims hold, the paper makes a useful conceptual contribution: it gives a common principled lens for several existing optimizers and a concrete recipe for deriving new memory-efficient ones. The detailed appendix proofs, the explicit structural taxonomy in Table 1, and the extensive ablations of Alice's components (tracking, switching, compensation) are genuine strengths. RACS and Alice are plausible and practical contributions, and the experiments up to 1B parameters are nontrivial. However, the central empirical claim is currently not verifiable because the paper reports internally inconsistent speed-up numbers, and the theoretical recoveries of Shampoo and SOAP are explicitly approximate (upper-bound or one-iteration alternating optimization) rather than exact solutions of the stated Frobenius objective. These issues affect the paper's headline conclusions and must be fixed before the contribution can be assessed reliably.

major comments (4)
  1. [Section 7.1, Table 2] The headline speed-up numbers are internally inconsistent. Table 2 reports Alice speed-ups of 2.22x, 2.00x, 2.45x, and 2.82x for 60M, 130M, 350M, and 1B, while the text in Section 7.1 states that Alice achieves 2.22x, 2.11x, 2.18x, and 2.15x using the cited Adam perplexities. Three of the four values differ, and the abstract's claim of 'better than 2x' is contradicted by the 2.00x entry in Table 2. The effective-throughput numbers are also not consistent with either set: for 1B, an Alice throughput of 45523 with a 2.82x step speed-up would give an effective throughput of about 128,375, not the reported 123,048, while a 2.15x speed-up would give about 97,874. The text further claims 'more than 2x faster convergence in wall-clock time' even though the 130M effective throughput ratio is 141148/82247 = 1.72x. Because this speed-up is the primary quantitative evidence for Alice's headline advantage, the paper must correct the tables/text, specify which Adam baseline is used for each speed-up, and recompute the effective throughput consistently.
  2. [Sections 3.2 and 3.5, Theorem 3.1 and Theorem 3.3] The recovery of Shampoo and SOAP is not an exact solution of the stated Frobenius approximation problem. Theorem 3.1 minimizes an upper bound (Eq. 4), not the objective in Eq. 2, and Theorem 3.3 combines that upper-bound minimization with a one-iteration alternating optimization. The abstract's statement that these optimizers 'can be viewed as solutions to FIM approximation (under the Frobenius norm)' is therefore too strong. The paper should consistently describe these as approximate solutions obtained by minimizing an upper bound or by one-step alternating optimization, and should state clearly that the exactness claim applies only to the diagonal, normalization, whitening, and Eigen-Adam blocks.
  3. [Section 5.2, Proposition 4] The theoretical justification for Alice's subspace-switching step relies on a very strong assumption: that gradients within a time block share the same eigenbasis as the tracked state Q*_ik, which makes the cross terms in Eq. (18) vanish. In real training this will not hold exactly, and when it fails the decomposition gains extra cross terms, so the claim that the residual is U_c Sigma_t U_c^T and that dominant entries in Sigma_t justify mixing in random complement basis is not established. The paper should either weaken this assumption, verify it empirically on the actual training runs, or explicitly present switching as a heuristic inspired by an idealized case.
  4. [Section 4, Proposition 3] The convergence proof of RACS's fixed-point iteration requires E[G^{\odot2}] to contain only strictly positive entries, so that the Perron-Frobenius theorem and the Birkhoff-Hopf contraction argument apply. LLM gradient blocks can contain zeros (e.g., saturated or sparse layers), in which case the assumption fails and the stated convergence of s and q to the principal singular vectors is not guaranteed. The paper should discuss this limitation and indicate what behavior is expected when some entries of E[G^{\odot2}] are zero, or should provide a proof under weaker assumptions.
minor comments (5)
  1. [Section 5.3, Eq. (19) and following text] The sentence says the discarded information is 'Mat(F^{1/2}_c g)', but the displayed decomposition contains the square-root pseudo-inverse with exponent -1/2; the exponent in the text should be corrected to -1/2.
  2. [Section 7.1 and Table 2] The model size is referred to as both '1B' and '1.3B' in different places; the labels should be made consistent so that the reader can map the table columns to the text.
  3. [Section 4, paragraph after Proposition 3] The text states that 'analytic solutions of s, q exist' before describing a five-step fixed-point iteration, but Proposition 3 provides a fixed-point characterization, not a closed-form analytic solution; the wording should be changed to avoid the contradiction.
  4. [Table 1] The table entries are difficult to parse, especially the RACS and Alice structure/computation cells (e.g., 'Approx. AliceS⊗Q'); reformatting the table with clearer notation and separating structural assumptions from computation/memory would improve readability.
  5. [Section 5.2, Algorithm 2] The input to the subspace-switching procedure is called 'Reconstructed state Q', but in Algorithm 2 it is then used as the state passed to the subspace-iteration routine; the notation should clarify whether Q is the low-rank EMA state or its reconstruction.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: FIM-approximation derivations are self-contained and the new optimizers are evaluated against external baselines.

full rationale

The derivation chain is self-contained. Section 3 defines the objective (Eq. 2) and solves it for each stated structural family: Prop. 1 gives Adam's diagonal second moment, Thm. 3.1 gives Shampoo's Kronecker preconditioners by minimizing an explicit upper bound, Prop. 2 gives normalization and whitening, Thm. 3.2 gives Eigen-Adam, and Thm. 3.3 gives SOAP. Each result is an algebraic consequence of the objective plus the stated structure, not an assumption of the target update. RACS and Alice are then derived (Prop. 3, Thm. 5.1) before being measured, and their reported perplexities are empirical outcomes rather than fitted predictions; hyperparameter tuning is standard practice and does not make the convergence claim circular. The only notable self-citation is to the authors' own SWAN work (Ma et al. 2024), used in Sec. 3.3 and Related Work to assert that normalization and whitening can replace Adam's internal states; this is background and context and is not load-bearing for the paper's theoretical claims. One reporting issue should be flagged but does not affect the circularity score: the Alice speed-up values are internally inconsistent (Table 2 row 'Speed-up in steps (Alice)' gives 2.22x, 2.00x, 2.45x, and 2.82x, while Sec. 7.1 states 'Alice achieves 2.22x, 2.11x, 2.18x, and 2.15x'), and the 130M value of exactly 2.00x contradicts the abstract's 'better than 2x faster convergence'. This weakens verifiability of the headline empirical claim, but it is a correctness and reporting concern, not a reduction of the derivation to its inputs.

Assumptions & free parameters 6 free parameters · 9 assumptions · 0 invented entities

The central derivation depends on several modeling choices that are not derived from first principles: the square-root natural-gradient update, the layer independence and EMA estimation of the empirical Fisher matrix, the Frobenius-norm objective, and the approximate solving procedures (upper bounds and one-iteration alternation). The new optimizers additionally rely on strict positivity of the squared-gradient matrix for RACS convergence and on a shared-eigenbasis assumption for Alice's switching justification.

free parameters (6)
  • RACS learning rate λ and scale α = λ=0.02; α=0.05 (60M to 350M), 0.02 (1B)
    Grid-searched per model size (Table 9) and needed for the reported convergence.
  • RACS EMA decay β = 0.9
    Chosen by hand; the ablation shows removing EMA degrades RACS (Fig. 5e).
  • Alice low-rank rank r = 128 (60M), 256 (130M/350M), 512 (1B)
    Grid-searched; controls memory and performance (Table 11).
  • Alice leading basis number l = 40 (60M to 350M), 160 (1B)
    Grid-searched; governs how many leading eigenvectors are kept in switching (Table 11).
  • Alice compensation scale α_c = 0.4 (60M to 350M), 0.2 (1B)
    Grid-searched; weights the compensation term in Algorithm 4.
  • Alice EMA decays β1, β2, β3 and update interval K = β1=0.9, β2=0.9, β3=0.999, K=200
    Chosen by hand following Adam and SOAP settings; the optimizer behavior depends on these.
assumptions (9)
  • domain assumption The square-root inverse natural gradient update W ← W − λ Mat(F^{-1/2} ∇L) is an appropriate optimization geometry (Eq. 1).
    Stated in Sec. 2.2; the entire framework derives optimizers as square-root NGD, not from convergence theory.
  • domain assumption The empirical FIM can be represented as E[g g^T] with EMA mini-batch gradient and layers treated independently.
    Sec. 2.2; layer independence ignores off-diagonal layer interactions.
  • ad hoc to paper Frobenius-norm minimization of FIM approximation is the right objective (Eq. 2).
    Sec. 3; this choice is a modeling assumption, not derived from optimization guarantees.
  • domain assumption For Shampoo, minimizing the upper bound (Eq. 4) is a valid surrogate for Eq. (2).
    Theorem 3.1 uses Powers-Stormer and Hölder inequalities; the bound may be loose.
  • ad hoc to paper One-iteration alternating optimization gives a useful approximation to the FIM solution for Eigen-Adam and SOAP.
    Theorems 3.2 and 3.3 solve a one-iteration alternating problem, not the full constrained minimization; no convergence guarantees are given for this alternating scheme.
  • domain assumption Strict positivity of E[G^2] entries for RACS convergence.
    Prop. 3 and Theorem D.1 rely on Perron-Frobenius and Birkhoff-Hopf contraction for positive matrices.
  • ad hoc to paper Gradients within a time block share the eigenbasis of the tracked tracking state for Alice's switching justification.
    Prop. 4 assumes G_t G_t^T shares the eigenbasis of Q*_ik, an idealized non-stationary approximation.
  • domain assumption Single-sample gradient estimates can replace the expectation E[·] inside the derived updates.
    Used in Proposition 3 discussion and in Algorithms 1 and 4; no unbiasedness or variance analysis is given.
  • ad hoc to paper Norm-growth limiter and random complement-basis sampling are valid stabilizers.
    Algorithm 1 and Algorithm 2 borrow or insert heuristics; their benefit is empirical only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension." pith.science (2026). https://pith.science/paper/ACZWA73V

@misc{pith2026250207752,
  author       = {Pith},
  title        = {Pith review of: Towards Efficient Optimizer Design for LLM via Structured Fisher Approximation with a Low-Rank Extension},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ACZWA73V}},
  note         = {Machine review of arXiv:2502.07752}
}
read the original abstract

Designing efficient optimizers for large language models (LLMs) with low-memory requirements and fast convergence is an important and challenging problem. This paper makes a step towards the systematic design of such optimizers through the lens of structured Fisher information matrix (FIM) approximation. We show that many state-of-the-art efficient optimizers can be viewed as solutions to FIM approximation (under the Frobenius norm) with specific structural assumptions. Building on these insights, we propose two design recommendations of practical efficient optimizers for LLMs, involving the careful selection of structural assumptions to balance generality and efficiency, and enhancing memory efficiency of optimizers with general structures through a novel low-rank extension framework. We demonstrate how to use each design approach by deriving new memory-efficient optimizers: Row and Column Scaled SGD (RACS) and Adaptive low-dimensional subspace estimation (Alice). Experiments on LLaMA pre-training (up to 1B parameters) validate the effectiveness, showing faster and better convergence than existing memory-efficient baselines and Adam with little memory overhead. Notably, Alice achieves better than 2x faster convergence over Adam, while RACS delivers strong performance on the 1B model with SGD-like memory.

Figures

Figures reproduced from arXiv: 2502.07752 by the authors.

Figure 1
Figure 1. 1B LLaMA C4 pretraining evaluation ppl. curve. ”+lm head” represents the last layer is [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Additional LLaMA C4 pretrain performance curve. (a), (b) and (c) represents the 60M, [PITH_FULL_IMAGE:figures/full_fig_p043_2.png] view at source ↗
Figure 3
Figure 3. Throughput of various methods. (a) this reports the absolute throughput, representing [PITH_FULL_IMAGE:figures/full_fig_p044_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The memory footprint of various optimizers. We use token batch size of [PITH_FULL_IMAGE:figures/full_fig_p045_4.png]
Figure 5
Figure 5. Figure 5: The pre-training curve to verify the effectiveness of the design choice. We consider 130M [PITH_FULL_IMAGE:figures/full_fig_p046_5.png]
Figure 6
Figure 6. Figure 6: The cosine similarity between eigenvectors per 200 steps. Since the update interval of [PITH_FULL_IMAGE:figures/full_fig_p047_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    cs.LG 2026-07 conditional novelty 7.0 of 10

    The top-r gradient subspace in GaLore-family optimizers is statistically non-identifiable beyond ~39 of 128 directions, and the fix is to transport optimizer state across refreshes rather than stabilize the basis.

  2. OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

    cs.LG 2026-07 conditional novelty 6.0 of 10

    A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

Reference graph

Works this paper leans on

36 extracted references · 16 canonical work pages · cited by 2 Pith papers

  1. [1]

    Scalable second order optimization for deep learning

    Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018,

    arXiv 2002

  2. [2]

    M1 0 0 0 M2 0 0 0 M3 # Example of Diagv(·) This will stack the vector element into a pure diagonal matrix: Diagv([a11, a22, a33]T ) =

    Example of Diag(·) This will extract the diagonals of a input matrix into a vector: M = " a11 a12 a13 a21 a22 a23 a31 a32 a33 # , Diag(M ) = [a11, a22, a33]T Example of DiagB(·) This simply stack the input matrices sequence into a larger block diagonal matrix: DiagB(M1, M2, M3) = " M1 0 0 0 M2 0 0 0 M3 # Example of Diagv(·) This will stack the vector elem...

    work page 2024

  3. [3]

    From the Theorem D.1, the iterative procedure for Q can be simply obtained by taking the diagonals of M: Q = Diag E GSGT ∥S∥2 F

    Proof. From the Theorem D.1, the iterative procedure for Q can be simply obtained by taking the diagonals of M: Q = Diag E GSGT ∥S∥2 F . Following the same proof strategy of Theorem D.1, we substitute Q into the update of S and re-write it into the vector format. First, let’s rewrite the update of S Si ∝ E[ X j=1 G2 jiQj] =⇒s = P T q ∥q∥2 2 where P = E[G⊙...

    work page 2018

  4. [4]

    Effects of last layer One crucial setup difference during evaluation for low-rank methods is whether the last layer is trained by full-rank Adam or not

    We found this empirical trick boosts the performance of Fira. Effects of last layer One crucial setup difference during evaluation for low-rank methods is whether the last layer is trained by full-rank Adam or not. Most previous work train the last layer, arguably one of the most important layer [Zhao et al., 2024c], using full-rank Adam. This effectively...

    work page 2000

  5. [5]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv

  6. [6]

    Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information

    Dongseong Hwang. Fadam: Adam is a natural gradient optimizer using diagonal empirical fisher information. arXiv preprint arXiv:2405.12807,

    arXiv

  7. [9]

    For 8-bit optimizers, we assume weights are stored in BF16, but optimizer states use FP8

    For memory estimation, we assume the use of BF16 format. For 8-bit optimizers, we assume weights are stored in BF16, but optimizer states use FP8. GaLore uses r = 1024 for 7B model. F.4 Memory estimation Following the setup of Zhao et al. [2024a], we provide the estimated GPU memory for each optimizers due to the difficulty of directly measuring their pra...

    work page arXiv 2000

  8. [10]

    All experiments are conduced on NVIDIA A100 GPUs. Table 7: The hyperparameters for GaLore and Fira learning rate update scale rank update interval60M 0.02 0.3 128 200130M0.02 0.3 256 200350M0.02 0.3 256 2001.3B 0.01 0.25 512 200 Table 8: The hyperparameters used for Adam optimizer. learning rateβ1 β2 correct bias 60M 0.001 0.9 0.999 True 130M 0.001 0.9 0....

    work page 2001

Show all 36 references
  1. [11]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983,

    arXiv

  2. [13]

    A new perspective on shampoo’s preconditioner

    Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on shampoo’s preconditioner. arXiv preprint arXiv:2406.17748,

    arXiv

  3. [14]

    Curvature-informed sgd via general purpose lie-group precondi- tioners

    Omead Pooladzandi and Xi-Lin Li. Curvature-informed sgd via general purpose lie-group precondi- tioners. arXiv preprint arXiv:2402.04553,

    arXiv

  4. [15]

    Flora: Low-rank core space for n-dimension

    Chongjie Si, Xuehui Wang, Xue Yang, Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xi- aokang Yang, and Wei Shen. Flora: Low-rank core space for n-dimension. arXiv preprint arXiv:2405.14739,

    arXiv

  5. [16]

    Connection of diagonal hessian estimates to natural gradients in stochastic optimization

    Shiqing Sun and James C Spall. Connection of diagonal hessian estimates to natural gradients in stochastic optimization. In 2021 55th Annual Conference on Information Sciences and Systems (CISS), pages 1–6. IEEE,

    2021

  6. [17]

    Understanding self-supervised learning with dual deep networks

    Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.00578,

    2010 arXiv

  7. [19]

    Soap: Improving and stabilizing shampoo using adam

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321,

    arXiv

  8. [20]

    Zhirong Yang and Jorma Laaksonen

    URL https://arxiv.org/abs/2412.11768. Zhirong Yang and Jorma Laaksonen. Principal whitened gradient for information geometry. Neural Networks, 21(2-3):232–240,

    arXiv

  9. [22]

    Large batch optimization for deep learning: Training bert in 76 minutes

    Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962,

    1904 arXiv

  10. [23]

    Adam-mini: Use fewer learning rates to gain more

    Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793,

    arXiv

  11. [24]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024a. 17 Pengxiang Zhao, Ping Li, Yingjie Gu, Yi Zheng, Stephan Ludger K ¨olker,...

    arXiv

  12. [26]

    for t = 1,

    Algorithm 5 Shampoo Optimizer Input: Lm = ϵIm, Rn = ϵIn, learning rate λ, optimization step T , loss function L. for t = 1, . . . , Tdo Gt = ∇Wt L Lm,t = Lm,t−1 + GtGT t Rn,t = Rn,t−1 + GT t Gt Wt = Wt−1 + λL − 1 4 m,tGtR − 1 4 n,t end for B.5 SOAP/AdaDiag++ SOAP/AdaDiag++ [An...

    2024

  13. [27]

    Note that our paper assumes Vec(·) is stacking columns of matrix whereas Gupta et al

    Since Adafactor is an approximation to Adam, they propose to *Lemma 8 in [Gupta et al., 2018]. Note that our paper assumes Vec(·) is stacking columns of matrix whereas Gupta et al

    2018

  14. [28]

    assumes stacking the rows, explaining the reverse order of presentation 21 use Adam instead of Adafactor in Shampoo’s eigen-space, to further improve the performance. They propose the following update rule: mt = β1mt−1 + (1 − β1)Gt (first moment) Lm,t = β3Lm,t−1 + (1 − β3)GtGT...

    2024

  15. [29]

    In practice, SW AN proposes to compute the(GGT )− 1 2 using Newton-Schulz iterations

    SW AN derives these two steps from investigating the LLM dynamics. In practice, SW AN proposes to compute the(GGT )− 1 2 using Newton-Schulz iterations. B.8 Newton Schulz iteration In many machine learning applications, like successive whitening and coloring transform [Cho et ...

    2019

  16. [30]

    At the same time, GaLore [Zhao et al., 2024a] popularizes the use of low-rank optimizers, which demonstrates on-par performance compared to full-rank Adam training

    explicitly show that LoRA is secretly a gradient compressor, which translates the modification to model architecture into a low-rank optimizer with randomly sampled matrix. At the same time, GaLore [Zhao et al., 2024a] popularizes the use of low-rank optimizers, which demonstr...

    2020

  17. [32]

    Then, the normalization step of Lamb and Lars can be viewed as a 1-sample approximation to FIM* under the structure considered in Sec

    However, if one vectorizes the matrix weight into a vector, and stacks those vectors into a larger matrix. Then, the normalization step of Lamb and Lars can be viewed as a 1-sample approximation to FIM* under the structure considered in Sec. 3.3. Muon Muon [Jordan et al., 2024...

    2024

  18. [33]

    U T U T c G ⊙2# =[U , Uc] U T G U T c G vuutE

    However, Sec. 3.3 does not provide an explanation for composing these two operators. Namely, the whitening operator is estimated using normalized gradients, rather than the raw gradient. We will leave the investigation of operator composition for future work. *This FIM is now ...

    2018

  19. [2008]

    Large batch training of convolutional networks

    Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888,

    arXiv

  20. [2014]

    Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition

    Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia, Val´erie Gar`es, Mounir Haddou, and Quang Huy Tran. Efficient approximations of the fisher matrix in neural networks using kronecker product singular value decomposition. arXiv preprint arXiv:2201.10285,

    arXiv

  21. [2016]

    Swan: Preprocessing sgd enables adam-level performance on llm training with significant memory reduction

    16 Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. Swan: Preprocessing sgd enables adam-level performance on llm training with significant memory reduction. arXiv preprint arXiv:2412.13148,

    arXiv

  22. [2017]

    Preconditioner on matrix lie group for sgd

    Xi-Lin Li. Preconditioner on matrix lie group for sgd. arXiv preprint arXiv:1809.10232,

    arXiv

  23. [2018]

    Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024a

    Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024a. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pha...

  24. [2019]

    On empirical comparisons of optimizers for deep learning

    D Choi. On empirical comparisons of optimizers for deep learning. arXiv preprint arXiv:1910.05446,

    1910 arXiv

  25. [2020]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971,

    arXiv

  26. [2021]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    arXiv

  27. [2023]

    Can we remove the square-root in adaptive gradient methods? a second-order perspective

    Wu Lin, Felix Dangel, Runa Eschenhagen, Juhan Bae, Richard E Turner, and Alireza Makhzani. Can we remove the square-root in adaptive gradient methods? a second-order perspective. arXiv preprint arXiv:2402.03496,

    arXiv

  28. [2024]

    Diederik P Kingma

    URL https: //kellerjordan.github.io/posts/muon/. Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.