Pith. sign in

REVIEW 3 major objections 5 minor 67 references

Low-rank Momentum Factorization for Memory Efficient Training

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MoFaSGD compresses the optimizer's first-order momentum into a low-rank SVD that is updated every step, and proves this costs nothing in asymptotic convergence while cutting memory to LoRA levels.

desk verdict MoFaSGD is a genuinely new low-rank momentum optimizer with a plausible empirical story, but the central O(1/sqrt(T)) convergence proof has a load-bearing subspace-inclusion error and an unjustified noise scaling, so the theory does not hold as submitted. read the letter →

arxiv 2507.08091 v1 pith:U7SHXPZN submitted 2025-07-10 cs.LG

Pouria Mahdavinia , Mehrdad Mahdavi This is my paper
classification cs.LG MSC 68T0790C2690C06
keywords low-rankmomentumfactorizationmemory-efficienttrainingtangentspaceprojectionspectrallynormalizedupdatesnon-convexstochasticoptimizationLLMfine-tuningSVDoptimizerstatecompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MoFaSGD is an optimizer designed for fine-tuning large models with scarce GPU memory. Its core idea is to store the first-order momentum, the running average of gradients that Adam-style optimizers keep in full, as a low-rank factored object $(U, \Sigma, V)$, updated online at every iteration by projecting the current gradient onto the tangent space of the previous factorization. The update direction itself is the product $U_{t+1} V_{t+1}^T$, which gives the spectral normalization effect of methods like Muon while keeping optimizer-state memory at LoRA-like levels. The paper proves an optimal $O(1/\sqrt{T})$ non-convex stochastic convergence rate and reports competitive fine-tuning results on LLM pre-training, GLUE, and instruction-tuning benchmarks.

What carries the argument

The load-bearing object is the tangent space $T_t$ of the low-rank momentum manifold at $(U_t, \Sigma_t, V_t)$: the collection of matrices $U_t M V_t^T + U_p V_t^T + U_t V_p^T$ with $U_p$ and $V_p$ orthogonal to the current factors. Projecting the gradient onto this tangent space yields the closed form $\hat{G}_t = U_t U_t^T G_t + G_t V_t V_t^T - U_t U_t^T G_t V_t V_t^T$, which Theorem 4.3 shows is the minimal-residual two-sided sketch. Because $\hat{G}_t$ is rank $r$, the sum $\hat{G}_t + \beta \hat{M}_{t-1}$ has rank at most $2r$; after a QR factorization of $[U_t, G_t V_t]$ and $[V_t, G_t^T U_t]$, a rank-$r$ SVD of a $2r \times 2r$ matrix updates the factors in $O((m+n)r^2 + r^3)$ time. This machinery carries the argument by letting the optimization subspace adapt continuously at low cost, while Theorem 4.5 uses the controlled projection residual to keep the momentum-factorization error from spoiling the descent rate.

What would settle it

Run MoFaSGD on a small model and, at several consecutive iterations, compute the largest principal angle between the subspace $\mathrm{Range}([U_{t-1}, G_{t-1} V_{t-1}])$ and $\mathrm{Range}(U_t)$. The proof of Lemma D.5 requires the former to be a subset of the latter; any iteration where the containment fails, which the rank-$2r$ SVD update does not prevent, invalidates the monotone-residual step. Alternatively, track $\|(I - U_t U_t^T) G_t (I - V_t V_t^T)\|_*$ across iterations: if it ever increases, the recursive compression bound that Theorem 4.5 depends on does not hold as written.

Watch

Extended reading notes

Core claim

The paper claims that the full-rank first-order momentum of an optimizer can be replaced by a rank-$r$ SVD factorization without sacrificing the asymptotic convergence rate. The key move is to project each incoming gradient onto the tangent space of the current momentum factors; because that projection is optimal among a natural class of two-sided sketches (Theorem 4.3), the residual of the low-rank momentum approximation stays controlled. The update of the factors themselves is cheap: since the projected gradient and the old momentum are each rank $r$, their sum lies in a $2r$-dimensional space and a small SVD of a $2r \times 2r$ matrix produces the new $U, \Sigma, V$. The same factors define the parameter update $W_{t+1} = W_t - \eta U_{t+1} V_{t+1}^T$, bypassing the subspace moment accumulation used by GaLore and making MoFaSGD a low-rank, memory-efficient variant of Muon. Theorem 4.5 then bounds the averaged nuclear-norm gradient by $O(\Delta/(\eta T) + \eta L + \sigma/\sqrt{T})$, which is optimal for smooth non-convex stochastic optimization.

Load-bearing premise

The proof of the convergence theorem assumes that the optimization subspace only grows at each step, meaning the new subspace must contain the previous subspace together with the projected gradient, so that the momentum compression error never increases. The algorithm's actual update rule guarantees the opposite inclusion, so the monotonicity the proof relies on is not assured.

Editorial extensions

If this is right

  • Optimizer-state memory for a weight matrix of size $m \times n$ shrinks from $O(mn)$ (AdamW) or $O(mr + nr)$ with full gradients (GaLore) to $O((m+n)r + r^2)$ for the stored factors, the same order as LoRA, while parameters are still updated in full.
  • Because the subspace is refined every iteration, there is no periodic full-matrix SVD and no staleness window; the ablation shows that increasing GaLore's update frequency degrades its performance, while MoFaSGD's per-step adaptation does not suffer this effect.
  • The method inherits the spectral-normalization behavior of Muon- and Shampoo-type preconditioners without storing second moments or computing matrix roots, so the per-iteration cost stays at the level of a standard optimizer.
  • The convergence guarantee is not degraded by the factorization: under nuclear-norm smoothness and a bounded-variance oracle, the averaged gradient norm reaches $O(1/\sqrt{T})$, matching the known lower bound for non-convex stochastic optimization.
  • As a low-rank variant of Muon, MoFaSGD offers a drop-in replacement for hidden-layer optimizers in transformer training with comparable quality to full-rank methods in the tested regimes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Applying the same tangent-space factorization to second-moment statistics would turn MoFaSGD into a fully adaptive (Adam-like) preconditioner at similar memory; the paper leaves this extension open.
  • The per-layer projection residual that the analysis controls suggests a concrete rule for adaptive rank allocation: give more rank to layers where $\|G_t - \hat{G}_t\|_F$ stays large.
  • Because the optimizer state is already a short list of factors, the low-rank gradient-buffer trick used for gradient accumulation could be extended to shard factors across devices in distributed fine-tuning, potentially cutting communication volume as well as memory.
  • The method's per-step subspace adaptation could also serve as a drop-in momentum module inside other low-rank subspace or PEFT pipelines, since it only consumes gradients and produces $(U, V)$ directions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MoFaSGD, a memory-efficient optimizer that maintains a low-rank SVD factorization of the first-order momentum and updates it online via tangent-space projections. The factors U_{t+1}, V_{t+1} are used directly in the parameter update W_{t+1}=W_t-η U_{t+1}V_{t+1}^T, yielding a LoRA-level memory footprint with per-iteration online subspace adaptation. The central theoretical claim is Theorem 4.5, which asserts an optimal O(1/√T) convergence rate for non-convex stochastic optimization under nuclear-norm smoothness and a first-moment noise bound. Empirically, the paper evaluates MoFaSGD on NanoGPT pre-training, GLUE fine-tuning, and Tulu3 instruction tuning, reporting competitive or better performance than GaLore and LoRA with comparable memory usage. The paper includes a memory profiling study, an ablation on GaLore's subspace update frequency, and a spectral analysis of AdamW momentum buffers that supports the low-rank momentum conjecture.

Significance. If the theoretical result were valid, the paper would make a significant contribution: it combines online low-rank momentum factorization with spectrally normalized updates, achieving a memory footprint comparable to LoRA while providing a provably optimal stochastic non-convex rate. The algorithmic idea is original and the empirical evaluation is unusually thorough for a memory-optimizer paper: it includes per-category memory breakdowns, wall-clock convergence curves, rank ablations, and a direct test of the low-rank momentum conjecture. The code is provided. However, the main theoretical guarantee is not established by the submitted proof. The proof of Lemma D.5 relies on a false subspace inclusion, and the stochastic noise scaling used throughout the analysis is not supported by the stated assumptions. These are load-bearing gaps, not presentation issues. The empirical results remain interesting, but the paper's central claim of an optimal convergence rate is unproven in the current form.

major comments (3)
  1. [Appendix D.3.1, Lemma D.5, Eq. (46)] The proof asserts that, from the update rule of Eq. (7)-(8), Range([U_{t-1}, G_{t-1}V_{t-1}]) is contained in Range(U_t). This inclusion is reversed. The update defines U_t = U'_{t-1}U''_{t-1} with U'_{t-1} = QR([U_{t-1}, G_{t-1}V_{t-1}]), so only Range(U_t) ⊆ Range([U_{t-1}, G_{t-1}V_{t-1}]) holds. The bracketed matrix spans up to 2r directions while U_t has only r columns, so the asserted reverse inclusion is dimensionally impossible unless the new gradient columns lie in the old range. This inclusion is exactly what lets Eq. (46) bound the current tangent-space residual by the previous residual; without it, the recursion in Eq. (47) and the bound in Eq. (50) do not follow, and Theorem 4.5 is unproven.
  2. [Appendix D.3.1, Lemma D.5, Eqs. (43)-(44)] The proof replaces M̂_t with Ĝ_t + βM̂_{t-1}, but M̂_t is the rank-r SVD of that 2r-rank matrix, not the matrix itself. The step from Eq. (43) to Eq. (44) omits the rank-r truncation error ∥M̂_t - (Ĝ_t + βM̂_{t-1})∥_*, which is not generally zero and is not bounded by the projection residual in the argument. Even if the subspace inclusion in Eq. (46) were corrected, the compression-error recursion would still need an additional term; the current proof does not account for it.
  3. [Assumption 4.2 and Lemmas D.4-D.5] Assumption 4.2 only bounds the first moment of the stochastic noise, E[∥∇L(W,ξ)-∇L(W)∥_*] ≤ σ, yet the proofs of Lemma D.4 (Eq. 39) and Lemma D.5 (Eq. 50) introduce a √B scaling, e.g., Tσ/((1-β)√B). With only a first-moment bound, summing T noise terms gives Tσ, not Tσ/√B; the √B factor requires a second-moment or sub-Gaussian condition that is neither stated nor derived. This is an independent gap that prevents the claimed O(1/√T) rate under the stated assumptions.
minor comments (5)
  1. [Lemma D.4 statement] The lemma says 'Under Assumptions 4.1 and 4.1'; the second reference should be Assumption 4.2.
  2. [Equation (7) and following text] The text says 'the inner matrix has rank at most r'; the 2r×2r matrix [βΣ_t-U_t^T G_t V_t, I; I, 0] generically has rank up to 2r, so this statement is unclear and should be corrected or clarified.
  3. [Theorem 4.3 proof] The notation U_{L,R} is used for different block matrices in Eq. (18), Eq. (24), and Eq. (27), which makes the proof hard to follow; the minimization over L and R is stated only implicitly. A clearer presentation of the SVD decomposition of L and R and the achievability of the lower bound would improve readability.
  4. [Figure 6b] The caption reads 'GaLore Update Frequency (τ) Ablation' but the legend and axis labels contain stray symbols (e.g., '= 300', '= 75'); these should be cleaned up.
  5. [Section 5.1] The phrase 'the 0.73B token budget, optimized for Muon's convergence speed' is a bit ambiguous; consider rewording to clarify that the budget is the standard NanoGPT speedrun budget.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detected; the convergence proof is self-contained, though it contains a non-circular proof gap in Lemma D.5.

full rationale

MoFaSGD's central derivation chain is self-contained: the algorithm is defined by Equations 7–9, and Theorem 4.5 is proved from Assumptions 4.1–4.2 via Lemmas D.3–D.5 plus a standard descent argument. The low-rank momentum conjecture is justified by external prior work (Feinberg et al. 2024; Zhao et al. 2024a) and independently validated on AdamW first-moment buffers from the Tulu3 run, which is external to MoFaSGD's own trajectory. No fitted parameter is renamed as a prediction, and no cited result by the present authors is load-bearing; the paper cites Shampoo, Muon, and GaLore only for motivation and positioning. Theorem 4.3's optimality claim is proven directly rather than assumed. The proof does contain a serious correctness gap: Lemma D.5 asserts Range([U_{t-1}, G_{t-1}V_{t-1}]) is contained in Range(U_t) 'based on Equation 7', but Equations 7–8 give the reverse containment Range(U_t) ⊆ Range([U_{t-1}, G_{t-1}V_{t-1}]), and the asserted inclusion fails in general. Separately, Assumption 4.2 bounds E‖G − ∇L‖_∗ by σ while Lemmas D.4 and D.5 introduce σ/√B without an explicit batch-size assumption. These are proof errors, not circular reductions: the theorem does not assume its conclusion, and no equation is defined in terms of the quantity it is used to predict. Accordingly, the circularity score is 0; the flagged issues belong in a correctness review, not a circularity analysis.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central theoretical claim rests on Assumptions 4.1-4.2, the low-rank momentum conjecture, and two ad hoc conditions used only in the proof: rank(G0) <= r and monotone subspace inclusion. The monotone inclusion is false, so the convergence guarantee is not established. Listing these makes the paper's actual evidential burden explicit.

free parameters (3)
  • rank r = 16, 32, 128 (pretraining); 4, 8 (GLUE); 8 (Tulu3)
    Rank of the momentum factorization, chosen per experiment; controls memory and compute trade-off.
  • momentum decay beta = 0.85 (NanoGPT), 0.95 (GLUE and Tulu3)
    Exponential decay factor for momentum; tuned by grid search; constrained to beta <= 1/3 in theory.
  • learning rate eta = 1e-3 to 1e-5 depending on task
    Step size; tuned per task; constrained to eta <= 1 in theory.
assumptions (5)
  • domain assumption L-smoothness with respect to nuclear norm (Assumption 4.1)
    Required for the descent lemma (Lemma D.3). Motivated by modular norm theory (Large et al. 2025), but not verified for the RoBERTa, GPT-2, or LLaMA losses used experimentally.
  • domain assumption Stochastic oracle satisfies E[||nabla L(W,xi)-nabla L(W)||_*] <= sigma (Assumption 4.2)
    Stated as a first-moment bound on the nuclear norm of the noise. The proof (Lemmas D.4, D.5) uses a sqrt(B) improvement that requires a second-moment or sub-Gaussian condition, which is not assumed.
  • domain assumption Gradient EMA exhibits low-rank structure (Section 4.1)
    Motivates the factorization. Experimentally supported for AdamW's first-moment buffer in Section 5.3, but the convergence proof does not use this beyond rank(G0) <= r.
  • ad hoc to paper rank(G0) <= r (Theorem 4.5)
    Makes the initial compression error e0 equal to zero in Lemma D.5. Not satisfied by typical full-rank gradients; the proof simply ignores e0 otherwise.
  • ad hoc to paper Monotone subspace inclusion: Range([U_{t-1}, G_{t-1}V_{t-1}]) is contained in Range(U_t), and analogously for V (Eq. 46 of Lemma D.5)
    This is the load-bearing step that makes the projection residual recursively non-increasing. The update rule in Eq. 7-8 gives the reverse containment, so this axiom is false in general.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Low-rank Momentum Factorization for Memory Efficient Training." pith.science (2026). https://pith.science/paper/U7SHXPZN

@misc{pith2026250708091,
  author       = {Pith},
  title        = {Pith review of: Low-rank Momentum Factorization for Memory Efficient Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7SHXPZN}},
  note         = {Machine review of arXiv:2507.08091}
}
read the original abstract

Fine-tuning large foundation models presents significant memory challenges due to stateful optimizers like AdamW, often requiring several times more GPU memory than inference. While memory-efficient methods like parameter-efficient fine-tuning (e.g., LoRA) and optimizer state compression exist, recent approaches like GaLore bridge these by using low-rank gradient projections and subspace moment accumulation. However, such methods may struggle with fixed subspaces or computationally costly offline resampling (e.g., requiring full-matrix SVDs). We propose Momentum Factorized SGD (MoFaSGD), which maintains a dynamically updated low-rank SVD representation of the first-order momentum, closely approximating its full-rank counterpart throughout training. This factorization enables a memory-efficient fine-tuning method that adaptively updates the optimization subspace at each iteration. Crucially, MoFaSGD leverages the computed low-rank momentum factors to perform efficient spectrally normalized updates, offering an alternative to subspace moment accumulation. We establish theoretical convergence guarantees for MoFaSGD, proving it achieves an optimal rate for non-convex stochastic optimization under standard assumptions. Empirically, we demonstrate MoFaSGD's effectiveness on large language model alignment benchmarks, achieving a competitive trade-off between memory reduction (comparable to LoRA) and performance compared to state-of-the-art low-rank optimization methods. Our implementation is available at https://github.com/pmahdavi/MoFaSGD.

Figures

Figures reproduced from arXiv: 2507.08091 by the authors.

Figure 1
Figure 1. Validation loss vs. training steps for MoFaSGD (left) and GaLore (right) across ranks [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Validation loss vs. wall-clock time across ranks. MoFaSGD scales better in convergence and [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Comparison of optimizer performance on GPT-2 using validation perplexity loss. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Empirical memory breakdown (GB) for LLaMA3.1-8B using different optimizers. spectrally normalized updates and dynamic momentum factorization enable more effective fine-tuning under memory constraints. We have also included the training loss curves in Appendix C.5, show…
Figure 5
Figure 5. Figure 5: MoFaSGD demonstrates superior sample efficiency and faster wall-clock convergence, achieving [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: (a) Average mass ratio of the AdamW first moment captured by top- [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: GPU memory trace during LLaMA3.1-8B training with MoFaSGD ( [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Training loss curves for post-training setups. [PITH_FULL_IMAGE:figures/full_fig_p024_8.png]
Figure 9
Figure 9. Figure 9: MoFaSGD (r = 8): Compact optimizer states and minimal gradient spikes. Total memory ∼29.4 GB [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]
Figure 10
Figure 10. Figure 10: LoRA (r = 8): Slightly higher activation and adapter memory, total ∼33.6 GB [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]
Figure 11
Figure 11. Figure 11: SWAN: Stateless optimizer, but full-sized gradient buffers lead to higher memory usage ( [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]
Figure 12
Figure 12. Figure 12: AdamW (BF16): Full-rank moments and gradient buffers dominate memory ( [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]
Figure 13
Figure 13. Figure 13: GaLore (Fused, r = 8): Low gradient and optimizer state memory; total ∼30.0 GB [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]
Figure 14
Figure 14. Figure 14: GaLore (Non-Fused, r = 8): Gradient accumulation inflates memory cost (∼44.5 GB). Quantitative Breakdown. The table below summarizes the memory footprint by category for all setups. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

67 extracted references · 23 canonical work pages

  1. [1]

    Memory efficient adaptive optimization

    Rohan Anil, Vineet Gupta, Tomer Koren, and Yoram Singer. Memory efficient adaptive optimization. Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  2. [2]

    Lower bounds for non-convex stochastic optimization

    Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 0 (1): 0 165--214, 2023

    2023

  3. [3]

    Modular duality in deep learning

    Jeremy Bernstein and Laker Newhouse. Modular duality in deep learning. arXiv preprint arXiv:2410.21265, 2024 a

    arXiv 2024

  4. [4]

    Old optimizer, new norm: An anthology

    Jeremy Bernstein and Laker Newhouse. Old optimizer, new norm: An anthology. arXiv.org, 2024 b . doi:10.48550/arxiv.2409.20325

    arXiv doi:10.48550/arxiv.2409.20325 2024

  5. [5]

    signsgd: Compressed optimisation for non-convex problems

    Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp.\ 560--569. PMLR, 2018

    work page 2018

  6. [6]

    Automatic gradient descent: Deep learning without hyperparameters

    Jeremy Bernstein, Chris Mingard, Kevin Huang, Navid Azizan, and Yisong Yue. Automatic gradient descent: Deep learning without hyperparameters. arXiv preprint arXiv:2304.05187, 2023

    arXiv 2023

  7. [7]

    Symbolic discovery of optimization algorithms

    Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36, 2024

    work page 2024

  8. [8]

    8-bit optimizers via block-wise quantization

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint arXiv:2110.02861, 2021

    arXiv 2021

Show all 67 references
  1. [9]

    Qlora: Efficient finetuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  2. [10]

    Adaptive subgradient methods for online learning and stochastic optimization

    John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12: 0 2121--2159, 2011

    2011

  3. [11]

    Combining axes preconditioners through kronecker approximation for deep learning

    Sai Surya Duvvuri, Fnu Devvrit, Rohan Anil, Cho-Jui Hsieh, and Inderjit S Dhillon. Combining axes preconditioners through kronecker approximation for deep learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  4. [12]

    Sketchy: Memory-efficient adaptive regularization with frequent directions

    Vladimir Feinberg, Xinyi Chen, Y Jennifer Sun, Rohan Anil, and Elad Hazan. Sketchy: Memory-efficient adaptive regularization with frequent directions. Advances in Neural Information Processing Systems, 36, 2024

    2024

  5. [13]

    Fast approximate natural gradient descent in a kronecker factored eigenbasis

    Thomas George, C \'e sar Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis. Advances in neural information processing systems, 31, 2018

    2018

  6. [14]

    Improving neural network training in low dimensional random bases

    Frithjof Gressmann, Zach Eaton-Rosen, and Carlo Luschi. Improving neural network training in low dimensional random bases. Advances in Neural Information Processing Systems, 33: 0 12140--12150, 2020

    2020

  7. [15]

    A kronecker-factored approximate fisher matrix for convolution layers

    Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference on Machine Learning, pp.\ 573--582. PMLR, 2016

    2016

  8. [16]

    Olmes: A standard for language model evaluations

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.08446, 2024

    2024 arXiv

  9. [17]

    Shampoo: Preconditioned stochastic tensor optimization

    Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp.\ 1842--1850. PMLR, 2018

    2018

  10. [18]

    Gradient descent happens in a tiny subspace

    Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754, 2018

    2018 arXiv

  11. [19]

    Flora: Low-rank adapters are secretly gradient compressors

    Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293, 2024

    2024 arXiv

  12. [20]

    Topics in matrix analysis

    Roger A Horn and Charles R Johnson. Topics in matrix analysis. Cambridge university press, 1994

    1994

  13. [21]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp.\ 2790--2799. PMLR, 2019

    2019

  14. [22]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  15. [23]

    modded-nanogpt: Speedrunning the nanogpt baseline, 2024 a

    Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, @fernbear.bsky.social, Boza Vlado, You Jiacheng, Franz Cesista, Braden Koszarsky, and @Grad62304977. modded-nanogpt: Speedrunning the nanogpt baseline, 2024 a . URL https://github.com/KellerJordan/modded-nanogpt

    2024

  16. [24]

    Muon: An optimizer for hidden layers in neural networks, 2024 b

    Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cecista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024 b . URL https://kellerjordan.github.io/posts/muon/

    2024

  17. [25]

    A rank stabilization scaling factor for fine-tuning with lora

    Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732, 2023

    2023 arXiv

  18. [26]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  19. [27]

    Accelerating neural network training: An analysis of the algoperf competition

    Priya Kasimbeg, Frank Schneider, Runa Eschenhagen, Juhan Bae, Chandramouli Shama Sastry, Mark Saroufim, Boyuan Feng, Less Wright, Edward Z Yang, Zachary Nado, et al. Accelerating neural network training: An analysis of the algoperf competition. arXiv preprint arXiv:2502.15015, 2025

    2025 arXiv

  20. [28]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam : A method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR), 2015

    2015

  21. [29]

    Vera: Vector-based random matrix adaptation

    Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454, 2023

    2023 arXiv

  22. [30]

    T " ulu 3: Pushing frontiers in open language model post-training

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  23. [31]

    Scalable optimization in the modular norm

    Tim Large, Yang Liu, Jacob Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. Advances in Neural Information Processing Systems, 37: 0 73501--73548, 2025

    2025

  24. [32]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  25. [33]

    Memory efficient optimizers with 4-bit states

    Bingrui Li, Jianfei Chen, and Jun Zhu. Memory efficient optimizers with 4-bit states. Advances in Neural Information Processing Systems, 36: 0 15136--15171, 2023

    2023

  26. [34]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  27. [35]

    Relora: High-rank training through low-rank updates

    Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High-rank training through low-rank updates. In The Twelfth International Conference on Learning Representations, 2023

    2023

  28. [36]

    On the limited memory bfgs method for large scale optimization

    Dong C Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45 0 (1): 0 503--528, 1989

    1989

  29. [37]

    Muon is scalable for llm training

    Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al. Muon is scalable for llm training. arXiv preprint arXiv:2502.16982, 2025

    2025 arXiv

  30. [38]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  31. [39]

    Fixing weight decay regularization in adam

    Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5, 2017

    2017 arXiv

  32. [40]

    Badam: A memory efficient full parameter training method for large language models

    Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter training method for large language models. arXiv preprint arXiv:2404.02827, 2024

    2024 arXiv

  33. [41]

    Came: Confidence-guided adaptive memory efficient optimization

    Yang Luo, Xiaozhe Ren, Zangwei Zheng, Zhuo Jiang, Xin Jiang, and Yang You. Came: Confidence-guided adaptive memory efficient optimization. arXiv preprint arXiv:2307.02047, 2023

    2023 arXiv

  34. [42]

    Adalomo: Low-memory optimization with adaptive learning rate

    Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu. Adalomo: Low-memory optimization with adaptive learning rate. arXiv preprint arXiv:2310.10195, 2023 a

    2023 arXiv

  35. [43]

    Full parameter fine-tuning for large language models with limited resources

    Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. arXiv preprint arXiv:2306.09782, 2023 b

    2023 arXiv

  36. [44]

    Swan: Preprocessing sgd enables adam-level performance on llm training with significant memory reduction

    Chao Ma, Wenbo Gong, Meyer Scetbon, and Edward Meeds. Swan: Preprocessing sgd enables adam-level performance on llm training with significant memory reduction. arXiv preprint arXiv:2412.13148, 2024

    2024 arXiv

  37. [45]

    New insights and perspectives on the natural gradient method

    James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, 21 0 (146): 0 1--76, 2020

    2020

  38. [46]

    Optimizing neural networks with kronecker-factored approximate curvature

    James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp.\ 2408--2417. PMLR, 2015

    2015

  39. [47]

    Microadam: Accurate adaptive optimization with low space overhead and provable convergence

    Ionut-Vlad Modoranu, Mher Safaryan, Grigory Malinovsky, Eldar Kurtic, Thomas Robert, Peter Richtárik, and Dan Alistarh. Microadam: Accurate adaptive optimization with low space overhead and provable convergence. arXiv.org, 2024. doi:10.48550/arxiv.2405.15593

    2024 doi arXiv

  40. [48]

    A new perspective on shampoo's preconditioner

    Depen Morwani, Itai Shapira, Nikhil Vyas, Eran Malach, Sham Kakade, and Lucas Janson. A new perspective on shampoo's preconditioner. arXiv preprint arXiv:2406.17748, 2024

    2024 arXiv

  41. [49]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35: 0 2...

    2022

  42. [50]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37: 0 30811--30849, 2025

    2025

  43. [51]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--16. IEEE, 2020

    2020

  44. [52]

    Adarankgrad: Adaptive gradient-rank and moments for memory-efficient llms training and fine-tuning

    Yehonathan Refael, Jonathan Svirsky, Boris Shustin, Wasim Huleihel, and Ofir Lindenbaum. Adarankgrad: Adaptive gradient-rank and moments for memory-efficient llms training and fine-tuning. arXiv preprint arXiv:2410.17881, 2024

  45. [53]

    Ldadam: Adaptive optimization from low-dimensional gradient statistics

    Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from low-dimensional gradient statistics. arXiv.org, 2024

    2024

  46. [54]

    Gradient multi-normalization for stateless and scalable llm training

    Meyer Scetbon, Chao Ma, Wenbo Gong, and Edward Meeds. Gradient multi-normalization for stateless and scalable llm training. arXiv preprint arXiv:2502.06742, 2025

    2025 arXiv

  47. [55]

    Adafactor: Adaptive learning rates with sublinear memory cost

    Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp.\ 4596--4604. PMLR, 2018

    2018

  48. [56]

    Tieleman and G

    T. Tieleman and G. Hinton. Lecture 6.5 - rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera: Neural Networks for Machine Learning, 2012

    2012

  49. [57]

    Practical low-rank communication compression in decentralized deep learning

    Thijs Vogels, Sai Praneeth Karimireddy, and Martin Jaggi. Practical low-rank communication compression in decentralized deep learning. Advances in Neural Information Processing Systems, 33: 0 14171--14181, 2020

    2020

  50. [58]

    Soap: Improving and stabilizing shampoo using adam

    Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. Soap: Improving and stabilizing shampoo using adam. arXiv preprint arXiv:2409.11321, 2024

    2024 arXiv

  51. [59]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  52. [60]

    How far can camels go? exploring the state of instruction tuning on open resources

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing...

    2023

  53. [61]

    A spectral condition for feature learning

    Greg Yang, James B Simon, and Jeremy Bernstein. A spectral condition for feature learning. arXiv preprint arXiv:2310.17813, 2023

    2023 arXiv

  54. [62]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021

    2021

  55. [63]

    Adalora: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  56. [64]

    Galore: Memory-efficient llm training by gradient low-rank projection

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024 a

    2024 arXiv

  57. [65]

    Adapprox: Adaptive approximation in adam optimization via randomized low-rank matrices

    Pengxiang Zhao, Ping Li, Yingjie Gu, Yi Zheng, Stephan Ludger K \"o lker, Zhefeng Wang, and Xiaoming Yuan. Adapprox: Adaptive approximation in adam optimization via randomized low-rank matrices. arXiv preprint arXiv:2403.14958, 2024 b

    2024 arXiv

  58. [66]

    Apollo: Sgd-like memory, adamw-level performance

    Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance. arXiv preprint arXiv:2412.05270, 2024

    2024 arXiv

  59. [67]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.