Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

Andi Han; Athanasios Glentis; Ioannis Tsaknakis; Jiaxiang Li; Mingyi Hong; Qiulin Shang; Quan Wei

REVIEW 4 major objections 5 minor 1 cited by

Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two techniques—refactorizing low-rank weights through an SVD-balanced split and periodically resetting optimizer momentum—allow a plain low-rank 1B pretraining run to beat GaLore and Fira in perplexity (15.01 vs 15.57 and 15.10) while…

desk verdict Useful benchmark and a nice empirical surprise about low-rank initialization, but the headline 'restarts' win is confounded by an unisolated learning-rate restart schedule. read the letter →

arxiv 2505.22922 v1 pith:PXUMLAQ4 submitted 2025-05-28 cs.LG cs.AI

Athanasios Glentis , Jiaxiang Li , Qiulin Shang , Andi Han , Ioannis Tsaknakis , Quan Wei , Mingyi Hong This is my paper

classification cs.LGcs.AI

keywords largelanguagemodelspre-trainingparameter-efficienttrainingmemory-efficientoptimizerslow-rankfactorizationweightrefactorizationmomentumresetGaLore

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that parameter- and memory-efficient methods can pretrain large language models almost as well as full-rank training, and identifies what closes the remaining gap. Its benchmark of LLaMA models from 60M to 1B on the C4 corpus reports that full-rank training with a properly tuned optimizer (Stable-SPAM) remains the best baseline, that plain low-rank pretraining works reasonably well when the low-rank factors are not zero-initialized, and that restoring high-rank update information is the main lever that improves low-rank methods. The proposed fixes are weight refactorization, which rebalances the two low-rank factors through their singular value decomposition so the product is unchanged but the Hessian conditioning improves, and momentum reset, which periodically zeroes the optimizer's first moment. Combining these on a 1B model gives validation perplexity 15.01, better than GaLore's 15.57 and Fira's 15.10, while using about 25% less memory. If true, the payoff is a lower memory floor for pretraining without giving up model quality.

What carries the argument

The paper's two techniques are the machinery. Weight refactorization periodically (every 200 steps) replaces $W=BA$ with $B'=U\sqrt{\Sigma}$ and $A'=\sqrt{\Sigma}V^{\top}$, where $W=U\Sigma V^{\top}$; the product is unchanged, but the singular values are split symmetrically between the factors. The paper's Lemma 4.1 bounds the condition number of the Hessian $\nabla^2 L(B,A)$ at a local minimizer and shows the bound is minimized when the singular-value split is balanced, which is exactly this refactorization. Momentum reset periodically sets the first moment of the optimizer to zero; the paper's Theorem 4.2 adapts the known SGD-with-momentum bound to the reset schedule and shows an acceleration regime when the iterate distance to the optimum shrinks as $\Omega(1/\sqrt{kT})$ with the number of reset cycles small relative to the cycle length. Both interventions change only the training trajectory, not the forward computation or the memory footprint.

What would settle it

Train the 1B LLaMA configuration for low-rank-restarts, GaLore, and Fira at least three times per method with fixed hyperparameters and different random seeds; if the typical inter-seed spread in final validation perplexity is as large as the measured gaps (0.09 over Fira, 0.56 over GaLore), the claimed ranking does not reproduce. A separate check of the theory: in a convex finite-sum problem where the $\Omega(1/\sqrt{kT})$ iterate-shrinkage premise provably holds, compare SGD with momentum reset at $T=200$, $K=500$ against plain SGD with momentum for the same total budget and verify the predicted $\log K$ speedup.

Watch

Extended reading notes

Core claim

On its own terms, the central discovery is that low-rank pretraining—previously reported to fail outright—becomes competitive with the best memory-efficient optimizers when two cheap interventions are added. Weight refactorization replaces the current factorization $W=BA$ with $B'=U\sqrt{\Sigma}$, $A'=\sqrt{\Sigma}V^{\top}$ from the singular value decomposition $W=U\Sigma V^{\top}$, leaving the network output identical while equalizing the singular values carried by the two factors; Lemma 4.1 shows this choice minimizes the bound on the Hessian condition number at a local minimum. Momentum reset sets the AdamW first moment to zero every 200 updates; Theorem 4.2 proves that for SGD with momentum this schedule converges faster than the standard schedule when the iterate-to-optimum distance decays as $\Omega(1/\sqrt{kT})$, and the experiments adopt the same period. On a 1B LLaMA trained on C4, plain low-rank with both interventions reaches validation perplexity 15.01, undercutting GaLore (15.57) and Fira (15.10) while using 3.66 GB versus 4.76 GB for weights and optimizer states. Applying the same techniques to SLTrain's sparse-plus-low-rank factorization yields 14.37, the best non-full-rank result in the benchmark.

Load-bearing premise

The load-bearing premise is that the reported perplexity differences are real: the 0.09-point edge of low-rank-restarts over Fira and the 0.56-point edge over GaLore at 1B come from single training runs with no error bars, so run-to-run variance of that size would erase the headline ordering.

Editorial extensions

If this is right

If the 1B result transfers to other scales, plain low-rank with these two tweaks becomes the cheapest competitive pretraining recipe in the benchmark: 3.66 GB of weights-plus-optimizer memory versus 4.76 GB for GaLore and Fira.
The benchmark contradicts the earlier claim that low-rank pretraining fails: with non-zero random initialization, plain low-rank reaches 18.22 perplexity on the 1B model.
Restoring high-rank information is the systematic performance lever: Fira beats GaLore, and SLTrain beats plain low-rank, across every model size tested.
The compute-scaling curves show final perplexity depends mostly on FLOPs and is nearly independent of whether weights are stored full-rank, low-rank, or sparse-plus-low-rank, so parameter-efficient structures do not shift the compute-perplexity frontier.
The two techniques compose: SLTrain-restarts reaches 14.37 perplexity at 1B, the best non-full-rank number reported, while full-rank Stable-SPAM remains the overall best at 13.97.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Extrapolating from the paper's conditioning lemma, the same SVD-balanced refactorization should help any factorization whose product is gauge-invariant, such as tensor or structured decompositions, since the argument relies only on the product remaining unchanged; that is a testable extension the paper does not run.
The 1B headline gaps (15.01 vs 15.57 vs 15.10) come from single runs without error bars, so the exact ordering could shift under multi-seed evaluation; the memory savings, however, follow from the parameterization and will not.
The experimental recipe couples momentum reset with dropping the learning rate to zero and rewarming on each restart, following the ReLoRA-style schedule, so the isolated effect of zeroing momentum alone is not measured; ablating reset with a constant learning rate would separate the two.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper gives you two things worth keeping. First, a head-to-head benchmark of six pretraining methods on LLaMA 60M to 1B, with code and detailed hyperparameters. Second, an empirical finding that plain low-rank training with Kaiming initialization gets a 1B perplexity of 18.22, contradicting GaLore's reported 142.53. That alone is worth a look.

The trouble is the headline claim in Section 4.3: 'low-rank-restarts' hitting 15.01 PPL and beating GaLore (15.57) and Fira (15.10) with 25% less memory. Appendix 7.2 says that 'for the proposed methods on every restart we drop the learning rate to zero and perform a quick linear warm up, similar to Lialin et al. (2024).' That's a third ingredient, a ReLoRA-style LR drop-and-warmup, bundled into the 'restarts' recipe. The GaLore and Fira baselines use their standard cosine schedule and don't get this treatment. The ablation in Figure 6 compares SLTrain vs momentum reset vs weight refactorization vs both, but never tests the LR schedule alone. So the comparison isn't apples-to-apples, and the improvement could be mostly coming from the borrowed LR-restart trick. The 0.09 PPL margin over Fira is a single run with no error bars, so it's within noise. The stress-test note is right, and it's a load-bearing flaw, not a nit.

The theory is also shaky. Section 4.2's momentum-reset bound uses a comparison that assumes T >> K, while the experiments use T=200 and about K=500 reset cycles at 1B, the opposite. And the reset bound carries a 2ηG² log K term that blows up with K, so the 'significantly faster' conclusion doesn't follow from the displayed inequalities.

What holds up: the benchmark methodology is mostly sound, the code is public, and the Kaiming-initialization observation is reproducible. But as written, the central empirical claim is unverified.

My recommendation: send it to review, but with a clear request to isolate the LR-restart schedule in the ablation and report multiple seeds. If the gain survives without the LR trick, the paper is solid. If not, the claim needs to be rewritten to advertise the recipe, not the two named techniques.

Referee Report

4 major / 5 minor

Summary. This manuscript combines a survey of parameter- and memory-efficient LLM pretraining with a benchmark of several representative methods (full-rank AdamW, full-rank Stable-SPAM, GaLore, Fira, LoRA, plain low-rank, and SLTrain) across LLaMA models from 60M to 1B parameters on the C4 dataset. The authors then propose two techniques, weight refactorization and momentum reset, which are applied periodically to the low-rank and SLTrain methods; the resulting recipes are called Low-Rank-restarts and SLTrain-restarts. The headline result is that Low-Rank-restarts at 1B reaches a validation perplexity of 15.01, lower than GaLore (15.57) and Fira (15.10), while using about 25% less memory. The paper also provides a theoretical analysis of the conditioning benefit of balanced refactorization (Lemma 4.1) and of momentum reset for SGD with momentum (Theorem 4.2).

Significance. If the headline empirical result held, the paper would offer a simple, memory-saving recipe that brings low-rank pretraining to the level of or above popular memory-efficient optimizers, together with a useful survey and a consistent benchmark across four model sizes. Strengths include a broad and current survey of efficient-pretraining methods, public code, a uniform experimental setup, and the observation that plain low-rank pretraining performs far better than previously reported at 1B scale (18.22 vs 142.53). However, the central claim is not currently supported because the 'restarts' recipes embed an additional ReLoRA-style learning-rate drop-and-warmup that the baselines do not receive, and the 1B comparisons are single-run point estimates without error bars. The theoretical arguments also rely on conditions that are not verified in the experimental setting. The manuscript has value but requires substantial additional experiments and revision.

major comments (4)

[Appendix 7.2 and Section 4.3] Appendix 7.2 states that 'for the proposed methods on every restart we drop the learning rate to zero and perform a quick linear warm up, similar to Lialin et al. (2024).' Therefore the Low-Rank-restarts and SLTrain-restarts entries in Table 3 contain a third, unadvertised component, a ReLoRA-style learning-rate restart schedule, that the GaLore, Fira, and SLTrain baselines do not receive. The ablation in Figure 6 compares SLTrain against momentum reset, weight refactorization, and both, but it does not test the learning-rate restart schedule alone, nor does it state whether the schedule was applied in the 'Both' condition. Since ReLoRA has demonstrated large gains from learning-rate restarts alone, the observed 18.22-to-15.01 improvement at 1B cannot be attributed to the two proposed techniques as claimed. Please add an ablation that isolates the LR-restart schedule and either apply the same schedule to the baselines or provide a clear argument for why it is not a confounding factor.
[Table 3 and Section 3.2] All perplexity values in Table 3, including the 1B rows, are single-run point estimates with no error bars or seed variance. The claimed ordering of Low-Rank-restarts (15.01) over Fira (15.10) rests on a 0.09 PPL gap that may be within run-to-run noise. Additionally, the caption of Figure 3 and the footnote on page 12 state that the 1B experiments did not receive a comprehensive hyperparameter search, so the 1B baselines may be under-tuned relative to the proposed method. Please report multiple seeds with standard deviations and tune the 1B baselines with comparable effort.
[Section 4.2, Theorem 4.2] The conclusion that momentum reset is 'significantly faster' than standard SGD-M depends on the iterate distance decreasing as Ω(1/sqrt(kT)) and on the assertion that 'usually T is much larger than K'. In the experiments the reset period is T=200 and the total number of resets at 1B is roughly K=500 (about 100K training steps), so T is not much larger than K. The Ω(1/sqrt(kT)) premise is not verified, and the analysis is for convex, L-smooth finite-sum SGD-M, whereas the experiments use AdamW on non-convex LLM training. This theoretical result therefore does not establish the benefit of momentum reset in the evaluated setting. Please either adapt the comparison to the actual T/K regime or explicitly frame the result as a heuristic.
[Section 4.1, Lemma 4.1] Lemma 4.1 justifies balanced refactorization by bounding the condition number at a local minimizer, but it assumes that the factors already have the form B*=UΣ^α and A*=Σ^{1-α}V^T. This is an ad-hoc assumption rather than a property of low-rank training, and the lemma does not indicate how far the realized factors are from this form. Since the empirical benefit of refactorization is the main evidence, please discuss when this assumption is plausible or provide a direct check on trained factors.

minor comments (5)

[Section 4.1, Lemma 4.1] The eigenvalue bound display appears to have a typo: the lower and upper bounds both use the same symbol λ, where presumably λ_min and λ_max are intended. The notation 'λ, λ' for the smallest and largest eigenvalues is also confusing and should be changed to λ_min and λ_max.
[Section 4.2] The sentence 'an aspect not covered which is not provided in Huang et al. (2025a)' is ungrammatical; it should be 'an aspect not covered by Huang et al. (2025a).'
[Abstract and Section 4.3] The abstract claims that the proposed techniques achieve 'a lower perplexity than GaLore and Fira', while Section 4.3 says 'yields a comparable perplexity ... to GaLore and Fira'. These statements should be reconciled; the table shows 15.01 < 15.10, but the margin is small enough that 'comparable' is more accurate without error bars.
[Figure 6] The caption says 'weight refactorization methods' where 'weight factorization methods' is presumably meant. The caption should also explicitly state whether the LR-restart schedule is applied in each ablation arm, since this is essential for interpreting the ablation.
[Table 3 and Appendix 7.2] Given that the proposed recipes use a ReLoRA-style LR restart schedule, the benchmark would be clearer if ReLoRA itself were included as a baseline or at least discussed in the comparison. Currently ReLoRA appears only as a citation in Appendix 7.2 and in the survey.

Circularity Check

1 steps flagged · score 2.0 of 10

No definitional circularity; the empirical benchmark is self-contained, but the 'restarts' recipes quietly include a ReLoRA-style LR drop-and-warmup that the ablation never isolates, so the attribution of the gains to weight refactorization and momentum reset is not established.

ansatz smuggled in via citation [Appendix 7.2 (Detailed settings) combined with Section 4.3 and Figure 6]
"In addition, for the proposed methods on every restart we drop the learning rate to zero and perform a quick linear warm up, similar to Lialin et al. (2024)."

By the paper's own configuration, the recipes labeled 'Low-Rank-restarts' and 'SLTrain-restarts' contain not only weight refactorization and momentum reset but also a third component: a ReLoRA-style LR drop-and-warmup. Section 4.3 attributes the perplexity gains to 'the two new techniques', and Figure 6 ablates only momentum reset, weight refactorization, and both, with no LR-schedule-only arm. The large improvements (18.22 to 15.01 for low-rank; 15.40 to 14.37 for SLTrain) could therefore be driven by the imported ReLoRA schedule rather than by the two named mechanisms. This is an attribution gap created by smuggling a known intervention in via citation, not an equation-level identity, so it is scored as a mild circularity-adjacent issue.

full rationale

The core benchmark results are self-contained empirical measurements obtained by running external methods, and no fitted constant is renamed as a prediction. The weight-refactorization lemma is a genuine Hessian-conditioning calculation, not a restatement of the benchmark conclusions. The momentum-reset theorem is a valid bound under stated smoothness/convexity assumptions; its 'if the iterate distance shrinks at Ω(1/sqrt(kT))' condition is an unverified premise rather than a definitional equivalence, and the experiments use T=200 with about K=500 resets, violating the stated 'T much larger than K' heuristic, but this is an assumption gap rather than circularity. The self-citations to SLTrain and Stable-SPAM are not load-bearing, because those methods are re-run and compared against GaLore, Fira, and full-rank baselines. The only in-scope issue is the LR drop-and-warmup embedded in every 'restarts' recipe without an ablation control, which clouds causal attribution of the headline gains to weight refactorization and momentum reset. That is a confound or smuggled ansatz rather than a definitional circularity, so the score remains low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on two hand-chosen hyperparameters (reset and refactorization periods) and on assumptions about the loss landscape that are standard in optimization but not verified for LLM pretraining. No new physical or mathematical entities are introduced.

free parameters (2)

momentum reset period = 200 iterations
Chosen by hand in Section 4.2 ('resetting the momentum every 200 updates yields the best performance'); no sweep or sensitivity analysis is shown.
weight refactorization period = 200 iterations
Chosen by hand in Section 4.1 ('we refactor the weights for low-rank and SLTrain every 200 update iterations'); no sweep is reported.

assumptions (4)

domain assumption Each loss component in the finite-sum is convex, L-smooth, and has bounded gradient variance (used in Theorem 4.2).
Invoked in Section 4.2 to apply Garrigos-Gower Theorem 7.4; not verified for LLM pretraining.
domain assumption The loss landscape is 'mild' (convex or overparameterized) so iterates converge toward a minimizer.
Stated in Section 4.2 ('if the loss landscape is mild...') to justify the benefit of momentum reset; this condition is asserted, not demonstrated for the experiments.
ad hoc to paper At the local minimizer, the loss Hessian is positive definite and the low-rank factors admit B*=U Sigma^alpha, A*=Sigma^{1-alpha} V^T.
Used in Lemma 4.1 (Section 4.1) to derive the alpha=1/2 conditioning result; restricts to a specific factorization family.
domain assumption Local conditioning analysis at a minimizer transfers to non-optimal iterates during training.
The refactorization is applied every 200 steps far from optimality, while Lemma 4.1 analyzes a local minimizer.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking." pith.science (2026). https://pith.science/paper/PXUMLAQ4

@misc{pith2026250522922,
  author       = {Pith},
  title        = {Pith review of: Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PXUMLAQ4}},
  note         = {Machine review of arXiv:2505.22922}
}

read the original abstract

Fueled by their remarkable ability to tackle diverse tasks across multiple domains, large language models (LLMs) have grown at an unprecedented rate, with some recent models containing trillions of parameters. This growth is accompanied by substantial computational challenges, particularly regarding the memory and compute resources required for training and fine-tuning. Numerous approaches have been explored to address these issues, such as LoRA. While these methods are effective for fine-tuning, their application to pre-training is significantly more challenging due to the need to learn vast datasets. Motivated by this issue, we aim to address the following questions: Can parameter- or memory-efficient methods enhance pre-training efficiency while achieving performance comparable to full-model training? How can the performance gap be narrowed? To this end, the contributions of this work are the following. (1) We begin by conducting a comprehensive survey that summarizes state-of-the-art methods for efficient pre-training. (2) We perform a benchmark evaluation of several representative memory efficient pre-training approaches to comprehensively evaluate their performance across model sizes. We observe that with a proper choice of optimizer and hyperparameters, full-rank training delivers the best performance, as expected. We also notice that incorporating high-rank updates in low-rank approaches is the key to improving their performance. (3) Finally, we propose two practical techniques, namely weight refactorization and momentum reset, to enhance the performance of efficient pre-training methods. We observe that applying these techniques to the low-rank method (on a 1B model) can achieve a lower perplexity than popular memory efficient algorithms such as GaLore and Fira, while simultaneously using about 25% less memory.

Figures

Figures reproduced from arXiv: 2505.22922 by the authors.

**Figure 1.** Estimated memory consumption of pre-training and inference for a 7B model with a token batch size of 256 on a single device. All methods use BF16 format and AdamW (without optimizer and activation checkpointing). For low-rank, LoRA and GaLore we assume a rank of 512 (with the original attention head dimension as 2048), and for SLTrain we use rank 512 and δ = 0.1. The weight factorization methods (low-rank and SLTrai… view at source ↗

**Figure 2.** Loss curve and evaluation perplexity before and after employing the momentum reset and [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

**Figure 3.** Performance of 60M, 130M and 350M Llama models after hyperparameter search (Wandb [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Perplexity, memory, and parameter size for pretraining LLaMA 1B on the C4 dataset with different methods. The radius and color of each circle scale with parameter size. Overall, the methods which have smaller, lighter circles on the left bottom corner are desirable for…

**Figure 5.** Figure 5: Scaling performance (in terms of evaluation perplexity) of different model structures with respect to compute (FLOPs). For the x-axis, the FLOPs are multiples of 1018. Here we include all the training trajectories (with different the hyperparameter settings) to constru…

**Figure 6.** Figure 6: Ablation study on the effect of the two proposed techniques. Note that here we use the same learning rate 0.003 on both methods, differently than the case where a hyperparameter search is performed (e.g., in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
cs.LG 2026-07 conditional novelty 6.0 of 10

A meta-pipeline plus LMO four-axis view yields a dual taxonomy of 108 optimizers, and a multi-objective LLM/vision benchmark shows no single family dominates the quality–cost–memory frontier.

Reference graph

Works this paper leans on

62 extracted references · 25 canonical work pages · cited by 1 Pith paper

[1]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023
[2]

R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

arXiv 2023
[3]

D. P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48 0 (3): 0 334--334, 1997

work page 1997
[4]

X. Chen, K. Feng, C. Li, X. Lai, X. Yue, Y. Yuan, and G. Wang. Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024

arXiv 2024
[5]

Exploring Quantization for Efficient Pre-Training of Transformer Language Models

K. Chitsaz, Q. Fournier, G. Mordido, and S. Chandar. Exploring quantization for efficient pre-training of transformer language models. arXiv preprint arXiv:2407.11722, 2024

work page Pith review arXiv 2024
[6]

A. Das. Natural galore: Accelerating galore for memory-efficient llm training and fine-tuning. arXiv preprint arXiv:2410.16029, 2024

work page Pith review arXiv 2024
[7]

Egashira, M

K. Egashira, M. Vero, R. Staab, J. He, and M. Vechev. Exploiting llm quantization. arXiv preprint arXiv:2405.18137, 2024

arXiv 2024
[8]

Garrigos and R

G. Garrigos and R. M. Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235, 2023

arXiv 2023

Show all 62 references

[9]

R. J. George, D. Pitt, J. Zhao, J. Kossaifi, C. Luo, Y. Tian, and A. Anandkumar. Tensor-galore: Memory-efficient training via gradient tensor decomposition. arXiv preprint arXiv:2501.02379, 2025

2025 arXiv
[10]

A. Han, J. Li, W. Huang, M. Hong, A. Takeda, P. Jawanpuria, and B. Mishra. SLT rain: a sparse plus low rank approach for parameter and memory efficient pretraining. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.ne...

2024
[11]

Y. Hao, Y. Cao, and L. Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprint arXiv:2402.03293, 2024 a

2024 arXiv
[12]

Y. Hao, Y. Cao, and L. Mou. NeuZip : Memory-efficient training and inference with dynamic compression of neural networks. arXiv preprint arXiv:2410.20650, 2024 b

2024 arXiv
[13]

K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026--1034, 2015

2015
[14]

Y. He, P. Li, Y. Hu, C. Chen, and K. Yuan. Subspace optimization for large language models with convergence guarantees. arXiv preprint arXiv:2410.11289, 2024

2024 arXiv
[15]

E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

2022
[16]

Huang, H

T. Huang, H. Hu, Z. Zhang, G. Jin, X. Li, L. Shen, T. Chen, L. Liu, Q. Wen, Z. Wang, et al. Stable-spam: How to train in 4-bit more stably than 16-bit adam. arXiv preprint arXiv:2502.17055, 2025 a

2025 arXiv
[17]

Huang, Z

T. Huang, Z. Zhu, G. Jin, L. Liu, Z. Wang, and S. Liu. SPAM : Spike-aware adam with momentum reset for stable LLM training. In The Thirteenth International Conference on Learning Representations, 2025 b . URL https://openreview.net/forum?id=L9eBxTCpQG

2025
[18]

S. R. Kamalakara, A. Locatelli, B. Venkitesh, J. Ba, Y. Gal, and A. N. Gomez. Exploring low rank training of deep neural networks. Technical report, arXiv preprint arXiv:2209.13569, 2022

2022 arXiv
[19]

Kaplan, S

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

2001 arXiv
[20]

Lialin, S

V. Lialin, S. Muckatira, N. Shivagunde, and A. Rumshisky. Relo RA : High-rank training through low-rank updates. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=DLJznSp6X3

2024
[21]

Liang, B

K. Liang, B. Liu, L. Chen, and Q. Liu. Memory-efficient llm training with online subspace descent. arXiv preprint arXiv:2408.12857, 2024

2024 arXiv
[22]

X. Liao, S. Li, Y. Xu, Z. Li, Y. Liu, and Y. He. Galore + : Boosting low-rank adaptation for llms with cross-head projection. arXiv preprint arXiv:2412.19820, 2024

2024 arXiv
[23]

Z. Liu, C. Zhao, I. Fedorov, B. Soran, D. Choudhary, R. Krishnamoorthi, V. Chandra, Y. Tian, and T. Blankevoort. Spinquant--llm quantization with learned rotations. arXiv preprint arXiv:2405.16406, 2024

2024 arXiv
[24]

Z. Liu, R. Zhang, Z. Wang, Z. Yang, P. Hovland, B. Nicolae, F. Cappello, and Z. Zhang. CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation . arXiv preprint arXiv:2502.10940, 2025

2025
[25]

S. B. Loeschcke, M. Toftrup, M. Kastoryano, S. Belongie, and V. Sn bjarnarson. Lo QT : Low rank adapters for quantized training. In 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICML 2024), 2024. URL h...

2024
[26]

Q. Luo, H. Yu, and X. Li. Badam: A memory efficient full parameter training method for large language models. arXiv preprint arXiv:2404.02827, 2024

2024 arXiv
[27]

X. Lv, N. Ding, K. Zhang, E. Hua, G. Cui, and B. Zhou. Scalable efficient training of large language models with low-dimensional projected attention. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14588--14599, 2024

2024
[28]

X. Ma, G. Fang, and X. Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

2023
[29]

J. K. Merikoski and R. Kumar. Inequalities for spreads of matrix sums and products. Applied Mathematics E-Notes, 4 0 (150-159): 0 9, 2004

2004
[30]

Miles, P

R. Miles, P. Reddy, I. Elezi, and J. Deng. Velo RA : Memory efficient training using rank-1 sub-token projections. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=bFoQXD7Uls

2024
[31]

Mo, L.-K

Z. Mo, L.-K. Huang, and S. J. Pan. Parameter and memory efficient pretraining via low-rank riemannian optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=i0zzO7Hslk

2025
[32]

Muhamed, O

A. Muhamed, O. Li, D. Woodruff, M. Diab, and V. Smith. GRASS : Compute efficient low-memory llm training with structured sparse gradients. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14978--15003, 2024

2024
[33]

Nguyen, W

D. Nguyen, W. Yang, R. Anand, Y. Yang, and B. Mirzasoleiman. Mini-batch coresets for memory-efficient training of large language models. arXiv preprint arXiv:2407.19580, 2024 a

2024 arXiv
[34]

Nguyen, L

S. Nguyen, L. Chen, B. Liu, and Q. Liu. H-fac: Memory-efficient optimization with factorized hamiltonian descent. arXiv preprint arXiv:2406.09958, 2024 b

2024 arXiv
[35]

R. Pan, X. Liu, S. Diao, R. Pi, J. Zhang, C. Han, and T. Zhang. LISA : Layerwise importance sampling for memory-efficient large language model fine-tuning. arXiv preprint arXiv:2403.17919, 2024

2024 arXiv
[36]

S. Qiu, A. Potapczynski, M. A. Finzi, M. Goldblum, and A. G. Wilson. Compute better spent: Replacing dense layers with structured matrices. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=ExHTFXEhc9

2024
[37]

Raffel, N

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 0 (140): 0 1--67, 2020

2020
[38]

Rajabi, N

S. Rajabi, N. Nonta, and S. Rambhatla. Subtrack your grad: Gradient subspace tracking for memory and time efficient full-parameter llm training. arXiv preprint arXiv:2502.01586, 2025

2025
[39]

A. V. Ramesh, V. Ganapathiraman, I. H. Laradji, and M. Schmidt. Blockllm: Memory-efficient adaptation of llms by selecting and optimizing the right coordinate blocks. arXiv preprint arXiv:2406.17296, 2024

2024 arXiv
[40]

Robert, M

T. Robert, M. Safaryan, I.-V. Modoranu, and D. Alistarh. Ldadam: Adaptive optimization from low-dimensional gradient statistics. arXiv preprint arXiv:2410.16103, 2024

2024 arXiv
[41]

Shamshoum, N

Y. Shamshoum, N. Hodos, Y. Sieradzki, and A. Schuster. CompAct : Compressed activations for memory-efficient llm training. arXiv preprint arXiv:2410.15352, 2024

2024 arXiv
[42]

N. Shazeer. Glu variants improve transformer. Technical report, arXiv preprint arXiv:2002.05202, 2020

2002 arXiv
[43]

D. Su, A. Gu, J. Xu, Y. Tian, and J. Zhao. Galore 2: Large-scale llm pre-training by gradient low-rank projection. arXiv preprint arXiv:2504.20437, 2025

2025 arXiv
[44]

M. Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

2023 arXiv
[45]

S. Wang, P. Zhou, J. Li, and H. Huang. 4-bit shampoo for memory-efficient network training. arXiv preprint arXiv:2405.18144, 2024 a

2024 arXiv
[46]

Z. Wang, J. Liang, R. He, Z. Wang, and T. Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint arXiv:2407.18242, 2024 b

2024 arXiv
[47]

Wei, C.-Y

Q. Wei, C.-Y. Yau, H.-T. Wai, Y. K. Zhao, D. Kang, Y. Park, and M. Hong. Roste: An efficient quantization-aware supervised fine-tuning approach for large language models. arXiv preprint arXiv:2502.09003, 2025

2025 arXiv
[48]

X. Wei, S. Moalla, R. Pascanu, and C. Gulcehre. Building on efficient foundations: Effective training of LLM s with structured feedforward layers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=WxLVYZbIew

2024
[49]

G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087--38099. PMLR, 2023

2023
[50]

J. Xiao, S. Sang, T. Zhi, J. Liu, Q. Yan, L. Luo, and B. Yuan. COAP : Memory-efficient training with correlation-aware gradient projection. arXiv preprint arXiv:2412.00071, 2024

2024 arXiv
[51]

Z. Yang, Z. Liu, S. Choudhary, X. Xie, C. Gao, S. Kunzmann, and Z. Zhang. Co MERA : Computing- and memory-efficient training via rank-adaptive tensor optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net...

2024
[52]

J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y. Cui, Z. Zhou, C. Gong, Y. Shen, et al. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420, 2023

2023 arXiv
[53]

Zhang and R

B. Zhang and R. Sennrich. Root mean square layer normalization. In NeurIPS, 2019

2019
[54]

Zhang, J

H. Zhang, J. Yin, G. Wang, Z. Liu, T. Zhang, A. Shrivastava, L. Yang, and V. Braverman. I3s: Importance sampling subspace selection for low-rank optimization in llm pretraining. arXiv preprint arXiv:2502.05790, 2025

2025
[55]

Zhang, M

Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

2023 arXiv
[56]

Zhang, C

Y. Zhang, C. Chen, Z. Li, T. Ding, C. Wu, Y. Ye, Z.-Q. Luo, and R. Sun. Adam-mini: Use fewer learning rates to gain more. arXiv preprint arXiv:2406.16793, 2024 a

2024 arXiv
[57]

Zhang, A

Z. Zhang, A. Jaiswal, L. Yin, S. Liu, J. Zhao, Y. Tian, and Z. Wang. Q-galore: Quantized galore with int4 projection and layer-adaptive low-rank gradients. arXiv preprint arXiv:2407.08296, 2024 b

2024 arXiv
[58]

J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian. Galore: Memory-efficient LLM training by gradient low-rank projection. In Forty-first International Conference on Machine Learning, 2024

2024
[59]

Zhou and S

K. Zhou and S. Wang. Revolutionizing large language model training through dynamic parameter adjustment. arXiv preprint arXiv:2406.06564, 2024

2024 arXiv
[60]

H. Zhu, Z. Zhang, W. Cong, X. Liu, S. Park, V. Chandra, B. Long, D. Z. Pan, Z. Wang, and J. Lee. Apollo: Sgd-like memory, adamw-level performance, 2024 a . URL https://arxiv.org/abs/2412.05270

2024 arXiv
[61]

X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 0 1556--1577, 2024 b

2024
[62]

Zmushko, A

P. Zmushko, A. Beznosikov, M. Tak \'a c , and S. Horv \'a th. FRUGAL : Memory-efficient optimization by reducing state overhead for scalable training. arXiv preprint arXiv:2411.07837, 2024

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Achiam, S

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

arXiv 2023

[2] [2]

R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023

arXiv 2023

[3] [3]

D. P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48 0 (3): 0 334--334, 1997

work page 1997

[4] [4]

X. Chen, K. Feng, C. Li, X. Lai, X. Yue, Y. Yuan, and G. Wang. Fira: Can we achieve full-rank training of llms under low-rank constraint? arXiv preprint arXiv:2410.01623, 2024

arXiv 2024

[5] [5]

Exploring Quantization for Efficient Pre-Training of Transformer Language Models

K. Chitsaz, Q. Fournier, G. Mordido, and S. Chandar. Exploring quantization for efficient pre-training of transformer language models. arXiv preprint arXiv:2407.11722, 2024

work page Pith review arXiv 2024

[6] [6]

A. Das. Natural galore: Accelerating galore for memory-efficient llm training and fine-tuning. arXiv preprint arXiv:2410.16029, 2024

work page Pith review arXiv 2024

[7] [7]

Egashira, M

K. Egashira, M. Vero, R. Staab, J. He, and M. Vechev. Exploiting llm quantization. arXiv preprint arXiv:2405.18137, 2024

arXiv 2024

[8] [8]

Garrigos and R

G. Garrigos and R. M. Gower. Handbook of convergence theorems for (stochastic) gradient methods. arXiv preprint arXiv:2301.11235, 2023

arXiv 2023