Pith. sign in

REVIEW 4 cited by

Subspace Optimization for Large Language Models with Convergence Guarantees

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.11289 v2 pith:CQAHT5DK submitted 2024-10-15 cs.LG math.OC

Yutong He , Pengrui Li , Yipeng Hu , Chuyan Chen , Kun Yuan This is my paper · ORCID
classification cs.LGmath.OC
keywords convergencegalorelargeoptimizationsubspacealgorithmsgoloregradient
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Subspace optimization algorithms, such as GaLore (Zhao et al., 2024), have gained attention for pre-training and fine-tuning large language models (LLMs) due to their memory efficiency. However, their convergence guarantees remain unclear, particularly in stochastic settings. In this paper, we reveal that GaLore does not always converge to the optimal solution and provide an explicit counterexample to support this finding. We further explore the conditions under which GaLore achieves convergence, showing that it does so when either (i) a sufficiently large mini-batch size is used or (ii) the gradient noise is isotropic. More significantly, we introduce GoLore (Gradient random Low-rank projection), a novel variant of GaLore that provably converges in typical stochastic settings, even with standard batch sizes. Our convergence analysis extends naturally to other subspace optimization algorithms. Finally, we empirically validate our theoretical results and thoroughly test the proposed mechanisms. Codes are available at https://github.com/pkumelon/Golore.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    cs.LG 2026-07 conditional novelty 7.0 of 10

    The top-r gradient subspace in GaLore-family optimizers is statistically non-identifiable beyond ~39 of 128 directions, and the fix is to transport optimizer state across refreshes rather than stabilize the basis.

  2. From PowerSGD to PowerSGD+: Low-Rank Gradient Compression for Distributed Optimization with Convergence Guarantees

    math.OC 2025-09 conditional novelty 6.0 of 10

    PowerSGD can provably fail to converge; the proposed PowerSGD+ with periodic SVD subspace resets converges under standard assumptions at O(1/sqrt(NT)).

  3. A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models

    cs.LG 2025-02 conditional novelty 5.0 of 10

    A randomized subspace optimizer cuts activation and optimizer-state memory during LLM training, with convergence guarantees and mostly comparable performance to GaLore and Adam.

  4. Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A benchmark and two low-cost tricks (weight refactorization and momentum reset) that make low-rank LLM pre-training competitive with GaLore and Fira at about 25% lower memory.

Pith tools