Pith. sign in

REVIEW 8 cited by

APOLLO: SGD-like Memory, AdamW-level Performance

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.05270 v4 pith:VPTAFFPP submitted 2024-12-06 cs.LG cs.AIcs.PF

Hanqing Zhu , Zhenyu Zhang , Wenyan Cong , Xi Liu , Sem Park , Vikas Chandra , Bo Long , David Z. Pan
show 2 more authors
Zhangyang Wang Jinwon Lee
This is my paper · ORCID
classification cs.LGcs.AIcs.PF
keywords memoryadamwperformancepre-trainingapollolearningoptimizerrate
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Large language models (LLMs) are notoriously memory-intensive during training, particularly with the popular AdamW optimizer. This memory burden necessitates using more or higher-end GPUs or reducing batch sizes, limiting training scalability and throughput. To address this, various memory-efficient optimizers have been proposed to reduce optimizer memory usage. However, they face critical challenges: (i) reliance on costly SVD operations; (ii) significant performance trade-offs compared to AdamW; and (iii) still substantial optimizer memory overhead to maintain competitive performance. In this work, we identify that AdamW's learning rate adaptation rule can be effectively coarsened as a structured learning rate update. Based on this insight, we propose Approximated Gradient Scaling for Memory-Efficient LLM Optimization (APOLLO), which approximates learning rate scaling using an auxiliary low-rank optimizer state based on pure random projection. This structured learning rate update rule makes APOLLO highly tolerant to further memory reductions while delivering comparable pre-training performance. Even its rank-1 variant, APOLLO-Mini, achieves superior pre-training performance compared to AdamW with SGD-level memory costs. Extensive experiments demonstrate that the APOLLO series performs on-par with or better than AdamW, while achieving greater memory savings by nearly eliminating the optimization states of AdamW. These savings provide significant system-level benefits: (1) Enhanced Throughput: 3x throughput on an 8xA100-80GB setup compared to AdamW by supporting 4x larger batch sizes. (2) Improved Model Scalability: Pre-training LLaMA-13B with naive DDP on A100-80GB GPUs without system-level optimizations. (3) Low-End GPU Friendly Pre-training: Pre-training LLaMA-7B on a single GPU using less than 12 GB of memory with weight quantization.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training

    cs.LG 2026-07 conditional novelty 7.0 of 10

    The top-r gradient subspace in GaLore-family optimizers is statistically non-identifiable beyond ~39 of 128 directions, and the fix is to transport optimizer state across refreshes rather than stabilize the basis.

  2. Gradient Multi-Normalization for Stateless and Scalable LLM Training

    cs.LG 2025-02 conditional novelty 7.0 of 10

    SinkGD is a stateless optimizer that balances gradients via square-root Sinkhorn iterations, matching or beating Adam and memory-efficient baselines on LLaMA pretraining with SGD-level memory.

  3. Gefen: Optimized Stochastic Optimizer

    cs.LG 2026-06 accept novelty 6.5 of 10

    Gefen automatically groups parameters for shared second moments and exact-DP-quantizes first moments, matching AdamW quality at roughly 1/8 the optimizer memory.

  4. Low-rank Momentum Factorization for Memory Efficient Training

    cs.LG 2025-07 reject novelty 6.0 of 10

    MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.

  5. E3D-Bench: A Benchmark for End-to-End 3D Geometric Foundation Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    E3D-Bench compares 16 3D geometric foundation models on depth, reconstruction, pose, and view-synthesis tasks with a unified evaluation toolkit.

  6. Taming LLMs by Scaling Learning Rates with Gradient Grouping

    cs.LG 2025-06 conditional novelty 5.0 of 10

    An optimizer wrapper that clusters per-layer momentum and scales learning rates by cluster-wise median deviations improves perplexity and accuracy across LLM and MLLM training, and lets LoRA pretraining approach full-...

  7. Geometrically Principled Randomized Optimization for Efficient LLM Training

    cs.LG 2025-10 reject novelty 4.0 of 10

    Randomized Grassmannian subspace updates, combined with Adam-state alignment and residual recovery, produce small evaluation-loss gains over prior low-rank LLM training methods.

  8. Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking

    cs.LG 2025-05 conditional novelty 4.0 of 10

    A benchmark and two low-cost tricks (weight refactorization and momentum reset) that make low-rank LLM pre-training competitive with GaLore and Fira at about 25% lower memory.

Pith tools