REVIEW 7 cited by
Flora: Low-Rank Adapters Are Secretly Gradient Compressors
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite large neural networks demonstrating remarkable abilities to complete different tasks, they require excessive memory usage to store the optimization states for training. To alleviate this, the low-rank adaptation (LoRA) is proposed to reduce the optimization states by training fewer parameters. However, LoRA restricts overall weight update matrices to be low-rank, limiting the model performance. In this work, we investigate the dynamics of LoRA and identify that it can be approximated by a random projection. Based on this observation, we propose Flora, which is able to achieve high-rank updates by resampling the projection matrices while enjoying the sublinear space complexity of optimization states. We conduct experiments across different tasks and model architectures to verify the effectiveness of our approach.
Forward citations
Cited by 7 Pith papers
-
Gradient Multi-Normalization for Stateless and Scalable LLM Training
SinkGD is a stateless optimizer that balances gradients via square-root Sinkhorn iterations, matching or beating Adam and memory-efficient baselines on LLaMA pretraining with SGD-level memory.
-
Low-rank Momentum Factorization for Memory Efficient Training
MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.
-
DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models
A truncated-SVD consensus step for decentralized LoRA is claimed to reach O(1/sqrt T) convergence, matching decentralized SGD, with supporting CLIP and LLAMA2-7B experiments.
-
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
GORP jointly trains LoRA and full-rank parameters inside a low-rank gradient subspace built from Adam first moments, reporting higher average accuracy and lower forgetting than O-LoRA and N-LoRA on LLM continual learn...
-
A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models
A randomized subspace optimizer cuts activation and optimizer-state memory during LLM training, with convergence guarantees and mostly comparable performance to GaLore and Adam.
-
Geometrically Principled Randomized Optimization for Efficient LLM Training
Randomized Grassmannian subspace updates, combined with Adam-state alignment and residual recovery, produce small evaluation-loss gains over prior low-rank LLM training methods.
-
Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
A benchmark and two low-cost tricks (weight refactorization and momentum reset) that make low-rank LLM pre-training competitive with GaLore and Fira at about 25% lower memory.
Discussion (0). Continue with ORCID to comment.