REVIEW 7 cited by
Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Fine-tuning is the primary methodology for tailoring pre-trained large language models to specific tasks. As the model's scale and the diversity of tasks expand, parameter-efficient fine-tuning methods are of paramount importance. One of the most widely used family of methods is low-rank adaptation (LoRA) and its variants. LoRA encodes weight update as the product of two low-rank matrices. Despite its advantages, LoRA falls short of full-parameter fine-tuning in terms of generalization error for certain tasks. We introduce Chain of LoRA (COLA), an iterative optimization framework inspired by the Frank-Wolfe algorithm, to bridge the gap between LoRA and full parameter fine-tuning, without incurring additional computational costs or memory overheads. COLA employs a residual learning procedure where it merges learned LoRA modules into the pre-trained language model parameters and re-initilize optimization for new born LoRA modules. We provide theoretical convergence guarantees as well as empirical results to validate the effectiveness of our algorithm. Across various models (OPT and llama-2) and seven benchmarking tasks, we demonstrate that COLA can consistently outperform LoRA without additional computational or memory costs.
Forward citations
Cited by 7 Pith papers
-
GaRA-SAM: Robustifying Segment Anything Model with Gated-Rank Adaptation
GaRA-SAM improves SAM's robustness to image corruption by using input-dependent gating to adjust the effective rank of low-rank adapters, beating prior methods on robust segmentation benchmarks.
-
One Rank at a Time: Cascading Error Dynamics in Sequential Learning
Errors from each rank-1 step in sequential low-rank learning compound through factors that grow when singular values are close, so early steps deserve more compute.
-
HRP: High-Rank Preheating for Superior LoRA Initialization
HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.
-
Continual Gradient Low-Rank Projection Fine-Tuning for LLMs
GORP jointly trains LoRA and full-rank parameters inside a low-rank gradient subspace built from Adam first moments, reporting higher average accuracy and lower forgetting than O-LoRA and N-LoRA on LLM continual learn...
-
A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models
A randomized subspace optimizer cuts activation and optimizer-state memory during LLM training, with convergence guarantees and mostly comparable performance to GaLore and Adam.
-
Geometrically Principled Randomized Optimization for Efficient LLM Training
Randomized Grassmannian subspace updates, combined with Adam-state alignment and residual recovery, produce small evaluation-loss gains over prior low-rank LLM training methods.
-
Mixture of Low Rank Adaptation with Partial Parameter Sharing for Time Series Forecasting
MoLA adapts a pre-trained short-horizon forecaster to multiple forecast steps via segment-specific mixtures of shared low-rank adapters, reporting modest mean-squared-error gains over the base models on most of eight ...
Discussion (0). Continue with ORCID to comment.