REVIEW 7 cited by
OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The advent of large language models (LLMs) has revolutionized natural language processing, enabling unprecedented capabilities in understanding and generating human-like text. However, the computational cost and convergence times associated with fine-tuning these models remain significant challenges. Low-Rank Adaptation (LoRA) has emerged as a promising method to mitigate these issues by introducing efficient fine-tuning techniques with a reduced number of trainable parameters. In this paper, we present OLoRA, an enhancement to the LoRA method that leverages orthonormal matrix initialization through QR decomposition. OLoRA significantly accelerates the convergence of LLM training while preserving the efficiency benefits of LoRA, such as the number of trainable parameters and GPU memory footprint. Our empirical evaluations demonstrate that OLoRA not only converges faster but also exhibits improved performance compared to standard LoRA across a variety of language modeling tasks. This advancement opens new avenues for more efficient and accessible fine-tuning of LLMs, potentially enabling broader adoption and innovation in natural language applications.
Forward citations
Cited by 7 Pith papers
-
ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation
Combining LoRA with snapshot ensembling yields a parameter-efficient uncertainty-aware segmentation ensemble that matches snapshot full-rank baselines, with feed-forward layers identified as the critical LoRA target.
-
Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
SALR combines static pruning of frozen weights with a trainable truncated-SVD low-rank residual adapter to match LoRA accuracy at 50% sparsity, cutting model size ~2x and giving ~1.7x inference speedup.
-
ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.
-
HRP: High-Rank Preheating for Superior LoRA Initialization
HRP initializes LoRA with the top singular vectors of a briefly preheated high-rank adapter, improving fine-tuning results over random initialization in experiments.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
QR-LoRA: QR-Based Low-Rank Adaptation for Efficient Fine-Tuning of Large Language Models
QR-LoRA trains only scalar coefficients on a pivoted QR basis of pretrained weights, reportedly matching full fine-tuning on GLUE with 601 parameters.
-
Dual Decomposition of Weights and Singular Value Low Rank Adaptation
DuDe combines DoRA's magnitude-direction decomposition with PiSSA's SVD-based initialization, reporting consistent but modest accuracy gains over LoRA, DoRA, and PiSSA on commonsense reasoning, GPQA, MMLU, and GSM8K.
Discussion (0). Continue with ORCID to comment.