REVIEW 7 cited by
Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
In this paper, we present Delta-LoRA, which is a novel parameter-efficient approach to fine-tune large language models (LLMs). In contrast to LoRA and other low-rank adaptation methods such as AdaLoRA, Delta-LoRA not only updates the low-rank matrices $\bA$ and $\bB$, but also propagate the learning to the pre-trained weights $\bW$ via updates utilizing the delta of the product of two low-rank matrices ($\bA^{(t+1)}\bB^{(t+1)} - \bA^{(t)}\bB^{(t)}$). Such a strategy effectively addresses the limitation that the incremental update of low-rank matrices is inadequate for learning representations capable for downstream tasks. Moreover, as the update of $\bW$ does not need to compute the gradients of $\bW$ and store their momentums, Delta-LoRA shares comparable memory requirements and computational costs with LoRA. Extensive experiments show that Delta-LoRA significantly outperforms existing low-rank adaptation methods. We further support these results with comprehensive analyses that underscore the effectiveness of Delta-LoRA.
Forward citations
Cited by 7 Pith papers
-
\k{appa}-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating
Selecting the top 50% of LoRA weight matrices by condition number halves trainable parameters and cuts fine-tuning time by about 16% while roughly matching full-LoRA accuracy.
-
K-Merge: Online Continual Merging of Adapters for On-device Large Language Models
K-Merge merges each incoming LoRA with its most similar stored adapter using an order-invariant running average, governed by a storage budget and (in K-Merge++) a similarity threshold, preserving task performance with...
-
TiMoE: Time-Aware Mixture of Language Experts
Training separate language models on disjoint time slices and masking future slices at inference improves temporal grounding on a new time-sensitive QA benchmark, with modest general-task losses.
-
ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints
LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
SingLoRA: Low Rank Adaptation Using a Single Matrix
SingLoRA replaces LoRA's two matrices A and B with one matrix A and the symmetric update AA^T, cutting adapter parameters roughly in half while claiming more stable fine-tuning.
-
MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning
MetaTT factorizes all transformer adapter weights into a single shared tensor-train, achieving LoRA-competitive accuracy with up to 30-40x fewer trainable parameters and a DMRG-inspired rank-adaptive optimizer.
Discussion (0). Continue with ORCID to comment.