REVIEW 7 cited by
Model merging with SVD to tie the Knots
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting. We release our code at: https://github.com/gstoica27/KnOTS.
Forward citations
Cited by 7 Pith papers
-
DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation
A hypernetwork generates per-column merging weights to combine source LoRA modules on CLIP, achieving state-of-the-art few-shot test-time domain adaptation.
-
Model soups need only one ingredient
A single checkpoint, edited by splitting each layer's update with SVD and reweighting the high- and low-energy parts, reaches soup-level OOD robustness without multi-model training.
-
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
Initializing LoRA's A matrix with the smallest-eigenvalue eigenvectors of other tasks' feature covariance reduces interference when merging task-specific LoRA models, improving average merged accuracy.
-
GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models
An LLM can memorize a knowledge graph into LoRA weights and answer relation/reasoning queries about it without graph context, but the evaluation partly trains on the test task.
-
Onsager Principle-Based Domain Embedding for Thermodynamically Consistent Cahn-Hilliard Model in Arbitrary Domain
As submitted, the paper is unverifiable: the abstract claims a thermodynamically consistent Cahn-Hilliard domain-embedding derivation, while the full text is a different paper on person re-identification.
-
ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.
-
Tensorized Clustered LoRA Merging for Multi-Task Interference
Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.
Discussion (0). Continue with ORCID to comment.