REVIEW 4 cited by
Task Singular Vectors: Reducing Task Interference in Model Merging
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Task Arithmetic has emerged as a simple yet effective method to merge models without additional training. However, by treating entire networks as flat parameter vectors, it overlooks key structural information and is susceptible to task interference. In this paper, we study task vectors at the layer level, focusing on task layer matrices and their singular value decomposition. In particular, we concentrate on the resulting singular vectors, which we refer to as Task Singular Vectors (TSV). Recognizing that layer task matrices are often low-rank, we propose TSV-Compress (TSV-C), a simple procedure that compresses them to 10% of their original size while retaining 99% of accuracy. We further leverage this low-rank space to define a new measure of task interference based on the interaction of singular vectors from different tasks. Building on these findings, we introduce TSV-Merge (TSV-M), a novel model merging approach that combines compression with interference reduction, significantly outperforming existing methods.
Forward citations
Cited by 4 Pith papers
-
Model soups need only one ingredient
A single checkpoint, edited by splitting each layer's update with SVD and reweighting the high- and low-energy parts, reaches soup-level OOD robustness without multi-model training.
-
Multi-objective Large Language Model Alignment with Hierarchical Experts
HoE claims to align a single LLM to any preference vector over multiple objectives using training-free LoRA experts, lightweight trained routers, and nearest-neighbor preference routing.
-
Composable Cross-prompt Essay Scoring by Merging Models
Merging LoRA adapters with Bayesian-optimized weights guided by a prior-encoded information maximization objective enables source-free cross-prompt essay scoring.
-
Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging
A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimoda...
Discussion (0). Continue with ORCID to comment.