REVIEW 6 cited by
ZipIt! Merging Models from Different Tasks without Training
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Typical deep visual recognition models are capable of performing the one task they were trained on. In this paper, we tackle the extremely difficult problem of combining distinct models with different initializations, each solving a separate task, into one multi-task model without any additional training. Prior work in model merging permutes one model to the space of the other then averages them together. While this works for models trained on the same task, we find that this fails to account for the differences in models trained on disjoint tasks. Thus, we introduce "ZipIt!", a general method for merging two arbitrary models of the same architecture that incorporates two simple strategies. First, in order to account for features that aren't shared between models, we expand the model merging problem to allow for merging features within each model by defining a general "zip" operation. Second, we add support for partially zipping the models up until a specified layer, naturally creating a multi-head model. We find that these two changes combined account for 20-60% improvement over prior work, making it more feasible to merge models trained on disjoint tasks without retraining.
Forward citations
Cited by 6 Pith papers
-
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
Initializing LoRA's A matrix with the smallest-eigenvalue eigenvectors of other tasks' feature covariance reduces interference when merging task-specific LoRA models, improving average merged accuracy.
-
Merge to Mix: Mixing Datasets via Model Merging
Merge to Mix shows that the performance of a parameter-averaged model predicts the performance of a model fine-tuned on any dataset mixture, enabling fast and accurate dataset mixture selection.
-
Forget the Data and Fine-Tuning! Just Fold the Network to Compress
Model folding compresses a network by k-means clustering similar neurons across adjacent layers and repairing activation statistics without data (Fold-AR, Fold-DIR), surpassing prior data-free methods at high sparsity.
-
Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
After truncating or expanding checkpoints to a shared shape, small-ratio weight averaging slightly improves average benchmark scores over strong Qwen sources, but headline gains are inflated by per-task best-ratio selection.
-
MorphGen: Morphology-Guided Representation Learning for Robust Single-Domain Generalization in Histopathological Cancer Classification
MorphGen uses supervised contrastive learning to align histopathology images with nuclear masks and applies SWA, reporting improved out-of-domain cancer classification accuracy on CAMELYON17, BCSS, and OCELOT.
-
NAN: A Training-Free Solution to Coefficient Estimation in Model Merging
NAN sets merging coefficients inversely proportional to each model's parameter norm and claims a least-squares justification, but the derivation yields a different formula and performance gains are inconsistent.
Discussion (0). Continue with ORCID to comment.