REVIEW 13 cited by
Deep Model Fusion: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Deep model fusion/merging is an emerging technique that merges the parameters or predictions of multiple deep learning models into a single one. It combines the abilities of different models to make up for the biases and errors of a single model to achieve better performance. However, deep model fusion on large-scale deep learning models (e.g., LLMs and foundation models) faces several challenges, including high computational cost, high-dimensional parameter space, interference between different heterogeneous models, etc. Although model fusion has attracted widespread attention due to its potential to solve complex real-world tasks, there is still a lack of complete and detailed survey research on this technique. Accordingly, in order to understand the model fusion method better and promote its development, we present a comprehensive survey to summarize the recent progress. Specifically, we categorize existing deep model fusion methods as four-fold: (1) "Mode connectivity", which connects the solutions in weight space via a path of non-increasing loss, in order to obtain better initialization for model fusion; (2) "Alignment" matches units between neural networks to create better conditions for fusion; (3) "Weight average", a classical model fusion method, averages the weights of multiple models to obtain more accurate results closer to the optimal solution; (4) "Ensemble learning" combines the outputs of diverse models, which is a foundational technique for improving the accuracy and robustness of the final model. In addition, we analyze the challenges faced by deep model fusion and propose possible research directions for model fusion in the future. Our review is helpful in deeply understanding the correlation between different model fusion methods and practical application methods, which can enlighten the research in the field of deep model fusion.
Forward citations
Cited by 13 Pith papers
-
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.
-
Personalized Image Aesthetic Assessment via Preference-rich Sample Mining and Cohort Merging
PRAC mines preference-rich images and merges LoRA adapters from aesthetically similar users to achieve state-of-the-art personalized aesthetic rating prediction.
-
Forgetting of task-specific knowledge in model merging-based continual learning
Controlled vision experiments show that linear weight merging preserves shared knowledge but rapidly erases task-specific (unshared) knowledge, and merging sequentially trained models is safer than merging parallel-tr...
-
Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs
Graft merges two domain-specialized multimodal models by combining channel-wise gating, entropy-based global weighting, and an activation compatibility score to improve fusion without retraining.
-
Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors
Weight interpolation between two DeepSeek models, restricted to routed expert tensors, yields a 671B Chimera model with near-R1 reasoning and roughly 40% lower output-token count.
-
Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning
First non-vacuous PAC-Bayes certificates for large vision and language models in the 100-example low-shot regime, obtained by reinterpreting model merging as a low-dimensional posterior.
-
Efficient and Accurate Method for Separating Variant Components from Invariant Background and Component Model Fusion for Fast RFIC Design Space Exploration
RFIC design sweeps are accelerated about 37x by splitting the field solution into an invariant-background part (solved once) and a small variant-component part, using layer-wise translation symmetry to build the coupl...
-
Semantic-guided LoRA Parameters Generation
SG-LoRA generates LoRA parameters for unseen tasks from text descriptions alone, using semantic expert selection plus a conditional VAE, matching or exceeding oracle fine-tuning on retrieval benchmarks.
-
Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
MergeLock applies random invertible matrix transformations to Transformer attention weights, preserving the model's own output while forcing any merged model's accuracy down to near random.
-
PSO-Merging: Merging Models Based on Particle Swarm Optimization
PSO-Merging applies particle swarm optimization over model weight space, seeded with original and sparsified experts, to build multitask models that outperform existing merging baselines on several language benchmarks.
-
ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.
-
FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognition
FCOS combines channel clustering and layer collapse diagnosis to prune AMR models by over 95% with minimal accuracy loss.
-
SeWA: Selective Weight Average via Probabilistic Masking
SeWA adaptively selects a sparse set of checkpoints for weight averaging via learned probabilistic masks, claiming better generalization with fewer averaged points.
Discussion (0). Continue with ORCID to comment.