Pith. sign in

REVIEW 14 cited by

Deep Model Fusion: A Survey

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.15698 v1 pith:WGQKDO72 submitted 2023-09-27 cs.LG cs.AI

Weishi Li , Yong Peng , Miao Zhang , Liang Ding , Han Hu , Li Shen This is my paper · ORCID
classification cs.LGcs.AI
keywords modelfusiondeepmodelsbetterdifferentlearningmethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Deep model fusion/merging is an emerging technique that merges the parameters or predictions of multiple deep learning models into a single one. It combines the abilities of different models to make up for the biases and errors of a single model to achieve better performance. However, deep model fusion on large-scale deep learning models (e.g., LLMs and foundation models) faces several challenges, including high computational cost, high-dimensional parameter space, interference between different heterogeneous models, etc. Although model fusion has attracted widespread attention due to its potential to solve complex real-world tasks, there is still a lack of complete and detailed survey research on this technique. Accordingly, in order to understand the model fusion method better and promote its development, we present a comprehensive survey to summarize the recent progress. Specifically, we categorize existing deep model fusion methods as four-fold: (1) "Mode connectivity", which connects the solutions in weight space via a path of non-increasing loss, in order to obtain better initialization for model fusion; (2) "Alignment" matches units between neural networks to create better conditions for fusion; (3) "Weight average", a classical model fusion method, averages the weights of multiple models to obtain more accurate results closer to the optimal solution; (4) "Ensemble learning" combines the outputs of diverse models, which is a foundational technique for improving the accuracy and robustness of the final model. In addition, we analyze the challenges faced by deep model fusion and propose possible research directions for model fusion in the future. Our review is helpful in deeply understanding the correlation between different model fusion methods and practical application methods, which can enlighten the research in the field of deep model fusion.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.

  2. Personalized Image Aesthetic Assessment via Preference-rich Sample Mining and Cohort Merging

    cs.CV 2026-07 conditional novelty 6.0 of 10

    PRAC mines preference-rich images and merges LoRA adapters from aesthetically similar users to achieve state-of-the-art personalized aesthetic rating prediction.

  3. Forgetting of task-specific knowledge in model merging-based continual learning

    cs.CV 2025-07 accept novelty 6.0 of 10

    Controlled vision experiments show that linear weight merging preserves shared knowledge but rapidly erases task-specific (unshared) knowledge, and merging sequentially trained models is safer than merging parallel-tr...

  4. Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Graft merges two domain-specialized multimodal models by combining channel-wise gating, entropy-based global weighting, and an activation compatibility score to improve fusion without retraining.

  5. Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Weight interpolation between two DeepSeek models, restricted to routed expert tensors, yields a 671B Chimera model with near-R1 reasoning and roughly 40% lower output-token count.

  6. Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    First non-vacuous PAC-Bayes certificates for large vision and language models in the 100-example low-shot regime, obtained by reinterpreting model merging as a low-dimensional posterior.

  7. No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Uniformly rescaling singular values of summed task updates, plus adding task-specific residual directions, improves multi-task model merging.

  8. Efficient and Accurate Method for Separating Variant Components from Invariant Background and Component Model Fusion for Fast RFIC Design Space Exploration

    physics.comp-ph 2026-02 conditional novelty 5.0 of 10

    RFIC design sweeps are accelerated about 37x by splitting the field solution into an invariant-background part (solved once) and a small variant-component part, using layer-wise translation symmetry to build the coupl...

  9. Semantic-guided LoRA Parameters Generation

    cs.LG 2025-09 conditional novelty 5.0 of 10

    SG-LoRA generates LoRA parameters for unseen tasks from text descriptions alone, using semantic expert selection plus a conditional VAE, matching or exceeding oracle fine-tuning on retrieval benchmarks.

  10. Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

    cs.LG 2025-09 conditional novelty 5.0 of 10

    MergeLock applies random invertible matrix transformations to Transformer attention weights, preserving the model's own output while forcing any merged model's accuracy down to near random.

  11. PSO-Merging: Merging Models Based on Particle Swarm Optimization

    cs.LG 2025-08 conditional novelty 5.0 of 10

    PSO-Merging applies particle swarm optimization over model weight space, seeded with original and sparsified experts, to build multitask models that outperform existing merging baselines on several language benchmarks.

  12. ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation

    cs.CV 2025-08 reject novelty 5.0 of 10

    ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.

  13. FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognition

    cs.LG 2025-05 conditional novelty 5.0 of 10

    FCOS combines channel clustering and layer collapse diagnosis to prune AMR models by over 95% with minimal accuracy loss.

  14. SeWA: Selective Weight Average via Probabilistic Masking

    cs.LG 2025-02 conditional novelty 5.0 of 10

    SeWA adaptively selects a sparse set of checkpoints for weight averaging via learned probabilistic masks, claiming better generalization with fewer averaged points.

Pith tools