Pith. sign in

REVIEW 8 cited by

DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.11617 v1 pith:G6CH3GWB submitted 2024-06-17 cs.CL

Pala Tej Deep , Rishabh Bhardwaj , Soujanya Poria This is my paper · ORCID
classification cs.CL
keywords mergingpointsmagprunemodelmodelsparametersbaselinecode
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the proliferation of domain-specific models, model merging has emerged as a set of techniques that combine the capabilities of multiple models into one that can multitask without the cost of additional training. In this paper, we propose a new model merging technique, Drop and rEscaLe via sampLing with mAgnitude (DELLA-Merging), that employs a novel pruning technique, MAGPRUNE, which shows significant advantages over DARE and TIES. MAGPRUNE first ranks the parameters in order of their magnitude and assigns higher dropout probabilities (p) to parameters with lower ranks corresponding to lower magnitudes. To approximate the original embeddings, MAGPRUNE employs a rescaling operation on the parameters that survive the random dropping by 1/(1 - p). On three different expert models considered for merging (LM, Math, Code) and corresponding benchmark datasets (AlpacaEval, GSM8K, MBPP), DELLA shows an average improvement of 2.4 points over baseline methods employing delta parameter pruning (an improvement of 3.6 points over TIES, 1.2 points over DARE), and 11.1 points over the no-pruning baseline (TA). We release the source code at: https://github.com/declare-lab/della.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A new taxonomy characterizes 48 post-training AI adaptation techniques on six axes and maps them to regulatory documentation requirements.

  2. AP-BMM: Approximating Capability-Cost Pareto Sets of LLMs via Asynchronous Prior-Guided Bayesian Model Merging

    cs.LG 2025-12 reject novelty 6.0 of 10

    The abstract claims asynchronous, GPU-efficient Bayesian model merging, while the full text presents a synchronous method with no wall-clock or asynchrony results.

  3. Training-free LLM Merging for Multi-task Learning

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Hi-Merging merges two task-specialized LLMs by pruning and scaling delta vectors at model and layer level, reporting gains over prior merging and multi-task fine-tuning on English and Chinese MCQA and QA tasks.

  4. Merge Hijacking: Backdoor Attacks to Model Merging of Large Language Models

    cs.CR 2025-05 conditional novelty 6.0 of 10

    Merge Hijacking is a backdoor attack that lets a malicious LLM checkpoint poison any model it is merged with while preserving normal behavior.

  5. Composable Cross-prompt Essay Scoring by Merging Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Merging LoRA adapters with Bayesian-optimized weights guided by a prior-encoded information maximization objective enables source-free cross-prompt essay scoring.

  6. Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

    cs.CL 2025-10 conditional novelty 5.0 of 10

    Mixing a pre-trained model's weights with its instruction-tuned version at an intermediate ratio yields models that are more accurate than the instructed model and substantially better calibrated.

  7. PSO-Merging: Merging Models Based on Particle Swarm Optimization

    cs.LG 2025-08 conditional novelty 5.0 of 10

    PSO-Merging applies particle swarm optimization over model weight space, seeded with original and sparsified experts, to build multitask models that outperform existing merging baselines on several language benchmarks.

  8. Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMs

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Neuron-Fusion selectively restores large-change neurons from a fine-tuned multimodal model and suppresses small changes, improving language retention with modest visual trade-offs.

Pith tools