Pith. sign in

REVIEW 5 major objections 7 minor 2 cited by

Rethinking Weight-Averaged Model-merging

T0 review · 5 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Weight-averaged model merging works because weights are class-specific templates and averaging acts as an implicit regularizer that shrinks weight magnitude and variance, explaining both its successes and its failures.

desk verdict Worth engaging for the empirical comparison, but the regularization explanation is a conjecture, not a proof. read the letter →

arxiv 2411.09263 v5 pith:E7UWY6ES submitted 2024-11-14 cs.LG cs.CV

Hu Wang , Congbo Ma , Ibrahim Almakky , Ian Reid , Gustavo Carneiro , Mohammad Yaqub This is my paper · ORCID
classification cs.LGcs.CV
keywords modelmergingweightaveragingsoupsinterpretabilitytemplatematchingregularizationfeatureensemblingvisiontransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish why weight-averaged model merging—combining the parameters of several trained models into one—works as well as it does. It argues that weights act as class-specific templates, so averaging weights is a meaningful linear combination akin to Mixup applied in weight space, and that averaging shrinks weight magnitude and variance, acting as an implicit regularizer. From these two mechanisms the paper derives concrete predictions: merging helps when models are neuron-compatible, is cheaper than ensembling, and fails for vision transformers and mismatched tasks, where uniform averaging collapses. A sympathetic reader would care because it turns a heuristic trick into a testable account of when untrained model combination is reliable.

What carries the argument

The load-bearing objects are: (1) the template interpretation of weights, where a row of the weight matrix is a class-specific pattern matched to an input by the inner product $w^\top x = \lVert w\rVert\lVert x\rVert\cos\theta$, so merging two weight matrices is a Mixup-like linear combination of templates; (2) the linear-equivalence identity $h = \tfrac{1}{2}(W_1+W_2)x = \tfrac{1}{2}(W_1x + W_2x)$, which holds only without non-linearities and breaks under activations such as ReLU; (3) the norm bound $\lVert (W_1+W_2)/2\rVert \le \max(\lVert W_1\rVert,\lVert W_2\rVert)$ from the triangle inequality and the variance identity $\operatorname{Var}((W_1+W_2)/2) = \frac{1}{4}(\sigma_1^2+\sigma_2^2) \le \max(\sigma_1^2,\sigma_2^2)$, which together show that averaging shrinks weight magnitude and variance; and (4) the Lipschitz bounds (Property 1 and Theorem 1) that connect smaller weight magnitudes and variances to smaller, stabler outputs. These pieces carry the argument from template structure to regularization to the observed architecture- and dataset-dependent behavior.

What would settle it

Train the same architecture on the same task from different random seeds, measure the linear-mode-connectivity barrier between every pair, and check whether the benefits of uniform weight averaging disappear exactly when the barrier is high. A sharper test is to permute the neurons of one model before averaging, which preserves its function but destroys neuron compatibility, and verify that merging accuracy falls to the same catastrophic levels as the PathMNIST and DeiT-Tiny collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that weight-averaged model merging is not arbitrary interpolation but a structured operation on learned templates. Each row of a classifier's weight matrix encodes a class prototype, and inner products match inputs to those prototypes; averaging two weight matrices therefore blends prototype sets the way Mixup blends images, giving the merged model a broader set of patterns to match. At the same time, averaging provably reduces the maximum weight magnitude and the variance of weights, and via the paper's Lipschitz bounds (Property 1 and Theorem 1) this shrinks the magnitude and variance of the network's outputs, making predictions less sensitive to input perturbations. The paper concludes that merging is implicitly a regularizer that trades expressiveness for stability, which is why it can beat individual models on compatible same-task models but degrades sharply for ViTs, for models trained on mismatched tasks, and on datasets like PathMNIST and CelebA where weight conflicts dominate.

Load-bearing premise

The entire template and regularization story assumes the models being averaged are neuron-compatible, meaning they sit in the same loss basin without permutation mismatch; the paper's own experiments show that when this fails, uniform averaging collapses, with accuracy dropping to 32.42% on PathMNIST and 8.91% for DeiT-Tiny on TinyImageNet.

Editorial extensions

If this is right

  • Because averaging shrinks weight magnitude and variance, merged models should be more stable than ensembles under weight perturbation, which the paper's scaling experiments (up to ×100) confirm.
  • Merging should be attempted only among neuron-compatible models; greedy soups, which select models by validation performance, recover much of the loss that uniform soups suffer on incompatible models such as PathMNIST (from 32.42% back to 88.48%).
  • For vision transformers, uniform weight averaging is ineffective and worsens as more models are added, so logit or feature ensembling is the safer strategy for those architectures.
  • Because merging acts as a regularizer, its benefits should be largest when individual models are overfit or noisy and smallest when they are already well-regularized.
  • The template view implies that merging across tasks produces composite templates (for example bicycle plus bottle), which is why cross-task soups can help on one task while sacrificing fine-grained class structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the regularization effect of merging could be repurposed as a no-training defense: averaging weights of models trained with different augmentations should suppress input sensitivity, a prediction that could be tested under adversarial or label-noise perturbations.
  • The template/Mixup analogy suggests a data-free probe for class similarity: interpolating two class templates in weight space and measuring which test images the merged template activates could reveal semantic overlap between classes.
  • One extension the paper leaves implicit is that permutation-alignment should be a mandatory pre-processing step before averaging ViT weights; the observed DeiT collapse predicts that aligned ViTs would recover soup gains.
  • The merging–ensembling gap peaks at a particular magnification factor (×100 on CIFAR-100, ×110 on CIFAR-10), suggesting a sweet spot where weight shrinkage still helps without destroying expressiveness; a practical follow-up is to rescale averaged weights to a target norm before deployment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper presents an interpretability-oriented analysis of weight-averaged model merging. The authors offer three perspectives: (1) interpreting learned weights as class templates, so that weight averaging acts like a Mixup operation in parameter space; (2) empirically comparing weight-space averaging (model merging) with feature-space averaging (model ensembling) across CNNs and ViTs on several datasets; and (3) arguing that merging implicitly regularizes by reducing weight magnitudes and variances, making predictions more robust. The paper concludes with three 'Proposals' that blend these observations into practical guidance. The empirical sections report accuracy comparisons for uniform and greedy soups versus logit/feature ensembles, plus experiments where parameter magnitudes are artificially scaled to study robustness.

Significance. If the central claims were established, the paper would offer a useful intuitive account of why model soups work, a broad empirical map of when merging versus ensembling is preferable, and a concrete robustness mechanism. The empirical scope is genuinely broad: multiple architectures (ResNet, DenseNet, VGG, ViT, DeiT), multiple datasets, and model counts from 2 to 10. The template visualizations are a compelling pedagogical device. However, the theoretical backbone is not sound as written: the variance derivation omits a covariance term, the magnitude argument proves only a non-increase bound rather than 'typical reduction', and the output-magnitude/variance conclusions are not implied by the stated upper bounds. The paper never directly measures whether real merged models have lower weight magnitude or variance than their constituents, so the regularization narrative—which is the paper's central explanatory claim—remains unsupported. The comparative empirical results would still be of interest, but the interpretability thesis needs substantial reworking.

major comments (5)
  1. [The Model Predictions and Weight Magnitudes, Eq. (8)] The variance formula Var((W1+W2)/2) = 1/4(σ1² + σ2²) silently omits the covariance term. For random matrices the correct expression is 1/4(σ1² + σ2² + 2Cov(W1,W2)). Since independently trained networks typically have correlated weights, the subsequent claim that averaging 'always' reduces variance relative to the larger member is not derived. The inequality (9) can be salvaged for any covariance with correlation ≤1, but the derivation as printed is invalid and the assumptions must be stated explicitly, or the general result must be proved.
  2. [Analysis of Weight Magnitude After Averaging, Eqs. (5)–(6)] Equations (5)–(6) establish only the non-increase bound ∥(W1+W2)/2∥ ≤ max(∥W1∥,∥W2∥). The text then asserts that averaging 'typically reduces' the maximum weight magnitude. A non-increase bound does not support a typical-reduction claim; equality occurs when the maximum is attained at the same entry in both matrices, and even near-equal maxima give a reduction only by happenstance. If the authors intend an empirical 'typically', they should measure the actual L∞ norms of merged versus individual models across their experimental settings rather than rely solely on the bound.
  3. [How Weight Magnitudes and Variance Effects Model Outputs, Property 1 and Theorem 1 (Eqs. (10)–(11))] Property 1 and Theorem 1 are upper bounds on output norm and variance under i.i.d. sub-Gaussian assumptions on the weights. The statement that 'the increment of model weights' magnitudes/variances will also magnify outputs' magnitudes/variances' does not follow from upper bounds; monotonicity is not a logical consequence, because cancellations in the weighted sum can make output variance decrease even when weight variance increases. The magnification experiments (Figure 7b, Table 10) artificially scale every parameter by ×90–×100 but never compare the actual weight magnitude/variance of the merged model against its constituents. Consequently, Proposal 3 and the abstract's claim that merging 'implicitly serves as a form of regularization' are not established by the supplied mathematics or experiments. To support this claim, the authors should directly measure norms and variances of real merged weights and test whether output stability actually improves in the un-scaled setting.
  4. [Linear Model Scenarios, Eq. (1), and the template narrative in Section 'The Patterns Contained in Model Weights'] The equivalence between weight averaging and feature averaging in Eq. (1) holds only under the assumption of 'no symmetric neuron mismatch issue.' The paper extends the 'meaningful linear combination' and Mixup-like template interpretation to deep networks without verifying that the trained models lie in the same loss basin or are permutation-compatible. The paper's own results show severe violations of this condition: uniform soups on PathMNIST collapse from 90.04% to 32.42% (Table 1), and DeiT-Tiny on TinyImageNet falls to 8.91% with 10 models (Supplementary Table 9). In these cases the template-matching explanation does not apply, yet the paper does not explicitly condition Proposal 1 on compatibility or discuss how its failures are predicted by the framework. Without an explicit compatibility condition and a discussion of the failure cases, the template narrative is not a falsifiable explanation of when merging will succeed.
  5. [Experiments and Supplementary Tables 1, 3–7, 9] All reported accuracies are single runs without error bars or significance tests. Since the paper's central empirical message is that ensembling consistently outperforms merging and that gaps widen with more models (Figures 3 and 7a), the lack of repeated-seed statistics makes it impossible to assess whether these differences are reliable. Additionally, many training details (optimizer hyperparameters, epochs, augmentation, pretraining initialization, and the exact construction of the 'Ens Features' variant) are deferred to supplementary material without enough specification in the main text to reproduce the comparisons. At minimum, the authors should report mean±std over multiple seeds for the headline tables and figures.
minor comments (7)
  1. [Throughout] The abbreviations 'Grd Ens Lgt', 'Ens Fts', 'Perf Ave' are used inconsistently across captions, tables, and text; please define each abbreviation at first use and use it uniformly.
  2. [Figure 3 and Table 1] The caption of Figure 3 refers to 'Performance Average' while the text and Table 1 use 'Perf Ave'; please unify the terminology.
  3. [Supplementary 'Theoretical Proofs'] The proof section refers to 'Eq. 14' but the main text numbers equations differently; cross-references between main text and supplementary equations should be corrected.
  4. [Section 'The Model Predictions and Weight Magnitudes'] In Figure 4, the scale of the axes and the statement that points are 'computed on the same data batch' are unclear; please describe how the scatter points are generated.
  5. [Supplementary Table 5] The entry '81.95' for 'Grd Ens Fts' on CIFAR-10 with 5 models appears inconsistent with the rest of the column (~92.6) and should be verified.
  6. [References] The citation for Property 1 and Theorem 1 to Wang et al. 2024 should be cited prominently in the main text, not only in the proof appendix, and the applicability of its security-oriented assumptions to trained vision models should be discussed.
  7. [Related Work and Introduction] The reference to model arithmetic uses Mikolov 2013 (word2vec) as a catch-all; please cite the specific model-arithmetic works (e.g., Ilharco et al. 2022, Zhou et al. 2024b) in that sentence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's mathematical identities, bounds, and empirical comparisons do not reduce to its conclusions.

full rationale

After walking the derivation chain, I find no circular step that reduces a prediction to its inputs. The mathematical content in Eqs. 1-4 is an identity showing that weight averaging equals feature averaging for linear layers, and Eqs. 5-9 are elementary bounds on the norm and variance of an average; none of these are fitted to accuracy numbers. Property 1 and Theorem 1 are attributed to Wang et al. 2024 but their proofs are reproduced in the appendix under explicit i.i.d. sub-Gaussian and Lipschitz assumptions, so even if that citation shares an author with the present paper, it is independent support because it does not assume the merging outcome it is used to explain. The magnification experiments are direct empirical interventions on scaled weights, not fitted parameters renamed as predictions, and the paper reports multiple negative results (e.g., uniform soups at 32.42% on PathMNIST and DeiT-Tiny at 8.91% on TinyImageNet) that would be impossible if the narrative were tautologically forced. Concerns that the regularization claim overreaches the upper bounds in Eqs. 10-11 are correctness and interpretation risks, not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No invented entities. The paper introduces no new particles, mediators, forces, or conserved quantities. Its theoretical content reduces to elementary inequalities plus imported bounds. The main unstated costs are the compatibility (basin) assumption and the treatment of trained weights as random variables, neither of which is independently verified in the deep-network experiments.

free parameters (1)
  • Magnitude factors = 90, 100, 110
    Hand-chosen scaling factors for the weight-magnitude experiments (Fig. 7b, Table 10). The conclusion that merging is more robust depends on the tested range; at smaller factors the gap between merging and ensembling is small, and the peak gap occurs at different factors for different datasets.
assumptions (4)
  • domain assumption No symmetric neuron mismatch between models being averaged (weights are permutation-compatible).
    Stated for the linear model in Eq. (1) ('Assuming that there is no symmetric neuron mismatch issue'), and implicitly relied upon when extending the template interpretation to deep networks. Violated in PathMNIST, CelebA, and DeiT experiments, where uniform soups collapse.
  • domain assumption W1 and W2 are independent random matrices when computing Var(W) = 1/4 (sigma1^2 + sigma2^2).
    Eq. (8) omits covariance terms; independence is not stated in the main text. Trained weight matrices are correlated through shared data, initialization, and optimization.
  • domain assumption Trained network weights behave as i.i.d. sub-Gaussian random variables.
    Property 1 and Theorem 1, imported from Wang et al. 2024, require i.i.d. sub-Gaussian weights. Real learned weights are structured and correlated; the paper applies the bound to trained models without justification.
  • standard math Activation functions are Lipschitz continuous with constant L.
    Used in Property 1 and Theorem 1; true for ReLU (L=1) and most common activations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Weight-Averaged Model-merging." pith.science (2026). https://pith.science/paper/E7UWY6ES

@misc{pith2026241109263,
  author       = {Pith},
  title        = {Pith review of: Rethinking Weight-Averaged Model-merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E7UWY6ES}},
  note         = {Machine review of arXiv:2411.09263}
}
read the original abstract

Model merging, particularly through weight averaging, has shown surprising effectiveness in saving computations and improving model performance without any additional training. However, the interpretability of why and how this technique works remains unclear. In this work, we reinterpret weight-averaged model merging through the lens of interpretability and provide empirical insights into the underlying mechanisms that govern its behavior. We approach the problem from three perspectives: (1) we analyze the learned weight structures and demonstrate that model weights encode structured representations that help explain the compatibility of weight averaging; (2) we compare averaging in weight space and feature space across diverse model architectures (CNNs and ViTs) and datasets, aiming to expose under which circumstances what combination paradigm will work more effectively; (3) we study the effect of parameter scaling on prediction stability, highlighting how weight averaging acts as a form of regularization that contributes to robustness. By framing these analyses in an interpretability context, our work contributes to a more transparent and systematic understanding of model merging for stakeholders interested in the safety and reliability of untrained model combination methods. The code is available at https://github.com/billhhh/Rethink-Merge.

Figures

Figures reproduced from arXiv: 2411.09263 by the authors.

Figure 1
Figure 1. Class-wise average images (a) and corresponding linear classifier visualizations (b) on the CIFAR-100 (Krizhevsky [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. We merge the linear classifiers for the first 5 and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Comparison of model merging and ensembling [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (18 more)
Figure 4
Figure 4. Figure 4: Scatter plot comparing the classification accuracies [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: An illustration showing how increasing the magni [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Average of all images for each class and linear classifiers visualization for each class in Tiny ImageNet (Le and Yang [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: (a) Performance Gaps between Logits Ensemble and Uniform Soups Across Different Configurations (excluded [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Per-class average images for all 100 classes of the CIFAR-100 dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Linear classifiers visualization for each class of CIFAR-100 dataset (all 100 classes). [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Merge linear classifiers of the first 50 and last 50 classes in the CIFAR-100 dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Kernel visualization of ResNet50 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Kernel visualization of VGG19 [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Per-class average images for the first 100 classes of the TinyImageNet dataset. [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Per class linear classifier visualization in TinyImageNet dataset (the first 100 classes). [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: Per-class average images for the last 100 classes of the TinyImageNet dataset. [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Per class linear classifier visualization in TinyImageNet dataset (the last 100 classes). [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: (a) Accuracy comparison of 2-model merging and [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: (a) Accuracy comparison of 5-model merging and [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 22
Figure 22. Figure 22: (a) Accuracy vs. Magnification Factor across [PITH_FULL_IMAGE:figures/full_fig_p022_22.png]
Figure 23
Figure 23. Figure 23: (a) Accuracy vs. Magnification Factor across [PITH_FULL_IMAGE:figures/full_fig_p023_23.png]
Figure 24
Figure 24. Figure 24: (a) Accuracy vs. Magnification Factor across [PITH_FULL_IMAGE:figures/full_fig_p023_24.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Task Vector Bases compresses T task vectors into M softmax-mixed basis vectors that preserve task arithmetic operations, with empirical gains over PCA and random selection.

  2. Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging

    cs.LG 2025-01 conditional novelty 6.0 of 10

    A continual model merging method, OPCM, sequentially projects each new task vector into a subspace orthogonal to the current merged model, achieving 5-8% higher average accuracy than baselines on CLIP-ViT tasks.

Reference graph

Works this paper leans on

51 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    K.; Hayase, J.; and Srinivasa, S

    Ainsworth, S. K.; Hayase, J.; and Srinivasa, S. 2022. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836

    arXiv 2022

  4. [4]

    Almakky, I.; Sanjeev, S.; Hashmi, A. U. R.; Qazi, M. A.; and Yaqub, M. 2024. MedMerge: Merging Models for Effective Transfer Learning to Medical Imaging Tasks. arXiv preprint arXiv:2403.11646

    work page Pith review arXiv 2024

  5. [5]

    Croce, F.; Rebuffi, S.-A.; Shelhamer, E.; and Gowal, S. 2023. Seasoning model soups for robustness to adversarial and natural distribution shifts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12313--12323

    work page 2023

  6. [6]

    M.; Gurevych, I.; and Khan, M

    Daheim, N.; M \"o llenhoff, T.; Ponti, E. M.; Gurevych, I.; and Khan, M. E. 2023. Model merging by uncertainty-based gradient matching. arXiv preprint arXiv:2310.12808

    arXiv 2023

  7. [7]

    Dekoninck, J.; Fischer, M.; Beurer-Kellner, L.; and Vechev, M. 2023. Controlled text generation via language model arithmetic. arXiv preprint arXiv:2311.14479

    arXiv 2023

  8. [8]

    Dosovitskiy, A. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    arXiv 2020

Show all 51 references
  1. [9]

    Entezari, R.; Sedghi, H.; Saukh, O.; and Neyshabur, B. 2021. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296

    2021 arXiv

  2. [10]

    P.; and Wilson, A

    Garipov, T.; Izmailov, P.; Podoprikhin, D.; Vetrov, D. P.; and Wilson, A. G. 2018. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31

    2018

  3. [11]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  4. [12]

    Huang, G.; Liu, Z.; Van Der Maaten, L.; and Weinberger, K. Q. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4700--4708

    2017

  5. [13]

    T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A

    Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

    2022 arXiv

  6. [14]

    Jang, D.-H.; Yun, S.; and Han, D. 2024. Model stock: All we need is just a few fine-tuned models. In European Conference on Computer Vision, 207--223. Springer

    2024

  7. [15]

    Jordan, K.; Sedghi, H.; Saukh, O.; Entezari, R.; and Neyshabur, B. 2022. Repair: Renormalizing permuted activations for interpolation repair. arXiv preprint arXiv:2211.08403

    2022 arXiv

  8. [16]

    Khanuja, S.; Johnson, M.; and Talukdar, P. 2021. Mergedistill: Merging pre-trained language models using distillation. arXiv preprint arXiv:2106.02834

    2021 arXiv

  9. [17]

    Krizhevsky, A. 2009. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto

    2009

  10. [18]

    Le, Y.; and Yang, X. 2015. Tiny imagenet visual recognition challenge. CS 231N, 7(7): 3

    2015

  11. [19]

    Li, W.; Peng, Y.; Zhang, M.; Ding, L.; Hu, H.; and Shen, L. 2023. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698

    2023 arXiv

  12. [20]

    Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015. Deep Learning Face Attributes in the Wild. In Proceedings of International Conference on Computer Vision (ICCV)

    2015

  13. [21]

    Ma, C.; Wang, H.; and Hoi, S. C. 2019. Multi-label thoracic disease image classification with cross-attention networks. In Medical Image Computing and Computer Assisted Intervention--MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13--17, 2019, Proceedings...

    2019

  14. [22]

    S.; and Raffel, C

    Matena, M. S.; and Raffel, C. A. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35: 17703--17716

    2022

  15. [23]

    Mikolov, T. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 3781

    2013 arXiv

  16. [24]

    Navon, A.; Shamsian, A.; Fetaya, E.; Chechik, G.; Dym, N.; and Maron, H. 2023. Equivariant deep weight space alignment. arXiv preprint arXiv:2310.13397

    2023 arXiv

  17. [25]

    Ortiz-Jimenez, G.; Favero, A.; and Frossard, P. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36: 66727--66754

    2023

  18. [26]

    Rame, A.; Couairon, G.; Dancette, C.; Gaya, J.-B.; Shukor, M.; Soulier, L.; and Cord, M. 2024. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Sanjeev, S.; Zhaksylyk, N.; Almakky, I.; Hashmi, A. U. R.; Qazi, M. A.; and Yaqub, M. 2024. FissionFusion: fast geometric generation and hierarchical souping for medical image analysis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, ...

    2024

  20. [28]

    Simonyan, K. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

    2014 arXiv

  21. [29]

    P.; Adilova, L.; Kamp, M.; Fischer, A.; Sch \"o lkopf, B.; and Hofmann, T

    Singh, S. P.; Adilova, L.; Kamp, M.; Fischer, A.; Sch \"o lkopf, B.; and Hofmann, T. 2024. Landscaping linear mode connectivity. arXiv preprint arXiv:2406.16300

    2024 arXiv

  22. [30]

    P.; and Jaggi, M

    Singh, S. P.; and Jaggi, M. 2020. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33: 22045--22055

    2020

  23. [31]

    Steiner, A.; Kolesnikov, A.; Zhai, X.; Wightman, R.; Uszkoreit, J.; and Beyer, L. 2021. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.10270

    2021 arXiv

  24. [32]

    Stoica, G.; Bolya, D.; Bjorner, J.; Ramesh, P.; Hearn, T.; and Hoffman, J. 2023. Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.03053

    2023 arXiv

  25. [33]

    Sung, Y.-L.; Li, L.; Lin, K.; Gan, Z.; Bansal, M.; and Wang, L. 2023. An empirical study of multimodal model merging. arXiv preprint arXiv:2304.14933

    2023 arXiv

  26. [34]

    Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J \'e gou, H. 2021. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 10347--10357. PMLR

    2021

  27. [35]

    Tversky, A.; and Gati, I. 1982. Similarity, separability, and the triangle inequality. Psychological review, 89(2): 123

    1982

  28. [36]

    Wang, H.; Almakky, I.; Ma, C.; Saeed, N.; and Yaqub, M. 2025. In-Model Merging for Enhancing the Robustness of Medical Imaging Classification Models. arXiv preprint arXiv:2502.20516

    2025 arXiv

  29. [37]

    Wang, H.; Yurochkin, M.; Sun, Y.; Papailiopoulos, D.; and Khazaeni, Y. 2020. Federated learning with matched averaging. arXiv preprint arXiv:2002.06440

    2020 arXiv

  30. [38]

    Wang, X.; Peng, Y.; Lu, L.; Lu, Z.; Bagheri, M.; and Summers, R. M. 2017. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision an...

    2017

  31. [39]

    Wang, Z.; Ma, Z.; Feng, X.; Sun, R.; Wang, H.; Xue, M.; and Bai, G. 2024. CoreLocker: Neuron-level Usage Control. In 2024 IEEE Symposium on Security and Privacy (SP), 222--222. IEEE Computer Society

    2024

  32. [40]

    Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A

    Wortsman, M.; Ilharco, G.; Gadre, S. Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A. S.; Namkoong, H.; Farhadi, A.; Carmon, Y.; Kornblith, S.; et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Inter...

    2022

  33. [41]

    A.; and Bansal, M

    Yadav, P.; Tam, D.; Choshen, L.; Raffel, C. A.; and Bansal, M. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36

    2024

  34. [42]

    Yang, E.; Shen, L.; Guo, G.; Wang, X.; Cao, X.; Zhang, J.; and Tao, D. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666

    2024 arXiv

  35. [43]

    Yang, E.; Wang, Z.; Shen, L.; Liu, S.; Guo, G.; Wang, X.; and Tao, D. 2023 a . Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575

    2023 arXiv

  36. [44]

    Yang, J.; Shi, R.; and Ni, B. 2021. MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis. In IEEE 18th International Symposium on Biomedical Imaging (ISBI), 191--195

    2021

  37. [45]

    Yang, J.; Shi, R.; Wei, D.; Liu, Z.; Zhao, L.; Ke, B.; Pfister, H.; and Ni, B. 2023 b . MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data, 10(1): 41

    2023

  38. [46]

    Yu, L.; Yu, B.; Yu, H.; Huang, F.; and Li, Y. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning

    2024

  39. [47]

    Zhang, H. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412

    2017 arXiv

  40. [48]

    Zhang, J.; Chi, J.; Li, Z.; Cai, K.; Zhang, Y.; and Tian, Y. 2024. Badmerging: Backdoor attacks against model merging. arXiv preprint arXiv:2408.07362

    2024 arXiv

  41. [49]

    Zhang, J.; Liu, J.; He, J.; et al. 2023. Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36: 12589--12610

    2023

  42. [50]

    Zhou, Y.; Song, L.; Wang, B.; and Chen, W. 2024 a . MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic. arXiv preprint arXiv:2406.11385

    2024 arXiv

  43. [51]

    Zhou, Z.; Chen, Z.; Chen, Y.; Zhang, B.; and Yan, J. 2024 b . On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm. In Forty-first International Conference on Machine Learning (ICML 2024)

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.