Pith. sign in

REVIEW 4 major objections 6 minor 64 references

SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A training-free, sample-wise rescaling scheme for merged models, built on the Representation Auto-Adaptation Hypothesis, delivers multi-task accuracy gains on vision and language benchmarks.

desk verdict A solid, useful training-free dynamic merging method whose reported gains are plausible but currently unverifiable because the key hyperparameters are undisclosed. read the letter →

arxiv 2506.18135 v1 pith:QKWNJEWB submitted 2025-06-22 cs.AI cs.CL

Zijun Chen , Zhanpeng Zhou , Bo Zhang , Weinan Zhang , Xi Sun , Junchi Yan This is my paper · ORCID
classification cs.AIcs.CL
keywords modelmergingmulti-tasklearningrepresentationsimilaritytaskarithmetictraining-freeinferencevectorstransformermodelsweightdisentanglement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to explain why model merging works and to use that explanation to make merging better. Its claim is that a merged model succeeds because it separates samples by task and, at deep layers, produces internal representations that match the corresponding fine-tuned expert — the Representation Auto-Adaptation Hypothesis. On that basis the paper builds SE-Merging, a training-free procedure that, for each test sample, computes the $\ell^2$ distance between the merged model's representation and each expert's, converts these distances into similarity scores, and rescales the merging coefficients to upweight the best-matching task vector before inference. The reported outcome is accuracy gains on vision and language benchmarks: +3.86 average points over AdaMerging on ViT-B/32, +0.57 on ViT-L/14, and +6.86 over TIES-Merging on GPT-2, without any additional training. If correct, this gives a simple, compatible plug-in that turns any task-vector-based merge into a per-sample dynamic router.

What carries the argument

The load-bearing object is the Representation Auto-Adaptation Hypothesis (Hypothesis 1) together with the rescaling rule of Algorithm 2. Given a test sample, the mechanism computes $r_{\mathrm{Merged}}=f^{(\ell)}(x;\theta_{\mathrm{Merged}})$ and $r_t=f^{(\ell)}(x;\theta_{PT}+\lambda\tau_t)$ for each task $t$, records the $\ell^2$ distances $d_t=\|r_{\mathrm{Merged}}-r_t\|_2$, converts them into similarities via a min-max normalized reversal, and produces per-sample coefficients $\lambda_t = \frac{\exp(s^{\mathrm{norm}}_t)}{\sum_j \exp(s^{\mathrm{norm}}_j)} T\lambda$. The layer $\ell$ is a hyperparameter; the paper's experiments show that task-specific representations concentrate at deep layers. This machinery turns the merged model itself into an implicit router, so no separate router network or training on test data is needed.

What would settle it

Take the paper's reported layer settings (deep layers such as 12 of ViT-B/32) and a held-out set of tasks not used for evaluation; measure the acc@1 of the similarity-based task identification. If acc@1 is well below 1.0 on that new task set, SE-Merging's gains should shrink or reverse. Additionally, check whether the layer $\ell$ was chosen by maximizing test accuracy: if so, part of the reported gain could be the result of test-set selection rather than the mechanism itself.

Watch

Extended reading notes

Core claim

The central claim is that model merging works because a merged model implicitly adapts to the correct expert for each input: for a sample $x_i$ from task $T_i$, the merged model's representation at layer $\ell$ stays close to that of the fine-tuned model $\theta_{PT}+\lambda\tau_i$ (Hypothesis 1). The paper supports the claim with t-SNE visualizations and with acc@1 measurements showing near-1.0 rates at deep layers (e.g., layer 12 of ViT-B/32 and layer 24 of ViT-L/14), and ties it to the weight-disentanglement condition. The discovery is that this hypothesis can be inverted into a method: instead of training a router, SE-Merging uses representation similarity of the merged model itself to identify the sample's task and then rescales the task-vector coefficients $\lambda_i$ accordingly before classifying. The paper claims this makes static merging methods dynamic and stronger, achieving state-of-the-art results among training-free approaches and outperforming the training-based AdaMerging.

Load-bearing premise

The method stands on the premise that, at the chosen (unreported) layer, the merged model's representation is closer to the correct expert than to any other expert for nearly every test sample; if that similarity ranking is often wrong, the rescaling upweights the wrong task vector and can hurt accuracy.

Editorial extensions

If this is right

  • Any task-vector merge (e.g., Task Arithmetic or TIES-Merging) can be upgraded to per-sample rescaling by adding a forward pass over the experts to compute similarities.
  • Training-free dynamic merging removes the need for routers or unlabeled test-set optimization (as used by AdaMerging), reducing computational cost.
  • The representation-bias measure drops after rescaling, indicating the merged model's inference process more closely mirrors the correct fine-tuned model.
  • The method extends to language models (GPT-2 on seven GLUE tasks) with gains beyond static baselines, suggesting general applicability beyond vision encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the hypothesis holds at deep layers for many architectures, the layer index could be selected automatically per model family (e.g., by a validation proxy) rather than per task set, making the method more robust to hidden test-set tuning.
  • The similarity-based router is essentially a zero-shot surrogate for the 'perfect router' used in prior dynamic merging work; comparing SE-Merging against the perfect-router upper bound would quantify how much task confusion still costs.
  • A natural stress-test is out-of-distribution or cross-domain samples, where deep-layer task separation may fail; the paper's analysis does not yet report such cases.
  • Because the rescaling is a soft weight over task vectors, SE-Merging could be combined with sparsification (like DARE) or conflict-reduction methods to further isolate the right expert's signal.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies model merging from a representation perspective. It proposes the Representation Auto-Adaptation Hypothesis: for a sample from task T_i, the internal representation of a task-arithmetic merged model is closer to the representation of the expert fine-tuned on T_i than to other experts. Building on this, the authors introduce SE-Merging, a training-free, sample-wise method that computes per-sample merging coefficients by comparing inner representations of the merged model with those of each fine-tuned model, then rescaling the task vectors accordingly. The method is evaluated on eight vision tasks with ViT-B/32 and ViT-L/14 and on seven GLUE tasks with GPT-2, reporting average gains of +3.86, +0.57, and +6.86 points over AdaMerging or TIES-Merging. The paper also gives a short theoretical discussion connecting the hypothesis to the existing Weight Disentanglement condition.

Significance. If the empirical claims hold, SE-Merging is a simple and attractive contribution: it is training-free, compatible with static merging methods, and it turns a plausible representation-level observation into a concrete inference-time algorithm. The paper is also useful in that it makes a falsifiable hypothesis explicit and evaluates that hypothesis quantitatively via acc@1. However, the central performance claims currently lack essential experimental reporting: the main free inputs (layer ell and base coefficient lambda) are not disclosed for the main tables, no error bars or seeds are provided, and the language experiments omit a leading adaptive baseline. These gaps are load-bearing because the whole method reduces to the accuracy of the router, so the claimed gains cannot yet be separated from in-sample hyperparameter selection or seed noise. The theoretical section is also weaker than the paper's contribution statement implies, since Equation (4) restates the hypothesis at the output level rather than deriving it.

major comments (4)
  1. [Algorithm 2; Tables 1–3] The two free inputs of the method, the representation layer ell and the base coefficient lambda, are never reported for the main results. Algorithm 2 (lines 1–2) takes ell and lambda as inputs, and Section 3's validation uses lambda=0.3 with ell=12 for ViT-B/32 and ell=24 for ViT-L/14, but Tables 1–3 do not state which values were used or how they were selected. Figure 3, which is computed on the same eight test sets used for Tables 1–2, shows acc@1 close to 1 only at selected deep layers. If ell (and lambda) were chosen by test-set accuracy, then the reported gains of +3.86, +0.57, and +6.86 partly reflect in-sample selection rather than a training-free advantage. Please report the chosen ell and lambda for every table, fix them a priori on a held-out split, or provide a sensitivity analysis showing that results are stable over a range of ell and lambda.
  2. [Tables 1–3] No seeds, runs, or error bars are reported for any result. Some of the claimed improvements are small on individual datasets (e.g., ViT-L/14 average +0.57, SVHN +0.90, DTD +1.28), and without variance estimates these differences cannot be distinguished from seed or run-to-run noise. Please report the mean and standard deviation over at least three independent runs, or explicitly state the single-run protocol used.
  3. [Section 2; Table 3] The language experiments omit AdaMerging, which is the strongest adaptive baseline in the vision tables, so the claim of a 'significant performance gain' in language tasks is only supported relative to static baselines. Please add AdaMerging (or a comparable adaptive training-based method) to Table 3, or restrict the claim to the static baselines actually compared. Additionally, Section 2 lists 'STS-2' as one of the seven GLUE tasks, but Table 3 shows SST-2 and no STS column; please clarify which tasks were used and fix the naming.
  4. [Section 3; Equation (4)] The theoretical support for Hypothesis 1 is circular. Equation (4) is exactly the output-level statement of Hypothesis 1 under the Weight Disentanglement definition, so it does not provide independent evidence for the hypothesis. The paper's contribution list claims a 'theoretical analysis' of the hypothesis, but the analysis assumes the conclusion. Please either remove the theoretical-analysis claim and present Section 3 as an interpretation, or provide a derivation that does not presuppose the hypothesis at the output level.
minor comments (6)
  1. [Algorithm 1, line 4] There is a typo: 'fine-tnued models' should be 'fine-tuned models'.
  2. [Section 4, Performance] The text says 'compare SE-Merging with ine-tuned individual models'; 'ine-tuned' should be 'fine-tuned'.
  3. [Section 2; Table 3] The GLUE task list in the text is inconsistent with Table 3: the text names STS-2 but the table contains SST-2, and the citation [42] is for SST. Please choose a consistent task list and dataset name.
  4. [Algorithm 2, lines 12–15] The symbol lambda is reused both as the initial scalar coefficient and as the per-sample rescaled coefficient lambda_t. Using a different symbol (e.g., alpha_t) for the per-sample coefficients would reduce confusion.
  5. [Figure 4] The caption says 'ℓ1 distance' but the y-axis label is shown as '𝑙! Distance'; please fix the label to read 'ℓ1 Distance'.
  6. [References] Reference [36] is formatted inconsistently with the rest of the bibliography; the author names should follow the standard 'Last, First' style used elsewhere.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the empirical accuracy gains are externally benchmarked; the only circular-adjacent passage is the theoretical 'insight', which restates the hypothesis via a trivial Weight Disentanglement decomposition.

  1. renaming known result [Section 3, 'Theoretical understandings: Insights into Hypothesis 1', Eqs. (3)-(4)]
    "A trivial way to satisfy Definition 1 is to enforce the following decomposition of model f: f(x;θPT+Σ_i αi τi) = Σ_i f(x;θPT+αi τi) 1(x∈Di) + f(x;θPT) 1(x∉∪i Di). Equation (4) indeed coincides with our Hypothesis 1, where for a sample x∈Di, the model automatically adapts to the corresponding finetuned model f(x;θPT+αi τi)."

    The purported theoretical understanding of Hypothesis 1 is obtained by assuming Weight Disentanglement and then writing the trivial routing decomposition. That decomposition is exactly the output-level form of Hypothesis 1: for a sample from Di, the merged model equals the corresponding fine-tuned model. Thus the 'explanation' restates the hypothesis rather than deriving it from independent first principles. The paper partially acknowledges this by noting Eq. (4) concerns only final outputs while the hypothesis concerns inner representations, but as stated the theoretical insight supplies no independent evidence for the inner-representation claim.

full rationale

The central empirical claim — that SE-Merging improves multi-task accuracy over AdaMerging, TIES-Merging, and other baselines — is evaluated on public benchmarks (SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD, and GLUE tasks) against external methods. Those accuracy numbers are not defined in terms of the method's own outputs, so the main reported gains do not reduce by construction to the inputs. The method itself is a well-defined training-free algorithm that computes representation distances, rescales coefficients, and evaluates; no table entry is a fitted parameter renamed as a prediction. The main circularity-adjacent passage is the theoretical section, where the 'insight' is essentially Hypothesis 1 restated under the trivial Weight Disentanglement decomposition; the paper explicitly calls this an initial attempt and, in the Limitations section, concedes that Weight Disentanglement is not rigorously justified. The undisclosed choice of layer ℓ and coefficient λ for the experiments is a reproducibility and potential test-set-selection concern, but the paper does not state that these were selected on the test labels, so it is not a demonstrated circular reduction. Self-citations such as Cross-Task Linearity [62] appear only as background and are not load-bearing. Overall, no core prediction is forced by the paper's inputs by construction; the minor theoretical tautology does not compromise the externally grounded empirical results.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

SE-Merging's central claim rests on two unproven modeling premises: weight disentanglement for the theory, and reliable layer-wise task identification for the method. The main tunable quantities (layer ℓ and base λ) are not reported, which is the most serious ledger gap.

free parameters (3)
  • representation layer ℓ
    Algorithm 2 takes layer ℓ as input; the paper's analysis shows acc@1 varies strongly by layer and the Discussion says the layer choice affects performance, but the main tables do not report which ℓ was used.
  • base scaling coefficient λ = 0.3 for the validation study; main results unreported
    SE-Merging starts from the Task Arithmetic merged model and rescales λ per sample. The reported accuracies depend on this base value, which is not stated for the final experiments.
  • similarity-to-coefficient transform = min-max normalization then softmax without temperature
    The mapping from ℓ2 distances to coefficients is chosen by hand in Algorithm 2; no ablation tests whether this specific transform is necessary.
assumptions (3)
  • domain assumption Weight Disentanglement (Definition 1, from prior work) holds for the fine-tuned models
    Section 3's theoretical insights explicitly assume this condition to connect Hypothesis 1 to task arithmetic; the authors concede in Section 6 that it is not rigorously justified for deep networks.
  • domain assumption Every test sample belongs to exactly one of the T known tasks
    Algorithm 2 picks a softmax over the T task similarities for every sample, so out-of-distribution inputs are forcibly assigned to a known task; this transductive setup is not declared or tested.
  • domain assumption At the chosen layer ℓ, the merged model's representation is closer to the correct expert than to all other experts for the vast majority of samples
    This is the empirical regularity that the router exploits; the paper validates it on the same datasets it later evaluates on, and no held-out guarantee is provided for other distribution shifts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging." pith.science (2026). https://pith.science/paper/QKWNJEWB

@misc{pith2026250618135,
  author       = {Pith},
  title        = {Pith review of: SE-Merging: A Self-Enhanced Approach for Dynamic Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKWNJEWB}},
  note         = {Machine review of arXiv:2506.18135}
}
read the original abstract

Model merging has gained increasing attention due to its intriguing property: interpolating the parameters of different task-specific fine-tuned models leads to multi-task abilities. However, despite its empirical success, the underlying mechanisms of model merging remain poorly understood. In this work, we delve into the mechanism behind model merging from a representation perspective. Our analysis reveals that model merging achieves multi-task abilities through two key capabilities: i) distinguishing samples from different tasks, and ii) adapting to the corresponding expert model for each sample. These two capabilities allow the merged model to retain task-specific expertise, enabling efficient multi-task adaptation. Building on these insights, we propose \texttt{SE-Merging}, a self-enhanced model merging framework that leverages these two characteristics to dynamically identify the corresponding task for each sample and then adaptively rescales the merging coefficients to further enhance task-specific expertise in the merged model. Notably, \texttt{SE-Merging} achieves dynamic model merging without additional training. Extensive experiments demonstrate that \texttt{SE-Merging} achieves significant performance improvements while remaining compatible with existing model merging techniques.

Figures

Figures reproduced from arXiv: 2506.18135 by the authors.

Figure 1
Figure 1. Overview of our self-enhanced model merging framework. (1) Left: Following Task arithmetic [17], we first obtain the merged model via task addition, i.e. θPT + λτ 1 + λτ 2, and the fine-tuned model via θPT + λτ 1 and θPT + λτ 2. Here, τ 1 = θ1 − θPT and τ 2 = θ2 − θPT denote the task vectors of task T1 and T2 respectively. θPT denotes the pre-trained model and θ1 and θ2 denote the models fine-tuned on task T1 and T2… view at source ↗
Figure 2
Figure 2. t-SNE visualization illustrating the two key capabilities. (a) Capability I: Distinguish samples from different tasks. The representations of the merged model on eight vision tasks, i.e. f (ℓ) (x; θMerged), where x ∈ ∪j∈[T]Dj , are seperated and located in different clusters. In addition, for SUN397 dataset, the representations f (ℓ) (xSUN397; θSUN397) and f (ℓ) (xSUN397; θMerged) are located in the same cluster. Th… view at source ↗
Figure 3
Figure 3. Quantitative validation of Hypothesis 1. We report the acc@1 for both ViT-B/32 and ViT￾L/14 across all tasks and different layers. The acc@1 metric is computed as described in Algorithm 1. As shown, at deeper layers, e.g., 12-th layer for ViT-B/32 and 24-th layer for ViT-L/14, acc@1 approaches 1.0 for each task, indicating that for most of the samples in the task, the representation of merged model is closely aligne… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of the ℓ1 distance (“representation bias” in Definition 2) between the representations of merged model and those of the corresponding fine-tuned model on both ViT-B/32 and ViT-L/14 over samples across all tasks. The green bars show the decrease of the rep…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 28 canonical work pages

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    2020

  2. [2]

    Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017. doi: 10.1109/JPROC.2017.2675998

    arXiv 2017

  3. [3]

    On lazy training in differentiable programming

    Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019

    work page 2019

  4. [4]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3606–3613, 2014. doi: 10.1109/CVPR.2014.461

    doi:10.1109/cvpr.2014.461 2014

  5. [5]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005

    2005

  6. [6]

    A survey on ensemble learning

    Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey on ensemble learning. Frontiers of Computer Science, 14:241–258, 2020

    work page 2020

  7. [7]

    Essentially no barriers in neural network energy landscape

    Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. InInternational conference on machine learning, pages 1309–1318. PMLR, 2018

    work page 2018

  8. [8]

    Parameter competition balancing for model merging

    Guodong DU, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, and Min Zhang. Parameter competition balancing for model merging. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

Show all 64 references
  1. [9]

    The role of permutation invariance in linear mode connectivity of neural networks.arXiv preprint arXiv:2110.06296, 2021

    Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks.arXiv preprint arXiv:2110.06296, 2021

    2021 arXiv

  2. [10]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. InInternational Conference on Machine Learning, pages 3259–3269. PMLR, 2020

    2020

  3. [11]

    Daniel Freeman and Joan Bruna

    C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In International Conference on Learning Representations, 2017

    2017

  4. [12]

    Loss surfaces, mode connectivity, and fast ensembling of dnns.Advances in neural information processing systems, 31, 2018

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns.Advances in neural information processing systems, 31, 2018

    2018

  5. [13]

    The third PASCAL recognizing textual entailment challenge

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Satoshi Sekine, Kentaro Inui, Ido Dagan, Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini, editors,Proceedings of the ACL-PASCAL Workshop on Textu...

    2007

  6. [14]

    An empirical evaluation of the t-sne algorithm for data visualization in structural engineering

    Parisa Hajibabaee, Farhad Pourkamali-Anaraki, and Mohammad Amin Hariri-Ardebili. An empirical evaluation of the t-sne algorithm for data visualization in structural engineering. In2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1674–...

    2021

  7. [15]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. doi: 10....

    2019

  8. [16]

    EMR-merging: Tuning- free high-performance model merging

    Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. EMR-merging: Tuning- free high-performance model merging. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  9. [17]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. InThe Eleventh International Conference on Learning Representations, 2023. 11

    2023

  10. [18]

    Neural tangent kernel: Convergence and generalization in neural networks.Advances in neural information processing systems, 31, 2018

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks.Advances in neural information processing systems, 31, 2018

    2018

  11. [19]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  12. [20]

    Scalable optimal transport methods in machine learning: A contemporary survey.IEEE transactions on pattern analysis and machine intelligence, 2024

    Abdelwahed Khamis, Russell Tsuchida, Mohamed Tarek, Vivien Rolland, and Lars Petersson. Scalable optimal transport methods in machine learning: A contemporary survey.IEEE transactions on pattern analysis and machine intelligence, 2024

    2024

  13. [21]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In2013 IEEE International Conference on Computer Vision Workshops, pages 554–561,

  14. [22]

    Explaining landscape connectivity of low-cost solutions for multilayer nets.Advances in neural information processing systems, 32, 2019

    Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets.Advances in neural information processing systems, 32, 2019

    2019

  15. [23]

    The mnist database of handwritten digits

    Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 2005

    2005

  16. [24]

    Understanding the loss surface of neural networks for binary classification

    Shiyu Liang, Ruoyu Sun, Yixuan Li, and Rayadurgam Srikant. Understanding the loss surface of neural networks for binary classification. InInternational Conference on Machine Learning, pages 2835–2843. PMLR, 2018

    2018

  17. [25]

    Twin-merging: Dynamic integration of modular expertise in model merging

    Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  18. [26]

    Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    2022

  19. [27]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y . Ng. Reading digits in natural images with unsupervised feature learning. InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011

    2011

  20. [28]

    On cross-layer alignment for model fusion of heterogeneous neural networks

    Dang Nguyen, Trang Nguyen, Khai Nguyen, Dinh Phung, Hung Bui, and Nhat Ho. On cross-layer alignment for model fusion of heterogeneous neural networks. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  21. [29]

    On connected sublevel sets in deep learning

    Quynh Nguyen. On connected sublevel sets in deep learning. InInternational conference on machine learning, pages 4790–4799. PMLR, 2019

    2019

  22. [30]

    On the loss landscape of a class of deep neural networks with no bad local valleys

    Quynh Nguyen, Mahesh Chandra Mukkamala, and Matthias Hein. On the loss landscape of a class of deep neural networks with no bad local valleys. InInternational Conference on Learning Representations, 2019

    2019

  23. [31]

    Task arithmetic in the tangent space: Improved editing of pre-trained models.Advances in Neural Information Processing Systems, 36, 2024

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025

  25. [33]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  26. [34]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  27. [35]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors,Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 238...

    2016 doi

  28. [36]

    First quora dataset release: question pairs (2017)

    Iyer Shankar, Dandekar Nikhil, and Csernai Kornel. First quora dataset release: question pairs (2017). URL https://www. quora. com/q/quoradata/First-Quora-Dataset-Release-Question-Pairs, 2017

    2017

  29. [37]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. InInternational Conference on Learning Representations, 2017

    2017

  30. [38]

    Relative entropic optimal transport: a (prior-aware) matching perspective to (unbalanced) classification

    Liangliang Shi, Haoyu Zhen, Gu Zhang, and Junchi Yan. Relative entropic optimal transport: a (prior-aware) matching perspective to (unbalanced) classification. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Proces...

    2023

  31. [39]

    OT-CLIP: Understanding and generalizing CLIP via optimal transport

    Liangliang Shi, Jack Fan, and Junchi Yan. OT-CLIP: Understanding and generalizing CLIP via optimal transport. InForty-first International Conference on Machine Learning, 2024

    2024

  32. [40]

    Double-bounded optimal transport for advanced clustering and classification

    Liangliang Shi, Zhaoqi Shen, and Junchi Yan. Double-bounded optimal transport for advanced clustering and classification. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 14982–14990, 2024

    2024

  33. [41]

    Model fusion via optimal transport.Advances in Neural Information Processing Systems, 33:22045–22055, 2020

    Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport.Advances in Neural Information Processing Systems, 33:22045–22055, 2020

    2020

  34. [42]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard,...

    2013

  35. [43]

    The german traffic sign recognition benchmark: A multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. InThe 2011 International Joint Conference on Neural Networks, pages 1453–1460, 2011. doi: 10.1109/IJCNN.2011.6033395

    2011

  36. [44]

    Fusionbench: A comprehensive benchmark of deep model fusion, 2024

    Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A comprehensive benchmark of deep model fusion, 2024

    2024

  37. [45]

    Merging multi-task models via weight-ensembling mixture of experts

    Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. InForty-first International Conference on Machine Learning, 2024

    2024

  38. [46]

    Bandeira, and Joan Bruna

    Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious valleys in one-hidden-layer neural network optimization landscapes.J. Mach. Learn. Res., 20:133:1–133:34, 2019

    2019

  39. [47]

    Springer, 2008

    Cédric Villani et al.Optimal transport: old and new, volume 338. Springer, 2008

    2008

  40. [48]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors,Proceedings of the 2018 EMNLP Workshop Blac...

    2018 doi

  41. [49]

    Mergenas: Merge operations into one for differentiable architecture search

    Xiaoxing Wang, Chao Xue, Junchi Yan, Xiaokang Yang, Yonggang Hu, and Kewei Sun. Mergenas: Merge operations into one for differentiable architecture search. In Christian Bessiere, editor,Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, ...

    2020 doi

  42. [50]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641, 2019. doi: 10.1162/tacl_a_ 00290

    2019 doi

  43. [51]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent, editors,Proceedings of the 2018 Conference of the North American Chapter of the Association for Compu...

    2018

  44. [52]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  45. [53]

    Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing in...

    2022

  46. [54]

    Ehinger, James Hays, Antonio Torralba, and Aude Oliva

    Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories.Int. J. Comput. Vision, 119(1):3–22, August 2016. ISSN 0920-5691. doi: 10.1007/s11263-014-0748-y

    2016 doi

  47. [55]

    Training-free heterogeneous model merging

    Zhengqi Xu, Han Zheng, Jie Song, Li Sun, and Mingli Song. Training-free heterogeneous model merging. arXiv preprint arXiv:2501.00061, 2024

    2024 arXiv

  48. [56]

    Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  49. [57]

    Representation surgery for multi-task model merging

    Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. InForty-first International Conference on Machine Learning, 2024

    2024

  50. [58]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  51. [59]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InForty-first International Conference on Machine Learning

  52. [60]

    Model assembly learning with heterogeneous layer weight merging.arXiv preprint arXiv:2503.21657, 2025

    Yi-Kai Zhang, Jin Wang, Xu-Xiang Zhong, De-Chuan Zhan, and Han-Jia Ye. Model assembly learning with heterogeneous layer weight merging.arXiv preprint arXiv:2503.21657, 2025

    2025 arXiv

  53. [61]

    Going beyond linear mode connectivity: The layerwise linear feature connectivity.Advances in Neural Information Processing Systems, 36:60853–60877, 2023

    Zhanpeng Zhou, Yongyi Yang, Xiaojiang Yang, Junchi Yan, and Wei Hu. Going beyond linear mode connectivity: The layerwise linear feature connectivity.Advances in Neural Information Processing Systems, 36:60853–60877, 2023

    2023

  54. [62]

    On the emergence of cross-task linearity in pretraining-finetuning paradigm

    Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence of cross-task linearity in pretraining-finetuning paradigm. InForty-first International Conference on Machine Learning, 2024. 14

    2024

  55. [2013]

    doi: 10.1109/ICCVW.2013.77

    2013 doi

  56. [2018]

    doi: 10.18653/v1/N18-1101

    Association for Computational Linguistics. doi: 10.18653/v1/N18-1101

    doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.