Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

MergeLock claims a fine-tuned model can be made unmergeable — functionally identical for its owner, yet near-random in any merged copy — by rewriting attention projections with random invertible matrices.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-05 12:25 UTC pith:3GYVRSAK

load-bearing objection MergeLock is a clean, useful extension of symmetry-based merging protection, but the 'cannot be recovered' claim only holds against a rotation-only adversary; the untested general-invertible alignment attack is the load-bearing hole. the 3 major comments →

arxiv 2509.01548 v1 pith:3GYVRSAK submitted 2025-09-01 cs.LG

Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

classification cs.LG
keywords model mergingmodel protectionattention symmetryinvertible matricestask arithmeticlinear mode connectivityparameter alignmentintellectual property
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that a developer can share a fine-tuned model safely by making it unmergeable: the model behaves exactly as before for anyone who uses it, but if anyone tries to merge it with another fine-tuned model, the merged result collapses to near-random accuracy. The mechanism is a symmetry of Transformer attention: for any invertible matrix A, inserting A into the query-key branch and A^(−1) into the value-output branch cancels internally, so every output stays identical while the weights move far from the parameter region where merging works. The paper reports that merging a protected model with an unprotected one degrades merged performance by over 95% in most settings, across vision and language models, three merging algorithms, and two model families, and that data-free alignment recovery restores only a few percentage points on average. If the claim holds, protection of open models becomes preventive rather than merely detective: the lock cannot be stripped without the secret key, while the key lets authorized users restore the original exactly.

Core claim

MergeLock's central claim: Transformer self-attention's gauge symmetry can sever a fine-tuned model from the shared basin that makes merging work, leaving its behaviour unchanged. The authors rewrite each head's query-key product as (XA)(A⁻¹K) and value-output product as (VB)(B⁻¹O), using per-head invertible matrices sampled randomly per layer. The inserted pairs cancel algebraically, so the model is functionally identical, yet the independent sampling pushes its weights out of the linear-mode-connectivity basin. The paper reports merged-model accuracy near random guessing — over 95% degradation in most settings — with orthogonal (Kabsch) alignment recovering only a few percentage points on

What carries the argument

The carrying object is the gauge symmetry of the attention bilinear form: the output depends on products XQ·(XK)ᵀ and XV·(XO)ᵀ, and inserting invertible pairs A, A⁻¹ (and B, B⁻¹) between the factors leaves it unchanged. Unlike the discrete permutation symmetry of MLPs, this symmetry is continuous and high-dimensional, so self-attention is the target. MergeLock samples A = RPD per layer and head — random R, permutation P, diagonal scaling D. The algebra guarantees the performance-preservation condition; the sampling guarantees unmergeability, verified by Frobenius-distance growth (Q&K distance to pretrained weights rises from ~0.3 to ~270 on SUN397) and a loss barrier in linear-mode-connectiv

Load-bearing premise

The claim that locked models cannot be cheaply restored assumes an attacker who aligns the protected model to another using orthogonal rotations only; the paper never tests an attacker who solves for a general invertible alignment that matches the query-key and value-output branches simultaneously, even though the method's own transformations are general invertible matrices.

What would settle it

Run the unconstrained version of the paper's own alignment objective: take a MergeLock-protected ViT-B/32 model and its unprotected counterpart, and per layer and head solve for general invertible matrices A and B minimizing the simultaneous least-squares mismatch ‖AᵀW_Q′ − W_Q‖² + ‖A⁻¹W_K′ − W_K‖² (and the matching value-output pair), then merge and measure accuracy. If any task recovers substantially above the roughly 5% average the paper reports, the claim that low-cost restoration is infeasible fails; the experiment costs about the same compute as the paper's own Kabsch alignment.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Merging any unprotected fine-tuned model with a MergeLock-protected model lands the merged model at near-random accuracy, under Task Arithmetic, Ties-Merging, and AdaMerging alike.
  • Data-free alignment attacks using orthogonal rotations (Kabsch) recover only a few percentage points on average, in contrast to the PaRaMS baseline where alignment restores most of the lost performance.
  • The protection carries across model scale (ViT-B/32, ViT-L/14) and to an encoder–decoder language model (Flan-T5), where transforming self-attention alone is sufficient.
  • Authorized use is preserved: the developer's secret transformation matrices invert the lock exactly, so normal sharing and usage workflows are unchanged.
  • The method shifts model protection from detection (watermarking, fingerprinting) to prevention, since a stolen-and-merged copy is useless rather than merely traceable.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The recovery claim is tested only against orthogonal-rotation alignment; the paper's own algebra allows arbitrary invertible A and B, so an attacker who fits unconstrained general invertible matrices to the query-key and value-output branches simultaneously is never tested, and nothing in the paper rules that attack out.
  • The reported single-digit average recoveries mask large per-cell recoveries in the paper's own tables (e.g., 51.8 and 60.8 percentage points on some vision task pairs after alignment), so 'cannot be effectively recovered' is really a claim about averages, not about every merge.
  • The same symmetry could be used adversarially: a malicious contributor can ship a pre-locked model that behaves normally on its own but silently poisons any merged model that includes it, turning the protection into a supply-chain attack surface.
  • A natural testable extension is layer and head selectivity — the paper locks all self-attention layers; measuring the minimum number of layers (or heads) needed to block merging would reveal both the margin of safety and the cheapest way to attack the lock.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes MergeLock, an active protection mechanism that modifies the self-attention QK and VO parameters of a fine-tuned Transformer using randomly sampled invertible matrices A = RPD and B = RP'D'. The claimed effect is exact output preservation while moving the protected model out of the shared parameter basin, so that merging it with another fine-tuned model produces a severely degraded model. Experiments are reported on ViT-B/32, ViT-L/14, and Flan-T5, with Task Arithmetic, Ties-Merging, and AdaMerging, and the paper reports performance drops of roughly 75-95 percentage points, with only small gains under a rotation-based alignment attack.

Significance. If the claims hold, MergeLock addresses a real and timely problem: preventing unauthorized model merging while preserving the utility of a shared model. The paper has clear strengths: a concrete threat model, a simple and original use of attention symmetries, coverage of multiple merging methods and architectures, and a comparison with the closest existing defense PaRaMS. However, two load-bearing issues prevent acceptance in the current form. First, the output-preservation derivation in Eqs. (8)-(9) is mathematically incorrect for the general invertible matrices used in the method. Second, the robustness claim is evaluated only against an adversary restricted to orthogonal alignment, even though the secret transformations are general invertible; the paper does not test or argue against a natural low-cost general-invertible alignment attack. These issues undermine the central claims that performance is preserved and that recovery is infeasible at low cost.

major comments (3)
  1. [§3.2, Eqs. (8)-(9)] The output-equivalence derivation is incorrect as stated for general invertible A and B. Using the row-vector convention implicit in Eq. (8), q k^T = (q A)(A^{-1} k^T), but the transpose of the second factor is (k A^{-T})^T, not (k A^{-1})^T. Concretely, the K-bias should transform as b_K A^{-T}, not b_K A^{-1}. Similarly, in Eq. (9), the output projection should be W_O B^{-T}, not W_O B^{-1}. As printed, the transformed layer is equal to the original only under extra conditions that do not hold for the A = RPD construction in §3.4, where D is a nontrivial diagonal scaling. The Performance Preservation Condition (Eq. 2) is therefore not established by the derivation. Please correct the transformation formulas or restrict the matrix class accordingly, and verify performance preservation empirically.
  2. [§4.2.2, Eqs. (10)-(11)] The robustness evaluation only considers alignment matrices constrained to be rotations, solved via Kabsch. The secret matrices A and B in §3.4 are general invertible, so an adversary is not restricted to rotations. A low-cost attack would solve, per head, an unconstrained least-squares / Procrustes problem such as min_X ||X^T W_Q^p - W_Q^t||_F^2 + ||X^{-1} W_K^p - W_K^t||_F^2, with an analogous problem for V and O. This has d^2 variables per head and no orthogonality constraint. No experiment or argument in the paper rules out this attack. Since the abstract and §5 claim that low-cost restoration is infeasible, the central robustness claim is currently supported only against a restricted threat model. Please either add this attack to the evaluation or provide a rigorous argument for why it fails.
  3. [Tables 1-5] Performance preservation is not empirically verified. All reported numbers are accuracies of merged models; none of the tables show the accuracy of the protected model by itself. Given the issue in Eqs. (8)-(9), the reader cannot tell whether Eq. (2) holds. This is especially urgent in Table 5, where MergeLock yields exactly 0.0 for every off-diagonal merged entry on Flan-T5, which is consistent with the protected model being non-functional rather than merely unmergeable. Please report the standalone accuracy of protected models (or otherwise verify output equivalence) in every experimental setting.
minor comments (6)
  1. [§3.4] The sampling distribution and scale for the random components R and D are not specified. Since these are the only free parameters of the protection, the paper should state them explicitly for reproducibility and to understand how they affect the magnitude of the parameter shift.
  2. [Tables 1-5] No error bars or standard deviations are reported. Because the protection depends on randomly sampled matrices, the 'over 95%' claim and the alignment-recovery numbers should be accompanied by statistics over several random keys.
  3. [§4.2.2] The displayed optimization in Eqs. (10)-(11) does not show the orthogonality constraint that the text then imposes on R. The constraint should be stated in the equations themselves.
  4. [Fig. 1 caption] The caption says 'A and B can be any matrixes', which conflicts with the corrected math: if A and B are general invertible, the inverse on the key/output side must be a transpose-inverse (A^{-T}, B^{-T}) or the bias/output formulas must change. Please reconcile the caption with the actual implementation.
  5. [Table 1, GTSRB row] There is a typo: '39.63' should presumably be '39.6'.
  6. [§2] The statement that MergeLock is 'considerably harder to reverse or align using purely mathematical methods' is asserted without support. In light of the untested general-invertible attack, either soften this claim or provide evidence.

Circularity Check

0 steps flagged

No significant circularity: the unmergeability claim is an empirically tested outcome, not a by-construction consequence of a fitted parameter or self-citation.

full rationale

The derivation chain is not circular. MergeLock's central claims are evaluated empirically against Task Arithmetic (Table 1), Ties-Merging (Table 3), and AdaMerging (Table 4), with A and B randomly sampled in Sec. 3.4 rather than fitted to the reported merge accuracies. No fitted parameter is renamed as a prediction: the secret transforms are random invertible matrices, and the unmergeability condition in Eq. 3 is a testable outcome, not an assumption of the result. The output-invariance property in Eqs. 8-9 is an algebraic identity derived independently of the target data and of the measured merge performance. The paper's self-citations (e.g., AdaMerging, the model-merging survey) appear only as baselines or related work; the main conclusion also holds against the external TA and Ties-Merging baselines, so these citations are not load-bearing. No uniqueness theorem is imported from the authors, and no ansatz is smuggled in through self-citation. Two caveats are non-circular: (i) the robustness evaluation in Sec. 4.2.2 (Eqs. 10-11) only tests a Kabsch-orthogonal alignment attack, leaving general invertible alignment untested, so the 'cannot be effectively recovered' claim is under-supported against that broader threat model; (ii) the Flan-T5 experiment leaves cross-attention modules unprotected. Both are completeness/correctness concerns, not reductions of the central claim to its own inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

MergeLock's central claim rests on three things the reader is asked to accept from prior literature or from unstated choices: the shared-basin assumption of model merging, a data-free attacker, and the exact cancellation of the invertible-matrix transformation. No new physical entity or fitted parameter is introduced; the main unstated choice is the distribution and scale of the random transformation matrices.

free parameters (1)
  • Sampling distribution and scale for random matrices R and D
    Sec. 3.4 defines A = RPD and B similarly but never specifies the distribution of R or the scaling range of D; the strength of unmergeability, and the difficulty of reversal, depend on these unstated choices.
axioms (3)
  • domain assumption Fine-tuned models from the same pretrained checkpoint lie in a shared loss basin, so small parameter distance is required for successful task-arithmetic merging.
    Invoked in Sec. 3.4 to justify why moving the protected model far from the pretrained parameters breaks merging; this is a prior empirical observation, not proven here.
  • domain assumption The attacker has no access to training data and only performs data-free alignment.
    Sec. 4.2.2 explicitly restricts the adversary to data-free alignment; the robustness conclusion does not cover attackers with data or fine-tuning.
  • standard math Attention transformations in Eqs. 8-9 cancel exactly, including bias terms, for every layer and head.
    Output equivalence requires exact invertibility and consistent application of A and A^{-1} to Q/K and B and B^{-1} to V/O; any approximation would break the performance preservation condition in Eq. 2.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 26871 in / 13864 out tokens · 156647 ms · 2026-08-05T12:25:41.314292+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing." pith.science (2026). https://pith.science/paper/3GYVRSAK

@misc{pith2026250901548,
  author       = {Pith},
  title        = {Pith review of: Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GYVRSAK}},
  note         = {Machine review of arXiv:2509.01548}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Model merging leverages multiple finetuned expert models to construct a multi-task model with low cost, and is gaining increasing attention. However, as a growing number of finetuned models become publicly available, concerns about the safety of model merging have emerged. Unauthorized merging may infringe on developers' rights and risk leaking sensitive personal information. Most existing methods focus on detecting whether a merged model originates from a specific source model, but fail to effectively prevent illegal merging. In this paper, we propose MergeLock, an active protection mechanism that disrupts model parameters to render them unmergeable, thereby directly preventing unauthorized model merging. Specifically, leveraging the inherent symmetry of the attention mechanism in Transformer-based models, we randomly sample two pairs of invertible matrices and apply them to the Query-Key (QK) and Value-Output (VO) branches. This transformation keeps the model's output unchanged while pushing it away from the shared parameter space of other finetuned models. Extensive experiments across both vision and language tasks demonstrate that MergeLock can degrade the performance of merged models by over 95% when a protected model is involved in most cases, demonstrating its effectiveness. Moreover, we further demonstrate that merged models protected by MergeLock cannot be effectively recovered using low-cost restoration methods, further enhancing robustness against unauthorized merging. The code is available at https://github.com/hetailang/Merge-Lock.

Figures

Figures reproduced from arXiv: 2509.01548 by Enneng Yang, Li Shen, Lu Yin, Shiwei Liu, Zihao Wang.

Figure 1
Figure 1. Figure 1: (a) Left: To prevent model merging, transformation matrices A, A−1 , B, B−1 are inserted into the query (Q), key (K), value (V) and output (O) branches of the self-attention layers. This transformation disrupts the parameter space to block effective merging, while preserving the orig￾inal model’s output equivalence. (b) Right: Visualization of our unmergeable strategy in the loss landscape. Normally, two f… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of the threats posed by unauthorized model merging and the protection pro [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (left) Frobenius distance of Q&K weights to pretrained model on SUN397. (right) Frobe￾nius distance of V&O weights to pretrained model on SUN397. • Component D is a diagonal scaling matrix that independently scales each dimension, further en￾larging the distance between models in parameter space without affecting the model’s output. The combination of these three matrices ensures that the transformed model… view at source ↗
Figure 4
Figure 4. Figure 4: Average accuracy of SUN397 (fine-tuned on ViT-B/32) after being merged with each of [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Linearly Mode Connectivity (LMC) curves between pairs of models across four dataset [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Efficient Decentralized Multi-task Dataset Valuation via Model Merging

    cs.CL 2026-07 conditional novelty 7.0

    Task-arithmetic model merging approximates multi-task coalition utilities well enough to recover Dataset Shapley rankings privately and without retraining.

  2. Locking Pretrained Weights via Deep Low-Rank Residual Distillation

    cs.LG 2026-05 unverdicted novelty 7.0

    DLR-Lock locks open-weight LLMs against unauthorized fine-tuning by swapping MLPs for deep low-rank residual networks that inflate backprop memory and complicate optimization, yet preserve original capabilities via mo...

  3. Generalizing the Geometry of Model Merging Through Frechet Averages

    cs.LG 2026-04 unverdicted novelty 7.0

    Model merging is reframed as Fréchet averaging on manifolds whose geometry respects architectural symmetries, generalizing Fisher merging and enabling better LoRA merges.

  4. Generalizing the Geometry of Model Merging Through Frechet Averages

    cs.LG 2026-04 unverdicted novelty 7.0

    Model merging is generalized as Fréchet averaging on symmetry-invariant manifolds, containing Fisher merging as a special case and offering a new approach for LoRA adapters.

  5. Making Models Unmergeable via Scaling-Sensitive Loss Landscape

    cs.AI 2026-01 conditional novelty 7.0

    TRAP² fine-tunes model updates so that standalone accuracy stays high while any re-scaling during merging reliably collapses task accuracy, across transformer and ConvNeXt backbones.

  6. On the Vulnerability of Parameter-Level Defenses to Model Merging

    cs.LG 2026-06 unverdicted novelty 5.0

    Parameter-level defenses for model merging are vulnerable to Anchor-Guided Attack because protected weights are dominated by the pretrained model, and a new defense ARF is introduced to counter it.

Reference graph

Works this paper leans on

71 extracted references · 40 canonical work pages · cited by 5 Pith papers · 3 internal anchors

  1. [1]

    Turning your weakness into a strength: Watermarking deep neural networks by backdooring

    Yossi Adi, Carsten Baum, Moustapha Ciss \' e , Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In William Enck and Adrienne Porter Felt (eds.), 27th USENIX Security Symposium, USENIX Security 2018, Baltimore, MD, USA, August 15-17, 2018 , pp.\ 1615--1631. USENIX Association, 2018. U...

  2. [2]

    Deep learning using rectified linear units (relu)

    Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR, abs/1803.08375, 2018. URL http://arxiv.org/abs/1803.08375

  3. [3]

    Ainsworth, Jonathan Hayase, and Siddhartha S

    Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T

  4. [4]

    Ipguard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary

    Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Ipguard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In Jiannong Cao, Man Ho Au, Zhiqiang Lin, and Moti Yung (eds.), ASIA CCS '21: ACM Asia Conference on Computer and Communications Security, Virtual Event, Hong Kong, June 7-11, 2021 , pp.\ 14--2...

  5. [5]

    Multitask learning

    Rich Caruana. Multitask learning. Machine Learning, 28: 0 41--75, 1997. URL https://api.semanticscholar.org/CorpusID:45998148

  6. [6]

    SWAD: domain generalization by seeking flat minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han - Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: domain generalization by seeking flat minima. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Informa...

  7. [7]

    Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks

    Zhao Chen, Vijay Badrinarayanan, Chen - Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 , volume 80 ...

  8. [8]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE , 105 0 (10): 0 1865--1883, 2017. doi:10.1109/JPROC.2017.2675998. URL https://doi.org/10.1109/JPROC.2017.2675998

  9. [9]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024

  10. [10]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014 , pp.\ 3606--3613. IEEE Computer Society, 2014. doi:10.1109/CVPR.2014.461. URL https://doi.org/10.1109/CVPR.2014.461

  11. [11]

    A unified architecture for natural language processing: deep neural networks with multitask learning

    Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In William W. Cohen, Andrew McCallum, and Sam T. Roweis (eds.), Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 , volume 307 of ACM International C...

  12. [12]

    Have you merged my model? on the robustness of large language model IP protection methods against model merging

    Tianshuo Cong, Delong Ran, Zesen Liu, Xinlei He, Jinyuan Liu, Yichen Gong, Qi Li, Anyu Wang, and Xiaoyun Wang. Have you merged my model? on the robustness of large language model IP protection methods against model merging. In Bo Li, Wenyuan Xu, Jieshan Chen, Yang Zhang, Jason Xue, Shuo Wang, Guangdong Bai, and Xingliang Yuan (eds.), Proceedings of the 1s...

  13. [13]

    The MNIST database of handwritten digit images for machine learning research [best of the web]

    Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. , 29 0 (6): 0 141--142, 2012. doi:10.1109/MSP.2012.2211477. URL https://doi.org/10.1109/MSP.2012.2211477

  14. [14]

    Multi-task learning for multiple language translation

    Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31,...

  15. [15]

    The role of permutation invariance in linear mode connectivity of neural networks

    Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=dNigytemkL

  16. [16]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp.\ 3259--3269. PMLR, 2020

  17. [17]

    Task singular vectors: Reducing task interference in model merging

    Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodol \` a . Task singular vectors: Reducing task interference in model merging. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025 , pp.\ 18695--18705. Computer Vision Found...

  18. [18]

    On the symmetries of deep learning models and their internal representations

    Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 202...

  19. [19]

    Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy

    Zhenyuan Guo, Yi Shi, Wenlong Meng, Chen Gong, Chengkun Wei, and Wenzhi Chen. Be cautious when merging unfamiliar llms: A phishing model capable of stealing privacy. CoRR, abs/2502.11533, 2025. doi:10.48550/ARXIV.2502.11533. URL https://doi.org/10.48550/arXiv.2502.11533

  20. [20]

    Stochastic weight averaging in parallel: Large-batch training that generalizes well

    Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=rygFWAEFwS

  21. [21]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens. , 12 0 (7): 0 2217--2226, 2019. doi:10.1109/JSTARS.2019.2918242. URL https://doi.org/10.1109/JSTARS.2019.2918242

  22. [22]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco T \' u lio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj

  23. [23]

    Vetrov, and Andrew Gordon Wilson

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018 , pp.\ 8...

  24. [24]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc - Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=FCnohuR6AnM

  25. [25]

    W. Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 32 0 (5): 0 922--923, 1976. doi:https://doi.org/10.1107/S0567739476001873. URL https://onlinelibrary.wiley.com/doi/abs/10.1107/S0567739476001873

  26. [26]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei - Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013 , pp.\ 554--561. IEEE Computer Society, 2013. doi:10.1109/ICCVW.2013.77. URL https://doi.org/10.1109/ICCVW.2013.77

  27. [27]

    Efficient backprop

    Yann LeCun, L \'e on Bottou, Genevieve B Orr, and Klaus-Robert M \"u ller. Efficient backprop. In Neural networks: Tricks of the trade, pp.\ 9--50. Springer, 1998

  28. [28]

    Adarank: Adaptive rank pruning for enhanced model merging

    Chanhyuk Lee, Jiho Choi, Chanryeol Lee, Donggyun Kim, and Seunghoon Hong. Adarank: Adaptive rank pruning for enhanced model merging. CoRR, abs/2503.22178, 2025. doi:10.48550/ARXIV.2503.22178. URL https://doi.org/10.48550/arXiv.2503.22178

  29. [29]

    Gshard: Scaling giant models with conditional computation and automatic sharding

    Denis Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020

  30. [30]

    Watermarking llms with weight quantization

    Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, and Xipeng Qiu. Watermarking llms with weight quantization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , pp.\ 3368--3378. Association for Computational Linguistics, 2023 a . doi:10.18653/V...

  31. [31]

    Deep model fusion: A survey

    Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023 b

  32. [32]

    Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pp.\ 1871--1880. Computer Vision Foundation / IEEE , 2019. doi:10.1109/CVPR.2019.00197. URL http://openaccess.thecvf.com/content\_CVPR\_2019/html/L...

  33. [33]

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018 , pp.\ 1930--1939. AC...

  34. [34]

    Bagdanov, and Joost van de Weijer

    Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces. CoRR, abs/2502.04959, 2025. doi:10.48550/ARXIV.2502.04959. URL https://doi.org/10.48550/arXiv.2502.04959

  35. [35]

    Merging models with fisher-weighted averaging

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022...

  36. [36]

    Equivariant architectures for learning in deep weight spaces

    Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, US...

  37. [37]

    Bissacco, Bo Wu, and A

    Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011. URL https://api.semanticscholar.org/CorpusID:16852518

  38. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine L...

  39. [39]

    Multi-task self-supervised learning for robust speech recognition

    Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Jo \ a o Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020 , pp.\ 6989--6993. IEEE , 2020. doi:10.1109...

  40. [40]

    Learning representations by back-propagating errors

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323 0 (6088): 0 533--536, 1986

  41. [41]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

  42. [42]

    Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging

    Li Shen, Anke Tang, Enneng Yang, Guibing Guo, Yong Luo, Lefei Zhang, Xiaochun Cao, Bo Du, and Dacheng Tao. Efficient and effective weight-ensembling mixture of experts for multi-task model merging. CoRR, abs/2410.21804, 2024. doi:10.48550/ARXIV.2410.21804. URL https://doi.org/10.48550/arXiv.2410.21804

  43. [43]

    The german traffic sign recognition benchmark: A multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. In The 2011 International Joint Conference on Neural Networks, IJCNN 2011, San Jose, California, USA, July 31 - August 5, 2011 , pp.\ 1453--1460. IEEE , 2011. doi:10.1109/IJCNN.2011.6033395. URL http...

  44. [44]

    Parameter-efficient multi-task model fusion with partial linearization

    Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=iynRvVVAmH

  45. [45]

    Progressive layered extraction (PLE): A novel multi-task learning (MTL) model for personalized recommendations

    Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (PLE): A novel multi-task learning (MTL) model for personalized recommendations. In Rodrygo L. T. Santos, Leandro Balby Marinho, Elizabeth M. Daly, Li Chen, Kim Falk, Noam Koenigstein, and Edleno Silva de Moura (eds.), RecSys 2020: Fourteenth ACM Conference on Recommende...

  46. [46]

    Least-squares estimation of transformation parameters between two point patterns

    Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Trans. Pattern Anal. Mach. Intell. , 13 0 (4): 0 376--380, 1991. doi:10.1109/34.88573. URL https://doi.org/10.1109/34.88573

  47. [47]

    Weight averaging for neural networks and local resampling schemes

    Joachim Utans. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pp.\ 133--138. Citeseer, 1996

  48. [48]

    Multi-task learning for dense prediction tasks: A survey

    Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 44 0 (7): 0 3614--3633, 2022. doi:10.1109/TPAMI.2021.3054719. URL https://doi.org/10.1109/TPAMI.2021.3054719

  49. [49]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

  50. [50]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

  51. [51]

    Localizing task information for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz - Jim \' e nez, Fran c ois Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=DWT9uiGjxT

  52. [52]

    Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracy

    Junhao Wei, Yu Zhe, and Jun Sakuma. Disrupting model merging: A parameter-level defense without sacrificing accuracy. CoRR, abs/2503.07661, 2025. doi:10.48550/ARXIV.2503.07661. URL https://doi.org/10.48550/arXiv.2503.07661

  53. [53]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song,...

  54. [54]

    Ehinger, James Hays, Antonio Torralba, and Aude Oliva

    Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN database: Exploring a large collection of scene categories. Int. J. Comput. Vis., 119 0 (1): 0 3--22, 2016. doi:10.1007/S11263-014-0748-Y. URL https://doi.org/10.1007/s11263-014-0748-y

  55. [55]

    Instructional fingerprinting of large language models

    Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instructional fingerprinting of large language models. In Kevin Duh, Helena G \' o mez - Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume ...

  56. [56]

    Raffel, and Mohit Bansal

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...

  57. [57]

    Raffel, and Mohit Bansal

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...

  58. [58]

    Adatask: A task-aware adaptive learning rate approach to multi-task learning

    Enneng Yang, Junwei Pan, Ximei Wang, Haibin Yu, Li Shen, Xihua Chen, Lei Xiao, Jie Jiang, and Guibing Guo. Adatask: A task-aware adaptive learning rate approach to multi-task learning. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative App...

  59. [59]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. CoRR, abs/2408.07666, 2024 a . doi:10.48550/ARXIV.2408.07666. URL https://doi.org/10.48550/arXiv.2408.07666

  60. [60]

    Representation surgery for multi-task model merging

    Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 b . URL https://openreview.net/forum?id=Sbl2keQEML

  61. [61]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 c . URL https://openreview.net/forum?id=nZP6NgD3QY

  62. [62]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024

  63. [63]

    Beyond the permutation symmetry of transformers: The role of rotation for model fusion

    Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, and Jundong Li. Beyond the permutation symmetry of transformers: The role of rotation for model fusion. CoRR, abs/2502.00264, 2025. doi:10.48550/ARXIV.2502.00264. URL https://doi.org/10.48550/arXiv.2502.00264

  64. [64]

    Symmetry in neural network parameter spaces

    Bo Zhao, Robin Walters, and Rose Yu. Symmetry in neural network parameter spaces. CoRR, abs/2506.13018, 2025. doi:10.48550/ARXIV.2506.13018. URL https://doi.org/10.48550/arXiv.2506.13018

  65. [65]

    Speech emotion recognition based on multi-task learning

    Huijuan Zhao, Zhijie Han, and Ruchuan Wang. Speech emotion recognition based on multi-task learning. In 5th IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security, BigDataSecurity/HPSC/IDS 2019, Washington, DC, US...

  66. [66]

    Learn from model beyond fine-tuning: A survey

    Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Learn from model beyond fine-tuning: A survey. CoRR, abs/2310.08184, 2023. doi:10.48550/ARXIV.2310.08184. URL https://doi.org/10.48550/arXiv.2310.08184

  67. [67]

    On the emergence of cross-task linearity in pretraining-finetuning paradigm

    Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence of cross-task linearity in pretraining-finetuning paradigm. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=qg6AlnpEQH

  68. [68]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  69. [69]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  70. [70]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  71. [71]

    w/ Alig

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...