REVIEW 3 major objections 6 minor 6 cited by
MergeLock claims a fine-tuned model can be made unmergeable — functionally identical for its owner, yet near-random in any merged copy — by rewriting attention projections with random invertible matrices.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-05 12:25 UTC pith:3GYVRSAK
load-bearing objection MergeLock is a clean, useful extension of symmetry-based merging protection, but the 'cannot be recovered' claim only holds against a rotation-only adversary; the untested general-invertible alignment attack is the load-bearing hole. the 3 major comments →
Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
MergeLock's central claim: Transformer self-attention's gauge symmetry can sever a fine-tuned model from the shared basin that makes merging work, leaving its behaviour unchanged. The authors rewrite each head's query-key product as (XA)(A⁻¹K) and value-output product as (VB)(B⁻¹O), using per-head invertible matrices sampled randomly per layer. The inserted pairs cancel algebraically, so the model is functionally identical, yet the independent sampling pushes its weights out of the linear-mode-connectivity basin. The paper reports merged-model accuracy near random guessing — over 95% degradation in most settings — with orthogonal (Kabsch) alignment recovering only a few percentage points on
What carries the argument
The carrying object is the gauge symmetry of the attention bilinear form: the output depends on products XQ·(XK)ᵀ and XV·(XO)ᵀ, and inserting invertible pairs A, A⁻¹ (and B, B⁻¹) between the factors leaves it unchanged. Unlike the discrete permutation symmetry of MLPs, this symmetry is continuous and high-dimensional, so self-attention is the target. MergeLock samples A = RPD per layer and head — random R, permutation P, diagonal scaling D. The algebra guarantees the performance-preservation condition; the sampling guarantees unmergeability, verified by Frobenius-distance growth (Q&K distance to pretrained weights rises from ~0.3 to ~270 on SUN397) and a loss barrier in linear-mode-connectiv
Load-bearing premise
The claim that locked models cannot be cheaply restored assumes an attacker who aligns the protected model to another using orthogonal rotations only; the paper never tests an attacker who solves for a general invertible alignment that matches the query-key and value-output branches simultaneously, even though the method's own transformations are general invertible matrices.
What would settle it
Run the unconstrained version of the paper's own alignment objective: take a MergeLock-protected ViT-B/32 model and its unprotected counterpart, and per layer and head solve for general invertible matrices A and B minimizing the simultaneous least-squares mismatch ‖AᵀW_Q′ − W_Q‖² + ‖A⁻¹W_K′ − W_K‖² (and the matching value-output pair), then merge and measure accuracy. If any task recovers substantially above the roughly 5% average the paper reports, the claim that low-cost restoration is infeasible fails; the experiment costs about the same compute as the paper's own Kabsch alignment.
If this is right
- Merging any unprotected fine-tuned model with a MergeLock-protected model lands the merged model at near-random accuracy, under Task Arithmetic, Ties-Merging, and AdaMerging alike.
- Data-free alignment attacks using orthogonal rotations (Kabsch) recover only a few percentage points on average, in contrast to the PaRaMS baseline where alignment restores most of the lost performance.
- The protection carries across model scale (ViT-B/32, ViT-L/14) and to an encoder–decoder language model (Flan-T5), where transforming self-attention alone is sufficient.
- Authorized use is preserved: the developer's secret transformation matrices invert the lock exactly, so normal sharing and usage workflows are unchanged.
- The method shifts model protection from detection (watermarking, fingerprinting) to prevention, since a stolen-and-merged copy is useless rather than merely traceable.
Where Pith is reading between the lines
- The recovery claim is tested only against orthogonal-rotation alignment; the paper's own algebra allows arbitrary invertible A and B, so an attacker who fits unconstrained general invertible matrices to the query-key and value-output branches simultaneously is never tested, and nothing in the paper rules that attack out.
- The reported single-digit average recoveries mask large per-cell recoveries in the paper's own tables (e.g., 51.8 and 60.8 percentage points on some vision task pairs after alignment), so 'cannot be effectively recovered' is really a claim about averages, not about every merge.
- The same symmetry could be used adversarially: a malicious contributor can ship a pre-locked model that behaves normally on its own but silently poisons any merged model that includes it, turning the protection into a supply-chain attack surface.
- A natural testable extension is layer and head selectivity — the paper locks all self-attention layers; measuring the minimum number of layers (or heads) needed to block merging would reveal both the margin of safety and the cheapest way to attack the lock.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MergeLock, an active protection mechanism that modifies the self-attention QK and VO parameters of a fine-tuned Transformer using randomly sampled invertible matrices A = RPD and B = RP'D'. The claimed effect is exact output preservation while moving the protected model out of the shared parameter basin, so that merging it with another fine-tuned model produces a severely degraded model. Experiments are reported on ViT-B/32, ViT-L/14, and Flan-T5, with Task Arithmetic, Ties-Merging, and AdaMerging, and the paper reports performance drops of roughly 75-95 percentage points, with only small gains under a rotation-based alignment attack.
Significance. If the claims hold, MergeLock addresses a real and timely problem: preventing unauthorized model merging while preserving the utility of a shared model. The paper has clear strengths: a concrete threat model, a simple and original use of attention symmetries, coverage of multiple merging methods and architectures, and a comparison with the closest existing defense PaRaMS. However, two load-bearing issues prevent acceptance in the current form. First, the output-preservation derivation in Eqs. (8)-(9) is mathematically incorrect for the general invertible matrices used in the method. Second, the robustness claim is evaluated only against an adversary restricted to orthogonal alignment, even though the secret transformations are general invertible; the paper does not test or argue against a natural low-cost general-invertible alignment attack. These issues undermine the central claims that performance is preserved and that recovery is infeasible at low cost.
major comments (3)
- [§3.2, Eqs. (8)-(9)] The output-equivalence derivation is incorrect as stated for general invertible A and B. Using the row-vector convention implicit in Eq. (8), q k^T = (q A)(A^{-1} k^T), but the transpose of the second factor is (k A^{-T})^T, not (k A^{-1})^T. Concretely, the K-bias should transform as b_K A^{-T}, not b_K A^{-1}. Similarly, in Eq. (9), the output projection should be W_O B^{-T}, not W_O B^{-1}. As printed, the transformed layer is equal to the original only under extra conditions that do not hold for the A = RPD construction in §3.4, where D is a nontrivial diagonal scaling. The Performance Preservation Condition (Eq. 2) is therefore not established by the derivation. Please correct the transformation formulas or restrict the matrix class accordingly, and verify performance preservation empirically.
- [§4.2.2, Eqs. (10)-(11)] The robustness evaluation only considers alignment matrices constrained to be rotations, solved via Kabsch. The secret matrices A and B in §3.4 are general invertible, so an adversary is not restricted to rotations. A low-cost attack would solve, per head, an unconstrained least-squares / Procrustes problem such as min_X ||X^T W_Q^p - W_Q^t||_F^2 + ||X^{-1} W_K^p - W_K^t||_F^2, with an analogous problem for V and O. This has d^2 variables per head and no orthogonality constraint. No experiment or argument in the paper rules out this attack. Since the abstract and §5 claim that low-cost restoration is infeasible, the central robustness claim is currently supported only against a restricted threat model. Please either add this attack to the evaluation or provide a rigorous argument for why it fails.
- [Tables 1-5] Performance preservation is not empirically verified. All reported numbers are accuracies of merged models; none of the tables show the accuracy of the protected model by itself. Given the issue in Eqs. (8)-(9), the reader cannot tell whether Eq. (2) holds. This is especially urgent in Table 5, where MergeLock yields exactly 0.0 for every off-diagonal merged entry on Flan-T5, which is consistent with the protected model being non-functional rather than merely unmergeable. Please report the standalone accuracy of protected models (or otherwise verify output equivalence) in every experimental setting.
minor comments (6)
- [§3.4] The sampling distribution and scale for the random components R and D are not specified. Since these are the only free parameters of the protection, the paper should state them explicitly for reproducibility and to understand how they affect the magnitude of the parameter shift.
- [Tables 1-5] No error bars or standard deviations are reported. Because the protection depends on randomly sampled matrices, the 'over 95%' claim and the alignment-recovery numbers should be accompanied by statistics over several random keys.
- [§4.2.2] The displayed optimization in Eqs. (10)-(11) does not show the orthogonality constraint that the text then imposes on R. The constraint should be stated in the equations themselves.
- [Fig. 1 caption] The caption says 'A and B can be any matrixes', which conflicts with the corrected math: if A and B are general invertible, the inverse on the key/output side must be a transpose-inverse (A^{-T}, B^{-T}) or the bias/output formulas must change. Please reconcile the caption with the actual implementation.
- [Table 1, GTSRB row] There is a typo: '39.63' should presumably be '39.6'.
- [§2] The statement that MergeLock is 'considerably harder to reverse or align using purely mathematical methods' is asserted without support. In light of the untested general-invertible attack, either soften this claim or provide evidence.
Circularity Check
No significant circularity: the unmergeability claim is an empirically tested outcome, not a by-construction consequence of a fitted parameter or self-citation.
full rationale
The derivation chain is not circular. MergeLock's central claims are evaluated empirically against Task Arithmetic (Table 1), Ties-Merging (Table 3), and AdaMerging (Table 4), with A and B randomly sampled in Sec. 3.4 rather than fitted to the reported merge accuracies. No fitted parameter is renamed as a prediction: the secret transforms are random invertible matrices, and the unmergeability condition in Eq. 3 is a testable outcome, not an assumption of the result. The output-invariance property in Eqs. 8-9 is an algebraic identity derived independently of the target data and of the measured merge performance. The paper's self-citations (e.g., AdaMerging, the model-merging survey) appear only as baselines or related work; the main conclusion also holds against the external TA and Ties-Merging baselines, so these citations are not load-bearing. No uniqueness theorem is imported from the authors, and no ansatz is smuggled in through self-citation. Two caveats are non-circular: (i) the robustness evaluation in Sec. 4.2.2 (Eqs. 10-11) only tests a Kabsch-orthogonal alignment attack, leaving general invertible alignment untested, so the 'cannot be effectively recovered' claim is under-supported against that broader threat model; (ii) the Flan-T5 experiment leaves cross-attention modules unprotected. Both are completeness/correctness concerns, not reductions of the central claim to its own inputs.
Axiom & Free-Parameter Ledger
free parameters (1)
- Sampling distribution and scale for random matrices R and D
axioms (3)
- domain assumption Fine-tuned models from the same pretrained checkpoint lie in a shared loss basin, so small parameter distance is required for successful task-arithmetic merging.
- domain assumption The attacker has no access to training data and only performs data-free alignment.
- standard math Attention transformations in Eqs. 8-9 cancel exactly, including bias terms, for every layer and head.
Cite this review
Pith. "Pith review of Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing." pith.science (2026). https://pith.science/paper/3GYVRSAK
@misc{pith2026250901548,
author = {Pith},
title = {Pith review of: Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing},
year = {2026},
howpublished = {\url{https://pith.science/paper/3GYVRSAK}},
note = {Machine review of arXiv:2509.01548}
}
read the original abstract
Model merging leverages multiple finetuned expert models to construct a multi-task model with low cost, and is gaining increasing attention. However, as a growing number of finetuned models become publicly available, concerns about the safety of model merging have emerged. Unauthorized merging may infringe on developers' rights and risk leaking sensitive personal information. Most existing methods focus on detecting whether a merged model originates from a specific source model, but fail to effectively prevent illegal merging. In this paper, we propose MergeLock, an active protection mechanism that disrupts model parameters to render them unmergeable, thereby directly preventing unauthorized model merging. Specifically, leveraging the inherent symmetry of the attention mechanism in Transformer-based models, we randomly sample two pairs of invertible matrices and apply them to the Query-Key (QK) and Value-Output (VO) branches. This transformation keeps the model's output unchanged while pushing it away from the shared parameter space of other finetuned models. Extensive experiments across both vision and language tasks demonstrate that MergeLock can degrade the performance of merged models by over 95% when a protected model is involved in most cases, demonstrating its effectiveness. Moreover, we further demonstrate that merged models protected by MergeLock cannot be effectively recovered using low-cost restoration methods, further enhancing robustness against unauthorized merging. The code is available at https://github.com/hetailang/Merge-Lock.
Figures
Forward citations
Cited by 6 Pith papers
-
Efficient Decentralized Multi-task Dataset Valuation via Model Merging
Task-arithmetic model merging approximates multi-task coalition utilities well enough to recover Dataset Shapley rankings privately and without retraining.
-
Locking Pretrained Weights via Deep Low-Rank Residual Distillation
DLR-Lock locks open-weight LLMs against unauthorized fine-tuning by swapping MLPs for deep low-rank residual networks that inflate backprop memory and complicate optimization, yet preserve original capabilities via mo...
-
Generalizing the Geometry of Model Merging Through Frechet Averages
Model merging is reframed as Fréchet averaging on manifolds whose geometry respects architectural symmetries, generalizing Fisher merging and enabling better LoRA merges.
-
Generalizing the Geometry of Model Merging Through Frechet Averages
Model merging is generalized as Fréchet averaging on symmetry-invariant manifolds, containing Fisher merging as a special case and offering a new approach for LoRA adapters.
-
Making Models Unmergeable via Scaling-Sensitive Loss Landscape
TRAP² fine-tunes model updates so that standalone accuracy stays high while any re-scaling during merging reliably collapses task accuracy, across transformer and ConvNeXt backbones.
-
On the Vulnerability of Parameter-Level Defenses to Model Merging
Parameter-level defenses for model merging are vulnerable to Anchor-Guided Attack because protected weights are dominated by the pretrained model, and a new defense ARF is introduced to counter it.
Reference graph
Works this paper leans on
-
[1]
Turning your weakness into a strength: Watermarking deep neural networks by backdooring
Yossi Adi, Carsten Baum, Moustapha Ciss \' e , Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In William Enck and Adrienne Porter Felt (eds.), 27th USENIX Security Symposium, USENIX Security 2018, Baltimore, MD, USA, August 15-17, 2018 , pp.\ 1615--1631. USENIX Association, 2018. U...
work page 2018
-
[2]
Deep learning using rectified linear units (relu)
Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR, abs/1803.08375, 2018. URL http://arxiv.org/abs/1803.08375
Pith/arXiv arXiv 2018
-
[3]
Ainsworth, Jonathan Hayase, and Siddhartha S
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T
work page 2023
-
[4]
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Ipguard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In Jiannong Cao, Man Ho Au, Zhiqiang Lin, and Moti Yung (eds.), ASIA CCS '21: ACM Asia Conference on Computer and Communications Security, Virtual Event, Hong Kong, June 7-11, 2021 , pp.\ 14--2...
-
[5]
Rich Caruana. Multitask learning. Machine Learning, 28: 0 41--75, 1997. URL https://api.semanticscholar.org/CorpusID:45998148
work page 1997
-
[6]
SWAD: domain generalization by seeking flat minima
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han - Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: domain generalization by seeking flat minima. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Informa...
work page 2021
-
[7]
Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks
Zhao Chen, Vijay Badrinarayanan, Chen - Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July 10-15, 2018 , volume 80 ...
work page 2018
-
[8]
Remote sensing image scene classification: Benchmark and state of the art
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE , 105 0 (10): 0 1865--1883, 2017. doi:10.1109/JPROC.2017.2675998. URL https://doi.org/10.1109/JPROC.2017.2675998
-
[9]
Scaling instruction-finetuned language models
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25 0 (70): 0 1--53, 2024
2024
-
[10]
Describing textures in the wild
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014 , pp.\ 3606--3613. IEEE Computer Society, 2014. doi:10.1109/CVPR.2014.461. URL https://doi.org/10.1109/CVPR.2014.461
-
[11]
A unified architecture for natural language processing: deep neural networks with multitask learning
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning. In William W. Cohen, Andrew McCallum, and Sam T. Roweis (eds.), Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008 , volume 307 of ACM International C...
-
[12]
Tianshuo Cong, Delong Ran, Zesen Liu, Xinlei He, Jinyuan Liu, Yichen Gong, Qi Li, Anyu Wang, and Xiaoyun Wang. Have you merged my model? on the robustness of large language model IP protection methods against model merging. In Bo Li, Wenyuan Xu, Jieshan Chen, Yang Zhang, Jason Xue, Shuo Wang, Guangdong Bai, and Xingliang Yuan (eds.), Proceedings of the 1s...
-
[13]
The MNIST database of handwritten digit images for machine learning research [best of the web]
Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. , 29 0 (6): 0 141--142, 2012. doi:10.1109/MSP.2012.2211477. URL https://doi.org/10.1109/MSP.2012.2211477
arXiv 2012
-
[14]
Multi-task learning for multiple language translation
Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31,...
-
[15]
The role of permutation invariance in linear mode connectivity of neural networks
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=dNigytemkL
work page 2022
-
[16]
Linear mode connectivity and the lottery ticket hypothesis
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp.\ 3259--3269. PMLR, 2020
2020
-
[17]
Task singular vectors: Reducing task interference in model merging
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodol \` a . Task singular vectors: Reducing task interference in model merging. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025 , pp.\ 18695--18705. Computer Vision Found...
work page 2025
-
[18]
On the symmetries of deep learning models and their internal representations
Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge. On the symmetries of deep learning models and their internal representations. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 202...
work page 2022
-
[19]
Be Cautious When Merging Unfamiliar LLMs: A Phishing Model Capable of Stealing Privacy
Zhenyuan Guo, Yi Shi, Wenlong Meng, Chen Gong, Chengkun Wei, and Wenzhi Chen. Be cautious when merging unfamiliar llms: A phishing model capable of stealing privacy. CoRR, abs/2502.11533, 2025. doi:10.48550/ARXIV.2502.11533. URL https://doi.org/10.48550/arXiv.2502.11533
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2502.11533 2025
-
[20]
Stochastic weight averaging in parallel: Large-batch training that generalizes well
Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=rygFWAEFwS
work page 2020
-
[21]
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens. , 12 0 (7): 0 2217--2226, 2019. doi:10.1109/JSTARS.2019.2918242. URL https://doi.org/10.1109/JSTARS.2019.2918242
-
[22]
Editing models with task arithmetic
Gabriel Ilharco, Marco T \' u lio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj
work page 2023
-
[23]
Vetrov, and Andrew Gordon Wilson
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018 , pp.\ 8...
work page 2018
-
[24]
Dataless knowledge fusion by merging weights of language models
Xisen Jin, Xiang Ren, Daniel Preotiuc - Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id=FCnohuR6AnM
work page 2023
-
[25]
W. Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 32 0 (5): 0 922--923, 1976. doi:https://doi.org/10.1107/S0567739476001873. URL https://onlinelibrary.wiley.com/doi/abs/10.1107/S0567739476001873
-
[26]
3d object representations for fine-grained categorization
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei - Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2013, Sydney, Australia, December 1-8, 2013 , pp.\ 554--561. IEEE Computer Society, 2013. doi:10.1109/ICCVW.2013.77. URL https://doi.org/10.1109/ICCVW.2013.77
-
[27]
Yann LeCun, L \'e on Bottou, Genevieve B Orr, and Klaus-Robert M \"u ller. Efficient backprop. In Neural networks: Tricks of the trade, pp.\ 9--50. Springer, 1998
work page 1998
-
[28]
Adarank: Adaptive rank pruning for enhanced model merging
Chanhyuk Lee, Jiho Choi, Chanryeol Lee, Donggyun Kim, and Seunghoon Hong. Adarank: Adaptive rank pruning for enhanced model merging. CoRR, abs/2503.22178, 2025. doi:10.48550/ARXIV.2503.22178. URL https://doi.org/10.48550/arXiv.2503.22178
-
[29]
Gshard: Scaling giant models with conditional computation and automatic sharding
Denis Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020
Pith/arXiv arXiv 2006
-
[30]
Watermarking llms with weight quantization
Linyang Li, Botian Jiang, Pengyu Wang, Ke Ren, Hang Yan, and Xipeng Qiu. Watermarking llms with weight quantization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , pp.\ 3368--3378. Association for Computational Linguistics, 2023 a . doi:10.18653/V...
-
[31]
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023 b
Pith/arXiv arXiv 2023
-
[32]
Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019 , pp.\ 1871--1880. Computer Vision Foundation / IEEE , 2019. doi:10.1109/CVPR.2019.00197. URL http://openaccess.thecvf.com/content\_CVPR\_2019/html/L...
-
[33]
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In Yike Guo and Faisal Farooq (eds.), Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018 , pp.\ 1930--1939. AC...
arXiv 2018
-
[34]
Bagdanov, and Joost van de Weijer
Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartlomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces. CoRR, abs/2502.04959, 2025. doi:10.48550/ARXIV.2502.04959. URL https://doi.org/10.48550/arXiv.2502.04959
-
[35]
Merging models with fisher-weighted averaging
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, 2022...
work page 2022
-
[36]
Equivariant architectures for learning in deep weight spaces
Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, US...
work page 2023
-
[37]
Bissacco, Bo Wu, and A
Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011. URL https://api.semanticscholar.org/CorpusID:16852518
2011
-
[38]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine L...
work page 2021
-
[39]
Multi-task self-supervised learning for robust speech recognition
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Jo \ a o Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020 , pp.\ 6989--6993. IEEE , 2020. doi:10.1109...
arXiv 2020
-
[40]
Learning representations by back-propagating errors
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Nature, 323 0 (6088): 0 533--536, 1986
work page 1986
-
[41]
Glu variants improve transformer
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020
Pith/arXiv arXiv 2002
-
[42]
Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging
Li Shen, Anke Tang, Enneng Yang, Guibing Guo, Yong Luo, Lefei Zhang, Xiaochun Cao, Bo Du, and Dacheng Tao. Efficient and effective weight-ensembling mixture of experts for multi-task model merging. CoRR, abs/2410.21804, 2024. doi:10.48550/ARXIV.2410.21804. URL https://doi.org/10.48550/arXiv.2410.21804
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2410.21804 2024
-
[43]
The german traffic sign recognition benchmark: A multi-class classification competition
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. In The 2011 International Joint Conference on Neural Networks, IJCNN 2011, San Jose, California, USA, July 31 - August 5, 2011 , pp.\ 1453--1460. IEEE , 2011. doi:10.1109/IJCNN.2011.6033395. URL http...
arXiv 2011
-
[44]
Parameter-efficient multi-task model fusion with partial linearization
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=iynRvVVAmH
work page 2024
-
[45]
Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (PLE): A novel multi-task learning (MTL) model for personalized recommendations. In Rodrygo L. T. Santos, Leandro Balby Marinho, Elizabeth M. Daly, Li Chen, Kim Falk, Noam Koenigstein, and Edleno Silva de Moura (eds.), RecSys 2020: Fourteenth ACM Conference on Recommende...
arXiv 2020
-
[46]
Least-squares estimation of transformation parameters between two point patterns
Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Trans. Pattern Anal. Mach. Intell. , 13 0 (4): 0 376--380, 1991. doi:10.1109/34.88573. URL https://doi.org/10.1109/34.88573
doi:10.1109/34.88573 1991
-
[47]
Weight averaging for neural networks and local resampling schemes
Joachim Utans. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pp.\ 133--138. Citeseer, 1996
work page 1996
-
[48]
Multi-task learning for dense prediction tasks: A survey
Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. , 44 0 (7): 0 3614--3633, 2022. doi:10.1109/TPAMI.2021.3054719. URL https://doi.org/10.1109/TPAMI.2021.3054719
-
[49]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
2017
-
[50]
Glue: A multi-task benchmark and analysis platform for natural language understanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018
Pith/arXiv arXiv 2018
-
[51]
Localizing task information for improved model merging and compression
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz - Jim \' e nez, Fran c ois Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=DWT9uiGjxT
work page 2024
-
[52]
Disrupting Model Merging: A Parameter-Level Defense Without Sacrificing Accuracy
Junhao Wei, Yu Zhe, and Jun Sakuma. Disrupting model merging: A parameter-level defense without sacrificing accuracy. CoRR, abs/2503.07661, 2025. doi:10.48550/ARXIV.2503.07661. URL https://doi.org/10.48550/arXiv.2503.07661
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.07661 2025
-
[53]
Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song,...
work page 2022
-
[54]
Ehinger, James Hays, Antonio Torralba, and Aude Oliva
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN database: Exploring a large collection of scene categories. Int. J. Comput. Vis., 119 0 (1): 0 3--22, 2016. doi:10.1007/S11263-014-0748-Y. URL https://doi.org/10.1007/s11263-014-0748-y
-
[55]
Instructional fingerprinting of large language models
Jiashu Xu, Fei Wang, Mingyu Derek Ma, Pang Wei Koh, Chaowei Xiao, and Muhao Chen. Instructional fingerprinting of large language models. In Kevin Duh, Helena G \' o mez - Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume ...
work page 2024
-
[56]
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...
work page 2023
-
[57]
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...
work page 2023
-
[58]
Adatask: A task-aware adaptive learning rate approach to multi-task learning
Enneng Yang, Junwei Pan, Ximei Wang, Haibin Yu, Li Shen, Xihua Chen, Lei Xiao, Jie Jiang, and Guibing Guo. Adatask: A task-aware adaptive learning rate approach to multi-task learning. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative App...
work page 2023
-
[59]
Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. CoRR, abs/2408.07666, 2024 a . doi:10.48550/ARXIV.2408.07666. URL https://doi.org/10.48550/arXiv.2408.07666
-
[60]
Representation surgery for multi-task model merging
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024 b . URL https://openreview.net/forum?id=Sbl2keQEML
work page 2024
-
[61]
Adamerging: Adaptive model merging for multi-task learning
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 c . URL https://openreview.net/forum?id=nZP6NgD3QY
work page 2024
-
[62]
Language models are super mario: Absorbing abilities from homologous models as a free lunch
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024
work page 2024
-
[63]
Beyond the permutation symmetry of transformers: The role of rotation for model fusion
Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen, and Jundong Li. Beyond the permutation symmetry of transformers: The role of rotation for model fusion. CoRR, abs/2502.00264, 2025. doi:10.48550/ARXIV.2502.00264. URL https://doi.org/10.48550/arXiv.2502.00264
-
[64]
Symmetry in neural network parameter spaces
Bo Zhao, Robin Walters, and Rose Yu. Symmetry in neural network parameter spaces. CoRR, abs/2506.13018, 2025. doi:10.48550/ARXIV.2506.13018. URL https://doi.org/10.48550/arXiv.2506.13018
-
[65]
Speech emotion recognition based on multi-task learning
Huijuan Zhao, Zhijie Han, and Ruchuan Wang. Speech emotion recognition based on multi-task learning. In 5th IEEE International Conference on Big Data Security on Cloud, IEEE International Conference on High Performance and Smart Computing, and IEEE International Conference on Intelligent Data and Security, BigDataSecurity/HPSC/IDS 2019, Washington, DC, US...
arXiv 2019
-
[66]
Learn from model beyond fine-tuning: A survey
Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Learn from model beyond fine-tuning: A survey. CoRR, abs/2310.08184, 2023. doi:10.48550/ARXIV.2310.08184. URL https://doi.org/10.48550/arXiv.2310.08184
-
[67]
On the emergence of cross-task linearity in pretraining-finetuning paradigm
Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. On the emergence of cross-task linearity in pretraining-finetuning paradigm. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=qg6AlnpEQH
work page 2024
-
[68]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
-
[69]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...
-
[70]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...
-
[71]
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...
arXiv 2015
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.