REVIEW 3 major objections 5 minor 6 cited by
Multi-Task Model Merging via Adaptive Weight Disentanglement
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Task vectors should be mutually orthogonal to minimize interference in model merging, the paper argues; its one-vector fix subtracts a learned redundant vector from every task vector to get there, while preserving per-task accuracy.
desk verdict A useful merging post-processing trick with a stronger empirical than theoretical story. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing identity is the first-order merging-gap approximation $G_i \approx k_i \sum_{j \neq i} \lambda_j \langle \tau_i, \tau_j \rangle$ (Eq. 9 in the paper), obtained by Taylor-expanding each task's loss around the pretrained weights $\Theta$ and substituting the gradient with $\nabla_{\Theta} L_i(\Theta) \approx k_i \tau_i$, $k_i < 0$. It converts 'no interference' into a geometric condition: pairwise orthogonal task vectors yield a zero merging gap, making orthogonality a sufficient condition for the Task Consistency Property. The mechanism that realizes this condition is a single trainable redundant vector $\delta$, shared across all tasks and initialized at zero; subtracting it produces the disentangled vectors $\hat{\tau}_i = \tau_i - \delta$. The objective $L = L_O + \alpha L_R$ balances two terms: $L_O$ penalizes the mean absolute cosine similarity between every pair of disentangled vectors, and $L_R = \|\delta\|$ keeps the perturbation small, with the appendix bounding the per-task replacement gap by $\|\nabla_{\Theta} L_i(\Theta)\| \cdot \|\delta\|$. Because only the weight vectors themselves enter the optimization of $\delta$, the procedure costs about two minutes on ViT-B/32 and eight minutes on ViT-L/14, and the outputs drop directly into existing merging recipes.
What would settle it
Estimate the true task gradients $\nabla_{\Theta} L_i(\Theta)$ on validation data and compute $\cos(\nabla_{\Theta} L_i(\Theta), \tau_i)$ for each task; values far from $\pm 1$ would show the scalar-multiple substitution is not the operative mechanism. The decisive test is to pick or construct two tasks with exactly orthogonal task vectors ($\cos(\tau_1, \tau_2) = 0$) whose loss gradients at $\Theta$ are strongly correlated: the paper's equation then predicts a zero merging gap, so measurable accuracy loss on either task after merging would falsify the claim that orthogonality is sufficient.
Extended reading notes
Core claim
The paper's central assertion is that task vectors used in model merging should be orthogonal to minimize interference among tasks. It formalizes the goal as a Task Consistency Property — the merged model should match, on each task, the performance obtained when only that task's own vector is added — and derives that a first-order Taylor expansion makes the per-task merging gap approximately $G_i \approx k_i \sum_{j \neq i} \lambda_j \langle \tau_i, \tau_j \rangle$, after replacing the unobservable gradient $\nabla_{\Theta} L_i(\Theta)$ with a negative scalar multiple of $\tau_i$. Mutual orthogonality of the task vectors therefore drives every gap to zero, which the paper presents as a sufficient condition, not a necessary one. On the practical side, AWD learns a single redundant vector $\delta$, initialized at zero, and exports disentangled vectors $\hat{\tau}_i = \tau_i - \delta$ by minimizing $L = L_O + \alpha L_R$, where $L_O$ is the mean absolute pairwise cosine similarity of the $\hat{\tau}_i$ and $L_R = \|\delta\|$ protects per-task performance; the appendix bounds the per-task replacement gap by $\|\nabla_{\Theta} L_i(\Theta)\| \cdot \|\delta\|$. Plugged into Task Arithmetic or AdaMerging, the disentangled vectors raise average accuracy by up to 6.1 points over plain Task Arithmetic on the eight-task vision benchmark, and the paper's control experiment that deliberately increases cosine similarity collapses accuracy, which it reads as causal evidence for the orthogonality mechanism.
Load-bearing premise
The load-bearing step is substituting each task's loss gradient at the pretrained weights with a negative scalar multiple of its task vector ($\nabla_{\Theta} L_i(\Theta) \approx k_i \tau_i$, $k_i < 0$), together with a first-order Taylor expansion of the loss; if the true gradient is not aligned with the task vector, orthogonal task vectors need not eliminate interference, and the paper's guarantee is void.
Editorial extensions
If this is right
- Any existing task-vector merger can be upgraded by a preprocessing pass that needs no task training data: AWD computes $\delta$ from the task vectors alone, so it applies where the fine-tuning data is unavailable or privacy-restricted.
- The benefit of orthogonalization grows with the number of merged tasks: on ViT-B/32 the gain over plain Task Arithmetic widens from 0.6 points with two tasks to 6.1 points with eight, matching the view that more tasks mean more pairwise interference to remove.
- Orthogonality, not parameter pruning, is the active ingredient: the paper's control experiment that reverses the objective and increases cosine similarity collapses average accuracy by 37.3% on ViT-B/32 and 29.0% on ViT-L/14 even though the reconstructed single-task models remain accurate.
- The approach transfers across model families — ViT-B/16, ViT-B/32, and ViT-L/14 for vision and RoBERTa-base/large on GLUE — and composes with both training-free and test-time-adaptation merging methods rather than replacing them.
Reading between the lines
- The derivation establishes a sufficient condition, not a characterization: non-orthogonal vectors can still interfere little if the unknown negative scalars $k_i$ happen to offset the inner products, so the paper leaves open how far orthogonality is also necessary.
- The gradient-substitution step is directly checkable: on any task where fine-tuning data is available, measuring $\cos(\nabla_{\Theta} L_i(\Theta), \tau_i)$ would show whether orthogonality is the operative mechanism or a proxy for a blunter effect such as shrinking the norm or variance of the summed vectors.
- The single shared $\delta$ presumes that all tasks share a common direction of redundancy; for heterogeneous task families a natural extension the paper does not explore is per-group or per-task redundant vectors, and a comparison would bound where the shared-vector assumption starts to cost accuracy.
- The loss-landscape finding — a wider low-loss basin after disentanglement — invites a testable tie-in with flat-minima intuitions: if merged-model performance tracks the flatness of the joint loss basin, then entanglement measures other than pairwise cosine similarity might predict merging success equally well.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Adaptive Weight Disentanglement (AWD), a post-hoc model-merging method that subtracts a single learned redundant vector δ from every task vector τ_i, producing disentangled task vectors that are more mutually orthogonal while preserving per-task performance. The paper first introduces a Task Consistency Property (the loss of the merged model on task i should equal the loss when only task vector i is added) and claims a proof that orthogonality of task vectors is sufficient for this property. AWD then minimizes the average absolute cosine similarity of the disentangled vectors plus an L2 norm penalty on δ. Experiments cover eight vision tasks (ViT-B/16, ViT-B/32, ViT-L/14) and eight GLUE tasks (RoBERTa-Base/Large), with AWD integrated into Task Arithmetic and AdaMerging, plus robustness, loss-landscape, and ablative analyses. Code is publicly available.
Significance. The empirical contribution is substantial and well executed: AWD consistently improves over Task Arithmetic, Ties-Merging, Consensus Merging, DARE, and AdaMerging across architectures and modalities, with reported gains of up to 2.8% (ViT-B/32) over Ties-Merging, and the computational cost is modest (2-8 minutes per task set, Table 5). The reversed-orthogonality control in Tables 6-7 of the supplementary is a genuine falsifiable test: preserving individual-task reconstruction while actively increasing cosine similarity collapses merged accuracy to roughly 32-56%, which strongly supports orthogonality as an empirically active mechanism rather than a tautological artifact. The paper also ships code and reports hyperparameter selection on validation sets. If the theoretical claim in Section 3.2 were correct, the paper would provide a principled sufficient condition for interference-free merging; as it stands, however, that proof is unsound (see major comments), so the paper's durable contribution is an effective and cheap heuristic with consistent empirical support, not an established theorem.
major comments (3)
- [§3.2, Eqs. (5)-(10); Supplementary §6.2, Eqs. (26)-(37)] The first-order Taylor truncation is inconsistent: the quadratic expansion of Li(Θ + Σ_j λj τj) contains cross terms (λi τi)⊤ Hi (Σ_{j≠i} λj τj) that are of exactly the same order in the task-vector magnitudes as the retained term Σ_{j≠i} ⟨∇Li(Θ), λj τj⟩ when the gradient is approximated by ∇Li(Θ) ≈ −Hi τi near a task optimum. Dropping one and keeping the other is not justified by the smallness of τ. The claim in Eq. (10) is false even in a quadratic model satisfying all the paper's assumptions. Take Θ=0, K=2, L1(θ)=½||θ−e1||², L2(θ)=½||θ−e2||², τ1=e1, τ2=e2, λ1=λ2=1. Then cos(τ1,τ2)=0 and the gradient substitution in Eq. (8) holds with k1=−1 (since ∇L1(0)=−τ1), yet G1 = L1(e1+e2) − L1(e1) = ½ > 0, contradicting Eq. (10). The first-order expansion gives G1 ≈ 0 only through an exact cancellation of the retained term with the dropped cross term; the true second-order gap ½(e2)⊤ H1 (e2) is positive and does not vanish when the task vectors are orthogonal. Since the paper's λ search range is {0.3, ..., 1.0}, this regime is inside the method's operating range. The implication 'orthogonality ⇒ zero merging gap' must be removed or replaced by a correct second-order analysis, and the abstract's claim of a theoretical proof must be qualified accordingly.
- [§3.2, Eq. (8)] The identification ∇Θ Li(Θ) ≈ ki τi with ki < 0 is load-bearing and unsupported. The task vector τi = Θ⋆_i − Θ is the total weight displacement along the fine-tuning trajectory; under gradient descent it equals a sum of gradients evaluated at many iterates, not the gradient at the single point Θ. A scalar proportionality between the gradient at Θ and the total displacement is a strong structural assumption, and it is the only bridge from the merging gap to cosine similarity in Eqs. (9)-(10). The paper should either validate it empirically (e.g., measure cos(∇Li(Θ), τi) on a subset of the eight vision tasks, which the pilot-experiment infrastructure permits) or present Eq. (8) explicitly as a heuristic. As written, the derivation converts an unverified approximation into a conclusion stated without caveat in the abstract.
- [Abstract and §1, Contributions] Given the two issues above, the statements 'theoretically proving that task vectors employed in model merging should be orthogonal' (Abstract) and 'we derive that the task vectors corresponding to different tasks should be mutually orthogonal' (Contribution 1) overstate what the manuscript establishes. What the paper actually demonstrates is (i) an empirical regularizer that lowers pairwise cosine similarity while preserving individual task performance, and (ii) a consistent, controlled correlation between reduced cosine similarity and improved merged accuracy (Fig. 6 and Tables 6-7 of the supplementary). The authors should reframe Section 3.2 as a motivating analysis with explicitly stated assumptions (for instance, a diagonal-Hessian or NTK-linearized model along the lines of [21]), or supply a genuine second-order derivation; the suggested sufficient condition in such an analysis is likely to be stronger than pairwise task-vector orthogonality (it should also involve the Hessians Hi). The empirical claims do not depend on the proof, so this revision is feasible within the paper's scope.
minor comments (5)
- [§3.1-§3.2] The sentence 'Property 1 and 23 are characteristics jointly manifested...' is garbled and should read 'Properties 1 and 2'; the cross-references between the main text and the supplementary (Eq. 23 as a restatement of Property 2, and the duplicated numbering of Eqs. 15-37) should be unified.
- [§3.3, §7.3, Table 5] The hyperparameter N (solution steps) is implied to be 1000 only by the caption of Table 5, and the learning rate β is not reported anywhere; both should be stated in Section 7.3. Also, 'we introduce an norm constraint' should read 'an L2 norm constraint'.
- [§4.2-§4.3] The paper should state explicitly whether the main results in Tables 1-3 use the full task-vector AWD or the linear-layer-only variant described in the pilot experiment; the text suggests the simplified variant is used for transformers, but the main tables and Section 3.3 are written as if the full vector is optimized.
- [Supplementary §6.1, Eqs. (15)-(22)] The Replacement Gap bound should be stated with an absolute value: from Eq. (21), the meaningful statement is |Ri| ≤ |∇Li(Θ)| ||δ||, not Ri ≤ |∇Li(Θ)| ||δ||; as written the inequality is either signed-inconsistent or trivial.
- [References] Entries [27] and [28] are the same paper (Tang et al., ICLR 2024) and should be merged; several equations in the main text render with garbled summation symbols (e.g., Eqs. (5)-(7)), which should be corrected in the camera-ready version.
Circularity Check
No significant circularity: orthogonality is derived from an explicit (if heuristic) approximation and then tested empirically against merged-model accuracy.
full rationale
The paper's derivation chain is not circular. It defines the merging gap G_i as the difference between the loss of the merged model and the loss of the single-task model, expands both to first order around the pretrained weights, approximates the gradient of each task loss at the pretrained model by a negative scalar multiple of the task vector (Eq. 8), and then observes that pairwise orthogonal task vectors make the resulting inner-product sum vanish. None of these equations defines the conclusion into the premises: the task vectors are not constructed to satisfy the task consistency property, and the orthogonality condition is a sufficient condition derived from an explicit approximation rather than an identity. The proposed method optimizes the average absolute cosine similarity of the disentangled vectors plus a norm penalty, and its effectiveness is evaluated by measuring merged-model accuracy on held-out validation/test tasks, which is not part of the optimization objective. The correlation reported in Section 8.2 is therefore an empirical test of whether more orthogonal vectors merge better, not a tautological restatement of the fitted objective. The paper also does not rely on load-bearing self-citations: the task arithmetic property and related empirical observations are attributed to external prior work (Ilharco et al., Ortiz-Jimenez et al.), not to the present authors' own prior results. The main weakness of the paper is a validity issue, not a circularity issue: the first-order Taylor truncation drops second-order cross terms that can be of the same order as the retained first-order terms, and the gradient-to-task-vector proportionality in Eq. 8 is a heuristic assumption. These concerns affect whether the theorem is correct under realistic losses, but they do not make the claimed prediction equivalent to its inputs by construction.
Assumptions & free parameters
free parameters (3)
- alpha (constraint coefficient) =
tuned over {1e-2, 1e-3, 1e-4, 1e-5, 1e-6}
- lambda (scaling coefficient for AWD Task Arithmetic) =
tuned over {0.3, 0.4, ..., 1.0}
- solution steps N and learning rate beta =
N=1000 steps in supplementary; beta not reported in main text
assumptions (4)
- domain assumption Task vectors are small enough that second-order and higher Taylor terms are negligible in the loss expansion (Section 3.2, Eq. 5-6; Supplementary 6.2).
- ad hoc to paper The loss gradient of task i at the pretrained model is proportional to the task vector: ∇ΘLi(Θ) ≈ kiτi with ki < 0 (Section 3.2, Eq. 8).
- domain assumption A single shared redundant vector δ can be subtracted from all task vectors to make them near-orthogonal while keeping task-specific performance (Section 3.3).
- ad hoc to paper Minimizing ||δ|| controls the replacement gap Ri because Ri ≤ |∇ΘLi(Θ)| ||δ|| (Supplementary Eq. 22).
Cite this review
Pith. "Pith review of Multi-Task Model Merging via Adaptive Weight Disentanglement." pith.science (2026). https://pith.science/paper/4FSB426Z
@misc{pith2026241118729,
author = {Pith},
title = {Pith review of: Multi-Task Model Merging via Adaptive Weight Disentanglement},
year = {2026},
howpublished = {\url{https://pith.science/paper/4FSB426Z}},
note = {Machine review of arXiv:2411.18729}
}
read the original abstract
Model merging has recently gained attention as an economical and scalable approach to incorporate task-specific weights from various tasks into a unified multi-task model. For example, in Task Arithmetic (TA), adding the fine-tuned weights of different tasks can enhance the model's performance on those tasks, while subtracting them leads to task forgetting. Although TA is highly effective, interference among task still hampers the performance of the merged model. Existing methods for handling conflicts between task generally rely on empirical selection, resulting in suboptimal performance. In this paper, we introduce an Adaptive Weight Disentanglement method. We begin by theoretically proving that task vectors employed in model merging should be orthogonal to minimize interference among tasks. Guided by this insight, we initialize redundant vectors such that, when subtracted from the original task vectors, the resulting vectors exhibit increased orthogonality. Additionally, we impose an norm constraint on the redundant vectors to preserve the performance of the task-specific models. Experimental results demonstrate the effectiveness of our proposed technique: it successfully extracts redundant vectors, and after their subtraction, the task vectors not only retain robust performance but also achieve superior fusion outcomes. Our code is available at \href{https://github.com/FarisXiong/AWD.git}{https://github.com/FarisXiong/AWD.git}.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 6 Pith papers
-
SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation
SAFE-Merge masks risk-prone parameter updates and recovers lost task information with a constrained low-rank correction, achieving the best H-score in data-free continual model merging benchmarks.
-
Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging
A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimoda...
-
Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
A continual model merging method, OPCM, sequentially projects each new task vector into a subspace orthogonal to the current merged model, achieving 5-8% higher average accuracy than baselines on CLIP-ViT tasks.
-
Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent
DOGE merges fine-tuned models by optimizing a data-free loss-gap proxy with gradient steps projected orthogonal to a shared task subspace, improving average accuracy over previous methods.
-
How to Merge Your Multimodal Models Over Time?
A systematic study of temporal model merging shows that initialization and deployment choices matter far more than the merging technique, with EMA-style weight interpolation as the best practice.
-
CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation
CABS+ prunes task vectors with a conflict-aware, balanced sparsification step and sets merging coefficients through CMA-ES search, improving efficiency and average merged-model performance.
Reference graph
Works this paper leans on
-
[27]
Parameter-efficient multi-task model fusion with partial linearization
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In Pro- ceedings of the Twelfth International Conference on Learn- ing Representations, 2024. 2, 4
work page 2024
-
[28]
Parameter-efficient multi-task model fusion with partial linearization
Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter-efficient multi-task model fusion with partial linearization. In Pro- ceedings of the Twelfth International Conference on Learn- ing Representations, 2024. 3, 8
work page 2024
-
[21]
Task arithmetic in the tangent space: Improved editing of pre-trained models
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Proceedings of the Confer- ence on Neural Information Processing Systems, 2023. 1, 2, 3, 4, 12
work page 2023
-
[1]
Git re-basin: Merging models modulo permutation symmetries
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srini- vasa. Git re-basin: Merging models modulo permutation symmetries. In Proceedings of the Eleventh International Conference on Learning Representations, 2023. 2
work page 2023
-
[2]
Rishabh Bhardwaj, Duc Anh Do, and Soujanya Poria. Lan- guage models are Homer simpson! safety re-alignment of fine-tuned language models through task arithmetic. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 14138–14149, 2024. 2
work page 2024
-
[3]
Remote sens- ing image scene classification: Benchmark and state of the art
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sens- ing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017. 6
work page 2017
-
[4]
Describing textures in the wild
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 3606–3613,
-
[5]
Knowledge neurons in pretrained transform- ers
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transform- ers. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 8493–8502. Association for Computational Linguistics, 2022. 7
work page 2022
Show all 49 references
-
[6]
Orthogonal gradient descent for continual learning
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In Inter- national Conference on Artificial Intelligence and Statistics, pages 3762–3773. PMLR, 2020. 2
2020
-
[7]
Adaptive orthogonal projection for batch and online contin- ual learning
Yiduo Guo, Wenpeng Hu, Dongyan Zhao, and Bing Liu. Adaptive orthogonal projection for batch and online contin- ual learning. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 36(6):6783–6791, 2022. 2
2022
-
[8]
Localize-and-stitch: Efficient model merging via sparse task arithmetic
Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. Localize-and-stitch: Efficient model merging via sparse task arithmetic. 2024. 2
2024
-
[9]
Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 6
2019
-
[10]
Editing models with task arithmetic
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In Proceedings of the Eleventh International Conference on Learning Representa- tions, 2023. 1, 2, 3, 4, 5, 6, 8, 13
2023
-
[11]
Dataless knowledge fusion by merging weights of language models
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxi- ang Cheng. Dataless knowledge fusion by merging weights of language models. In Proceedings of the Eleventh Inter- national Conference on Learning Representations, 2023. 3, 6
2023
-
[12]
REPAIR: REnormalizing permuted activations for interpolation repair
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In Proceedings of the Eleventh International Conference on Learning Representa- tions, 2023. 2
2023
-
[13]
3d object representations for fine-grained categorization
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, pages 554–561, 2013. 6
2013
-
[14]
The mnist database of handwritten digits
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998. 6
1998
-
[15]
Towards continual learning desiderata via hsic-bottleneck orthogonalization and equian- gular embedding
Depeng Li, Tianqi Wang, Junwei Chen, Qining Ren, Kenji Kawaguchi, and Zhigang Zeng. Towards continual learning desiderata via hsic-bottleneck orthogonalization and equian- gular embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 38(12):13464–13473, 2024. 2
2024
-
[16]
Model-protected multi-task learning
Jian Liang, Ziqi Liu, Jiayu Zhou, Xiaoqian Jiang, Changshui Zhang, and Fei Wang. Model-protected multi-task learning. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 44(2):1002–1019, 2022. 1
2022
-
[17]
Roberta: A robustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 2019. 7
2019
-
[18]
Twin-merging: Dynamic integration of modular expertise in model merging
Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. 2024. 7, 13, 14, 16
2024
-
[19]
Merging models with fisher-weighted averaging
Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. In Proceedings of the Inter- national Conference on Neural Information Processing Sys- tems, pages 17703–17716, 2022. 3, 6 9
2022
-
[20]
Learning to route among specialized experts for zero- shot generalization
Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero- shot generalization. In Proceedings of the International Con- ference on Machine Learning, 2024. 1
2024
-
[22]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. 2021. 1, 6, 13
2021
-
[23]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learn- ing Research, 21(1), 2020. 1
2020
-
[24]
Continual learning with scaled gradient projection
Gobinda Saha and Kaushik Roy. Continual learning with scaled gradient projection. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 9677–9685, 2023. 2
2023
-
[25]
Multitask prompted training enables zero-shot task general- ization
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyo...
2022
-
[26]
The german traffic sign recognition bench- mark: a multi-class classification competition
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition bench- mark: a multi-class classification competition. In Interna- tional Joint Conference on Neural Networks , pages 1453–
-
[29]
Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. UL2: Unifying language learning paradigms. In Proceedings of the Eleventh International Confere...
2023
-
[30]
Is fine-tuning needed? pre-trained language models are near perfect for out-of-domain detection
Rheeya Uppaal, Junjie Hu, and Yixuan Li. Is fine-tuning needed? pre-trained language models are near perfect for out-of-domain detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12813–12832. Asso...
2023
-
[31]
Knowledge fusion of large language models
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In Proceedings of the Twelfth International Confer- ence on Learning Representations, 2024. 2
2024
-
[32]
GLUE: A multi-task benchmark and analysis platform for natural language un- derstanding
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language un- derstanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for ...
2018
-
[33]
Localizing task infor- mation for improved model merging and compression
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Franc ¸ois Fleuret, and Pascal Frossard. Localizing task infor- mation for improved model merging and compression. In Proceedings of the 41st International Conference on Ma- chine Learning , pages 50268–50287. PMLR, 2024. 2, 3, 6
2024
-
[34]
A comprehensive survey of continual learning: Theory, method and application
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5362–5383, 2024. 2
2024
-
[35]
Or- thogonal subspace learning for language model continual learning
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Or- thogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.14152, 2023. 2
2023 arXiv
-
[36]
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without ...
2022
-
[37]
Sun database: Exploring a large col- lection of scene categories
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Tor- ralba, and Aude Oliva. Sun database: Exploring a large col- lection of scene categories. International Journal of Com- puter Vision, 119:3–22, 2016. 6
2016
-
[38]
Active finetuning: Exploiting an- notation budget in the pretraining-finetuning paradigm
Yichen Xie, Han Lu, Junchi Yan, Xiaokang Yang, Masayoshi Tomizuka, and Wei Zhan. Active finetuning: Exploiting an- notation budget in the pretraining-finetuning paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 23715–23724, 2023. 1
2023
-
[39]
Ties-merging: resolving interfer- ence when merging models
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raf- fel, and Mohit Bansal. Ties-merging: resolving interfer- ence when merging models. In Proceedings of the Inter- national Conference on Neural Information Processing Sys- tems, pages 7093–7115, 2023. 1, 2, 3, 4, 5, 6, 12
2023
-
[40]
Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xi- aochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. 2024. 1, 3
2024
-
[41]
Representation 10 surgery for multi-task model merging
Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiao- jun Chen, Xingwei Wang, and Dacheng Tao. Representation 10 surgery for multi-task model merging. In Proceedings of the 41st International Conference on Machine Learning , pages 56332–56356. PMLR, 2024. 6, 7
2024
-
[42]
Adamerging: Adap- tive model merging for multi-task learning
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adap- tive model merging for multi-task learning. In Proceedings of the Twelfth International Conference on Learning Repre- sentations, 2024. 2, 5, 6, 7
2024
-
[43]
On the dimensionality of word embedding
Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. In Proceedings of the International Conference on Neural Information Processing Systems . Curran Asso- ciates, Inc., 2018. 7
2018
-
[44]
Language models are super mario: Absorbing abilities from homologous models as a free lunch
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InProceedings of the In- ternational Conference on Machine Learning. PMLR, 2024. 2, 3, 4, 5, 12
2024
-
[45]
Reading digits in natural images with unsuper- vised feature learning
Netzer Yuval. Reading digits in natural images with unsuper- vised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. 6
2011
-
[46]
Contin- ual learning of context-dependent processing in neural net- works
Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Contin- ual learning of context-dependent processing in neural net- works. Nature Machine Intelligence , 1(8):364–372, 2019. 2 11 Multi-Task Model Merging via Adaptive Weight Disentanglement Supplementary Material
2019
-
[47]
Further Theoretical Derivation 6.1. Analysis of Invariance Optimization Objective Inspired by previous works [39, 44], we empirically incor- porate the norm of δ as Invariance objective to ensure that each disentangled task vector performs well on its corre- sponding specific ...
-
[48]
Calculate Resources and Environment
Experimental Settings 7.1. Calculate Resources and Environment. All of our experiments were conducted on NVIDIA A100 40GB and NVIDIA H800 80GB. Due to the specific config- uration of AdaMerging, we used the NVIDIA H800 80GB for the ViT-L/14 variant, while other experiments wer...
2000
-
[49]
Performance on ViT-B/16 Table 4 presents the performance of various merging meth- ods applied to ViT-B/16
Additional Experimentals 8.1. Performance on ViT-B/16 Table 4 presents the performance of various merging meth- ods applied to ViT-B/16. Our approach achieves a 6.0% improvement over Task Arithmetic and a 1.8% gain com- pared to AdaMerging on corresponding settings. These re- ...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.