Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that knowledge conflicts in model merging come from task-vector components aligned with task-loss gradients, and that a trust-region mask fixing this improves merged-model accuracy.

desk verdict A training-free mask on task vectors that buys a few accuracy points, with a plausible but under-tested story about why it works. read the letter →

arxiv 2501.15065 v1 pith:EPU62UVU submitted 2025-01-25 cs.LG cs.AI

Wenju Sun , Qingyong Li , Wen Wang , Yangli-ao Geng , Boyang Li This is my paper · ORCID
classification cs.LGcs.AI
keywords modelmergingtaskarithmeticknowledgeconflicttrustregionvectorgradientorthogonalitytraining-freemulti-tasklearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When multiple fine-tuned models are combined by adding their task vectors to a pre-trained model, the tasks can interfere, degrading performance. The paper defines this as 'knowledge conflict' and traces it to task-vector components that align with the gradient of each task's loss at the pre-trained weights. It proposes Task Arithmetic in the Trust Region (TATR), a training-free method that merges only the parameter dimensions where the product |gradient × task vector| is small, i.e., directions nearly orthogonal to the gradient. On eight image-classification datasets, TATR raises Task Arithmetic's average accuracy from 69.1% to 72.8% with a ViT-B/32 backbone, and also improves Ties-Merging, AdaMerging, and Surgery when used as a plug-in. The central claim is that discarding gradient-aligned components, rather than keeping gradient-descent ones, is what resolves cross-task interference.

What carries the argument

The central object is the trust-region mask defined by Eq. (5): a parameter dimension $n$ belongs to the trust region when $\sum_{i \neq j} |\nabla_\theta L_j(\theta_{\text{pre}})[n] \cdot \Delta_i[n]| < \epsilon$, with $\epsilon$ set by a proportion threshold $\tau$ on the ranked sensitivity. The mask is applied by element-wise multiplication with the task vectors before adding them to the pre-trained weights (Algorithm 1). The mechanism exploits overparameterization: many parameter directions are near-orthogonal to task gradients, so a small mask can discard the conflicting aligned components without losing task-specific knowledge. Gradients are approximated from small exemplar sets or, in the zero-shot variant, by the task vectors themselves.

What would settle it

Compare, for each task pair, the true change in task j's loss when task i's vector is added against the first-order prediction $\langle \nabla L_j(\theta_{\text{pre}}), \Delta_i \rangle$; TATR's mask is only justified if small predicted products correspond to small true loss changes. Alternatively, apply TATR to a small, heavily trained model with little parameter redundancy where near-orthogonal gradient directions are rare, and check whether the accuracy gains over Task Arithmetic disappear or reverse.

Watch

Extended reading notes

Core claim

Knowledge conflicts in task arithmetic are primarily caused by components of task vectors that align with the gradient of task-specific losses at the pre-trained model; even gradient-descent-aligned components harm performance because the first-order approximation overshoots local optima. Therefore the paper defines a trust region of parameter dimensions where the absolute product $|\nabla_\theta L_j(\theta_{\text{pre}})[n] \cdot \Delta_i[n]|$ is below a threshold, and merges only within it. TATR uses per-task exemplar samples to estimate gradients, ranks dimension sensitivity, masks away a small proportion of parameters (typically under 1%), and can be layered on top of existing task-arithmetic variants. Empirically this yields consistent accuracy gains across eight datasets and three backbones, with the largest gains when combined with test-time adaptation methods.

Load-bearing premise

The trust-region mask is built from first-order gradient×task-vector products at the pre-trained weights, yet the paper itself shows that first-order Taylor expansions fail for full-size task vectors; if those products mis-rank the true cross-task interference, the mask discards the wrong dimensions.

Editorial extensions

If this is right

  • If the trust-region diagnosis is right, any task-arithmetic-style merge can be improved by masking gradient-aligned dimensions, without retraining.
  • The mask also reduces interference for test-time adaptation methods like AdaMerging and Surgery, so those methods can train their coefficients in a less conflicting parameter space.
  • The drop in accuracy from masking only ~1% of parameters suggests conflicts are concentrated in a small fraction of dimensions.
  • The zero-shot variant, using task vectors as gradient surrogates, offers a fully training-free path to conflict reduction when exemplars are unavailable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to apply the trust-region mask iteratively, recomputing gradients at the merged point; if the first-order assumption is weak, this could strengthen the mask beyond the paper's single-shot version.
  • The orthogonal-directions argument connects to parameter-level redundancy results; one could test TATR on smaller, less overparameterized models where orthogonal directions are scarce, which would clarify whether the mechanism is the stated orthogonality or just sparsification.
  • The method's success with one-shot exemplar sets suggests gradient estimation is robust, so a practical extension would be to use the mask for continual learning or model editing where only a few samples per task are available.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies knowledge conflicts in Task Arithmetic (TA) model merging, defining a conflict as the degradation in one task's loss when another task's fine-tuned model is included in the merged model. The authors argue, via a first-order Taylor expansion around the pre-trained weights, that conflict originates mainly from task-vector components aligned with the gradient of the task-specific loss. They propose TATR, a training-free mask that keeps only parameter dimensions for which the accumulated product |\nabla L_j(\theta_{pre})| \odot |\Delta_i| is small, discarding the remaining dimensions before merging. TATR is evaluated as a standalone method and as a plug-in for Ties-Merging, AdaMerging, and Surgery on eight image-classification datasets with CLIP ViT-B/32, ViT-B/16, and ViT-L/14, reporting consistent average-accuracy gains (e.g., 72.8% vs. 69.1% for Task Arithmetic on ViT-B/32). The paper also includes a zero-shot variant that approximates the gradient by the task vector magnitude, loss-landscape visualizations, and ablations of positive, negative, NTK-based, and zero-shot sensitivity definitions.

Significance. If the empirical claims hold, TATR is a simple, inexpensive, plug-and-play masking scheme with consistent gains across multiple backbones and datasets. The one-shot and zero-shot variants make it particularly attractive for practical model merging, and the ablation in Table 6, where positive and negative masks fail catastrophically, provides meaningful evidence that the sign of the masking criterion matters. However, the central causal claim---that gradient-orthogonal dimensions are precisely the safe ones---is not yet convincingly established. The missing random-mask and magnitude-mask controls, together with the acknowledged failure of first-order Taylor approximation in the very regime where the mask is defined, leave the theoretical justification underdetermined. The contribution is therefore best viewed as a promising empirical heuristic whose mechanism requires further validation.

major comments (3)
  1. [Section 6.2, Table 1; Algorithm 1] The reported gains are not contrasted with a control that discards the same fraction tau of dimensions randomly or by parameter magnitude. Since the selected tau is below 1% (Figure 3b), TATR alters only a tiny fraction of the model, and any small structured perturbation of the task-sum could shift accuracy by a few points. Without random-mask and magnitude-mask baselines matched to the same tau, the +3.7 point improvement over Task Arithmetic on ViT-B/32 cannot be attributed specifically to gradient-orthogonality rather than to generic sparsity. This control is essential for the paper's central claim.
  2. [Section 4, Eq. (4) and Figure 2(b); Section 5, Eq. (8) and Algorithm 1] The paper states that the first-order Taylor expansion fails for large task vectors because the negative component overshoots local optima, yet the masking criterion in Eq. (8) is precisely the first-order product |\nabla L_j(\theta_{pre})| \odot |\Delta_i|, with the sign collapsed by taking absolute values. The move from the signed scalar product in Eq. (4) to the absolute per-dimension product in Eq. (8) is an additional heuristic, not a consequence of the Taylor analysis. If first-order sensitivity is unreliable in the regime of real task vectors, the theory does not explain why the absolute first-order product should correctly rank conflict-inducing dimensions. Please either provide a theoretical justification for the absolute-value step or add an experiment that directly compares the predictions of the signed and absolute first-order criteria, e.g., by showing that the signed version fails precisely on dimensions that the absolute version correctly excludes.
  3. [Section 6.1, Implementation details; Figure 3(b)] The hyperparameter tau is described as 'tuned' over [0.1%, 0.2%, 0.5%, 1.0%, 2.0%, 5.0%], but the manuscript does not specify the selection protocol, in particular whether a validation split is used or whether the reported test accuracies are the best over the grid. If tau is selected on the test data, the headline numbers incorporate fitted selection and the 'training-free' characterization is weakened. Please state the selection criterion explicitly, report results for a fixed tau (e.g., tau = 0.5%) as a sensitivity check, and clarify the relationship between the choice of tau and the test-set accuracies shown in Figure 3(b).
minor comments (6)
  1. [Table 6] The positive and negative sensitivity variants are defined with an average over task pairs before thresholding; because positive and negative signed products can cancel in this average, the catastrophic failure of these variants does not by itself establish that every dimension's signed product is harmful. Please report, or at least discuss, the signed product without the averaging.
  2. [Definition 1, Eq. (3)] Knowledge conflict is defined through loss differences, while all experimental results report accuracy. The connection between the two (e.g., using cross-entropy loss and noting that lower loss is expected to correspond to higher accuracy) should be stated explicitly.
  3. [Eq. (6)] The expectation is placed outside the absolute value in the gradient estimator, which is nonstandard for Fisher-type estimates; since |E[g]| <= E[|g|] and the two estimators can rank dimensions differently, this design choice should be justified or at least discussed.
  4. [Figure 4] The figure is not legible in the provided version, and the axis labels and layer indexing are unclear; the text's claims about periodic sensitivity and bias-layer sensitivity cannot be verified from the figure as rendered. Please redraw with clear axes and legends.
  5. [Section 6.3, Figure 3(b)] The text states that excluding less than 1% of parameters is sufficient, but it does not state whether the same tau is used across all datasets and backbones or how the value was chosen for the main tables; please make this explicit.
  6. [Section 2.2, Related Work] The relationship between TATR's trust-region mask and existing magnitude-based pruning in Ties-Merging is not discussed; given the small tau, a comparison with low-magnitude pruning at the same sparsity level would help the reader understand the novelty of the gradient-based criterion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the trust-region mask is derived from gradients and task vectors, and the central claim is supported by independent loss-landscape visualizations and ablations; the remaining concerns are experimental, not circular.

full rationale

The derivation chain is coherent rather than circular. Definition 1 formalizes knowledge conflict through actual loss changes, and Eq. (4) approximates that conflict by a first-order Taylor expansion, yielding the product of the task-specific gradient and the task vector. The trust region in Definition 2 and Eq. (8) is then defined as the set of dimensions where the absolute value of that same product is small, and Eq. (10) merges only those components. This is an operationalization of the analysis, not an equivalence: the mask is a function of the gradients and task vectors, and the merged model is not equal to any quantity being fitted. The paper's key empirical claim that orthogonal (small-product) components are less conflict-inducing is supported by loss-landscape visualizations in Figure 2 and by ablations over alternative sensitivity definitions in Table 6, so the conclusion does not reduce to the definition of the mask alone. The zero-shot variant uses task vectors to estimate gradients, but the paper explicitly labels this an approximation with estimation errors and reports it as a separate heuristic; the mask is not definitionally identical to the output. Tuning the proportion threshold tau is a standard hyperparameter-selection step, and there is no evidence in the text that the headline accuracies are obtained by fitting a quantity that is then relabeled as a prediction. The absence of a random-mask or magnitude-mask control, and the paper's own observation that first-order Taylor expansions fail for large task vectors, are legitimate scientific concerns about causal attribution and theoretical justification, but they are not circularity. Self-citations such as Sun et al. (2023) appear only in related work and are not load-bearing for the proposed method. Overall, no circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No new physical entities are postulated. The method introduces one hyperparameter (tau) and relies on four modeling assumptions, listed above, that are not derived from first principles.

free parameters (1)
  • tau (proportion threshold for trust region) = tuned in [0.1%, 0.2%, 0.5%, 1.0%, 2.0%, 5.0%], best value selected by performance on the reported test sets
    Controls the fraction of parameter dimensions retained for merging. The paper does not describe a validation split, so the selected value directly influences the reported accuracies.
assumptions (4)
  • ad hoc to paper First-order Taylor expansion of task losses around theta_pre accurately orders parameter dimensions by conflict risk over the full task-vector displacement.
    Used to define the trust region (Eq. 5) and the sensitivity Omega_Trust (Eq. 8), despite the paper's own statement in Section 4 that first-order approximation fails for large task vectors.
  • domain assumption Pre-trained models are overparameterized enough that a sufficient number of near-orthogonal parameter directions exist to carry task knowledge after masking.
    Invoked in Section 5 to justify that discarding gradient-collinear dimensions does not destroy task-specific knowledge.
  • domain assumption The exemplar-set gradient approximation (Eq. 6), with as few as one sample per task, estimates the true gradient well enough to build the mask.
    Used in Algorithm 1 and empirically supported by Figure 3a, but not theoretically guaranteed.
  • ad hoc to paper The zero-shot gradient estimate |gradient L_k(theta_pre)| is approximately equal to |Delta_k| as a surrogate for gradient magnitude.
    Used to construct the zero-shot variant (Eq. 7); the paper acknowledges this is approximate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts." pith.science (2026). https://pith.science/paper/EPU62UVU

@misc{pith2026250115065,
  author       = {Pith},
  title        = {Pith review of: Task Arithmetic in Trust Region: A Training-Free Model Merging Approach to Navigate Knowledge Conflicts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EPU62UVU}},
  note         = {Machine review of arXiv:2501.15065}
}
abstract

Multi-task model merging offers an efficient solution for integrating knowledge from multiple fine-tuned models, mitigating the significant computational and storage demands associated with multi-task training. As a key technique in this field, Task Arithmetic (TA) defines task vectors by subtracting the pre-trained model ($\theta_{\text{pre}}$) from the fine-tuned task models in parameter space, then adjusting the weight between these task vectors and $\theta_{\text{pre}}$ to balance task-generalized and task-specific knowledge. Despite the promising performance of TA, conflicts can arise among the task vectors, particularly when different tasks require distinct model adaptations. In this paper, we formally define this issue as knowledge conflicts, characterized by the performance degradation of one task after merging with a model fine-tuned for another task. Through in-depth analysis, we show that these conflicts stem primarily from the components of task vectors that align with the gradient of task-specific losses at $\theta_{\text{pre}}$. To address this, we propose Task Arithmetic in Trust Region (TATR), which defines the trust region as dimensions in the model parameter space that cause only small changes (corresponding to the task vector components with gradient orthogonal direction) in the task-specific losses. Restricting parameter merging within this trust region, TATR can effectively alleviate knowledge conflicts. Moreover, TATR serves as both an independent approach and a plug-and-play module compatible with a wide range of TA-based methods. Extensive empirical evaluations on eight distinct datasets robustly demonstrate that TATR improves the multi-task performance of several TA-based model merging methods by an observable margin.

Figures

Figures reproduced from arXiv: 2501.15065 by the authors.

Figure 1
Figure 1. Illustration of knowledge con￾flicts between task vectors. In scenario (a), the two task vectors contain large￾magnitude components in opposite di￾rections. In scenario (b), the difference in vector magnitudes causes the merged model to be dominated by one task. Both lead to suboptimal performance in one or more tasks. In this paper, we propose a novel trust-region criterion for model merging, Task Arithmetic in the… view at source ↗
Figure 2
Figure 2. (a) Performance comparison across eight datasets (Cf. Section 6.1) when merging negative [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average accuracy (%) of TATR on eight tasks versus the number of exemplars (a) and [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The average sensitivity of each dataset to task vectors across layers. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Loss landscape for each dataset and the components of the cumulative task vector from [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Loss landscape for each datasets and the components of the cumulative task vector from [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging

    cs.AI 2025-05 conditional novelty 6.0 of 10

    CAT Merging selectively removes conflict-prone components from task vectors, using feature-based projection and masking, and improves average multi-task accuracy by up to 2.5% over prior training-free merging methods.

  2. Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent

    cs.LG 2025-01 conditional novelty 6.0 of 10

    DOGE merges fine-tuned models by optimizing a data-free loss-gap proxy with gradient steps projected orthogonal to a shared task subspace, improving average accuracy over previous methods.

Reference graph

Works this paper leans on

60 extracted references · 44 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Ensemble of averages: Improving model selection and boosting performance in domain generalization

    Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In Advances in Neural Information Processing Systems, volume 35, pp.\ 8265--8277, 2022

    work page 2022

  3. [3]

    Swad: Domain generalization by seeking flat minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems, volume 34, pp.\ 22405--22418, 2021

    work page 2021

  4. [4]

    Weighted training for cross-task learning

    Shuxiao Chen, Koby Crammer, Hangfeng He, Dan Roth, and Weijie J Su. Weighted training for cross-task learning. In International Conference on Learning Representations, 2022 a . URL https://openreview.net/forum?id=ltM1RMZntpu

    work page 2022

  5. [5]

    The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy

    Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 12020--12030, June 2022 b

    work page 2022

  6. [6]

    G rad N orm: Gradient normalization for adaptive loss balancing in deep multitask networks

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. G rad N orm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, volume 80, pp.\ 794--803, 10--15 Jul 2018

    work page 2018

  7. [7]

    Just pick a sign: Optimizing deep multitask models with gradient sign dropout

    Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In Advances in Neural Information Processing Systems, volume 33, pp.\ 2039--2050, 2020

    work page 2020

  8. [8]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105 0 (10): 0 1865--1883, 2017. doi:10.1109/JPROC.2017.2675998

    arXiv 2017

Show all 60 references
  1. [9]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, June 2014

    2014

  2. [10]

    Analyzing redundancy in pretrained transformer models

    Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),...

    2020 doi

  3. [11]

    Mssm: A multiple-level sparse sharing model for efficient multi-task learning

    Ke Ding, Xin Dong, Yong He, Lei Cheng, Chilin Fu, Zhaoxin Huan, Hai Li, Tan Yan, Liang Zhang, Xiaolu Zhang, and Linjian Mo. Mssm: A multiple-level sparse sharing model for efficient multi-task learning. In International ACM SIGIR Conference on Research and Development in Infor...

    2021

  4. [12]

    Theory of statistical estimation

    Ronald Aylmer Fisher. Theory of statistical estimation. In Mathematical proceedings of the Cambridge philosophical society, volume 22, pp.\ 700--725, 1925

    1925

  5. [13]

    Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning

    Yuan Gao, Haoping Bai, Zequn Jie, Jiayi Ma, Kui Jia, and Wei Liu. Mtl-nas: Task-agnostic neural architecture search towards general-purpose multi-task learning. In IEEE Conference on Computer Vision and Pattern Recognition, June 2020

    2020

  6. [14]

    Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning

    Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. In Advances in Neural Information Processing S...

    2021

  7. [15]

    Metabalance: Improving multi-task recommendations via adapting gradient magnitudes of auxiliary tasks

    Yun He, Xue Feng, Cheng Cheng, Geng Ji, Yunsong Guo, and James Caverlee. Metabalance: Improving multi-task recommendations via adapting gradient magnitudes of auxiliary tasks. In International ACM Web Conference, pp.\ 2205–2215, 2022. doi:10.1145/3485447.3512093. URL https://d...

    2022

  8. [16]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12 0 (7): 0 2217--2226, 2019. ...

    2019

  9. [17]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  10. [18]

    Revisiting scalarization in multi-task learning: A theoretical perspective

    Yuzheng Hu, Ruicheng Xian, Qilong Wu, Qiuling Fan, Lang Yin, and Han Zhao. Revisiting scalarization in multi-task learning: A theoretical perspective. In Advances in Neural Information Processing Systems, volume 36, pp.\ 48510--48533, 2023

    2023

  11. [19]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023 a

    2023

  12. [20]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023 b . URL https://openreview.net/forum?id=6t0Kwf8-jrj

    2023

  13. [21]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, pp.\ 876--885, 2018

    2018

  14. [22]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018

    2018

  15. [23]

    Rotograd: Gradient homogenization in multitask learning

    Adri \'a n Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=T8wHz4rnuGL

    2022

  16. [24]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=FCnohuR6AnM

    2023

  17. [25]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  18. [26]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops, June 2013

    2013

  19. [27]

    MNIST handwritten digit database

    Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann.lecun.com/exdb/mnist/

    2010

  20. [28]

    Conflict-averse gradient descent for multi-task learning

    Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, volume 34, pp.\ 18878--18890, 2021 a

    2021

  21. [29]

    Conflict-averse gradient descent for multi-task learning

    Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Advances in Neural Information Processing Systems, volume 34, pp.\ 18878--18890, 2021 b

    2021

  22. [30]

    Adversarial multi-task learning for text classification

    Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classification. In Association for Computational Linguistics, pp.\ 1--10, July 2017. doi:10.18653/v1/P17-1001. URL https://aclanthology.org/P17-1001

    2017 doi

  23. [31]

    Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition, June 2019

    2019

  24. [32]

    Auto-lambda: Disentangling dynamic task relationships

    Shikun Liu, Stephen James, Andrew Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=KKeCMim5VN

    2022

  25. [33]

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H. Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In International ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp.\ 1930–1939, 2018. doi:1...

    1930

  26. [34]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems, volume 35, pp.\ 17703--17716, 2022

    2022

  27. [35]

    Multi-task distillation: Towards mitigating the negative transfer in multi-task learning

    Ze Meng, Xin Yao, and Lifeng Sun. Multi-task distillation: Towards mitigating the negative transfer in multi-task learning. In IEEE International Conference Image Processing, pp.\ 389--393, 2021. doi:10.1109/ICIP42928.2021.9506618

    2021

  28. [36]

    Multi-task learning as a bargaining game

    Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. In International Conference on Machine Learning, volume 162, pp.\ 16428--16446, 17--23 Jul 2022. URL https://proceedings.mlr.press/...

    2022

  29. [37]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp.\ 4, 2011

    2011

  30. [38]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems, volume 36, pp.\ 66727--66754, 2023

    2023

  31. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  32. [40]

    An overview of gradient descent optimization algorithms

    Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2017. URL https://arxiv.org/abs/1609.04747

    2017 arXiv

  33. [41]

    Multi-task learning as multi-objective optimization

    Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018

    2018

  34. [42]

    The german traffic sign recognition benchmark: A multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: A multi-class classification competition. In International Joint Conference on Neural Networks, pp.\ 1453--1460, 2011

    2011

  35. [43]

    Towards plastic and stable exemplar-free incremental learning: A dual-learner framework with cumulative parameter averaging

    Wenju Sun, Qingyong Li, Wen Wang, and Yangli-ao Geng. Towards plastic and stable exemplar-free incremental learning: A dual-learner framework with cumulative parameter averaging. arXiv preprint arXiv:2310.18639, 2023

    2023 arXiv

  36. [44]

    Adashare: Learning what to share for efficient deep multi-task learning

    Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pp.\ 8728--8740, 2020

    2020

  37. [45]

    Merging multi-task models via weight-ensembling mixture of experts

    Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. In International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=nLRKnO74RB

    2024

  38. [46]

    Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations

    Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In ACM Conference on Recommender Systems, pp.\ 269–278, 2020. doi:10.1145/3383313.3412236. URL https://doi.org...

    2020

  39. [47]

    Hacking task confounder in meta-learning

    Jingyao Wang, Yi Ren, Zeen Song, Jianqi Zhang, Changwen Zheng, and Wenwen Qiang. Hacking task confounder in meta-learning. In IJCAI, 2023

    2023

  40. [48]

    Localizing task information for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Fran c ois Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=DWT9uiGjxT

    2024

  41. [49]

    All of statistics: A concise course in statistical inference, 2013

    L Wasserman. All of statistics: A concise course in statistical inference, 2013

    2013

  42. [50]

    Mixture of lo RA experts

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lo RA experts. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=uWvKBCYh4S

    2024

  43. [51]

    Sun database: Exploring a large collection of scene categories

    Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119: 0 3--22, 2016

    2016

  44. [52]

    TIES -merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES -merging: Resolving interference when merging models. In Advances in Neural Information Processing Systems, 2023

    2023

  45. [53]

    Cross-task knowledge distillation in multi-task recommendation

    Chenxiao Yang, Junwei Pan, Xiaofeng Gao, Tingyu Jiang, Dapeng Liu, and Guihai Chen. Cross-task knowledge distillation in multi-task recommendation. In AAAI Conference on Artificial Intelligence, volume 36, pp.\ 4318--4326, Jun. 2022. doi:10.1609/aaai.v36i4.20352

    2022 doi

  46. [54]

    Adatask: A task-aware adaptive learning rate approach to multi-task learning

    Enneng Yang, Junwei Pan, Ximei Wang, Haibin Yu, Li Shen, Xihua Chen, Lei Xiao, Jie Jiang, and Guibing Guo. Adatask: A task-aware adaptive learning rate approach to multi-task learning. AAAI Conference on Artificial Intelligence, 37 0 (9): 0 10745--10753, Jun. 2023. URL https:/...

    2023

  47. [55]

    Representation surgery for multi-task model merging

    Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. In International Conference on Machine Learning, 2024 a . URL https://openreview.net/forum?id=Sbl2keQEML

    2024

  48. [56]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=nZP6NgD3QY

    2024

  49. [57]

    Gradient surgery for multi-task learning

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in Neural Information Processing Systems, volume 33, pp.\ 5824--5836, 2020

    2020

  50. [58]

    Composing parameter-efficient modules with arithmetic operation

    Jinghan Zhang, shiqi chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operation. In Advances in Neural Information Processing Systems, volume 36, pp.\ 12589--12610, 2023 a

    2023

  51. [59]

    A survey on negative transfer

    Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. IEEE/CAA Journal of Automatica Sinica, 10 0 (2): 0 305--329, 2023 b . doi:10.1109/JAS.2022.106004

    2023

  52. [60]

    A survey on multi-task learning

    Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 34 0 (12): 0 5586--5609, 2022. doi:10.1109/TKDE.2021.3070203

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.