Pith. sign in

REVIEW 4 major objections 6 minor 49 references

Parameter-Efficient Interventions for Enhanced Model Merging

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Distributed low-rank edits beat one big adapter in model merging

desk verdict Solid, useful post-merge repair paper with a genuine new application of interventions; the main system comparison holds, but the mini-intervention claim is internally inconsistent and a distributed-Surgery control is missing. read the letter →

arxiv 2412.17023 v1 pith:5VTGOBVZ submitted 2024-12-22 cs.CV

Marcin Osial , Daniel Marczak , Bartosz Zieliński This is my paper · ORCID
classification cs.CV
keywords modelmergingrepresentationbiasparameter-efficientinterventionslow-rankadaptationmulti-tasklearningvisiontransformersurgerydistillation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the performance loss of merged multi-task vision models comes from representation bias that accumulates across all layers, not just at the output. It proposes IntervMerge, which attaches tiny task-specific low-rank intervention modules after the attention of every transformer block and trains them with a distillation loss against the original task models using a small sample of task data. It also introduces mini-interventions that edit only a slice of the representation, cutting extra parameters further. On eight image-classification tasks with a CLIP ViT-B/32 encoder, IntervMerge applied to AdaMerging reaches 88.96% average accuracy at rank 1, beating the previous state-of-the-art Surgery at rank 64 (87.5%) while using 3.5 times fewer extra parameters. If correct, this means widely-distributed, parameter-light representation repairs are more effective than a single high-capacity adapter at the final layer.

What carries the argument

The central object is the low-rank intervention module $\Phi^t_b$, an additive residual edit to the representation produced by the self-attention block, parameterized by two rank-$r$ projection matrices plus a bias, applied to the class token (or a slice of it) in every block and for each task. It is trained with a distillation loss that pushes the merged model's per-task representations toward those of the task-specific models, distributing the correction across depth rather than concentrating it at the last layer, which stops early errors from propagating and stabilizes later representations. The mini-intervention variant $\Phi^t_b(z_b[j:p])$ restricts the edit to a contiguous slice of the representation, and shifting that slice across blocks produces a parameter-efficient way to spread knowledge throughout the network.

What would settle it

Train IntervMerge with zero calibration data (e.g., using only the merged model's own predictions as pseudo-targets) and show that accuracy falls below the base merge; or, with data available, draw the calibration sample from a different distribution than the test split (e.g., one class per task removed) and show that the gains reported in Table 1 vanish. Alternatively, compare a single rank-64 intervention at the final layer against rank-1 interventions at every layer under the same total parameter budget on a broader suite; if the former wins, the depth-distribution hypothesis fails.

Watch

Extended reading notes

Core claim

The central discovery is that representation bias in a merged model can be corrected more effectively by lightweight, task-specific interventions applied at every block of the network than by a single high-rank adapter at the end. IntervMerge inserts, after the multi-head self-attention of each ViT block, a module $\Phi^t_b$ acting on the [CLS] token: $\Phi^t_b(z_b) = z_b + W_2^T(W_1 z_b + b - W_2 z_b)$, with low-rank matrices $W_1, W_2$ of rank $r$ (default $r=1$). Mini-interventions generalize this to editing only a contiguous slice $z_b[j:p]$ of the representation, reducing parameters to as little as 3k while still outperforming Surgery. Using the same distillation loss as Surgery, the modules are trained on a tiny held-out sample (1–10% of the test set) and can be combined with any base merge method; with AdaMerging the rank-1 IntervMerge reaches 88.96% average accuracy versus 87.5% for Surgery at rank 64, and the method also improves weaker merges such as Weight Averaging and Ties-Merging.

Load-bearing premise

The method assumes that a small sample of each task's data (1–10% of the test set) is available at merge time and is representative enough to train the intervention modules through the distillation loss; if that data is missing or unrepresentative, the interventions will overfit and the merged model gains nothing over the base merge.

Editorial extensions

If this is right

  • IntervMerge beats Surgery on all five base merging methods tested, with improvements of 2.56 points over Weight Averaging, 4.57 over Task Arithmetic, and 5.02 over Task-wise AdaMerging (Table 1).
  • When interventions are applied to every one of the 12 ViT blocks, average accuracy is highest (88.83%); halving the number of blocks to every second block costs less than a point, while concentrating all interventions in the last block drops accuracy by about 3 points.
  • Intervening on the class token is the best token choice for image classification, reaching 89.49% average accuracy versus 70.82% for patch tokens (Table 2).
  • With the same total parameter budget, shifting a 64-element mini-intervention across blocks (85.63%) slightly exceeds the best single-block full intervention (85.48%), so distributed small edits are at least as good as a carefully chosen single location.
  • With only 1% of the test set available for calibration, IntervMerge reaches 85.55% average accuracy versus 82.8% for Surgery, and the gap widens as more data is added (Table 8).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If distributed low-rank corrections are the right model of representation bias, the same recipe should transfer to decoder-only language models, where the role of the class token would be played by a fixed summary position or a learned prompt; testing that would separate the vision-specific part of the claim from the general mechanism.
  • The shift heuristic in mini-interventions implies the bias has a layer-dependent, structured component; learning the slice positions per task (rather than fixing a shift) might recover most of the full-intervention accuracy at the mini-intervention parameter cost.
  • The data-visibility results suggest the method is useful precisely when calibration data is scarce; the extreme zero-data case is untested, and a plausible outcome is that without any task data IntervMerge falls back to the base merge and could underperform Surgery on some tasks.
  • A practical consequence implicit in the paper is a new serving configuration: one shared backbone plus several 3k-parameter per-task modules instead of full per-task models, which is attractive for memory-limited deployment on edge devices.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes IntervMerge, a post-merging repair method for multi-task ViT classifiers. Task-specific low-rank intervention modules, inspired by ReFT, are inserted after the MHSA sublayer in every transformer block and trained by distillation between the merged model and the individual task models, optionally using a small amount of data. A 'mini-intervention' variant edits only a contiguous slice of the representation to reduce the parameter overhead. The method is evaluated on eight image classification datasets with ViT-B/32 and ViT-L/14 (the latter in the supplement), comparing against base merging methods and the Surgery baseline. The headline result is that rank-1 IntervMerge with AdaMerging reaches 88.96% average accuracy with 147k extra parameters, surpassing rank-64 Surgery at 87.5% with 524k extra parameters.

Significance. If the comparisons are properly controlled, the contribution is practically useful: it would show that distributing low-rank representation interventions across blocks can repair representation bias more cheaply than a single final-layer adapter. The paper also provides a careful ablation of token choice, block placement, rank, and edit length, which is valuable design guidance for the merging community. The use of three-seed standard deviations in Tables 2, 4, and 7 is a strength. However, the central attribution of the improvement is currently under-supported because the headline comparison varies the intervention formula and the number of intervened layers simultaneously, and a few reported claims are not consistent with the tables. The method is simple enough to reproduce, but the paper does not state whether code will be released.

major comments (4)
  1. [§5, Table 1] The claim that rank-1 IntervMerge outperforms rank-64 Surgery by 1.46% with 3.5x fewer parameters is not an apples-to-apples comparison of the intervention methods. The two models differ in the intervention function (residual ReFT-style map vs. the ReLU adapter) and in placement (all 12 blocks vs. only the final representation). Since Table 3 shows that the number of intervened blocks matters (85.48% for one middle block vs. 88.83% for all 12), the advantage may reflect placement rather than the proposed intervention design. Please add a controlled comparison, for example Surgery applied at every block with a comparable parameter budget and training protocol, or IntervMerge restricted to the last block with a matched budget.
  2. [§5, paragraph after Table 1] The statement that the 64-part mini-intervention with the h + RT(b) formula 'achieves an average accuracy 3% higher than Surgery, using fewer parameters' is contradicted by Table 1, where AdaMerging w/IntervMerge (mini-interv.) has 83.45% with 3k parameters while AdaMerging w/Surgery has 86.1% with 131k parameters. Please clarify which Surgery configuration is being referenced (e.g., the rank-1 row in Table 6), and report the numbers for that configuration in Table 1 or adjust the claim.
  3. [§3.1, Eq. (3.1); §5.5, Table 8] The objective in Eq. (3.1) is written as a loss on the test datasets, and Table 8 explicitly uses 1-10% of the available test set to fit the interventions. If the same test split is also used for hyperparameter selection (Tables 2-5) and final evaluation, the reported accuracies may be optimistically biased. Please state clearly whether the available subset is disjoint from the evaluation set; if it is not, add experiments with a separate validation split or a subset of the training set, and add a limitation note in Section 6.
  4. [§5, Tables 1 and 6] The headline comparisons are single runs without error bars, while Tables 2, 4, and 7 report ± values over three seeds. Because several differences in the tables are small (e.g., Table 4 rows 87.59 vs. 87.54, Table 7 rows 89.53 vs. 89.56), the paper should report mean and standard deviation over at least three seeds for Tables 1 and 6, or at least for the key rank-1 IntervMerge vs. rank-16/64 Surgery comparisons.
minor comments (6)
  1. [§3.2.1, Eq. (3.4)] The specified dimensions W1, W2 ∈ R^{k×r} are inconsistent with the product W2^T(W1 z_b + b - W2 z_b); W1 and W2 should be in R^{r×k} and b in R^r for the expression to be well-defined. Please correct the dimensions.
  2. [§3.2.2, Eq. (3.5)] The edited segment is written as z_b[j:p] in the equation but as z_b[j:k] in the prose; p is not defined. Please unify the notation.
  3. [Table 5] The range '0-786' appears to be a typo for '0-768', and the range '200-184' appears reversed; please fix these entries.
  4. [Table 3] The note that intervened blocks are 'marked in green' does not carry over to the typeset table, and the exact block indices are not provided; please list them explicitly.
  5. [Table 7] The notation RT(b) should be written as R^T b (transpose of R applied to b) to avoid ambiguity with a matrix product.
  6. [Tables 1 and 6] The two tables report slightly different accuracies for the same AdaMerging w/IntervMerge rank-1 configuration (88.96 vs. 88.90); please clarify whether these come from different settings or different runs.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; IntervMerge is validated against external baselines and no equation reduces to its own inputs.

full rationale

The paper's central comparison is empirical: IntervMerge modules are trained with the distillation loss (Eq. 3.2) and evaluated on the same eight benchmarks used for Surgery and AdaMerging, so the headline gain is an externally anchored measurement rather than a fitted constant. The intervention formula (Eq. 3.4) is adapted from ReFT and used as a design choice, but it is not invoked as proof of success; performance is measured. Tables 1-8 tune design choices (block count, token, rank, edit length) on the benchmark itself, which is model selection rather than a circular derivation. The only author self-citation (Magmax, ref. [22]) appears as related-work context and carries no load in the argument. The skeptic's point that the 12-block rank-1 IntervMerge vs. single-block rank-64 Surgery comparison varies both placement and architecture is a legitimate experimental-confounds concern, but it is not circularity: the claim is not forced by construction, and the paper includes ablations of block count (Table 3) and rank (Table 6). Use of a small subset of test inputs for distillation is a data-assumption limitation, not a derivation that reduces to its own output. No equation is equivalent to another by definition, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The method trades a small number of tunable design choices (rank, block count, token, part size) for the ability to repair arbitrary merged models without retraining. These choices are all validated by ablations on the same benchmark, so they should be treated as fitted selection, not as free-floating constants. No new physical or architectural entity is introduced; the intervention modules are borrowed from ReFT.

free parameters (5)
  • rank r = 1 (default), up to 64
    The rank of the low-rank projections W1, W2 determines the parameter count and accuracy (Tables 1 and 6). The default r=1 is selected by hand, and the paper reports improvements with r=4 on AdaMerging.
  • number of intervened blocks N = 12
    The paper sets N=12 (all ViT-B/32 blocks) after ablating 1, 2, 3, 4, 6, 12 blocks in Table 3; the all-12 configuration gives the best average.
  • token selection for intervention = class token
    Table 2 shows the class token gives 89.49 avg vs 88.79 for first/last token and 70.82 for patch tokens; the class token is chosen for all subsequent experiments.
  • mini-intervention part size and position = shifting 64-element segments
    Tables 4-5 show 64-element shifted edits achieve 85.63 vs 85.48 for optimal single-block full edit; this configuration is used for the mini-intervention claims.
  • distillation hyperparameters = matched to Surgery
    The paper reuses Surgery's iterations, batch size, optimizer, and data-subset fraction (Section 4), so these are inherited rather than independently validated.
assumptions (3)
  • domain assumption Representation bias in the merged model is the primary cause of performance loss and can be corrected by task-specific interventions on intermediate representations.
    The paper motivates this in Section 1 and 3.2 without a formal theorem; it is supported only by empirical results.
  • domain assumption The distillation loss (Eq. 3.2) on a small data subset is a sufficient objective for aligning the merged model to each task-specific model.
    Section 3.2 states the loss without analysis of its optimization landscape or generalization.
  • ad hoc to paper Intervening after Multi-Head Self-Attention is at least as good as intervening elsewhere; the paper only evaluates this placement (Section 6 lists it as a limitation).
    The position after MHSA is fixed by design and not compared to after-MLP positions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Parameter-Efficient Interventions for Enhanced Model Merging." pith.science (2026). https://pith.science/paper/5VTGOBVZ

@misc{pith2026241217023,
  author       = {Pith},
  title        = {Pith review of: Parameter-Efficient Interventions for Enhanced Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VTGOBVZ}},
  note         = {Machine review of arXiv:2412.17023}
}
read the original abstract

Model merging combines knowledge from task-specific models into a unified multi-task model to avoid joint training on all task data. However, current methods face challenges due to representation bias, which can interfere with tasks performance. As a remedy, we propose IntervMerge, a novel approach to multi-task model merging that effectively mitigates representation bias across the model using taskspecific interventions. To further enhance its efficiency, we introduce mini-interventions, which modify only part of the representation, thereby reducing the additional parameters without compromising performance. Experimental results demonstrate that IntervMerge consistently outperforms the state-of-the-art approaches using fewer parameters.

Figures

Figures reproduced from arXiv: 2412.17023 by the authors.

Figure 1
Figure 1. IntervMerge consistently demonstrates su￾perior performance compared to the state-of-the-art Surgery approach in multi-task model merging. This advantage is particularly evident when utilizing our effi￾cient mini-intervention mechanism, which achieves bet￾ter results than Surgery while employing three times fewer parameters. It is important to note that Inter￾vMerge may exhibit more additional parameters than Surger… view at source ↗
Figure 2
Figure 2. Various solutions of MTL have different issues. Multiple individually trained models (a) require storing [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the mini-intervention approach, [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Representations of various methods obtained [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Utilizing a stitched network demonstrates that [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Accuracy across considered datasets fluctu [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 32 canonical work pages

  1. [1]

    Evolutionary optimization of model merging recipes

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187 , 2024

    arXiv 2024

  2. [2]

    Multitask learning

    Rich Caruana. Multitask learning. Machine learning , 28:41–75, 1997

    work page 1997

  3. [3]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865– 1883, 2017

    work page 2017

  4. [4]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606–3613, 2014

    2014

  5. [5]

    Model breadcrumbs: Scaling multi-task model merging with sparse masks

    MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. arXiv preprint arXiv:2312.06795 , 2023

    arXiv 2023

  6. [6]

    Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models

    Nikolaos Dimitriadis, Pascal Frossard, and Fran¸ cois Fleuret. Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models. In Interna- tional Conference on Machine Learning , pages 8015–

    work page

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021

    2021

  8. [8]

    Essentially no barriers in neural network energy landscape

    Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning , pages 1309–1318. PMLR, 2018

    work page 2018

Show all 49 references
  1. [9]

    Sharpness-aware minimization for efficiently improving generalization

    Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020

    2010 arXiv

  2. [10]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning , pages 3259–3269. PMLR, 2020

    2020

  3. [11]

    Loss sur- faces, mode connectivity, and fast ensembling of dnns

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss sur- faces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018

    2018

  4. [12]

    Arcee’s mergekit: A toolkit for merging large language mod- els

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Bene- dict, Mark McQuade, and Jacob Solawetz. Arcee’s mergekit: A toolkit for merging large language mod- els. arXiv preprint arXiv:2403.13257 , 2024

    2024 arXiv

  5. [13]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover clas- sification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover clas- sification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217– 2226, 2019

    2019

  6. [14]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco T´ ulio Ribeiro, Mitchell Worts- man, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023

    2023

  7. [15]

    Patching open-vocabulary models by interpolating weights

    Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Ko- rnblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In NeurIPS, 2022

    2022

  8. [16]

    Garipov, D

    Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, D. Vetrov, and A. Wilson. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence (UAI) , 2018

    2018

  9. [17]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849, 2022

    2022 arXiv

  10. [18]

    3d object representations for fine-grained catego- rization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei- Fei. 3d object representations for fine-grained catego- rization. In Proceedings of the IEEE international con- ference on computer vision workshops , pages 554–561, 2013

    2013

  11. [19]

    The mnist database of handwritten digits

    Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/ , 1998

    1998

  12. [20]

    Deep model fusion: A survey

    Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv: 2309.15698 , 2023

    2023 arXiv

  13. [21]

    Kobyzev, Mehdi Rezagholizadeh, Ahmad Rashid, A

    Peng Lu, I. Kobyzev, Mehdi Rezagholizadeh, Ahmad Rashid, A. Ghodsi, and P. Langlais. Improving gener- alization of pre-trained language models via stochastic weight averaging. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022

    2022

  14. [22]

    Magmax: Leveraging model merging for seamless continual learning

    Daniel Marczak, Bart lomiej Twardowski, Tomasz Trzci´ nski, and Sebastian Cygert. Magmax: Leveraging model merging for seamless continual learning. 2024

    2024

  15. [23]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, page 4, Granada, Spain, 2011

    2011

  16. [24]

    What is being transferred in transfer learn- ing? Advances in neural information processing sys- tems, 33:512–523, 2020

    Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? Advances in neural information processing sys- tems, 33:512–523, 2020

    2020

  17. [25]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jim´ enez, Alessandro Favero, and Pas- cal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In NeurIPS, 2023

    2023

  18. [26]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. In International conference on ma- chine learni...

    2021

  19. [27]

    Model ratatouille: Recycling diverse models for out- of-distribution generalization

    Alexandre Ram´ e, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, L´ eon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out- of-distribution generalization. In International Confer- ence on Machine Learning, pages 28656–28679. PMLR, 2023

    2023

  20. [28]

    Rewarded soups: towards pareto-optimal alignment by interpolating weights fine- tuned on diverse rewards

    Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine- tuned on diverse rewards. Advances in Neural Informa- tion Processing Sy...

    2024

  21. [29]

    Warp: On the benefits of weight averaged rewarded policies

    Alexandre Ram´ e, Johan Ferret, Nino Vieillard, Robert Dadashi, L´ eonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douil- lard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies. arXiv preprint arXiv:2406.16768, 2024

    2024 arXiv

  22. [30]

    Diverse weight averaging for out-of- distribution generalization

    Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of- distribution generalization. NeurIPS, 2022

    2022

  23. [31]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaf- fin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 , 2021

    arXiv

  24. [32]

    The german traffic sign recognition benchmark: a multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks, pages 1453–1460. IEEE, 2011

    2011

  25. [33]

    Weight averaging for neural net- works and local resampling schemes

    Joachim Utans. Weight averaging for neural net- works and local resampling schemes. In Proc. AAAI- 96 Workshop on Integrating Multiple Learned Models. AAAI Press, pages 133–138. Citeseer, 1996

    1996

  26. [34]

    Multi-task learning for dense prediction tasks: A survey

    Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey. IEEE TPAMI, 2021

    2021

  27. [35]

    Lo- calizing task information for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz- Jimenez, Fran¸ cois Fleuret, and Pascal Frossard. Lo- calizing task information for improved model merging and compression. ICML, 2024

    2024

  28. [36]

    Learning neural network subspaces

    Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In International Conference on Machine Learning , pages 11217–11227. PMLR, 2021

    2021

  29. [37]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy with- out increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy with- out increasing ...

    2022

  30. [38]

    Robust fine- tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer visio...

    2022

  31. [39]

    Under- standing and improving information transfer in multi- task learning

    Sen Wu, Hongyang Zhang, and Christopher R´ e. Under- standing and improving information transfer in multi- task learning. ICLR, 2020

    2020

  32. [40]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Reft: Representation finetuning for language models. arXiv preprint arXiv: 2404.03592 , 2024

    2024 arXiv

  33. [41]

    Sun database: Explor- ing a large collection of scene categories

    Jianxiong Xiao, Krista A Ehinger, James Hays, Anto- nio Torralba, and Aude Oliva. Sun database: Explor- ing a large collection of scene categories. International Journal of Computer Vision , 119:3–22, 2016

    2016

  34. [42]

    Lm-cocktail: Resilient tuning of language models via model merging

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Xingrun Xing. Lm-cocktail: Resilient tuning of language models via model merging. arXiv preprint arXiv:2311.13534 , 2023

    2023 arXiv

  35. [43]

    TIES-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In NeurIPS, 2023

    2023

  36. [44]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  37. [45]

    Repre- sentation surgery for multi-task model merging

    Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xi- aojun Chen, Xingwei Wang, and Dacheng Tao. Repre- sentation surgery for multi-task model merging. ICML, 2024

    2024

  38. [46]

    Adamerg- ing: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guib- ing Guo, Xingwei Wang, and Dacheng Tao. Adamerg- ing: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations, 2024

    2024

  39. [47]

    Unleashing the power of multi- task learning: A comprehensive survey spanning tra- ditional, deep, and pretrained foundation model eras

    Jun Yu, Yutong Dai, Xiaokang Liu, Jin Huang, Yishan Shen, Ke Zhang, Rong Zhou, Eashan Adhikarla, Wenx- uan Ye, Yixin Liu, et al. Unleashing the power of multi- task learning: A comprehensive survey spanning tra- ditional, deep, and pretrained foundation model eras. arXiv prepr...

    2024 arXiv

  40. [48]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024

    2024

  41. [49]

    Cross-task linearity emerges in the pretraining-finetuning paradigm

    Zhanpeng Zhou, Zijun Chen, Yilan Chen, Bo Zhang, and Junchi Yan. Cross-task linearity emerges in the pretraining-finetuning paradigm. arXiv preprint arXiv:2402.03660, 2024. Copyright © 2025 by SIAM Unauthorized reproduction of this article is prohibited

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.