Pith. sign in

REVIEW 3 major objections 4 minor 40 references

Forgetting of task-specific knowledge in model merging-based continual learning

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that linear model merging preserves shared knowledge but rapidly degrades unshared task-specific knowledge, making naive merging risky for continual learning.

desk verdict The shared-vs-task-specific forgetting result is real and cleanly shown; the incremental-vs-parallel advantage claim is partly inherited from an asymmetric training budget and needs a matched comparison. read the letter →

arxiv 2507.23311 v1 pith:KNAOW6DZ submitted 2025-07-31 cs.CV

Timm Hess , Gido M van de Ven , Tinne Tuytelaars This is my paper · ORCID
classification cs.CV
keywords modelmergingweightinterpolationcontinuallearningcatastrophicforgettingsharedknowledgetask-specificshortcutvisualcues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a dichotomy in how linear model merging treats different kinds of knowledge: what is shared between models survives or improves, while what is unique to one task is rapidly erased. The authors build controlled vision experiments in which 5x5 colored or grayscale noise patches on CIFAR-100 act as separate 'tasks', letting them isolate shared from task-specific knowledge. They find that interpolation between specialized models preserves the common base knowledge, sometimes pushing no-cue accuracy above the pre-trained baseline, but collapses sensitivity to a task's own cue from above 98.5 percent toward chance-level general accuracy. They also find that merging models taken from a sequential, incremental training process consistently beats merging parallel-trained specialists. If the paper is right, simple post-hoc weight averaging is unsuitable for continual learning whenever preserving unique past-task skills matters.

What carries the argument

The central object is a pair of controlled visual cues—a solid-color HSV patch and a grayscale noise patch, both 5x5—superimposed on CIFAR-100 images during training with probability $p_C = 0.5$. Two protocols instantiate the knowledge types: the shared protocol uses the same color cue at the same position for both tasks, and the task-specific protocol uses the color cue at top-left versus the noise cue at bottom-right so the tasks share as little task-specific information as possible. All endpoint models start from the same pre-trained weights $\theta_{\mathrm{PT}}$, and merging is the linear interpolation $\bar{\theta}(\alpha) = \alpha\theta_{T1} + (1-\alpha)\theta_{T2}$, evaluated by applying each cue deterministically ($p_C = 1.0$) to probe cue sensitivity and with no cue ($p_C = 0.0$) to probe general shared knowledge. This design carries the argument because it makes the shared versus unshared distinction observable in a single controlled accuracy curve instead of relying on semantic task splits.

What would settle it

Run the task-specific protocol with the two cues swapped in position or applied at a different patch size; if interpolating at the midpoint keeps color-cue accuracy near the endpoint level instead of collapsing toward the general accuracy of roughly 60 percent, then the claimed dichotomy is an artifact of this particular cue placement rather than a general property of merging.

Watch

Extended reading notes

Core claim

On the paper's own terms: during linear interpolation of two models adapted from a common pre-trained checkpoint, shared knowledge components are largely preserved and can be enhanced, while unshared task-specific components are significantly degraded because averaging interferes with divergent parameter adaptations. This holds for both training regimes, but merging incrementally trained models consistently outperforms merging parallel-trained models, suggesting that sequential adaptation, even without explicit continual-learning mechanisms, leaves models on trajectories that are more amenable to beneficial merging. The consequence for continual learning is that naive merging risks catastrophic forgetting of unique past-task knowledge and is suited mainly to scenarios that prioritize general shared capabilities or highly similar tasks.

Load-bearing premise

The clean separation between shared and task-specific knowledge depends on the assumption that the injected 5x5 cues, and their non-overlapping positions, create purely task-unique behavior with no hidden shared component that would change the forgetting curves.

Editorial extensions

If this is right

  • Naive linear merging is not a safe standalone continual-learning mechanism when past tasks depend on unique, unshared skills.
  • Merging checkpoints from an incremental training run preserves knowledge better than merging independently trained specialists, so sequential training trajectories are more merge-friendly.
  • Shared and general knowledge can survive or improve through interpolation; in the main experiments, no-cue CIFAR-100 accuracy rose to a peak of 62.69 percent, above the pre-trained 61.3 percent baseline.
  • Task-specific cue accuracy, starting near 98.5 percent, falls sharply as the interpolation coefficient moves toward the other endpoint, with the drop steeper in the parallel setting.
  • Even in the incremental scenario, unshared task-specific knowledge is still rapidly degraded, so the benefit is not enough to retain unique past-task skills.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same interpolation curve separates shared from task-specific knowledge, the protocol doubles as a diagnostic: a capability that collapses at the midpoint of an interpolation is task-specific and will be lost in any naive merge.
  • The paper frames forgetting as undesirable for continual learning, but the mechanism is the same one used to erase biases in language models; whether the loss is a bug or a feature depends entirely on whether the erased knowledge is wanted, which the paper leaves to the application.
  • A natural next experiment, not run here, would scale the cue protocol to transformer-based vision models and longer task sequences to test whether the shared-versus-specific dichotomy persists.
  • The incremental advantage suggests that co-designing sequential training to keep checkpoints merge-compatible, for example by limiting drift of shared features, could preserve more unique skills than post-hoc averaging alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies linear weight interpolation (model merging) between models trained on CIFAR-100 with injected synthetic visual cues. It introduces a controlled methodology for instantiating shared versus task-specific knowledge and compares two training regimes: incremental (sequential adaptation) and parallel (independent specialists). The main reported findings are that shared knowledge is largely preserved or even enhanced under interpolation, that unshared task-specific knowledge degrades rapidly, and that merging incrementally trained models outperforms merging parallel-trained models. The claims are supported by accuracy measurements on the full CIFAR-100 test set, with three seeds and standard errors, plus additional chunking and reverse-order experiments.

Significance. If the claims hold, this is a useful controlled demonstration in computer vision of a knowledge-type asymmetry in linear model merging, complementing prior LLM work by Zaman et al. and informing when naive merging is appropriate for continual learning. The paper's strengths are its explicit synthetic-cue protocol for isolating knowledge types, its endpoint sanity checks, its reporting of means with standard errors, and its release of code. However, the headline incremental-versus-parallel comparison is confounded by an asymmetric training budget, and part of the shared-knowledge protocol involves retraining on the same task; these issues affect the central message and need to be addressed before the claims can be accepted as stated.

major comments (3)
  1. [§4, Table 1, Figure 1e] The comparison underlying the headline 'incremental vs. parallel' claim is confounded by an asymmetric training budget. In the task-specific protocol (TColor→TNoise), the incremental α=1 endpoint has undergone 100 adaptation epochs and seen all data from both tasks, whereas each parallel endpoint has undergone only 50 epochs on a single task. Table 1 shows that the no-cue accuracy at α=1 is already 62.08% for incremental versus 60.83% for parallel, and the color-cue accuracy at α=1 is 64.97% versus 59.43%; the incremental curve's superiority is thus mostly inherited from the endpoint, not produced by the interpolation. The reported incremental peak of 62.69% is only 0.61 points above its best endpoint, and no significance test for the interpolation-specific advantage is reported. Please equalize the training budget (or task exposure) across the two arms, or explicitly report endpoint-normalized curves (e.g., accuracy relative to the corresponding α=0 and α=1 endpoints) and temper the abstract claim accordingly.
  2. [§3, Figure 1d] The shared-knowledge protocol TColor→TColor is also affected by the same asymmetry in a stronger form: in the incremental arm the second endpoint is obtained by training again on the same color-cue task, so that endpoint has 100 epochs on the identical task and data, while the parallel endpoints each have 50. The monotonic improvement along the incremental path in Figure 1d therefore partly reflects repeated training rather than a property of merging shared knowledge. I suggest either redesigning the shared protocol so the two incremental stages use distinct data (e.g., different chunks) with the same cue, or restricting the shared-knowledge claims to the parallel arm and the no-cue shared-knowledge measure, which do not have this confound.
  3. [§3, Appendix D.2] In the task-specific protocol, cue identity is confounded with spatial position: the color cue is always in the top-left and the noise cue always in the bottom-right. Because the reverse-order experiment keeps this same position-cue mapping, it does not disentangle whether the observed rapid degradation reflects task-specific knowledge per se or merely the fact that the two endpoints are specialized to different image locations (or different low-level patch statistics). A cross-over experiment with positions swapped between the cues, or an additional shared protocol with the same cue at two different positions, would make the shared-versus-task-specific interpretation substantially more robust.
minor comments (4)
  1. [Abstract, §4] The word 'consistently' overstates the evidence, which comes from one main protocol, one reverse-order setup, and one chunking setup, each with three seeds; 'in our experiments' would be more precise.
  2. [Figure 1 vs Figure 2] The caption of Figure 1 says the pre-trained reference is a solid black line, while in Figure 2 the pre-trained reference is a black diamond; consider using consistent marker styles across the two figures.
  3. [Appendix D.2, Table 3] Table 3 uses 'Alpha' as a column header while the text and other tables use the symbol α; unify the notation.
  4. [§6] The limitations paragraph mentions scaling to vision transformers but gives no citation for the claim that merging benefits are known for large models; a reference to model soups or task arithmetic (both already cited elsewhere) would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all headline claims are direct measurements on held-out test sets, with no fitted parameter or derived quantity that reduces to its input.

full rationale

The paper's central claims (shared knowledge preserved, task-specific knowledge degrades, incremental merging outperforms parallel merging) are empirical observations obtained by interpolating fixed endpoint checkpoints and evaluating on held-out CIFAR-100 test sets with deterministic cue application (p_C=1.0 or 0.0). No parameter is fitted to the reported accuracies, and no prediction is computed from the definitions of shared vs. task-specific; those definitions merely select which probe accuracy is read out. The self-citations (Hess et al. 2023, 2024; van de Ven et al. 2025) supply background framing and motivation, but the result curves in Figure 1 and Tables 1-3 are independent of those citations. The incremental-vs-parallel contrast is confounded by asymmetric training budgets (the incremental endpoint has two 50-epoch adaptations and sees both tasks, while parallel specialists each see one task), so the superiority of the incremental curve may be partly inherited from endpoint quality rather than from merging compatibility; however, this is a potential confound or interpretation risk, not a circularity, because the comparison is still a direct measurement rather than a quantity made equal to its input by construction. Accordingly, no circular step meeting the evidentiary standard can be exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper is empirical; it introduces no new mathematical entities or fitted constants. The only quantities are experimentally chosen (cue size, probability, positions), which are design choices rather than free parameters being fit to support the claim.

assumptions (3)
  • domain assumption Linear weight interpolation is a meaningful operation for these models due to shared pre-training (linear mode connectivity).
    The paper assumes that averaging weights of models fine-tuned from a common initialization yields interpretable intermediate models, citing Goodfellow et al. 2015; Frankle & Carbin 2019; Ainsworth et al. 2023. This underlies all interpolation experiments.
  • domain assumption Accuracy on cue-present test sets measures task-specific knowledge.
    Section 3 and Figure 1: the evaluation protocol uses p_C=1.0 for cue-specific probes and p_C=0.0 for shared knowledge. This assumes the cue's behavioral effect is a valid proxy for internal knowledge.
  • domain assumption Non-overlapping cue positions prevent information transfer between tasks in the unshared protocol.
    Section 3: cues are placed top-left and bottom-right to 'minimize information transfer'. This is a design assumption that the tasks are truly unshared.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Forgetting of task-specific knowledge in model merging-based continual learning." pith.science (2026). https://pith.science/paper/KNAOW6DZ

@misc{pith2026250723311,
  author       = {Pith},
  title        = {Pith review of: Forgetting of task-specific knowledge in model merging-based continual learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KNAOW6DZ}},
  note         = {Machine review of arXiv:2507.23311}
}
read the original abstract

This paper investigates the linear merging of models in the context of continual learning (CL). Using controlled visual cues in computer vision experiments, we demonstrate that merging largely preserves or enhances shared knowledge, while unshared task-specific knowledge rapidly degrades. We further find that merging models from an incremental training process consistently outperforms merging models trained in parallel.

Figures

Figures reproduced from arXiv: 2507.23311 by the authors.

Figure 1
Figure 1. Experimental Protocol and Main Results. (a) Example of shared and task-specific knowledge instantiated with visual cues. The shared knowledge protocol uses the same cue for both tasks, while the task-specific knowledge protocol uses distinct cues. (b) Schematic illustration of ‘incremental’ (blue) and ‘parallel’ (orange) training. Both scenarios start from a common pre-trained model (Task 0, yellow) and adapt models… view at source ↗
Figure 2
Figure 2. Weight-Interpolation Results in the CIFAR-100 ‘Chunking’ Setup. (a) Accuracy (y-axis) vs. interpola￾tion coefficient α (x-axis) for the task-specific knowledge protocol in the chunking setup. Performance is evaluated on the color (left) and noise (right) visual cues, comparing incremental (blue circles) and parallel (orange diamonds) training. The endpoint models are specialized on chunk 2 with TColor (α = 0) and ch… view at source ↗
Figure 3
Figure 3. Weight-Interpolation Results for Reverse Order Cue Adaptation. Accuracy (y-axis) vs. interpolation coefficient α (x-axis). The α = 0 endpoint is specialized on TNoise and the α = 1 endpoint on TColor. Results compare ‘incremental’ (blue circles) and ‘parallel’ (orange diamonds) training. (a) The task-specific panels (TNoise, TColor) show the performance on the respective cue-specific test sets. (b) The shared CIFAR-… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 29 canonical work pages

  1. [1]

    Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa

    Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023

    work page 2023

  2. [2]

    Where is the truth? the risk of getting confounded in a continual world

    Florian Peter Busch, Roshni Kamath, Rupert Mitchell, Wolfgang Stammer, Kristian Kersting, and Martin Mundt. Where is the truth? the risk of getting confounded in a continual world. In International Conference on Machine Learning, 2025

    work page 2025

  3. [3]

    A continual learning survey: Defying forgetting in classification tasks

    Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ale s Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 0 (7): 0 3366--3385, 2022

    work page 2022

  4. [4]

    Essentially no barriers in neural network energy landscape

    Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, pp.\ 1309--1318. PMLR, 2018

    work page 2018

  5. [5]

    How to merge your multimodal models over time? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 20479--20491, 2025

    Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Albanie, and Matthias Bethge. How to merge your multimodal models over time? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 20479--20491, 2025

    work page 2025

  6. [6]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019

    2019

  7. [7]

    Loss surfaces, mode connectivity, and fast ensembling of DNNs

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of DNNs . Advances in Neural Information Processing Systems, 31, 2018

    work page 2018

  8. [8]

    Shortcut learning in deep neural networks

    Robert Geirhos, J \"o rn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2 0 (11): 0 665--673, 2020

    2020

Show all 40 references
  1. [9]

    Goodfellow, Oriol Vinyals, and Andrew M

    Ian J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems. In International Conference on Learning Representations, 2015

    2015

  2. [10]

    Two complementary perspectives to continual learning: Ask not only what to optimize, but also how

    Timm Hess, Tinne Tuytelaars, and Gido M van de Ven. Two complementary perspectives to continual learning: Ask not only what to optimize, but also how. In Proceedings of the 1st ContinualAI Unconference, volume 249 of Proceedings of Machine Learning Research, pp.\ 37--61, 2023

    2023

  3. [11]

    Knowledge accumulation in continually learned representations and the issue of feature forgetting

    Timm Hess, Eli Verwimp, Gido M van de Ven, and Tinne Tuytelaars. Knowledge accumulation in continually learned representations and the issue of feature forgetting. Transactions on Machine Learning Research, 2024

    2024

  4. [12]

    Patching open-vocabulary models by interpolating weights

    Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems, 35: 0 29262--29277, 2022

    2022

  5. [13]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023

    2023

  6. [14]

    Averaging weights leads to wider optima and better generalization

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, 2018

    2018

  7. [15]

    Continual learning with weight interpolation

    Jedrzej Kozal, Jan Wasilewski, Bartosz Krawczyk, and Micha Wo \'z niak. Continual learning with weight interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp.\ 4187--4195, 2024

    2024

  8. [16]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  9. [17]

    Chunking: Continual learning is not just about distribution shift

    Thomas L Lee and Amos Storkey. Chunking: Continual learning is not just about distribution shift. In Proceedings of The 3rd Conference on Lifelong Learning Agents, volume 274 of Proceedings of Machine Learning Research, pp.\ 915--937, 2025

    2025

  10. [18]

    Deep model fusion: A survey

    Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023

    2023 arXiv

  11. [19]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, 30, 2017

    2017

  12. [20]

    MagMax : Leveraging model merging for seamless continual learning

    Daniel Marczak, Bart omiej Twardowski, Tomasz Trzci \'n ski, and Sebastian Cygert. MagMax : Leveraging model merging for seamless continual learning. In European Conference on Computer Vision, pp.\ 379--395, 2024

    2024

  13. [21]

    Weighted ensemble models are strong continual learners

    Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and St \'e phane Lathuili \`e re. Weighted ensemble models are strong continual learners. In European Conference on Computer Vision, pp.\ 306--324. Springer, 2024

    2024

  14. [22]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35: 0 17703--17716, 2022

    2022

  15. [23]

    Catastrophic interference in connectionist networks: The sequential learning problem

    Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp.\ 109--165. Elsevier, 1989

    1989

  16. [24]

    Linear mode connectivity in multitask and continual learning

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations, 2021

    2021

  17. [25]

    What is being transferred in transfer learning? Advances in Neural Information Processing Systems, 33: 0 512--523, 2020

    Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in Neural Information Processing Systems, 33: 0 512--523, 2020

    2020

  18. [26]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36: 0 66727--66754, 2023

    2023

  19. [27]

    Continual lifelong learning with neural networks: A review

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113: 0 54--71, 2019

    2019

  20. [28]

    Effect of scale on catastrophic forgetting in neural networks

    Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022

    2022

  21. [29]

    A practitioner's guide to real-world continual multimodal pretraining

    Vishaal Udandarao, Karsten Roth, Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals, Olivier Henaff, Samuel Albanie, Zeynep Akata, and Matthias Bethge. A practitioner's guide to real-world continual multimodal pretraining. Advances in Neural Information Processing S...

    2024

  22. [30]

    Continual learning and catastrophic forgetting

    Gido M van de Ven , Nicholas Soures, and Dhireesha Kudithipudi. Continual learning and catastrophic forgetting. In John Wixted (ed.), Learning and Memory: A Comprehensive Reference (Third Edition), volume 1, pp.\ 153--168. Academic Press, Oxford, 2025

    2025

  23. [31]

    A comprehensive survey of continual learning: Theory, method and application

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46 0 (8): 0 5362--5383, 2024

    2024

  24. [32]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  25. [33]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision...

    2022

  26. [34]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36: 0 7093--7115, 2023

    2023

  27. [35]

    Model merging in LLMs , MLLMs , and beyond: Methods, theories, applications and opportunities

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in LLMs , MLLMs , and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666, 2024

    2024 arXiv

  28. [36]

    Fuse to forget: Bias reduction and selective memorization through model fusion

    Kerem Zaman, Leshem Choshen, and Shashank Srivastava. Fuse to forget: Bias reduction and selective memorization through model fusion. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.\ 18763--18783, Miami, Florida, USA, 2024. Associ...

    2024

  29. [37]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  30. [38]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  31. [39]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  32. [40]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.