Pith. sign in

REVIEW 3 major objections 4 minor 46 references

Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model merging, not fine-tuning, makes few-shot learning certifiable.

desk verdict The model-merging/PAC-Bayes connection is real and interesting, but the reported certificates rely on a 10-sample Monte Carlo estimate of train error, so the headline bounds are not rigorous as written. read the letter →

arxiv 2505.15798 v1 pith:GQFVTQL3 submitted 2025-05-21 cs.LG

Taehoon Kim , Henry Gouk , Minyoung Kim , Timothy Hospedales This is my paper · ORCID
classification cs.LG
keywords modelmergingPAC-Bayesboundsgeneralisationcertificatesfew-shotlearningnon-vacuousfusionlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model-merging learners fit only a small set of coefficients that combine fixed pre-trained source models, so the PAC-Bayes complexity term scales with the number of merge coefficients, not with the size of the source networks. The paper argues that this makes low-shot learning certifiable: standard PAC-Bayes bounds with a diagonal Gaussian posterior over merging weights give non-vacuous generalisation certificates from as few as 100 training examples, even when the source models are an 88M-parameter vision transformer or a 7B-parameter language model. It further shows that optimising the PAC-Bayes certificate itself as the training objective, and building a data-dependent prior from part of the support set, tightens the bounds and can turn vacuous guarantees into useful ones. The result matters because it opens a route to mathematical guarantees in low-data, high-stakes settings where held-out validation is unreliable.

What carries the argument

The central object is the re-parameterisation of a downstream learner as a distribution over merging coefficients. The paper sets a Gaussian prior $P=\mathcal{N}(\mu_P,\lambda_1 I)$ with uniform mean $\mu_P=1/M$ over $M$ source models and a diagonal Gaussian posterior $Q=\mathcal{N}(\mu_Q,\lambda_2 I)$, typically with both variances fixed at 0.05. The identity that carries the argument is the closed-form KL divergence $$\mathrm{KL}(Q\parallel P)=\frac{d}{2}\left(\frac{\lambda_2}{\lambda_1}-1-\ln\frac{\lambda_2}{\lambda_1}\right)+\frac{1}{2\lambda_1}\|\mu_Q-\mu_P\|^2 ,$$ which grows linearly in the number of merging parameters $d$ and quadratically in the distance between posterior and prior means, and is independent of the size of the source networks. The second mechanism is the paper's Theorem 2 bound, which numerically inverts the KL divergence between Bernoulli random variables to convert the support-set loss $\hat L(Q)$ into a certificate $C$ on the population risk $L(Q)$. For larger $d$, the paper uses a data-dependent prior built from half of the support set so that the posterior mean stays close to the prior mean. A finite-hypothesis-class variant replaces Gaussians with a uniform categorical distribution over discretised merging weights, giving deterministic certificates that are looser but avoid a randomised classifier.

What would settle it

Run the certification protocol on a diverse collection of low-shot tasks and check coverage on large held-out query sets: under the claimed confidence $\delta=0.05$, at most 5% of certified tasks should have measured test error above the certified bound; a significantly larger mis-coverage rate would falsify the fixed-variance diagonal-Gaussian certificates. A more targeted calculation is to take a task whose optimal merge weights are far from the prior mean and verify that the resulting KL term pushes the numerical certificate above 1, showing where the approach breaks.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that non-vacuous generalisation guarantees for contemporary large models in the low-shot regime do not require new theory or new algorithms; they are already achievable by reinterpreting model-merging learners in PAC-Bayes terms. The load-bearing observation is that a merger whose only downstream parameters are coefficients of a weighted sum over fixed source models has tiny dimension, so the KL divergence between a Gaussian posterior over those coefficients and a uniform prior stays small regardless of source-model size. Instantiating this with diagonal Gaussians of fixed variance 0.05, the authors demonstrate certificates: off-the-shelf Task Arithmetic, Ties-Merging, and task-wise AdaMerging give non-vacuous bounds on image tasks, while layer-wise AdaMerging becomes certifiable after replacing its objective with the PAC-Bayes bound. For a 7B-parameter language model adapted by task arithmetic from 100 examples per task, the same procedure gives non-vacuous bounds on the Big-Bench Hard tasks where merging succeeds. The paper presents this as the first demonstration of non-trivial generalisation guarantees for large models in a 100-example regime.

Load-bearing premise

The certificates assume the merging weights are well captured by a diagonal Gaussian posterior with variance fixed at 0.05 and a prior mean that is either uniform or built from half the support set; if the genuinely good merging weights sit far from that Gaussian, the KL term grows and the bound becomes vacuous.

Editorial extensions

If this is right

  • Off-the-shelf Task Arithmetic, Ties-Merging, and task-wise AdaMerging already produce non-vacuous PAC-Bayes certificates on five image-classification datasets with an 88M-parameter vision transformer, without changing their learning objectives.
  • Optimising the PAC-Bayes bound as the training objective converts the vacuous layer-wise AdaMerging certificate into a non-vacuous one, for example from a bound of 1.000 to 0.663 on DTD, with only a modest test-error penalty on most datasets.
  • Using a data-dependent prior built from half of the support set further tightens bounds, for example improving the layer-wise AdaMerging EuroSAT certificate from 1.000 to 0.483.
  • The same procedure certifies a 7B-parameter language model adapted by task arithmetic with 100 examples per task on the Big-Bench Hard tasks where merging succeeds, while zero-shot use of the base model frequently gives vacuous bounds.
  • With a few thousand training examples, the certified generalisation gap shrinks enough that test error is certified to be within about five percentage points of training error, and a discretised finite-hypothesis-class variant yields reasonable deterministic certificates as well.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same mechanism should transfer to any parameter-efficient learning interface whose parameter count is small and whose prior can be placed near a good posterior, such as a handful of adapter or prompt weights; the paper only demonstrates merging weights, but the KL-vs-dimension trade-off is generic.
  • Because the bound is computed on the same support set that trains the posterior, the method offers a competing workflow to held-out validation, and Figure 3's comparison suggests it uses scarce data more efficiently than splitting off a validation set; this could be leveraged in regulated settings that require published performance guarantees.
  • The certificates are relative to a fixed pool of source models; adding, removing, or replacing a source model changes the merge-weight space and the prior, so a newly merged model would need a fresh certificate rather than inheriting the old one.
  • A natural testable extension is to replace the fixed-variance diagonal Gaussian with a richer posterior family or to learn the variance; if that materially tightens the certificate, the reported bounds are conservative, whereas if it cannot be made to fit, the fixed Gaussian is the load-bearing idealisation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes that PAC-Bayes generalisation certificates can be obtained for model-merging-based few-shot learners. The central idea is that learning downstream tasks by fusing a small number of pretrained models involves only a low-dimensional merging parameter vector, so the KL term in standard PAC-Bayes bounds is small even when the base models are very large. The authors reinterpret off-the-shelf merging algorithms as producing the mean of a Gaussian posterior, and also propose directly optimising a PAC-Bayes objective. Experiments with CLIP-ViT-B/32 on image classification and Mistral-7B on BBH report non-vacuous bounds with 100 training examples. The paper also studies data-dependent priors and compares with a validation-set bound.

Significance. The conceptual observation is valuable: if learning is restricted to a low-dimensional merging space, the effective complexity of the learner is small and PAC-Bayes bounds can be tight even with very little data. The paper correctly uses standard PAC-Bayes machinery, including a proper data split for the data-dependent prior, and it is transparent in providing full results in appendices. If the certificate computation is made rigorous, the work would be a notable advance over prior non-vacuous certification results, which have required substantially more data. The current manuscript, however, contains a load-bearing technical gap in how the empirical risk entering the bound is computed, and the empirical claims are stronger than the full tables support.

major comments (3)
  1. [Appendix A.2 / Eq. (6), Theorem 2, Eq. (9)] The certificates in Tables 1–3 are not valid high-probability bounds as printed because the quantity \hat L(Q) plugged into Eq. (9) is not the exact expectation required by Theorem 2. Appendix A.2 states that train error is obtained by Monte Carlo approximation using 10 sampled weights from the posterior. A 10-sample average is an unbiased estimate of E_{θ∼Q}[\hat L(θ)], not the expectation itself, and it can lie below the true value. No concentration correction such as a Hoeffding or empirical-Bernstein upper confidence bound is applied before this estimate is used in Theorem 2 or Eq. (10). Consequently, the reported PB Bound and Upper Bound columns are functions of a random estimator whose optimism is uncontrolled. This is a correctness gap in the central claim and must be fixed, for example by replacing \hat L(Q) with a high-probability upper bound on the expected empirical risk, or by computing the expectation exactly.
  2. [Section 5.3 / Table 3 vs. Appendix C.2 / Table 5] The headline claim that Task Arithmetic yields non-vacuous low-shot certificates for Mistral-7B is based on a selected subset of BBH tasks. Appendix Table 5 shows that many BBH tasks have vacuous certificates (e.g., Web of Lies, Temporal Sequences, Dyck Languages) and several tasks where Task Arithmetic does not improve test error over the zero-shot baseline (e.g., Word Sorting, Date Understanding, Object Counting). The statement in Section 5.3 that Task Arithmetic 'is able to make substantial gains over zero-shot application' is therefore too strong. Although the selection is disclosed in Section 5.1 and full results appear in the appendix, the abstract and conclusion state the claim without this qualification. The authors should either present all tasks in the main text, report an aggregate success rate, or clearly frame the result as per-task certification rather than a general property of the merging learner.
  3. [Section 4.2 / Eq. (10)] The text says the proposed objective minimises the upper bound of L(Q) in Theorem 2, but Eq. (10) is the conventional square-root bound from Theorem 1, not the Langford–Seeger bound from Theorem 2. If the optimisation is actually performed with Eq. (10), then the 'Bound Optimization' experiments are not directly minimising the PB Bound reported in Tables 1–3. The authors should either correct the reference to Theorem 2 or explicitly state that Eq. (10) is used as a tractable surrogate for the Seeger bound.
minor comments (4)
  1. [Appendix B, first paragraph] The sentence 'If the prior is chosen with a large enough variance then the KL divergence with a relatively concentrated posterior does become too large' appears to have a missing negation; a larger prior variance should make the KL divergence smaller, not larger.
  2. [Table 3 / Section 5.3] The notation '(0.887(−0.113))' in the PB Bound column of Table 3 is never defined; the reader cannot tell whether the parenthesised value is the gap relative to the test error, the gap relative to the zero-shot bound, or something else.
  3. [Throughout] The spelling of AdaMerging is inconsistent: the paper uses both 'AdaMerging' and 'Adamerging'. Please standardise the spelling.
  4. [Section 3.1, 'The Test Set Bound'] The 'test set bound' is said to use a held-out validation dataset, which is confusing terminology because a test set bound that is computed on a validation split is not a bound on the independent test set. Please clarify the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No substantive circularity: the PAC-Bayes certificates are instantiations of external bounds over a low-dimensional merging parameter space; a motivational self-citation and a Monte-Carlo validity caveat do not make the derivation circular.

full rationale

The derivation is self-contained rather than circular. Section 4.1 selects a diagonal-Gaussian prior/posterior over fusion weights, Appendix B provides the closed-form KL divergence, and Theorems 1-2 are external PAC-Bayes bounds (Langford-Seeger/Seeger) instantiated with a measured training loss and the computed KL term. The low dimensionality of the merging parameter space explains why the certified gap can be small, but the bound values in Tables 1-3 are computed outcomes, not asserted targets; non-vacuity is an empirical finding. Equation (10) is a standard PAC-Bayes training objective, not a concealed restatement of the claimed certificate. Data-dependent priors preserve validity through explicit train/prior splitting (Section 5.2), so the improved certificates are not built in by construction. Test errors come from external benchmarks, so no fitted parameter is renamed as a prediction. The only author self-citation, [38], motivates why validation is unreliable in low-shot regimes; it is not load-bearing for the bound computation and therefore is not circularity. I flag one non-circular validity gap for the record: Appendix A.2 says train/test errors were 'computed using Monte-Carlo approximation using 10 sampled weights from posterior', while Theorem 2 and Eq. (6) require the exact expectation \hat L(Q); no concentration or confidence correction is applied before plugging the estimate into the bound, so the tables' 'PB Bound' values are not rigorous 1-delta guarantees as written. This correctness concern, and the fixed-variance Gaussian modelling choice, do not make the derivation circular.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim uses only standard PAC-Bayes bounds and a parametric assumption over merge weights; no new physical or mathematical entities are introduced.

free parameters (1)
  • Gaussian prior/posterior variance (lambda1, lambda2) = 0.05
    Set to 0.05 for all experiments in Section 5.1. This directly controls KL(Q||P) and thus the certified gap. The authors check robustness across 0.025-0.2 in Table 6, but it remains a hand-chosen hyperparameter.
assumptions (3)
  • standard math PAC-Bayes bounds (Theorems 1 and 2, Langford-Seeger and Seeger) hold for any posterior Q and prior P that do not depend on the certification data.
    The paper relies on these theorems to convert KL plus empirical risk into a population risk bound (Sections 3.1 and 4.1).
  • domain assumption Support set examples are drawn IID from the target task distribution.
    Required for PAC-Bayes bounds to apply; stated implicitly by the few-shot learning setup (Section 3).
  • ad hoc to paper The merging posterior is a diagonal Gaussian with fixed variance, and the prior is a Gaussian with mean 1/M.
    Section 4.1 posits N(mu_P, lambda1 I) and N(mu_Q, lambda2 I). The tightness of all reported certificates depends on this distributional choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning." pith.science (2026). https://pith.science/paper/GQFVTQL3

@misc{pith2026250515798,
  author       = {Pith},
  title        = {Pith review of: Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQFVTQL3}},
  note         = {Machine review of arXiv:2505.15798}
}
read the original abstract

Certifying the IID generalisation ability of deep networks is the first of many requirements for trusting AI in high-stakes applications from medicine to security. However, when instantiating generalisation bounds for deep networks it remains challenging to obtain non-vacuous guarantees, especially when applying contemporary large models on the small scale data prevalent in such high-stakes fields. In this paper, we draw a novel connection between a family of learning methods based on model fusion and generalisation certificates, and surprisingly show that with minor adjustment several existing learning strategies already provide non-trivial generalisation guarantees. Essentially, by focusing on data-driven learning of downstream tasks by fusion rather than fine-tuning, the certified generalisation gap becomes tiny and independent of the base network size, facilitating its certification. Our results show for the first time non-trivial generalisation guarantees for learning with as low as 100 examples, while using vision models such as VIT-B and language models such as mistral-7B. This observation is significant as it has immediate implications for facilitating the certification of existing systems as trustworthy, and opens up new directions for research at the intersection of practice and theory.

Figures

Figures reproduced from arXiv: 2505.15798 by the authors.

Figure 1
Figure 1. Impact of learning with bound-optimization and data-dependent prior. CLIP-ViT-B/32 on [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Comparison of mistral-7B vs. mistral-7B+Task Arithmetic [17]. CG Gap refers to the certified generalisation gap. (a) BBH benchmark: train error vs. certified generalisa￾tion gap. Grey zone indicates vacuous region. (b) TweetEval hate speech detection: performance across metrics. 40 60 80 100 Total Data Size 0.5 0.6 0.7 Certificate eurosat DDP Half Val 40 60 80 100 Total Data Size 0.85 0.90 0.95 1.00 gtsrb DDP Half V… view at source ↗
Figure 3
Figure 3. Change of Performance Certificates with different total data size on 5 image classifica [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Change of Actual/Certified Generalisation Gap by different total data size (100, 500, 1000, [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Change of Metrics (Train Accuracy, Test Accuracy, PAC-Bayes Bound, PAC-Bayes Upper [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Relative change in Test error, Validation error, and PAC-Bayes bound for discrete compared [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 28 canonical work pages

  1. [1]

    https://huggingface.co/Dolphin-2.1-Mistral-7B

    Dolphin-2.1-mistral-7b. https://huggingface.co/Dolphin-2.1-Mistral-7B . Ac- cessed: 2025-01-30

    work page 2025

  2. [2]

    https://huggingface.co/MetaMath-Mistral-7B

    Metamath-mistral-7b. https://huggingface.co/MetaMath-Mistral-7B . Accessed: 2025-01-30

    work page 2025

  3. [3]

    https://huggingface.co/ Speechless-Code-Mistral-7B-v1.0

    Speechless-code-mistral-7b-v1.0. https://huggingface.co/ Speechless-Code-Mistral-7B-v1.0 . Accessed: 2025-01-30

    work page 2025

  4. [4]

    Evolutionary optimization of model merging recipes

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, pages 1–10, 2025

    work page 2025

  5. [5]

    User-friendly introduction to pac-bayes bounds

    Pierre Alquier. User-friendly introduction to pac-bayes bounds. arXiv preprint arXiv:2110.11216, 2021

    arXiv 2021

  6. [6]

    SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter

    Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Fran- cisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In Proceed- ings of the 13th International Workshop on Semantic Evaluation, pages 54–63, Minneapolis, Min...

    work page 2019

  7. [7]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, 2017

    work page 2017

  8. [8]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014

    2014

Show all 46 references
  1. [9]

    Model breadcrumbs: Scaling multi-task model merging with sparse masks

    MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. ECCV, 2024

    2024

  2. [10]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  3. [11]

    Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data

    Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. UAI, 2017

    2017

  4. [12]

    Andrew Y . K. Foong, Wessel Bruinsma, David R. Burt, and Richard E. Turner. How Tight Can PAC-Bayes be in the Small Data Regime? In Advances in Neural Information Processing Systems, 2021

    2021

  5. [13]

    Arcee’s mergekit: A toolkit for merging large language models

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee’s mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257, 2024

    2024 arXiv

  6. [14]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019

    2019

  7. [15]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  8. [16]

    Lo- rahub: Efficient cross-task generalization via dynamic lora composition

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269, 2023

    2023 arXiv

  9. [17]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023. 10

    2023

  10. [18]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  11. [19]

    Fantastic generalization measures and where to find them

    Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2020

    2020

  12. [20]

    3d object representations for fine- grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013

    2013

  13. [21]

    Tutorial on practical prediction theory for classification

    John Langford. Tutorial on practical prediction theory for classification. Journal of machine learning research, 6(3), 2005

    2005

  14. [22]

    Bounds for averaging classifiers

    John Langford and Matthias Seeger. Bounds for averaging classifiers. School of Computer Science, Carnegie Mellon University, 2001

    2001

  15. [23]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998

    1998

  16. [24]

    Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks

    Joshua Lee, Prasanna Sattigeri, and Gregory Wornell. Learning new tricks from old dogs: Multi-source transfer learning from pre-trained networks. Advances in neural information processing systems, 32, 2019

    2019

  17. [25]

    Deep model fusion: A survey

    Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023

    2023 arXiv

  18. [26]

    Sanae Lotfi, Marc Anton Finzi, Yilun Kuang, Tim G. J. Rudner, Micah Goldblum, and An- drew Gordon Wilson. Non-vacuous generalization bounds for large language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  19. [27]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    2022

  20. [28]

    Trustworthy machine learning

    Bálint Mucsányi, Michael Kirchhof, Elisa Nguyen, Alexander Rubinstein, and Seong Joon Oh. Trustworthy machine learning. arXiv preprint arXiv:2310.08215, 2023

    2023 arXiv

  21. [29]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, page 4. Granada, 2011

    2011

  22. [30]

    Exploring generalization in deep learning

    Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. Advances in neural information processing systems, 30, 2017

    2017

  23. [31]

    A survey on transfer learning

    Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359, 2009

    2009

  24. [32]

    Tighter risk certificates for neural networks

    Maria Perez-Ortiz, Omar Rivasplata, John Shawe-Taylor, and Csaba Szepesvári. Tighter risk certificates for neural networks. Journal of Machine Learning Research, 22(227):1–40, 2021

    2021

  25. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  26. [34]

    Rapin and O

    J. Rapin and O. Teytaud. Nevergrad - A gradient-free optimization platform.https://GitHub. com/FacebookResearch/Nevergrad, 2018

    2018

  27. [35]

    Pac-bayesian meta- learning: From theory to practice

    Jonas Rothfuss, Martin Josifoski, Vincent Fortuin, and Andreas Krause. Pac-bayesian meta- learning: From theory to practice. arXiv preprint arXiv:2211.07206, 2022

    2022 arXiv

  28. [36]

    Pac-bayesian generalisation error bounds for gaussian process classification

    Matthias Seeger. Pac-bayesian generalisation error bounds for gaussian process classification. Journal of machine learning research, 3(Oct):233–269, 2002. 11

    2002

  29. [37]

    Understanding machine learning: From theory to algorithms

    Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  30. [38]

    Evaluating the evaluators: Are validation methods for few-shot learning fit for purpose? Transactions on Machine Learning Research, 2024

    Luísa Shimabucoro, Ruchika Chavhan, Timothy Hospedales, and Henry Gouk. Evaluating the evaluators: Are validation methods for few-shot learning fit for purpose? Transactions on Machine Learning Research, 2024

    2024

  31. [39]

    A com- prehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities

    Yisheng Song, Ting Wang, Puyu Cai, Subrota K Mondal, and Jyoti Prakash Sahoo. A com- prehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Comput. Surv., 2023. Just Accepted

    2023

  32. [40]

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32: 323–332, 2012

    2012

  33. [41]

    Challenging big- bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  34. [42]

    Fusionbench: A compre- hensive benchmark of deep model fusion

    Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A compre- hensive benchmark of deep model fusion. arXiv preprint arXiv:2406.03280, 2024

    2024

  35. [43]

    Sun database: Large-scale scene recognition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485–3492. IEEE, 2010

    2010

  36. [44]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In NeurIPS, 2024

    2024

  37. [45]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024

    2024

  38. [46]

    + Bound” indicate that we use the PAC-Bayes bound as the optimisation objective. We also provide an additional “Upper Bound

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2017. 12 A Additional Experimental Setup A.1 Experimental Details For Task Arithmetic [17] and Ties-Merging [44], they use only...

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.