Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Why Do More Experts Fail? A Theoretical Analysis of Model Merging

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves a finite upper bound on how many expert models can be usefully merged, explains the saturation through Gaussian Width and Approximate Kinematics arguments, and offers a heavy-tailed reparameterization that extends the…

desk verdict The empirical saturation story and the RHT trick are worth a look, but the theory as written does not support the headline claims. read the letter →

arxiv 2505.21226 v2 pith:EOB54FRI submitted 2025-05-27 cs.LG

Zijing Wang , Xingle Xu , Yongkang Liu , Yiqun Zhang , Peiqin Lin , Shi Feng , Xiaocui Yang , Daling Wang
show 1 more author
Hinrich Schütze
This is my paper · ORCID
classification cs.LG MSC 68T0760D05
keywords modelmergingGaussianWidthdiminishingreturnssaturationLoRAexpertsheavy-taileddistributionparameterredundancymulti-tasklearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to prove why model merging — combining several task-specific expert models into one multi-task model — stops paying off once too many experts are added. Its central claim is that the effective parameter space of a merged model saturates: the variance of the merged weights converges to a nonzero floor set by expert correlation, the Gaussian Width of the merged subspace grows only like the square root of a cumulative eigenvalue sum, and Approximate Kinematics places a redundancy threshold beyond which the loss cannot remain in its sublevel set. If the theory is right, practitioners gain a principled stopping rule for how many experts to merge and an explanation for why even the best current methods level off after roughly four to six experts. The paper also offers a remedy, the Reparameterized Heavy-Tailed (RHT) method, which reshapes the parameter distribution through a heavy-tailed transformation and, the authors prove, strictly enlarges the functional coverage of the merged model. Experiments on 12 benchmarks with GENOME and Model Swarms on two base models support the saturation and the RHT improvement.

What carries the argument

The load-bearing object is the Gaussian Width of the merged model's sublevel set: for the quadratic sublevel set $S(\epsilon) = \{\theta : (\theta-\theta^*)^\top H (\theta-\theta^*) \le 2\epsilon\}$ it evaluates to $w(S(\epsilon)) \approx \sqrt{2\epsilon\,\mathrm{Tr}(H^{-1})}$, and the concavity of the square root converts a linear growth of the inverse-eigenvalue sum into strictly diminishing marginal returns. The second object is Approximate Kinematics Theory, whose phase transition — a subspace and a convex cone intersect with near-certainty once their statistical dimensions sum to roughly the ambient dimension — yields the parameter-redundancy threshold of Theorem 3. The third is the RHT transformation itself, a two-step map (subtract a Gaussian, then amplify component-wise with a sub-linear power law), which turns Gaussian parameters into heavy-tailed ones and, through the Jacobian-weighted coverage integral, enlarges the function space the merged model can express.

What would settle it

On a fixed base model, merge 2, 4, 6, 8, and 10 LoRA experts on the six general-purpose tasks and measure two quantities: the trace of the merged-weight covariance, which Theorem 1 predicts follows $\sigma^2(\rho + (1-\rho)/n)$ and saturates at $\sigma^2\rho$, and the per-expert marginal performance gain, which Theorem 2 predicts decreases monotonically along the curve $\sqrt{2\epsilon \sum_{i=1}^M 1/\lambda_i}$ with exactly one new inverse-eigenvalue term per expert. If the covariance trace does not flatten, or if marginal gains stay flat or increase past the claimed saturation count, the ceiling mechanism is falsified. The cheapest version: merge 20+ LoRA experts on one benchmark battery and check whether marginal gains strictly decrease at the predicted square-root rate.

Watch

Extended reading notes

Core claim

The paper's central claim is that merged-model performance is bounded by the geometry of the parameter space, not by the quality of the merging algorithm. Theorem 1 states that if expert updates are approximately Gaussian with equal variance $\sigma^2$ and pairwise correlation $\rho$, the merged variance is $\sigma^2_{\mathrm{merge}} = \sigma^2(\rho + (1-\rho)\sum_i \alpha_i^2)$, which converges to the floor $\sigma^2\rho$ as the expert count grows; requiring each expert to reduce variance by at least $\Delta$ caps the useful number of experts at $n \le \sigma^2(1-\rho)/\Delta$. Theorem 2 states that the Gaussian Width of the merged sublevel set is $w(S_M) \approx \sqrt{2\epsilon \sum_{i=1}^M 1/\lambda_i}$, so the marginal gain $\Delta w_M = w(S_M) - w(S_{M-1})$ strictly decreases by concavity of the square root. Theorem 3 uses Approximate Kinematics to show that once the number of non-zero merged parameters $k$ exceeds $D - \sum_{i=1}^{D-k} r_i^2/(\|\theta^* - \theta_k\|_2^2 + r_i^2)$, the loss cannot be kept within the sublevel set and degradation becomes inevitable. Theorems 4 and 5 establish that subtracting a Gaussian copy from the weights and applying the component-wise map $T(w'_i) = \operatorname{sign}(w'_i)|w'_i|^\gamma(1+\alpha e^{-\beta|w'_i|})$ yields a heavy-tailed distribution with tail exponent $\kappa = 1/\gamma$, and Theorem 6 proves that heavy-tailed parameters strictly increase the coverage $C(\mathcal{F})$ of the function space, which is the stated mechanism behind RHT's improved merging.

Load-bearing premise

The load-bearing premise is that the $M$-th added expert contributes exactly one inverse-Hessian eigenvalue $1/\lambda_M$ to the Gaussian Width sum of Theorem 2; the paper asserts this identification without deriving it, and if a new expert can add several eigenvalue terms at once, or add them in a different order, the strictly concave diminishing-returns conclusion does not follow even for a perfectly quadratic loss.

Editorial extensions

If this is right

  • Merging more than the saturation count of LoRA experts yields no further performance gain, and beyond the Theorem-3 redundancy threshold performance degrades; current methods saturate between roughly four and six experts.
  • The adaptive termination condition $\Delta = \mathbb{E}[\rho^2_{i-1} - \rho^2_i]$ provides a computable stopping rule: keep merging only while the next expert's variance reduction clears the threshold.
  • Because the bound $n \le \sigma^2(1-\rho)/\Delta$ tightens with expert correlation $\rho$, lowering $\rho$ — for instance by preferring near-orthogonal domain subspaces, as the pairwise fusion experiments show — raises the number of models that can be merged.
  • RHT shifts the saturation point: the heavy-tailed reparameterization strictly increases coverage, so the same merging algorithm keeps improving past where the vanilla method plateaus, as the MMLU, MATH, and MGSM trend lines show.
  • The number of principal components explaining roughly 95% of merged weight variance predicts where performance peaks, giving a cheap diagnostic for when to stop adding experts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The theorem structure suggests a sharper test than the paper runs: numerically estimate the Hessian eigenvalues of the merged loss and verify that each added expert contributes exactly one term $1/\lambda_i$ to the Gaussian Width sum — that indexing, rather than square-root concavity alone, is what carries Theorem 2.
  • If the correlation-driven bound is the true bottleneck, then training LoRA experts with explicit orthogonality regularization should measurably raise the saturation count; the paper's domain-similarity analysis is a partial check of this, but not a direct causal experiment.
  • The RHT idea is not specific to merging: the same sub-linear amplification could extend the coverage of any low-rank parameter distribution, such as few-shot LoRA ensembles or adapter stacks, which is a testable transfer the paper does not claim.
  • The local quadratic approximation of the loss means the Gaussian Width estimate is only valid near the optimum; tracking the empirical Hessian along the merging trajectory would show whether the square-root law survives outside that regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper develops a theoretical account of why model merging saturates when many LoRA experts are combined. It claims four main results: Theorem 1 gives an upper bound on the number of mergeable experts from variance saturation; Theorem 2 uses Gaussian Width to claim strictly concave diminishing marginal returns; Theorem 3 derives a parameter-redundancy threshold beyond which performance must degrade; and Theorems 4-6 propose a Reparameterized Heavy-Tailed method (RHT) that provably increases coverage of the merged model's function space. Experiments with GENOME and Model Swarms on general-purpose and knowledge-intensive benchmarks show saturation around 4-6 experts and improved scaling with RHT on some tasks.

Significance. If the theorems were correct, the paper would supply a principled stopping rule for model merging and a geometric explanation of a widely observed saturation phenomenon, and RHT would be a useful practical recipe. The paper deserves credit for a broad experimental matrix (two base models, two merging algorithms, and 12 benchmarks), for releasing a code repository, and for attempting to connect high-dimensional geometry to LLM merging. However, the central theoretical chain is not established: Theorem 2 rests on an unproven spectral identification, Theorem 5's tail analysis is internally inconsistent, and Theorems 1 and 3 equate variance or geometric coverage with task performance without a supporting argument. As it stands, the paper's headline contributions are not supported by the proofs.

major comments (4)
  1. [C.2, Eq. (41)] In the proof of Theorem 2, Eq. (40) gives w(S(ε)) ≈ sqrt(2ε Tr(H^{-1})) for a D-dimensional ellipsoid. Eq. (41) then replaces Tr(H^{-1}) by Σ_{i=1}^M 1/λ_i. This replacement is not derived: no argument shows that merging M LoRA experts makes exactly M eigenvalues of the Hessian relevant, nor that these are the first M eigenvalues, nor even that the eigenvalues are ordered. With rank-r LoRA experts, the span of the merged updates has dimension at most M r, so the sum over M terms is not justified by the geometry. Moreover, Eq. (7) requires the added terms 1/λ_i to be non-increasing; concavity of sqrt alone only gives ∆w_M > ∆w_{M+1} if the increments are ordered. Without this, the strictly concave diminishing-returns claim, the paper's central theoretical result, is unproven, and the same unproven M-dependence propagates into the saturation discussion in Section 3.1.
  2. [C.5, Eqs. (10)-(11)] Theorem 5's conclusion that the transformation induces heavy tails is contradicted by its own density. Eq. (10) has the large-|y| form |y|^{1/γ-1} exp(- |y|^{2/γ} / (2(σ²+σ_g²))). Because 0<γ<1, the exponent 2/γ is greater than 2, so the tail decays faster than a Gaussian, not slower, and the text's claim that the tail decays more slowly than a Gaussian is false. Eq. (11) then asserts a power-law tail P(|W|>|y|) ~ |y|^{-κ}, which is inconsistent with the stretched-exponential tail in Eq. (10). Consequently, the heavy-tailed premise of Theorem 6 and the RHT method's coverage argument are unsupported, and the empirical RHT gains in Figure 6 cannot be attributed to the stated mechanism.
  3. [C.3, Eq. (50)] Theorem 3's proof does not establish Eq. (50). The text moves from Eq. (48)'s Gaussian width of an ellipsoid to a projected Gaussian width sum involving r_i²/(||θ* - θ_k||² + r_i²) by invoking an unexplained modification from [12]. No derivation is given for why the statistical dimension of the projection cone equals this sum, and the threshold in Eq. (8) is then asserted to mark inevitable performance degradation. Even if the geometric statement were true, it concerns the existence of points in a sublevel set, not the benchmark accuracy of the merged model; the leap from '0 ∈ S(θ', ε)' to 'performance degradation becomes inevitable' is unsupported.
  4. [2, Eq. (4)] Theorem 1 derives an upper bound on the number of merged experts from the saturation of the variance of the merged parameter vector (Eqs. (27)-(31)). But the variance of the parameter vector is not shown to determine task performance. The quantity σ²_merge can saturate while the model's loss or accuracy continues to change, so Eq. (4) is a bound on variance reduction, not a bound on 'model merging' in the sense used in the abstract. In addition, the bound n ≤ σ²(1-ρ)/Δ contains the user-chosen threshold Δ and the fitted correlation ρ; without a concrete rule for setting Δ from the optimization objective, it does not predict the observed saturation point (e.g., 4 for Swarms and 6 for GENOME in Section 3.1).
minor comments (4)
  1. [2 and Appendix C] The ambient dimension is denoted d in Section 2 and D in the proofs, while D is also used for the datasets Dknowd and Dgend; the notation should be unified.
  2. [2.3, Eq. (12)] The coverage C(F) = ∫_X |{f(x): f∈F}| dx is not well-defined: the inner set is a subset of the output space, and the measure with respect to which its size is taken is unspecified.
  3. [C.5, Eq. (55)] Theorem 5 assumes T is invertible and applies the change-of-variables formula, but the component-wise map sign(x)|x|^γ(1+αe^{-β|x|}) is not globally invertible with a differentiable inverse at 0; the proof needs to address the non-smooth points.
  4. [C.2, Eq. (39)] The step from Eq. (38) to Eq. (40) says Jensen's inequality gives an approximation, but Jensen actually yields E[||H^{-1/2}g||] ≤ sqrt(E||H^{-1/2}g||²) = sqrt(Tr(H^{-1})); the direction of the inequality should be stated correctly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the derivation does not reduce to its own inputs; main gaps are mathematical, not definitional.

full rationale

The paper's central derivation is not circular by the standards of the requested analysis. Theorem 1 (Section 2, Eqs. 1-4) follows algebraically from the stated assumptions that LoRA increments are Gaussian with equal variance and equal pairwise correlation; the resulting bound n ≤ σ²(1−ρ)/Δ depends on the user-chosen Δ and is not fitted to the observed saturation point. No fitted value is renamed as a prediction. Theorem 2 (Section 2.1 and Appendix C.2) uses the standard Gaussian-width computation for an ellipsoidal sublevel set. The transition from Eq. 40 to Eq. 41, where the full Hessian trace is replaced by a sum over M experts, is mathematically under-supported, and the claim that concavity alone implies Δw_M > Δw_{M+1} requires an eigenvalue ordering that is not established. These are correctness gaps, not circular reductions: the formula is not defined into existence, and the conclusion is not statistically forced by the inputs. Theorem 3 (Appendix C.3) invokes external lemmas from Amelunxen et al. [2] and Larsen et al. [12]; no load-bearing self-citation chain appears. Theorems 4-6 (Section 2.3) are qualitative and do not derive their conclusion by importing the target claim. Self-citations ([16], [36]) appear in related work and in the selection of baseline merging methods, but they do not carry the theoretical argument; the empirical validation is in-sample and qualitative, which is a weakness but not a circular one. The Limitations section explicitly acknowledges the LoRA and homogeneous-architecture assumptions, further indicating that the claims are conditional rather than definitional.

Assumptions & free parameters 7 free parameters · 6 assumptions · 1 invented entities

The central claim rests on several assumptions borrowed from the paper's own experiments: Gaussian LoRA increments with equal variance and correlation, a direct link between variance and performance, an unproven eigenvalue-counting identification in Theorem 2, an unverified application of Approximate Kinematics to a non-cone set, and a heavy-tailed claim contradicted by Eq 10. These are the main ledger entries.

free parameters (7)
  • σ² (variance of LoRA increments) = estimated from histograms, numeric value not given
    Assumed equal across all experts in Theorem 1; measured in Section 3.3 from the same LoRA setup used for validation.
  • ρ (pairwise expert correlation) = not reported
    Assumed equal for all pairs in Eq 2; the bound n ≤ σ²(1-ρ)/Δ depends on it, but no value is estimated or used for prediction.
  • Δ (minimum variance reduction) = not specified
    User-chosen threshold in Eq 4 that sets the upper bound; no rule for choosing it is given.
  • ε (performance tolerance) = not specified
    Appears in the Gaussian width expressions, Eq 5 and Eq 40; arbitrary tolerance parameter.
  • γ (RHT exponent) = not specified, only bounded 0<γ<1
    Controls the claimed tail behavior in Theorem 5; no selection criterion or value is reported.
  • α, β (RHT nonlinearity parameters) = not specified
    Appear in Eq 9; no values or tuning details are given.
  • σ_g² (RHT noise variance) = not specified
    Appears in RHT step 1 and Theorem 4; no value is reported.
assumptions (6)
  • domain assumption LoRA expert increments θ_i are i.i.d. Gaussian N(0, σ²I) with equal pairwise correlation ρ.
    Invoked in Theorem 1 (Eq 1-3); justified by the authors' own histograms in Section 3.3, not by an independent external benchmark.
  • ad hoc to paper Merged model performance is determined by the variance of the merged parameter vector.
    Theorem 1 equates variance saturation with performance saturation without connecting σ²_merge to any loss or accuracy measure.
  • ad hoc to paper Each added expert contributes one inverse-Hessian eigenvalue term to the Gaussian width sum in Eq 5.
    The proof in Appendix C.2 never derives the M-dependence; this identification is the entire basis for the diminishing-returns claim.
  • ad hoc to paper The sublevel set S(θ',ε) and its projection cone satisfy the conditions of Approximate Kinematics Lemma 1.
    Appendix C.3 applies Lemma 1 to a sublevel set that is not shown to be a convex cone; the statistical dimension relation is asserted rather than verified.
  • ad hoc to paper The nonlinear transformation T with 0<γ<1 produces heavy tails with tail index κ=1/γ.
    This is false: Eq 10 contains exp(-|y|^{2/γ}) with 2/γ>2, which gives faster-than-Gaussian decay, not a power-law tail.
  • standard math Standard probability facts: linear combinations of Gaussians are Gaussian, the change-of-variables formula, and concavity of sqrt.
    Used in Theorems 1, 2, 4, and 5; these facts are correct but insufficient to support the paper's conclusions.
invented entities (1)
  • Function-space coverage C(F)
    purpose: Defined in Eq 12 to prove that heavy-tailed parameter distributions expand the merged model's function space.
    The definition integrates |{f(x): f∈F}| over x, but the inner cardinality is undefined for a continuum of functions, and no independent measurement of coverage is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Why Do More Experts Fail? A Theoretical Analysis of Model Merging." pith.science (2026). https://pith.science/paper/EOB54FRI

@misc{pith2026250521226,
  author       = {Pith},
  title        = {Pith review of: Why Do More Experts Fail? A Theoretical Analysis of Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EOB54FRI}},
  note         = {Machine review of arXiv:2505.21226}
}
read the original abstract

Model merging dramatically reduces storage and computational resources by combining multiple expert models into a single multi-task model. Although recent model merging methods have shown promising results, they struggle to maintain performance gains as the number of merged models increases. In this paper, we investigate the key obstacles that limit the scalability of model merging when integrating a large number of expert models. First, we prove that there is an upper bound on model merging. Further theoretical analysis reveals that the limited effective parameter space imposes a strict constraint on the number of models that can be successfully merged. Gaussian Width shows that the marginal benefit of merging additional models diminishes according to a strictly concave function. This implies that the effective parameter space becomes rapidly saturated as the number of merged models increases. Furthermore, using Approximate Kinematics Theory, we prove the existence of a unique optimal threshold beyond which adding more models does not yield significant performance improvements. At the same time, we introduce a straightforward Reparameterized Heavy-Tailed method (RHT) to extend the coverage of the merged model, thereby enhancing its performance. Empirical results on 12 benchmarks, including both knowledge-intensive and general-purpose tasks, validate our theoretical analysis. We believe that these results spark further research beyond the current scope of model merging. The source code is in the Github repository: https://github.com/wzj1718/ModelMergingAnalysis.

Figures

Figures reproduced from arXiv: 2505.21226 by the authors.

Figure 1
Figure 1. Cumulative variance compari￾son across different expert models. To further verify this phenomenon, we perform principal component analysis (PCA) [29] on the weights of various expert models used in the experiments (see [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Heatmap of Cosine Similarities Between Sentence Embeddings of Dknowd and Dgend. phy chem bio fin med phy chem bio fin med 0.00 0.81 0.80 0.72 0.71 0.81 0.00 0.79 0.70 0.69 0.80 0.79 0.00 0.70 0.68 0.72 0.70 0.70 0.00 0.41 0.71 0.69 0.68 0.41 0.00 Subspace Angle Similarity Matrix 1.0 0.5 0.0 0.5 1.0 1.5 1.0 0.5 0.0 0.5 1.0 v =1.61 v =1.32 v =1.23 LoRA Direction Comparison (PCA Projection) physics biology chemistry ph… view at source ↗
Figure 4
Figure 4. KL divergence distribution and perplexity comparison. (Left) The KL divergence distri [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Weight distributions of the base model vs. LoRA fine-tuning on different domains. The [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Model merging trends with RHT enhancement. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Incremental parameter distribution in LoRA. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Histogram of expert model weight distributions under model merging. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sharpness-aware Model Merging with Salience Recovery for LLM-based Cross-Domain Sequential Recommendation

    cs.IR 2026-07 reject novelty 4.0 of 10

    SharpRec combines sharpness-aware fine-tuning with a nonlinear parameter reshape to merge LoRA adapters for cross-domain recommendation, but the reshape's claimed heavy-tail effect is mathematically backward.

Reference graph

Works this paper leans on

38 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [12]

    How many degrees of freedom do we need to train deep networks: a loss landscape perspective

    Brett W Larsen, Stanislav Fort, Nic Becker, and Surya Ganguli. How many degrees of freedom do we need to train deep networks: a loss landscape perspective.arXiv preprint arXiv:2107.05802, 2021

    work page Pith review arXiv 2021

  2. [1]

    Evolutionary optimization of model merging recipes.Nature Machine Intelligence, pages 1–10, 2025

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes.Nature Machine Intelligence, pages 1–10, 2025

    2025

  3. [2]

    Living on the edge: Phase transitions in convex programs with random data.Information and Inference: A Journal of the IMA, 3(3):224–294, 2014

    Dennis Amelunxen, Martin Lotz, Michael B McCoy, and Joel A Tropp. Living on the edge: Phase transitions in convex programs with random data.Information and Inference: A Journal of the IMA, 3(3):224–294, 2014

    work page 2014

  4. [3]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  5. [4]

    Revisiting weight averaging for model merging.arXiv preprint arXiv:2412.12153, 2024

    Jiho Choi, Donggyun Kim, Chanhyuk Lee, and Seunghoon Hong. Revisiting weight averaging for model merging.arXiv preprint arXiv:2412.12153, 2024

    arXiv 2024

  6. [5]

    Model swarms: Col- laborative search to adapt llm experts via swarm intelligence.arXiv preprint arXiv:2410.11163, 2024

    Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, et al. Model swarms: Col- laborative search to adapt llm experts via swarm intelligence.arXiv preprint arXiv:2410.11163, 2024

    arXiv

  7. [6]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    arXiv 2009

  8. [7]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    arXiv 2021

Show all 38 references
  1. [8]

    Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    2022

  2. [9]

    Lo- rahub: Efficient cross-task generalization via dynamic lora composition.arXiv preprint arXiv:2307.13269, 2023

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition.arXiv preprint arXiv:2307.13269, 2023

    2023 arXiv

  3. [10]

    Camels in a changing climate: Enhancing lm adaptation with tulu 2.arXiv preprint arXiv:2311.10702, 2023

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2.arXiv preprint arXiv:2311.10702, 2023

    2023 arXiv

  4. [11]

    The singular value decomposition: Its computation and some applications.IEEE Transactions on automatic control, 25(2):164–176, 1980

    Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some applications.IEEE Transactions on automatic control, 25(2):164–176, 1980

    1980

  5. [13]

    The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning.arXiv preprint arXiv:2104.08691, 2021

    2021 arXiv

  6. [14]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  7. [15]

    Gpt understands, too.AI Open, 5:208–215, 2024

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too.AI Open, 5:208–215, 2024

    2024

  8. [16]

    Hift: A hierarchical full parameter fine-tuning strategy.arXiv preprint arXiv:2401.15207, 2024

    Yongkang Liu, Yiqun Zhang, Qian Li, Tong Liu, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Schütze. Hift: A hierarchical full parameter fine-tuning strategy.arXiv preprint arXiv:2401.15207, 2024

    2024 arXiv

  9. [17]

    Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models.arXiv preprint arXiv:2402.12851, 2024

    Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models.arXiv preprint arXiv:2402.12851, 2024. 10

    2024 arXiv

  10. [18]

    Pack of llms: Model fusion at test-time via perplexity optimization.arXiv preprint arXiv:2404.11531, 2024

    Costas Mavromatis, Petros Karypis, and George Karypis. Pack of llms: Model fusion at test-time via perplexity optimization.arXiv preprint arXiv:2404.11531, 2024

    2024 arXiv

  11. [19]

    Orthogonal adaptation for modular customization of diffusion models

    Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein. Orthogonal adaptation for modular customization of diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7964–7973, 2024

    2024

  12. [20]

    Ensemble learning.Ensemble machine learning: Methods and applications, pages 1–34, 2012

    Robi Polikar. Ensemble learning.Ensemble machine learning: Methods and applications, pages 1–34, 2012

    2012

  13. [21]

    Acceleration of stochastic approximation by averaging

    Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4):838–855, 1992

    1992

  14. [22]

    Lora soups: Merging loras for practical skill composition tasks.arXiv preprint arXiv:2410.13025, 2024

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks.arXiv preprint arXiv:2410.13025, 2024

    2024 arXiv

  15. [23]

    Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing Systems, 36:71095–71134, 2023

    Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing System...

    2023

  16. [24]

    Language models are multilingual chain-of-thought reasoners.arXiv preprint arXiv:2210.03057, 2022

    Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush V osoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multilingual chain-of-thought reasoners.arXiv preprint arXiv:2210.03057, 2022

    arXiv

  17. [25]

    Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  18. [26]

    Unlocking the potential of model merging for low-resource languages

    Mingxu Tao, Chen Zhang, Quzhe Huang, Tianyao Ma, Songfang Huang, Dongyan Zhao, and Yansong Feng. Unlocking the potential of model merging for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8705–8720, 2024

    2024

  19. [27]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  20. [28]

    Estimation in high dimensions: a geometric perspective

    Roman Vershynin. Estimation in high dimensions: a geometric perspective. InSampling Theory, a Renaissance: Compressive Sensing and Other Developments, pages 3–66. Springer, 2015

    2015

  21. [29]

    Principal component analysis.Chemometrics and intelligent laboratory systems, 2(1-3):37–52, 1987

    Svante Wold, Kim Esbensen, and Paul Geladi. Principal component analysis.Chemometrics and intelligent laboratory systems, 2(1-3):37–52, 1987

    1987

  22. [30]

    Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36:7093–7115, 2023

    2023

  23. [31]

    What matters for model merging at scale?arXiv preprint arXiv:2410.03617, 2024

    Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale?arXiv preprint arXiv:2410.03617, 2024

    2024 arXiv

  24. [32]

    Adamerging: Adaptive model merging for multi-task learning.arXiv preprint arXiv:2310.02575, 2023

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning.arXiv preprint arXiv:2310.02575, 2023

    2023 arXiv

  25. [33]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InForty-first International Conference on Machine Learning, 2024

    2024

  26. [34]

    Emotion detection on tv show transcripts with sequence- based convolutional neural networks

    Sayyed M Zahiri and Jinho D Choi. Emotion detection on tv show transcripts with sequence- based convolutional neural networks. InAAAI Workshops, volume 18, pages 44–52, 2018. 11

    2018

  27. [35]

    Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    2023

  28. [36]

    Nature-inspired population-based evolution of large language models.arXiv preprint arXiv:2503.01155, 2025

    Yiqun Zhang, Peng Ye, Xiaocui Yang, Shi Feng, Shufei Zhang, Lei Bai, Wanli Ouyang, and Shuyue Hu. Nature-inspired population-based evolution of large language models.arXiv preprint arXiv:2503.01155, 2025

    2025 arXiv

  29. [37]

    Llamafactory: Unified efficient fine-tuning of 100+ language models.arXiv preprint arXiv:2403.13372, 2024

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models.arXiv preprint arXiv:2403.13372, 2024

    2024 arXiv

  30. [38]

    particle

    Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE Transactions on Image Processing, 30:8008–8018, 2021. 12 A Experimental Setup Our experiments are strictly performed on high-performance computing hardware, NVIDIA-A800- SXM4-80GB, to e...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.