REVIEW 4 major objections 5 minor 1 cited by
MERGE$^3$: Efficient Evolutionary Merging on Consumer-grade GPUs
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read MERGE3 claims a 50-fold cut in evolutionary merging cost by predicting full-dataset accuracy from a small random sample via Item Response Theory, bringing quality multilingual merges to one consumer GPU.
desk verdict Useful empirical estimator for fast evolutionary merging, but the theory has a real proof error and the 'preserving performance' claim overstates Table 7. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Assumption 1, linear inheritance of latent abilities: if a merged model is formed by a linear combination of endpoint parameters, its latent ability vector is $\tilde{\gamma} = \sum_i \lambda_i \gamma_i$ for the same interpolation coefficients. Substituting this identity into the IRT response model turns the probability of a correct answer into a logistic function of a weighted sum of the pre-estimated endpoint abilities, so that during evolution the only parameters fitted per candidate are the scalar coefficients $\lambda_i$, estimated by maximum likelihood on the small subset rather than by re-running the whole model over the full dataset. The MP-IRT estimator (Equation 4) blends observed correctness on the evaluated subset with IRT-predicted correctness on the remaining items, weighted by subset size; GMP-IRT then interpolates MP-IRT with the raw subset accuracy. The theoretical guarantee that subset-optimal solutions remain within $\epsilon$ of full-dataset optima, and the empirical finding that abilities estimated under the assumption are closer to ground-truth ability vectors than general-purpose IRT estimates, both trace back to this identity.
What would settle it
Take two endpoints, form merged models across a grid of interpolation coefficients, and compare MP-IRT's predicted per-item correctness on the unevaluated portion of the dataset with the measured correctness of each actual merged model. If Assumption 1 is doing the work, prediction errors should be small and roughly uniform; if it is failing, errors should concentrate on exactly the items where the endpoints disagree — one answers correctly and the other does not — and grow as that disagreement sharpens. A direct version of this test is to run the MERGE3 pipeline on a pair of models whose merge is known to degrade both tasks, where the paper itself expects the assumption to bend, and check whether MP-IRT systematically overpredicts gains.
Extended reading notes
Core claim
The central claim is that two new estimators, MP-IRT and GMP-IRT, predict a merged model's full-dataset accuracy from its responses on a tiny random subset (as small as roughly 1.5% of the data) accurately enough that an evolutionary search driven by these estimates finds merges close to those found by searching on the full dataset. Formally, the paper proves MP-IRT is asymptotically unbiased and shows that if the estimator is $\epsilon$-stable in expectation, then minimizing the estimated fitness on the subset yields a solution whose expected gap to the true full-dataset optimum is at most $\epsilon$. Empirically, the estimators beat the prior P-IRT and GP-IRT baselines in absolute error across five benchmarks, and the end-to-end system matches a full-dataset-evolved baseline on Japanese GSM8K while using roughly 50x fewer FLOPs. The paper also reports that merging an English math model with language-specific models raises translated-GSM8K accuracy 10–20 points over endpoints and standard baselines in each language, and that one evolved multilingual model beats each of its language-specific endpoints by up to 19 points on ARC.
Load-bearing premise
The entire efficiency argument rests on the premise that a merged model's skill profile is exactly a weighted average of the skill profiles of the models being merged, so the accuracy guarantee is only as strong as that linear relationship; when weight combination creates genuinely new non-linear interactions or destructive interference, the predicted full-dataset accuracy can drift from reality and the search can chase the wrong merges.
Editorial extensions
If this is right
- Fitness-evaluation cost drops from roughly $4 \times 10^6$ TFLOPs to about $0.077 \times 10^6$ TFLOPs, and the same Japanese GSM8K merge task that would take about 62 days of fitness evaluation on a single NVIDIA 4090 completes in 10–21 hours.
- Merging an English math-specialized model with language-specific models transfers mathematical reasoning: accuracy on translated GSM8K rises 10–20 points over the endpoints and over standard merging baselines in Italian, Romanian, German, and Dutch.
- A single model evolved over four languages outperforms every individually fine-tuned endpoint on translated ARC, with the largest gain (19 points) in Dutch, the weakest baseline language.
- On Japanese GSM8K, MERGE3-100 (10% of the fitness data) reaches 0.42 accuracy versus 0.49 for the full-dataset evolutionary baseline, while MERGE3-20 (2% of the data) reaches 0.34.
- The self-merging control — merging the Italian model with itself — produces no gain over the Italian endpoint, which the paper reads as evidence that the improvements are genuine cross-lingual transfer rather than prompt-template fitting.
Reading between the lines
- Because the comparison is at a fixed budget (175 trials versus 1,000), a natural test is to spend the saved FLOPs on more evolutionary trials: if accuracy climbs toward the full-dataset baseline of 0.49, the residual gap is search budget, not estimator bias.
- The same recipe — estimate endpoint abilities once on the full benchmark, then predict merged-model performance from interpolation coefficients — should transfer to objectives where full-dataset scoring is even more prohibitive, such as safety, alignment, or factuality evaluations, and to model families beyond the Mistral-7B fine-tunes used here.
- The paper itself restricts the linear-inheritance assumption to merges that show positive gains; a per-item error analysis of MP-IRT on items where endpoints disagree most sharply would show exactly where the assumption starts to bend, and could motivate a non-linear correction term.
- Uniform random sampling matching more complex subset-selection strategies in the paper's ablations suggests the IRT correction, not the sampler, is doing the work; if so, the estimators should port to new benchmarks and model families without per-dataset tuning of the extraction step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces MERGE3, an evolutionary model-merging pipeline aimed at consumer GPUs. It replaces full-dataset fitness evaluation with (i) a random subset \bar D and (ii) two IRT-based estimators, MP-IRT and GMP-IRT, that predict accuracy on the unobserved portion of D using linear combinations of endpoint ability vectors. Experiments cover cross-lingual GSM8K transfer and multilingual ARC merging, with a claimed ~50x reduction in fitness FLOPs. The theoretical section (Sec. 5) aims to show that optimizing the estimator preserves near-optimal full-data accuracy; an open-source library is promised.
Significance. The empirical demonstration that random subsets of 20-100 examples plus IRT estimates can guide evolutionary merging at 50x lower cost is potentially useful for practitioners. The release of Mergenetic and the FLOPs notebook are concrete contributions. However, the paper's central 'preserving performance' claim is currently stronger than the evidence: Fig. 6 and Table 7 show a 0.07-0.15 accuracy gap versus full-data evolution, and the estimator validation is on fixed leaderboard models, not the search distribution. If the authors add rank-correlation and agreement validation on search-visited candidates and revise the theoretical statements, the contribution would be solid.
major comments (4)
- [§5.3, Theorem 6 (and proof D.4)] The main theoretical guarantee is not proven: Theorem 6 assumes that \hat Z^mp-IRT is ε-stable in expectation and then concludes near-optimality, while the proof in D.4 asserts this stability 'under standard regularity conditions' without deriving it. Proposition 5 only gives asymptotic unbiasedness conditional on Assumption 1, known item parameters, and \hat λ convergence; it does not yield a finite-sample bound on |F(θ;D) - \hat Z(θ;\bar D)|. Please provide an explicit bound on ε as a function of |\bar D| or restate Theorem 6 as a consistency result rather than a preservation guarantee.
- [§3.2 and §4.1] Assumption 1 (Eq. 2) is load-bearing for Eq. (4), which places roughly 98% of the weight on predicted probabilities for unobserved items. The defense that only positive-gain merges matter is circular in the evolutionary loop: MP-IRT itself decides which candidates are kept, so the search cannot be restricted a priori to the linearity regime. Figure 3 validates absolute error on six pre-selected Open LLM Leaderboard merges; no rank correlation, top-1 agreement, or comparison on candidates actually generated by the evolutionary search is reported. Please report Spearman correlation and top-k agreement between MP-IRT/GMP-IRT estimates and full-dataset accuracy on the search trajectory.
- [Abstract, Fig. 6, Table 7] The claim that MERGE3 preserves performance is contradicted by the paper's own numbers: Table 7 reports accuracies of 0.34 (MERGE3-20) and 0.42 (MERGE3-100) versus 0.49 for full-data EvoLLM-JP-7B, and Fig. 6 shows the same gap. The accurate claim is a 50x cost reduction with competitive-but-lower accuracy. Please revise the abstract, introduction, and conclusions accordingly and discuss the accuracy-efficiency tradeoff.
- [B.3.1] The evaluation protocol draws fitness subsets from the GSM8K test set and evaluates on the remaining test items. Evolving on a portion of the test set and reporting accuracy on the remainder weakens the generalization claim, especially without a separate validation split. Please move fitness extraction to a training/validation split or justify why test-set selection does not inflate the reported accuracies.
minor comments (5)
- [D.4] The heading says 'Proof of Theorem 2' but the statement being proved is Theorem 6.
- [Eq. (3)] Equation (3) writes λ1γ1 + λ2γ2 while Assumption 1 is stated for n endpoints; please clarify the two-endpoint specialization.
- [Introduction / B.3.1] The Introduction says merging a math model with 'three language-specific models', while B.3.1 describes four cross-lingual merges (Japanese, Romanian, German, Dutch); please reconcile the counts.
- [Eq. (4)] The symbol \hat τ in Eq. (4) is used before it is defined in the text; define it explicitly.
- [Figure 1] The statement that point size reflects data amount is hard to read in grayscale; use labeled markers instead of reliance on size alone.
Circularity Check
Empirical speedup/accuracy comparisons are self-contained, but two load-bearing defenses are circular: Theorem 6 assumes the ε-stability it claims to prove, and the 'positive-gain' defense of Assumption 1 uses the assumption-laden MP-IRT estimator to define the set of positive-gain merges.
-
self definitional
[Section 5.3, Theorem 6 and Appendix D.4]
"Suppose that, as | ¯D| → ∞, ˆZ mp-IRT converges in probability to Z (the true performance on D), and that ˆZ mp-IRT is ϵ-stable in expectation for sufficiently large | ¯D|. Then the expected global optimum of ˆZ mp-IRT on ¯D differs from that of Z on D by at most ϵ. As | ¯D| → ∞, ϵ → 0."
The conclusion is exactly what Theorem 4 already derives from the hypothesis 'F(·; ¯D) is ϵ-stable in expectation'. Theorem 6 adds no independent derivation: its proof says Proposition 5 'implies' ε-stability under 'standard regularity conditions', but Proposition 5 only proves pointwise asymptotic unbiasedness for fixed models, conditional on Assumption 1 and λ̂ → λ; it does not establish the uniform-in-θ ε-stability needed for the optimum-gap result. The formal 'performance preservation' guarantee therefore reduces, by construction, to the unproved assumption that MP-IRT is ε-stable, rather than proving that optimizing on ¯D tracks the full-dataset optimum.
-
other
[Section 3.2, paragraph after Eq. (5)]
"Although model merging can sometimes degrade performance due to weight interference—suggesting non-linear ability interactions— our assumption is empirically supported as we are interested only in evolved models that show a positive performance gain."
In Algorithm 1, every candidate's fitness is the MP-IRT/GMP-IRT estimate from Eq. (4), whose predictions use Eq. (3), which is built on Assumption 1. Hence 'evolved models that show a positive performance gain' are exactly the candidates that score well under the assumption-laden estimator; the set is not defined independently of the assumption. The §4.1 validation cited as support measures absolute error on six fixed Open LLM Leaderboard merges and compares two IRT-based ability estimates; it does not test candidates on the evolutionary search distribution. The defense that only positive-gain (linear-inheritance) merges matter is therefore circular: the estimator decides which merges are kept, and the kept set is then used as evidence for the estimator's core assumption.
full rationale
The empirical core of MERGE3 is not circular. The 50× cost reduction is a measured throughput comparison, and the central 'preserving performance' result is benchmarked against EvoLLM-JP-7B, which evaluates fitness on the full GSM8K dataset; the six-merge estimator validation compares MP-IRT/GMP-IRT to independently computed full-dataset accuracy, providing external grounding for the small absolute errors. There is no load-bearing self-citation: the IRT machinery is attributed to Polo et al. (2024) and the evolutionary baseline to Akiba et al. (2025), both external, while the Rodolà co-authored Related Work citations are not used to justify the estimator. The circularity is localized to the argumentative defenses: Theorem 6's formal guarantee assumes the ε-stability it claims to establish, and the response to non-linear interference defines the relevant 'positive-gain' models using the very estimator that assumes linear ability inheritance. These are partial circularities in the paper's justification of its core assumption, but the final merged models' test-set accuracies are measured independently, so the central empirical claim retains independent content. Score 4 reflects partial circularity without full reduction of the empirical result to a fitted input.
Assumptions & free parameters
free parameters (6)
- IRT latent dimension d =
15
- GMP-IRT interpolation weight c =
not stated numerically
- Fitness subset size k =
20, 30, 50, or 100 per experiment
- Merge strategy per language =
SLERP for Romanian and Dutch, TIES+DARE for Japanese and German
- Evolutionary hyperparameters =
population 25, iterations 7
- Interpolation coefficients lambda =
MLE estimates per candidate
assumptions (4)
- domain assumption Assumption 1: gamma_tilde = sum lambda_i gamma_i (linear inheritance of latent abilities)
- domain assumption IRT item parameters alpha_i and beta_i estimated from Open LLM Leaderboard responses transfer to the endpoint models and languages used
- domain assumption Endpoint models lie in the same basin and are mergeable
- standard math MLE consistency of lambda on random subsets under standard regularity conditions
Cite this review
Pith. "Pith review of MERGE$^3$: Efficient Evolutionary Merging on Consumer-grade GPUs." pith.science (2026). https://pith.science/paper/BEZMONTN
@misc{pith2026250210436,
author = {Pith},
title = {Pith review of: MERGE$^3$: Efficient Evolutionary Merging on Consumer-grade GPUs},
year = {2026},
howpublished = {\url{https://pith.science/paper/BEZMONTN}},
note = {Machine review of arXiv:2502.10436}
}
abstract
Evolutionary model merging enables the creation of high-performing multi-task models but remains computationally prohibitive for consumer hardware. We introduce MERGE$^3$, an efficient framework that makes evolutionary merging feasible on a single GPU by reducing fitness computation costs 50$\times$ while preserving performance. MERGE$^3$ achieves this by Extracting a reduced dataset for evaluation, Estimating model abilities using Item Response Theory (IRT), and Evolving optimal merges via IRT-based performance estimators. Our method enables state-of-the-art multilingual and cross-lingual merging, transferring knowledge across languages with significantly lower computational overhead. We provide theoretical guarantees and an open-source library, democratizing high-quality model merging.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation
StatsMerging predicts per-layer merging coefficients from weight statistics and teacher pseudo-labels, achieving 94.5% average accuracy across eight vision tasks, 5.1 points above WEMoE.
Reference graph
Works this paper leans on
-
[1]
In particular, for θ = θ∗, F (θ∗; D) − F (θ∗; ¯D) ≤ ϵ
By ϵ-stability, for all θ ∈ Θ: F (θ; D) − F (θ; ¯D) ≤ ϵ. In particular, for θ = θ∗, F (θ∗; D) − F (θ∗; ¯D) ≤ ϵ. Hence F (θ∗; ¯D) ≥ F (θ∗; D) − ϵ and F (θ∗; ¯D) ≤ F (θ∗; D) + ϵ
-
[2]
Because θ∗ is the minimizer of F (·; D), F (ˆθ; D) ≥ F (θ∗; D)
Since ˆθ is the minimizer of F (·; ¯D), we have F (ˆθ; ¯D) ≤ F (θ∗; ¯D). Because θ∗ is the minimizer of F (·; D), F (ˆθ; D) ≥ F (θ∗; D)
-
[3]
The first term is ≤ 0 (since ˆθ is a minimizer on ¯D), and the second term is ≤ ϵ
To bound ˆm − m, we can add and subtract F (θ∗; ¯D) to have ˆm − m = F (ˆθ; ¯D) − F (θ∗; ¯D) + F (θ∗; ¯D) − F (θ∗; D) . The first term is ≤ 0 (since ˆθ is a minimizer on ¯D), and the second term is ≤ ϵ. Hence ˆm − m ≤ 0 + ϵ = ϵ
-
[4]
The first term is ≤ 0 (since θ∗ is a minimizer on D), and the second term is ≤ ϵ
Analogously, to bound m − ˆm, we can rewrite m − ˆm = F (θ∗; D) − F (ˆθ; D) + F (ˆθ; D) − F (ˆθ; ¯D) . The first term is ≤ 0 (since θ∗ is a minimizer on D), and the second term is ≤ ϵ. Thus, m − ˆm ≤ 0 + ϵ = ϵ. 4https://github.com/MrYxJ/ calculate-flops.pytorch
-
[5]
Hence F (θ∗; D) − F (ˆθ; ¯D) ≤ ϵ, completing the proof
Combining these inequalities: −ϵ ≤ ˆm − m ≤ ϵ =⇒ | m − ˆm| ≤ ϵ. Hence F (θ∗; D) − F (ˆθ; ¯D) ≤ ϵ, completing the proof. D.2. Proof of Theorem 4 Proof. By hypothesis, for every θ ∈ Θ, E ¯D h F (θ; D) − F (θ; ¯D) i ≤ ϵ. Using Jensen’s inequality for the absolute value, E ¯D F (θ; D) − F (θ; ¯D) ≤ E ¯D h F (θ; D) − F (θ; ¯D) i ≤ ϵ. Hence, −ϵ ≤ E ¯D F (θ; ¯D)...
-
[6]
for optimizing the objective function through evo- lutionary algorithms, and Lm-Evaluation-Harness (Gao et al., 2024) for implementing some of the fitness functions. In table 2 we outline the supported merging methods, while in table 3 we outline the currently available evolutionary al- gorithms. We believe this library is a significant contribution as it...
work page 2024
-
[8]
In each experiment, the pop- ulation size was fixed to 25 and the number of iterations to 7
to translate the datasets. In each experiment, the pop- ulation size was fixed to 25 and the number of iterations to 7. To check the correctness of the solution, following Akiba et al. (2025), we used a regex to extract the last numeri- cal value returned in the model’s answer and compare it with the ground truth. The solution is also checked to be in the...
work page 2025
-
[36]
URL https://aclanthology.org/2024. emnlp-industry.36/. Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. The Eleventh International Conference on Learning Representations, 2022. Jiang, A., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D., de las Casas, D., Bre...
arXiv 2024
Show all 14 references
-
[229]
acl-long.229/
URL https://aclanthology.org/2022. acl-long.229/. Lord, F., Novick, M., and Birnbaum, A. Statistical theories of mental test scores. 1968. Matena, M. and Raffel, C. Merging models with fisher- weighted averaging. Ortiz-Jimenez, G., Favero, A., and Frossard, P. Task arith- meti...
2022
-
[2007]
operator to generate offspring solutions by combin- ing parent solutions. To maintain diversity and explore the search space, we applied Polynomial Mutation (Deb et al., 2007), which introduces small perturbations to offspring solutions and enhances the algorithm’s ability to ...
2007
-
[2020]
doi: 10.1109/access.2020
ISSN 2169-3536. doi: 10.1109/access.2020. 2990567. URL http://dx.doi.org/10.1109/ ACCESS.2020.2990567. Brzezi´nska, J. Item response theory models in the measure- ment theory. Communications in Statistics-Simulation and Computation, 49(12):3299–3313, 2020. B¨ack, T. and Schwef...
2020
-
[2024]
for merging the models, Pymoo (Blank & Deb,
-
[2025]
under the same conditions as MERGE 3 to assess its accuracy, following the prompting structure outlined by 13 Efficient Evolutionary Merging on Consumer-grade GPUs Akiba et al. (2025). B.3.2. M ULTI-L INGUAL TRANSFER In this experiment, we tackle the ARC dataset in multiple la...
2025
-
[7115]
Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D
Curran Associates, Inc., 2023. Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Con- ference on Learning Representations. Yang, E., Shen, L., Guo, G., Wang, X., Cao, X...
2023 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.