REVIEW 4 major objections 5 minor 1 cited by
Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read Mediator claims that a layer-by-layer choice — average the layers where fine-tuned model updates agree, route the layers where they clash — can merge several fine-tuned LLMs into one model that outperforms both static merging and full…
desk verdict Useful engineering recipe for memory-efficient LLM merging, but the centerpiece 'adaptive conflict-based split' doesn't move the needle — Table 10 shows it buys memory savings, not accuracy, and the conflict metric is computed on raw weights rather than task arithmetic. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Three objects carry the argument. First, the per-layer sign-conflict ratio $d_l = \sum_{i,j} \mathbb{I}(\mathrm{sgn}(w^l_i w^l_j) = -1) / |\theta^l|$ (Equation 3): it counts, after denoising the task arithmetics, the fraction of parameter pairs with opposite signs within a layer, and it is the only quantity that decides a layer's fate. Second, the Gaussian fit $d_l \sim \mathcal{N}(\mu, \sigma)$ with cutoff at $\mu + \sigma$: layers below the cutoff are averaged, layers above are routed, so the entire average-versus-route split is one threshold on a measured statistic rather than a learned policy. Third, the dense-plus-sparse expert decomposition $\theta^l = \bar{\theta}^l + \sum_{\tau} h(\tau|x) \hat{\Delta}^l_\tau$: averaged layers provide the dense core $\bar{\theta}^l$, routed layers store denoised, sparsified deltas $\hat{\Delta}^l_\tau$, and a small classifier feeds temperature-scaled probabilities $h(\tau|x) = \mathrm{softmax}(\pi_\kappa(\tau|x)/\beta)$ with $\beta = 1.5$ to select and blend the top-$k$ experts once per sample. The decomposition is what converts the conflict measurement into memory savings: with roughly 14 percent of expert parameters retained, routed-layer storage falls by a factor that reaches about 7$\times$ at eight experts.
What would settle it
One experiment would settle whether the conflict statistic carries the result: take a pair of fine-tuned models, artificially inject heavy sign disagreement into their middle layers while leaving the outer layers nearly consistent, and check whether Mediator then routes the middle layers and keeps its advantage. A second, cheaper check is to shuffle the measured per-layer conflict ratios before applying the Gaussian cutoff; if merged accuracy is unchanged, the layer-wise measurement adds nothing beyond a fixed averaging-versus-routing budget. Both tests reuse the paper's released code, model set, and hyperparameters.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that parameter conflict between fine-tuned LLMs is not spread uniformly through the network but concentrated in the outer layers, and that a hybrid of averaging and routing can exploit this structure. Concretely, Mediator computes, for each layer, the sign-conflict ratio $d_l$ of Equation 3 — the proportion of parameters whose updates in two models' task arithmetics point in opposite directions after denoising — fits a Gaussian $d_l \sim \mathcal{N}(\mu, \sigma)$ over layers, averages every layer with $d_l < \mu + \sigma$ into a shared dense backbone, and routes the remaining layers by reconstructing them from the average layer plus a top-$k$, temperature-softened blend of sparsified task-arithmetic deltas. Because task arithmetic concentrates near zero, keeping about 14 percent of each expert's parameters preserves the routing gains, which is what turns routing from a storage luxury into a memory-efficient option. The evidence presented is a sweep over four model families and scales, an eight-task scale-up, ablations on temperature, compression ratio, and the averaging decision, and case studies of generated answers, with the claimed result that Mediator posts the highest average score in every setting while recovering most of each specialist's peak performance.
Load-bearing premise
The load-bearing premise is that the fraction of parameters whose fine-tuned updates point in opposite directions inside a layer truly measures how harmful averaging that layer would be, and that a single statistical cutoff based on the spread of those fractions correctly sorts layers into safe-to-average and must-route ones; if sign disagreements do not track the interference that actually degrades merged performance, the split reduces to a tuned heuristic.
Editorial extensions
If this is right
- A merged model can recover most of each specialist's peak performance on its own task while beating every static and dynamic baseline on average, so deployment can replace a collection of fine-tuned experts with one model.
- Memory scales as one dense backbone plus a small fraction of each expert, letting a capability equivalent to four 7B models run inside roughly 24 GB, and the paper reports the 7B$\times$4 case fits a single RTX 4090.
- Task-level routing suffices: with a once-per-sample router, post-training time stays between 1.3 and 2.0 hours, versus 7.5 to 12.5 hours for token-level BTX, and inference stays within 0.2–0.4 seconds of the base model.
- Out-of-distribution inputs are handled by soft expert blending guided by task uncertainty, so inputs that mix several tasks (e.g., code plus math) get a convex combination of the relevant experts rather than a forced choice.
- Ablations bound the cost of the averaging choice: dropping layer-wise merging costs only 0.06–0.27 percent accuracy, and compressing experts to about 14 percent retention is the reported optimum.
Reading between the lines
- The paper leaves open whether the same sign-conflict statistic is useful beyond merging — for instance, as a layer-importance diagnostic that says which layers of a fine-tuned model hold general knowledge and should be frozen during parameter-efficient adaptation.
- The OOD mechanism is tested only against domains the router has seen or that interpolate between them; an extension not attempted here is merging experts from genuinely new domains or unknown task labels, where the fallback to the base model for low-confidence inputs is the only safety net.
- Because temperature and top-$k$ are chosen globally, a natural follow-up the paper does not try is per-input or per-layer temperature adaptation, which could push the reported out-of-distribution gains further.
- The router needs labeled fine-tuning data to train; replacing it with a routing decision derived directly from the conflict statistics would make Mediator fully calibration-free, but the paper does not test that reduction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Mediator, a hybrid model-merging framework that averages LLM layers with low measured parameter conflict and routes layers with high conflict using task-level sparse experts derived from task arithmetics, with temperature-scaled uncertainty-based expert mixing for out-of-distribution inputs. The method is evaluated on LLaMA-3B/8B and Qwen-4B/7B backbones fine-tuned on math, QA, coding, and other tasks, and is compared against TIES, PCB, Twin-merging, and BTX. The central claim is that Mediator consistently outperforms these baselines in average task accuracy while incurring less system cost than routing-based alternatives.
Significance. If the empirical claims held as stated, Mediator would be a practically useful contribution: it is evaluated on modern LLM backbones, uses CoT-augmented fine-tuning, reports post-training and inference costs, and demonstrates scalability to eight experts. The system-optimization discussion (CPU prefetching, CUDA atomicAdd, batched arithmetic inference) is a concrete plus, and the limitations section is candid about missing theory and loading bottlenecks. However, the load-bearing adaptive-layer-merge story is not supported by the paper's own ablation, the conflict metric is defined on raw weights rather than the task arithmetic that motivates it, and the headline gains rest on point estimates with hyperparameters tuned on the same evaluation tasks. These issues materially weaken the claimed contribution and need to be addressed before the central claims can be accepted.
major comments (4)
- [Section 3, Eq. (3)] The conflict ratio d_l is computed on the raw fine-tuned weights θ_l, whereas the motivating discussion and the sparsity argument in Section 4.2 concern task arithmetic Δτ = θτ − θ. Figure 4 shows that ‖Δτ‖ is orders of magnitude smaller than ‖θ‖, so the sign of θτ is dominated by the base model θ for almost all coordinates; d_l therefore mostly counts rare base-sign flips, concentrated in embeddings and the head, rather than interference among task-specific updates. The paper does not show that d_l rank-correlates with the actual per-layer accuracy loss under averaging. Please either redefine d_l on Δτ or provide a per-layer ablation/correlation study showing that the chosen metric predicts which layers should be averaged.
- [Table 10] The ablation 'w/o layer-wise merging' shows that removing the conflict-based split changes Qwen-1.5 performance by only −0.06% and Qwen-2.5 by −0.27%, i.e., the adaptive layer split is essentially neutral. Since this split is a primary contribution of the paper (Section 4.1, Algorithm 1, and the first contribution bullet), the experimental results do not demonstrate that the adaptive merging component is load-bearing for the headline gains; those gains could come entirely from the task-level sparse-expert router. The manuscript should either show settings where the conflict-based split contributes substantially, or reframe the contribution to focus on the routing/sparsification design.
- [Tables 9, 11 and Tables 2–4] The key hyperparameters β and the expert compression ratio are selected on the same evaluation tasks used in the main comparison tables (Table 9 tunes β and Table 11 tunes the compression ratio on the same GSM8K/TriviaQA/HumanEval/etc. benchmark set). The paper reports no error bars or repeated-seed variation, and the margins over BTX are small in places (e.g., Qwen-7B: 71.00 vs. 69.61; LLaMA-8B: 71.80 vs. 70.52). This creates a risk that the reported advantage is partly a tuned-configuration artifact. Please add variance estimates and an evaluation protocol that tunes hyperparameters on a held-out split.
- [Table 7] The 'memory-efficient' claim is not clearly supported against the Twin-merging baseline: Table 7 lists identical memory for Mediator and Twin-merging on LLaMA-3B (10G), LLaMA-8B (35G), and Qwen-4B (13G), with Mediator slightly higher on Qwen-7B (33G vs. 32G). Since the abstract and introduction emphasize memory reduction (e.g., 'reduces storage from 50% to 7%'), the paper needs to clarify what is included in these numbers (GPU vs. CPU storage, expert compression ratio) and explain why Mediator does not improve on Twin in this table despite the claimed compression advantage.
minor comments (5)
- [Section 4.3, Eq. (4)] Eq. (4) exponentiates πκ(τ|x), which is already a probability, but the text says the logits hκ are temperature-scaled. Please use the logits in the softmax and define the notation consistently.
- [Algorithm 1 vs. Definition 3.1] Algorithm 1 defines Δτ = θτ − θ̄ and Δ⋆ = θ − θ̄, while Definition 3.1 defines task arithmetic as θτ − θ. The anchor for the 'sparse expert' decomposition should be consistent; otherwise the reconstructed model in Algorithm 2 (θ̄ + Σ h Δ̂) is not the same as θ + Σ h Δ̂.
- [Tables 2–4 and Appendix G.2] The backbone names are inconsistent across the paper ('Llama 3.1 3B' vs. 'Llama-3.2-3B', 'LLaMA 3.2 8B' vs. 'LLaMA 3.1 8B'). Please standardize the model names and ensure the appendix tables match the main tables.
- [Figure 2] The caption says '(Qwen 2.5 7B)' but the legend shows four models (Qwen-4B, Llama-3B, Qwen-7B, Llama-8B). Please clarify which model(s) the displayed conflict curves correspond to.
- [Appendix F] Sections F.2 and F.3 both carry the title 'CoT based Data Augmentation'; the second one appears to be a duplicate heading and should be renamed or merged.
Circularity Check
No derivational circularity; minor evaluation-selection circularity in choosing temperature and compression ratio on the same benchmark tables.
-
fitted input called prediction
[Section 5.2 (Tables 9 and 11); Appendix C.2]
"For router based selection, we use the temperature parameter beta in Equation 4 as 1.5 to convert the prediction rates into concrete merging parameters for each expert, which achieves the best experimental results. Results show that the optimal performance is obtained when 14% parameters are left."
The final configuration is selected by maximizing the same GSM8K/TriviaQA/WinoGrande/HumanEval/MMLU averages that are later reported as Mediator's headline gains (e.g., 51.40 on Qwen-4B and 71.80 on LLaMA-8B). Therefore the claim that Mediator consistently achieves significant performance improvements on these benchmarks is partly a selected optimum over beta and compression ratio, not an independent prediction of performance at a fixed configuration. The method's architecture is still independently compared against TIES/PCB/Twin/BTX, so this is a bounded, evaluation-level circularity rather than a derivation that reduces to its inputs.
full rationale
The core derivation chain is not circular: d_l is an empirical sign-conflict statistic computed from the finetuned models; the mu+sigma split is a fixed decision rule, not fitted to evaluation scores; expert decomposition theta_tau = theta + (theta_tau - theta) is an algebraic identity; and sparse-expert retention plus task-level routing are validated against external baselines (TIES, PCB, Twin-merging, BTX) rather than derived from Mediator's own claims. The paper's self-citations are background related-work references and are not load-bearing for the central mechanism. The one genuine circular element is the selection of hyperparameters (beta=1.5, compression ratio 14%) on the same evaluation tables used to report the headline performance; this is a mild tuning/evaluation circularity, not a self-definitional derivation. The skeptic's concern that d_l is measured on raw weights rather than task arithmetic, and that Table 10 shows the layer-wise averaging contributes almost nothing, is a correctness/ablation weakness, not a circularity, so it does not raise the circularity score further.
Assumptions & free parameters
free parameters (4)
- expert compression ratio c =
14%
- temperature beta =
1.5
- layer routing threshold =
mu + sigma of Gaussian fit
- denoising cutoff for task arithmetic =
mu_UPD +/- sigma_UPD
assumptions (6)
- domain assumption Opposite signs of denoised task arithmetic elements measure parameter conflict.
- ad hoc to paper Layer conflict ratios follow a Gaussian distribution.
- domain assumption A fine-tuned model can be reconstructed as base plus sparse task arithmetic.
- domain assumption Attention layers store task-shared knowledge and should always be averaged.
- domain assumption A small classifier trained on prompts from three tasks estimates posterior task probability for unseen inputs.
- domain assumption Task-level routing is preferable to token-level routing for merged SFT LLMs.
Cite this review
Pith. "Pith review of Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing." pith.science (2026). https://pith.science/paper/GHMKN7PD
@misc{pith2026250204411,
author = {Pith},
title = {Pith review of: Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing},
year = {2026},
howpublished = {\url{https://pith.science/paper/GHMKN7PD}},
note = {Machine review of arXiv:2502.04411}
}
read the original abstract
Model merging aggregates Large Language Models (LLMs) finetuned on different tasks into a stronger one. However, parameter conflicts between models leads to performance degradation in averaging. While model routing addresses this issue by selecting individual models during inference, it imposes excessive storage and compute costs, and fails to leverage the common knowledge from different models. In this work, we observe that different layers exhibit varying levels of parameter conflicts. Building on this insight, we average layers with minimal parameter conflicts and use a novel task-level expert routing for layers with significant conflicts. To further reduce storage costs, inspired by task arithmetic sparsity, we decouple multiple fine-tuned experts into a dense expert and several sparse experts. Considering the out-of-distribution samples, we select and merge appropriate experts based on the task uncertainty of the input data. We conduct extensive experiments on both LLaMA and Qwen with varying parameter scales, and evaluate on real-world reasoning tasks. Results demonstrate that our method consistently achieves significant performance improvements while requiring less system cost.
Figures
Figures from the paper (5 more)
Forward citations
Cited by 1 Pith paper
-
Local Mixtures of Experts: Essentially Free Test-Time Training via Model Merging
TTMM pre-trains many local LoRA experts on data clusters and merges the most relevant few at test time, approximating test-time training with a 100x speedup and near-TTT perplexity.
Reference graph
Works this paper leans on
-
[1]
Layer-wise training dynamics and optimization. This direction discusses the layer-wise training dynamics to help shed some light on the paramter conflicts and the layer-wise adaptivity
-
[2]
URL https: //doi.org/10.48550/arXiv.2306.00978
doi: 10.48550/arXiv.2306.00978. URL https: //doi.org/10.48550/arXiv.2306.00978. Liu, D., Wang, Z., Wang, B., Chen, W., Li, C., Tu, Z., Chu, D., Li, B., and Sui, D. Checkpoint merging via bayesian optimization in llm pretraining. arXiv preprint arXiv:2403.19390, 2024a. Liu, H., Zhang, Y ., Luo, Y ., and Yao, A. C. Augmenting Math Word Problems via Iterativ...
-
[3]
OOD Detection & Generalization. This direction reviews some works of the OOD Detection and Generalization, shedding light on deployment of the model merging on the out-of-distribution data
-
[4]
Model compression. This direction shortly review some works about the model compression, in which many methods can be directly applied into our framework to further reduce the memory costs. Note that in our paper we propose a general framework instead of a new model compression method. Different model compression methods can be combined into our framework
-
[5]
Data Synthesis. This direction reviews some works about how to generate new synthetic data to improve the model merging performance. In our framework, we exploit the CoT to generate new synthetic data to improve the finetuning performance on downstream tasks, which is a real-world downstream task instead of traditional model fine-tuning using the in-domai...
work page 1996
-
[9]
Bayesian deep learning. This direction reviews some works of the Bayesian deep learning, discussing the uncertainty and Bayesian model averaging. 1Normally, each transformer layer occupies the same memory. 21 Mediator: Memory-efficient LLM Merging with Less Parameter Conflicts and Uncertainty Based Routing
-
[13]
The importance measurement is based on the calibration dataset
Require Calibration Dataset. The importance measurement is based on the calibration dataset. In the LLM era, the pretrained dataset is significantly large, it is difficult to collect the complete pretrained dataset and measure the importance of the parameters on it
-
[14]
Computation Costs. Because that the importance measurement is based on the calibration dataset, the computation costs is almost similar to conduct the complete forward process of the different models. In traditional small models, such a computaton cost is acceptable. However, in the LLM era, the model size is significantly large, such a computation cost i...
Show all 18 references
-
[15]
While methods in these importance based weighted averaging methods can improve the merging performance, they still face the parameter conflicts between different models
Unaddressed Parameter Conflicts. While methods in these importance based weighted averaging methods can improve the merging performance, they still face the parameter conflicts between different models. Because of the highly 22 Mediator: Memory-efficient LLM Merging with Less ...
2023
-
[17]
Structured pruning is hardware-agnostic, facilitating accelerated inference but may degrade performance due to the removal of critical components, often necessitating fine-tuning
iteratively prunes and grows weights to minimize reconstruction error in sparse models. Structured pruning is hardware-agnostic, facilitating accelerated inference but may degrade performance due to the removal of critical components, often necessitating fine-tuning. Loss-base...
2019
-
[1282]
Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A
PMLR, 2017. Mitra, A., Khanpour, H., Rosset, C., and Awadallah, A. Orca-math: Unlocking the potential of slms in grade school math, 2024. URL https://arxiv.org/ abs/2402.14830. Molchanov, P., Mallya, A., Tyree, S., Frosio, I., and Kautz, J. Importance estimation for neural net...
2017 arXiv
-
[2007]
org/CorpusID:16660598
URL https://api.semanticscholar. org/CorpusID:16660598. Stojkovic, J., Choukse, E., Zhang, C., Goiri, I., and Torrellas, J. Towards greener llms: Bringing energy-efficiency to the forefront of llm inference, 2024. URL https: //arxiv.org/abs/2403.20306. Sukhbaatar, S., Golovnev...
2024 arXiv
-
[2016]
Romeo and Juliet
for the multiclass logistic loss for ν ∈ [0, 1]. Following (Kleijn & der Vaart, 2012; Xie et al., 2022), the task parameter τ is assumed to have the continuity, where the KL divergence is assumed to haver the 2nd-order Taylor expansion. Then, we have the following theorem and ...
1990
-
[2018]
You, J., Chung, J.-W., and Chowdhury, M
doi: https://doi.org/10.1145/3196398.3196408. You, J., Chung, J.-W., and Chowdhury, M. Zeus: Under- standing and optimizing gpu energy consumption of dnn training, 2022. URL https://arxiv.org/abs/ 2208.06102. Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y . Extend model merging ...
2022
-
[2022]
overlapping local updates
builds upon the concept of “overlapping local updates”, introducing a learning strategy that harmonizes the high parallelism characteristic of layer-wise training with the superior predictive accuracy associated with end-to-end (E2E) learning. Besides, classification-based los...
2018
-
[2023]
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D
URL https://proceedings.mlr.press/ v202/frantar23a.html. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //ope...
2023 arXiv
-
[2024]
Zhao, Y ., Huang, J., Hu, J., Wang, X., Mao, Y ., Zhang, D., Jiang, Z., Wu, Z., Ai, B., Wang, A., Zhou, W., and Chen, Y
URL https://openreview.net/forum? id=1ndDmZdT4g. Zhao, Y ., Huang, J., Hu, J., Wang, X., Mao, Y ., Zhang, D., Jiang, Z., Wu, Z., Ai, B., Wang, A., Zhou, W., and Chen, Y . Swift:a scalable lightweight infrastructure for fine-tuning, 2024. URL https://arxiv.org/abs/ 2408.05517. ...
2024 arXiv
-
[7261]
Zelikman, E., Wu, Y ., Mu, J., and Goodman, N
PMLR, 2019b. Zelikman, E., Wu, Y ., Mu, J., and Goodman, N. STaR: Bootstrapping Reasoning With Reasoning. Advances in Neural Information Processing Systems , 35:15476– 15488, 2022. Zeng, W., Xu, C., Zhao, Y ., Lou, J.-G., and Chen, W. Au- tomatic instruction evolving for large...
2022 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.