Channel Merging: Preserving Specialization for Merged Experts

Bohan Zhuang; Ganggui Ding; Jing Liu; Linlin Ou; Mingyang Zhang; Xinyi Yu

REVIEW 3 major objections 4 minor 1 cited by

Channel Merging: Preserving Specialization for Merged Experts

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Channel Merging claims that merging fine-tuned LLM experts at the channel level rather than the whole-model level eliminates most parameter conflicts, so a zoo of N experts can be stored as K groups (K<N) and each expert reconstructed on…

desk verdict A genuinely new channel-level merging idea with solid empirical gains, but the mechanism is under-validated and the paper needs a round of hardening. read the letter →

arxiv 2412.15283 v1 pith:EGCS5VV2 submitted 2024-12-18 cs.CL cs.AIcs.LG

Mingyang Zhang , Jing Liu , Ganggui Ding , Xinyi Yu , Linlin Ou , Bohan Zhuang This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords modelmerginglargelanguagemodelschannel-wisetaskarithmeticparameter-efficientinferenceexpertroutingK-Meansclusteringcosinesimilarity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual model-merging trade-off—performance loss from parameter conflicts versus storage cost from keeping every expert—can be avoided by merging at a finer granularity. Instead of averaging whole expert models, it clusters per-channel delta parameters across experts by cosine similarity, merges only within each cluster, and stores an index that lets each expert be reconstructed at inference time. It claims this reduces storage from N full experts to K full groups (K

What carries the argument

The central object is the per-output-channel clustering of delta parameters. For the i-th output channel, the N experts' delta vectors $δ_i = {δ_i^{t_1}, ..., δ_i^{t_N}} ∈ R^{N × I}$ are grouped by K-Means into K clusters $C_i^1, ..., C_i^K$ using cosine similarity, and cluster members are merged with task arithmetic: $Θ_i^k = P_i + λ ∑_{δ_i^{t_n} ∈ C_i^k} δ_i^{t_n}$. Each expert stores an index set $S^{t_n}$ recording which cluster holds each channel; during inference the activated expert's parameters are reconstructed by concatenating the relevant group parameters. The storage analysis reduces the total from $NΨ$ to $KΨ$ parameters, a factor of $K/N$, because the index overhead scales only with the number of channels.

What would settle it

Construct two fine-tuned experts on the same base model whose channel delta vectors are highly cosine-similar but functionally opposed (e.g., one channel encodes a feature that the other suppresses), then compute the distance between the merged layer's output and each expert's original output on that expert's own data; if the merged channel reproduces neither expert's behavior, the similarity proxy is refuted. Concretely, a reader could pick channels with cosine similarity above 0.9 and measure per-channel activation distances before and after merging as in Eq. (3).

Watch

Extended reading notes

Core claim

The central claim is that channel-level merging preserves specialist performance that model-level merging loses. For each output channel, the delta vectors of all N experts are clustered by cosine similarity into K groups using K-Means; within each group the deltas are averaged onto the pretrained weights. Because only highly similar parameters are averaged, parameter conflicts largely disappear, and because each expert keeps an index of which group each of its channels came from, activating an expert is just a lookup-and-concatenate operation. The paper reports that with K=2 and N=4 experts, the total parameter count drops to 53% of a full ensemble (14.3B vs 26.8B) while matching or exceeding unmerged experts on English reasoning, math, code, and Chinese benchmarks, and matching an ensemble on general tasks when paired with a router.

Load-bearing premise

The method assumes that two channels whose fine-tuning changes point in nearly the same direction can be averaged without losing either expert's behavior, even though the paper never checks the averaged channels against the original outputs.

Editorial extensions

If this is right

A zoo of N task-specialized LLMs can be deployed with roughly K/N of the original storage, and the ratio improves as N grows because K stays constant.
Channel-level merging can be layered on top of existing pruning-based merging methods: the paper shows DARE-CM and TIES-CM outperform their model-level counterparts on every reported task.
Adding a lightweight task router to select which expert to reconstruct preserves the specialized-task gains while improving general-task accuracy over any single expert.
The method's storage advantage holds even when the number of experts increases, unlike partial-merging approaches such as BYOM that must keep separate parameters for each expert.
The ablation across granularities (channel, layer, model) indicates that finer granularity is the reason for the reduced performance degradation, not the specific clustering algorithm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the storage ratio K/N improves as N grows, the method's advantage should become more pronounced with larger expert zoos, provided the clustering assumption still holds; the paper's experiments only go up to six experts.
The paper motivates cosine similarity by citing a link to activation similarity, but clusters raw parameters rather than activations; a natural stronger variant would cluster channels by their functional effect on layer outputs, which the paper does not test.
The task router is trained with supervision from the experts' original fine-tuning datasets, so out-of-distribution queries are an untested regime; a misrouted query would reconstruct the wrong expert and could negate the accuracy benefit.
The layer-wise similarity analysis is performed on only two model families; whether the method transfers to other base models or to experts with non-overlapping fine-tuning distributions is an open question the paper does not address.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper before the rush of merging papers hits: it takes partial merging down to the channel level. Instead of merging whole models or keeping whole layers, it clusters each output channel's delta vectors across experts via K-Means, merges within clusters, and stores an index of which cluster each expert's channel went to. At inference you reconstruct the activated expert by concatenating the right cluster slots. That is a real idea, and the storage math is clear: K groups per channel gives K/N of the ensemble footprint. The authors also do the right ablations: channel vs layer vs model granularity, KMeans vs random grouping, cosine vs Euclidean vs Manhattan, group count, and pruning ratio. The channel-level results beat model-level merging across the board, and the router results at 53% parameters are worth taking seriously. The limitations section is honest about needing a shared pretrained base and about parameter growth vs one-size-fits-all. This is a useful paper for anyone working on multi-expert serving or model merging. Now the soft spots. First, no error bars or significance tests anywhere, and some numbers are suspicious. DARE-CM jumps TriviaQA from 58.39 to 64.49, above the unmerged baseline, with no mechanistic story. That suggests either the method is doing something more than preserving specialization, or the evaluation is noisy. Second, K and lambda look selected on the test benchmarks. K=2 is used for N=4, lambda=0.5, and the pruning ratio is tuned to 30% using the same datasets. That is hyperparameter fitting by another name. Third, the paper never addresses the cluster-size problem. With K=2 and N=4, clusters can be 2-and-2, but they could also be 1-and-3, and lambda=0.5 then halves a singleton expert's delta instead of preserving it. The paper reports no cluster-size statistics and no variant with per-cluster lambda. The stress-test note is right that cosine similarity is only a proxy; the appendix compares distance metrics, not whether reconstructed outputs match the originals. The similarity heatmap measures index overlap, not function. Fourth, ME-Switch, an earlier paper by largely the same group, is cited but never positioned against; the router story overlaps heavily and the paper should say what is new beyond it. None of this kills the approach—the empirical pattern is consistent and the ablations show the clustering matters—but the paper currently argues from correlation. A serious referee should ask for error bars, a validation split for hyperparameters, a cluster-size report, and a functional test (logit or output similarity) of the reconstruction. I would send this to review rather than desk-reject; it deserves a response, but it needs a substantive revision before it is publishable as is.

Referee Report

3 major / 4 minor

Summary. The paper proposes Channel Merging, a method to merge multiple task-specific fine-tuned LLMs into a compact set of parameters. For each output channel, the delta vectors of the experts are clustered into K groups via K-Means based on cosine similarity; parameters within each cluster are merged with task arithmetic, and a small index per expert records the group assignment. At inference, the parameters of the activated expert are reconstructed by concatenating the appropriate group parameters. The paper reports that this approach matches unmerged experts on specialized reasoning, math, code, and Chinese benchmarks, and that with a task-specific router it reaches performance comparable to a full model ensemble while storing about 53% of the parameters. The method is evaluated against DARE, TIES, and BYOM on the Mistral-7B family.

Significance. The paper addresses a practically important problem: deploying multiple specialized LLMs without loading all of them into GPU memory. The channel-level clustering idea is simple and the storage-reduction argument is clear (K/N parameter ratio). The experiments cover four diverse domains and include useful ablations on granularity, clustering method, and number of groups. If the technical issues around the merging rule and the interpretation of the results are resolved, the method could be a practical contribution to model-merging and multi-expert deployment. However, the current support for the central claim that similarity-based channel averaging preserves expert specialization is incomplete, and one equation-level inconsistency needs clarification.

major comments (3)

[Method, Eq. (3) and Table 6] The merging rule Θk_i = P_i + λ Σ δt_i with a fixed λ=0.5 is not an averaging operation when clusters have sizes other than two. In particular, a singleton cluster would produce P_i + 0.5δ, i.e., a halved delta, yet Table 6 reports that with K=4 (where each cluster trivially contains one expert per channel) the results are exactly identical to the unmerged baseline. This is only possible if λ is effectively 1, or if pruning and scaling are not applied, for singleton clusters. The paper must clarify the exact formula used in this case, report cluster-size statistics for the K=2 setting, and either adapt λ to the cluster size (e.g., 1/|C|) or justify the fixed value empirically.
[Merging with Channel Similarity; Table 5] The load-bearing premise is that clustering channels by cosine similarity of delta vectors preserves expert function better than random grouping. Table 5 shows that Random grouping already achieves strong performance (e.g., Instruction avg 69.23 vs 69.88 for KMeans; Math avg 43.96 vs 44.95), so the benefit of similarity-based clustering over simply storing two mixed models is small on several tasks. The paper should report a more direct functional test, such as the agreement between the reconstructed expert's per-channel outputs and the original expert's outputs, or at least discuss why the similarity proxy matters beyond the storage reduction. Without this, the claim that 'preserving specialization' is the mechanism is not fully supported.
[Tables 2 and 3] The experimental support for the headline claims is weakened by the absence of error bars or significance tests. In Table 3 the difference between DARE-CM+router and the full ensemble is 0.01 on MMLU+CMMLU (54.42 vs 54.43) and 0.24 on AGIEval (38.01 vs 38.25), which is likely within run-to-run variation; the text calls these results 'comparable,' which is fair, but the paper also claims improvements over individual baselines with similar-sized gaps. In Table 2, DARE-CM beats the unmerged baseline on TriviaQA by 6.10 points (64.49 vs 58.39) with no mechanism explained. The authors should provide multiple seeds, error bars, or a statistical test, and explicitly discuss unexplained large gains.

minor comments (4)

[Method, Eq. (5)] The text says 'where L denotes the concatenation operation' but the displayed equation uses a different symbol (a large operator that is not defined). Please align the notation and define the concatenation operator explicitly.
[Model size reduction analysis] The sentence 'the total number of parameters necessary is effectively diminished to KΨ/NΨ = K/N' is mathematically the ratio of new to old parameter counts, not the reduced count itself. Rephrase to avoid confusion, e.g., 'reduced from NΨ to KΨ, a factor of K/N.'
[Table 4 and surrounding text] The terms 'Layer' and 'Model' granularities in Table 4 are not defined precisely. It should be stated clearly what is clustered at each granularity, e.g., whether the clustering is applied per layer to the entire weight matrix, or globally across layers. Without this, the ablation is difficult to interpret.
[Appendix, 'The similarity between merged experts'] The heatmap in Figure 5 measures the fraction of channels where two experts share the same group index, which is an index-overlap measure, not a measure of functional or output similarity. Please clarify this in the caption and text to avoid overstating what the figure establishes.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: channel merging is an empirical compression method; storage ratio is arithmetic and performance is measured on held-out benchmarks.

full rationale

No circularity found. The paper's central claim—that channel-level clustering and merging of expert delta vectors preserves performance at reduced storage—is an empirical claim. Eq. (3) constructs cluster parameters from the trained deltas, and Eq. (5) reconstructs each expert by concatenating the cluster parameters assigned to it; for K<N this reconstruction is a genuine approximation, not an identity, so 'matching unmerged models' is not true by construction. The K/N storage reduction is a direct accounting consequence of storing K group-parameter sets rather than N expert models and is arithmetic, not a fitted prediction. The only learned component is the task-specific router (Eq. 6), trained on queries labeled by their source fine-tuning task and evaluated on separate general benchmarks (MMLU, CMMLU, AGIEval); this is a standard held-out protocol. The citation to Liu et al. 2024 (ME-Switch) for the router paradigm is a self-citation but is not load-bearing—the router is an optional component and its effectiveness is measured, not assumed. The unverified proxy assumption that cosine similarity of delta vectors implies functional compatibility (Section 'Similarity Inconsistency'; Eq. 3) is a correctness and validation risk, not a circularity: nothing in the paper defines the target metric in terms of that similarity. Stated limitations (same pretrained backbone; possible parameter increase vs. one-size-fits-all) are constraints rather than circular steps.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central method uses three tuned hyperparameters (K, lambda, pruning ratio) and assumes the standard task-arithmetic merge operator plus a valid cosine-similarity grouping. No new physical entities are introduced. The main modeling risk is that the similarity-preservation assumption is not verified against held-out data.

free parameters (3)

K (number of clusters) = 2
Number of merged parameter groups. K=2 selected in ablations (Table 6) by trading storage and accuracy on evaluation benchmarks.
lambda (task arithmetic scaling) = 0.5
Scaling factor in Eq. (3), set to 0.5 following task arithmetic convention; not swept in the paper.
Pruning ratio (DARE/TIES) = 30%
Fraction of delta parameters pruned before merging. Chosen in Table 10 by evaluating 0%, 30%, 50%, 70% on benchmark test sets.

assumptions (4)

standard math Task arithmetic combines delta parameters as weighted sums (Eq. 1)
Borrowed from Ilharco et al. (2023) as the merge operator; the paper does not re-derive it.
standard math K-Means with cosine similarity groups similar channel deltas
The grouping algorithm is assumed to find clusters that minimize intra-cluster variance in cosine space; used without modification.
domain assumption Experts are all fine-tuned from the same pre-trained base model
Stated in the Limitations; channel-wise delta comparison is only meaningful within one model family.
domain assumption The router's training labels (source task datasets) generalize to unseen benchmark queries
The router is trained on 50k samples from the fine-tuning datasets and then applied to MMLU/CMMLU/AGIEval; the paper assumes distributional overlap.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Channel Merging: Preserving Specialization for Merged Experts." pith.science (2026). https://pith.science/paper/EGCS5VV2

@misc{pith2026241215283,
  author       = {Pith},
  title        = {Pith review of: Channel Merging: Preserving Specialization for Merged Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EGCS5VV2}},
  note         = {Machine review of arXiv:2412.15283}
}

read the original abstract

Lately, the practice of utilizing task-specific fine-tuning has been implemented to improve the performance of large language models (LLM) in subsequent tasks. Through the integration of diverse LLMs, the overall competency of LLMs is significantly boosted. Nevertheless, traditional ensemble methods are notably memory-intensive, necessitating the simultaneous loading of all specialized models into GPU memory. To address the inefficiency, model merging strategies have emerged, merging all LLMs into one model to reduce the memory footprint during inference. Despite these advances, model merging often leads to parameter conflicts and performance decline as the number of experts increases. Previous methods to mitigate these conflicts include post-pruning and partial merging. However, both approaches have limitations, particularly in terms of performance and storage efficiency when merged experts increase. To address these challenges, we introduce Channel Merging, a novel strategy designed to minimize parameter conflicts while enhancing storage efficiency. This method clusters and merges channel parameters based on their similarity to form several groups offline. By ensuring that only highly similar parameters are merged within each group, it significantly reduces parameter conflicts. During inference, we can instantly look up the expert parameters from the merged groups, preserving specialized knowledge. Our experiments demonstrate that Channel Merging consistently delivers high performance, matching unmerged models in tasks like English and Chinese reasoning, mathematical reasoning, and code generation. Moreover, it obtains results comparable to model ensemble with just 53% parameters when used with a task-specific router.

Figures

Figures reproduced from arXiv: 2412.15283 by the authors.

**Figure 1.** This diagram contrasts various methods of handling multiple experts in LLMs. Panel (a) illustrates the conventional [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Layer-wise proportion of channel similarity between Instruction expert and other experts in (a) Mistral-7B model [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** An illustration of our Channel Merging method. The process involves two core parts: Channel-wise Merging and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Experimental results on four different task categories as the number of experts varies from one to six. ’Channel’ and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Heatmap of channel similarities between different [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion
cs.CL 2025-01 reject novelty 5.0 of 10

InfiFusion fuses multiple large language models into one pivot model using enhanced universal logit distillation, and reports that the fused model outperforms all source models on 11 benchmarks with a fraction of the ...

Reference graph

Works this paper leans on

54 extracted references · 33 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

bert-base-multilingual-cased

2023. bert-base-multilingual-cased . https://huggingface.co/google-bert/bert-base-multilingual-cased

work page 2023
[4]

Chinese-Mistral-7B-Instruct-v0.1

2023. Chinese-Mistral-7B-Instruct-v0.1 . https://huggingface.co/itpossible/Chinese-Mistral-7B-Instruct-v0.1

work page 2023
[5]

CollectiveCognition-v1.1-Mistral-7B

2023. CollectiveCognition-v1.1-Mistral-7B . https://huggingface.co/teknium/CollectiveCognition-v1.1-Mistral-7B

work page 2023
[6]

2023. Dolphin . https://huggingface.co/datasets/cognitivecomputations/dolphin

work page 2023
[7]

Dolphin-2.2.1-Mistral-7B

2023. Dolphin-2.2.1-Mistral-7B . https://huggingface.co/cognitivecomputations/dolphin-2.2.1-mistral-7b

work page 2023
[8]

Hercules-2.5-Mistral-7B

2023. Hercules-2.5-Mistral-7B . https://huggingface.co/Locutusque/Hercules-2.5-Mistral-7B

work page 2023

Show all 54 references

[9]

MetaMath-Mistral-7B

2023. MetaMath-Mistral-7B . https://huggingface.co/meta-math/MetaMath-Mistral-7B

2023
[10]

Speechless-Code-Mistral-7b-V1.0

2023. Speechless-Code-Mistral-7b-V1.0. https://huggingface.co/uukuguy/speechless-code-mistral-7b-v1.0

2023
[11]

Wizard-LM-Chinese-instruct-evol

2023 a . Wizard-LM-Chinese-instruct-evol

2023
[12]

WizardLM-evol-instruct-V2-196k

2023 b . WizardLM-evol-instruct-V2-196k . https://huggingface.co/datasets/MaziyarPanahi/WizardLM_evol_instruct_V2_196k

2023
[13]

K.; Hayase, J.; and Srinivasa, S

Ainsworth, S. K.; Hayase, J.; and Srinivasa, S. 2023. Git re-basin: Merging models modulo permutation symmetries. Proc. Int. Conf. Learn. Repren

2023
[14]

Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

2021 arXiv
[15]

Chen, L.; Zaharia, M.; and Zou, J. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176

2023 arXiv
[16]

Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

2021 arXiv
[17]

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

2021 arXiv
[18]

Contributors, O. 2023. Opencompass: A universal evaluation platform for foundation models. GitHub repository

2023
[19]

M.; Gurevych, I.; and Khan, M

Daheim, N.; Möllenhoff, T.; Ponti, E. M.; Gurevych, I.; and Khan, M. E. 2024. Model Merging by Uncertainty-Based Gradient Matching. Proc. Int. Conf. Learn. Repren

2024
[20]

Dodge, J.; Ilharco, G.; Schwartz, R.; Farhadi, A.; Hajishirzi, H.; and Smith, N. 2020. Fine-Tuning Pretrained Language Models: Weight Initializations. Data Orders, and Early Stopping. arXiv

2020
[21]

Entezari, R.; Sedghi, H.; Saukh, O.; and Neyshabur, B. 2022. The role of permutation invariance in linear mode connectivity of neural networks. Proc. Int. Conf. Learn. Repren

2022
[22]

Goddard, C.; Siriwardhana, S.; Ehghaghi, M.; Meyers, L.; Karpukhin, V.; Benedict, B.; McQuade, M.; and Solawetz, J. 2024. Arcee's MergeKit: A Toolkit for Merging Large Language Models. arXiv preprint arXiv:2403.13257

2024 arXiv
[23]

Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; and Steinhardt, J. 2021 a . Measuring massive multitask language understanding. Proc. Int. Conf. Learn. Repren

2021
[24]

Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021 b . Measuring mathematical problem solving with the math dataset. Proc. Adv. Neural Inf. Process. Syst

2021
[25]

Huang, Y.; Bai, Y.; Zhu, Z.; Zhang, J.; Zhang, J.; Su, T.; Liu, J.; Lv, C.; Zhang, Y.; Fu, Y.; et al. 2024. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Proc. Adv. Neural Inf. Process. Syst

2024
[26]

T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A

Ilharco, G.; Ribeiro, M. T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A. 2023. Editing models with task arithmetic. Proc. Int. Conf. Learn. Repren

2023
[27]

Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023 a . Mistral 7B. arXiv preprint arXiv:2310.06825

2023 arXiv
[28]

Jiang, D.; Ren, X.; and Lin, B. Y. 2023. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561

2023 arXiv
[29]

T.; et al

Jiang, W.; Lin, B.; Shi, H.; Zhang, Y.; Kwok, J. T.; et al. 2023 b . Effective and Parameter-Efficient Reusing Fine-Tuned Models. arXiv preprint arXiv:2310.01886

2023 arXiv
[30]

S.; and Zettlemoyer, L

Joshi, M.; Choi, E.; Weld, D. S.; and Zettlemoyer, L. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1601--1611

2017
[31]

Kim, D.; Park, C.; Kim, S.; Lee, W.; Song, W.; Kim, Y.; Kim, H.; Kim, Y.; Lee, H.; Kim, J.; et al. 2023. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312.15166

2023 arXiv
[32]

B.; Granitzer, M.; and Lemmerich, F

Klabunde, M.; Amor, M. B.; Granitzer, M.; and Lemmerich, F. 2023. Towards Measuring Representational Similarity of Large Language Models. In UniReps: the First Workshop on Unifying Representations in Neural Models

2023
[33]

Li, H.; Zhang, Y.; Koto, F.; Yang, Y.; Zhao, H.; Gong, Y.; Duan, N.; and Baldwin, T. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212

2023 arXiv
[34]

Liu, J.; Gong, R.; Zhang, M.; He, Y.; Cai, J.; and Zhuang, B. 2024. ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models. arXiv:2406.09041

2024 arXiv
[35]

Lu, K.; Yuan, H.; Lin, R.; Lin, J.; Yuan, Z.; Zhou, C.; and Zhou, J. 2023. Routing to the expert: Efficient reward-guided ensemble of large language models. arXiv preprint arXiv:2311.08692

2023 arXiv
[36]

Luo, Z.; Xu, C.; Zhao, P.; Sun, Q.; Geng, X.; Hu, W.; Tao, C.; Ma, J.; Lin, Q.; and Jiang, D. 2024. WizardCoder: Empowering Code Large Language Models with Evol-Instruct. Proc. Int. Conf. Learn. Repren

2024
[37]

Mason-Williams, G.; and Dahlqvist, F. 2024. What Makes a Good Prune? Maximal Unstructured Pruning for Maximal Cosine Similarity. Proc. Int. Conf. Learn. Repren

2024
[38]

S.; and Raffel, C

Matena, M. S.; and Raffel, C. A. 2022 a . Merging models with fisher-weighted averaging. Proc. Adv. Neural Inf. Process. Syst

2022
[39]

S.; and Raffel, C

Matena, M. S.; and Raffel, C. A. 2022 b . Merging Models with Fisher-Weighted Averaging. Proc. Adv. Neural Inf. Process. Syst

2022
[40]

E.; Adi, Y.; Liu, J.; Remez, T.; Rapin, J.; et al

Roziere, B.; Gehring, J.; Gloeckle, F.; Sootla, S.; Gat, I.; Tan, X. E.; Adi, Y.; Liu, J.; Remez, T.; Rapin, J.; et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

2023 arXiv
[41]

Shnitzer, T.; Ou, A.; Silva, M.; Soule, K.; Sun, Y.; Solomon, J.; Thompson, N.; and Yurochkin, M. 2023. Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789

2023 arXiv
[42]

Stoica, G.; Bolya, D.; Bjorner, J.; Ramesh, P.; Hearn, T.; and Hoffman, J. 2024. Zipit! merging models from different tasks without training. Proc. Int. Conf. Learn. Repren

2024
[43]

Talmor, A.; Herzig, J.; Lourie, N.; and Berant, J. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo...

2019
[44]

Tang, A.; Shen, L.; Luo, Y.; Yin, N.; Zhang, L.; and Tao, D. 2024. Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. arXiv preprint arXiv:2402.00433

2024 arXiv
[45]

Toshniwal, S.; Moshkov, I.; Narenthiran, S.; Gitman, D.; Jia, F.; and Gitman, I. 2024. OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset. arXiv preprint arXiv: Arxiv-2402.10176

2024 arXiv
[46]

Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[47]

Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A

Wortsman, M.; Ilharco, G.; Gadre, S. Y.; Roelofs, R.; Gontijo-Lopes, R.; Morcos, A. S.; Namkoong, H.; Farhadi, A.; Carmon, Y.; Kornblith, S.; et al. 2022. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. Proc. In...

2022
[48]

Wu, C.; Zhang, X.; Zhang, Y.; Wang, Y.; and Xie, W. 2023. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454

2023 arXiv
[49]

A.; and Bansal, M

Yadav, P.; Tam, D.; Choshen, L.; Raffel, C. A.; and Bansal, M. 2024. Ties-merging: Resolving interference when merging models. Proc. Adv. Neural Inf. Process. Syst., 36

2024
[50]

Yang, E.; Wang, Z.; Shen, L.; Liu, S.; Guo, G.; Wang, X.; and Tao, D. 2024. Adamerging: Adaptive model merging for multi-task learning. Proc. Int. Conf. Learn. Repren

2024
[51]

T.; Li, Z.; Weller, A.; and Liu, W

Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

2023 arXiv
[52]

Yu, L.; Yu, B.; Yu, H.; Huang, F.; and Li, Y. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. Proc. Int. Conf. Mach. Learn

2024
[53]

Zhong, W.; Cui, R.; Guo, Y.; Liang, Y.; Lu, S.; Wang, Y.; Saied, A.; Chen, W.; and Duan, N. 2024. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. In Findings of the Association for Computational Linguistics: NAACL 2024

2024
[54]

Zhou, C.; and Yuqi, B. 2024. Chinese-Mistral: An Efficient and Effective Chinese Large Language Model. https://github.com/THU-ESIS/Chinese-Mistral

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

bert-base-multilingual-cased

2023. bert-base-multilingual-cased . https://huggingface.co/google-bert/bert-base-multilingual-cased

work page 2023

[4] [4]

Chinese-Mistral-7B-Instruct-v0.1

2023. Chinese-Mistral-7B-Instruct-v0.1 . https://huggingface.co/itpossible/Chinese-Mistral-7B-Instruct-v0.1

work page 2023

[5] [5]

CollectiveCognition-v1.1-Mistral-7B

2023. CollectiveCognition-v1.1-Mistral-7B . https://huggingface.co/teknium/CollectiveCognition-v1.1-Mistral-7B

work page 2023

[6] [6]

2023. Dolphin . https://huggingface.co/datasets/cognitivecomputations/dolphin

work page 2023

[7] [7]

Dolphin-2.2.1-Mistral-7B

2023. Dolphin-2.2.1-Mistral-7B . https://huggingface.co/cognitivecomputations/dolphin-2.2.1-mistral-7b

work page 2023

[8] [8]

Hercules-2.5-Mistral-7B

2023. Hercules-2.5-Mistral-7B . https://huggingface.co/Locutusque/Hercules-2.5-Mistral-7B

work page 2023