{"id":"33776669-52b9-428a-956b-c3f878a32a38","arxiv_id":"2412.15283","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Channel Merging clusters similar channel parameters from fine-tuned LLMs and reconstructs the selected expert at inference, matching unmerged accuracy with about 53% of the ensemble's parameters.","lead":"This paper proposes Channel Merging, a way to store several fine-tuned language models as shared low-level parameter groups and rebuild the right expert on the fly during inference. It cuts parameter storage by nearly half while keeping accuracy close to running the models separately.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Core proxy untested: cosine similarity of delta vectors is used to justify averaging, but no evidence shows merged channels preserve expert function.","rationale":"The reader's CONDITIONAL verdict hinges on the same unvalidated assumption. I agree that it is the most load-bearing point: if cosine similarity is not a faithful proxy for functional compatibility, the method's mechanism is unsupported, even if the empirical numbers happen to look good. I considered other potential issues: the 53% parameter figure is arithmetically consistent with K=2 groups, the index is negligible relative to model size, and the limitation that experts must share a backbone is explicitly stated. The fixed-λ concern is a concrete sub-instance of the proxy problem; it is testable independently but does not replace the functional check. The proposed test directly measures whether Eq. (3) followed by Eq. (5) reconstructs the expert's behavior on its own domain. If the test passes on this setup, the central claim gains support; if it fails, the reported performance needs an alternative explanation, and the paper's title claim of 'preserving specialization' would be misleading.","tokens_in":14359,"tokens_out":14755,"duration_ms":132945,"concrete_test":"Use the exact Mistral-7B setup from Table 2 (N=4 experts, K=2, λ=0.5). After Channel Merging, for each expert sample 500 queries from its fine-tuning domain (e.g., GSM8K for MetaMath, HumanEval for Speechless-Code, CommonSenseQA for Dolphin, CMMLU for Chinese-Mistral). Generate responses from the original expert and the reconstructed expert with identical hyperparameters (temperature, max tokens). Compare the output distributions: compute top-1 agreement rate and mean KL divergence between the two models' next-token distributions over the generated sequences. Also compute the average L2 norm difference between original and reconstructed weight matrices per layer. If top-1 agreement is >98% and KL divergence is <0.05, the cosine-similarity proxy is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Channel Merging preserves specialization at K/N storage—rests on the implicit premise that averaging delta vectors within K-Means clusters (Eq. 3) yields channel parameters functionally equivalent to each expert's original. The paper motivates this only with an observational analysis of cosine-similarity proportions (Figure 2) and an ablation showing KMeans beats random grouping (Table 5). Neither verifies that the reconstructed expert behaves like the original. The proxy is scale-sensitive: cosine similarity ignores the norm of the delta vectors, while Eq. (3) merges by summation with a fixed λ=0.5. If two deltas point in similar directions but have very different magnitudes, the merged vector's norm is dominated by the larger delta, and the smaller expert's channel is distorted. Moreover, λ=0.5 is only a proper averaging factor if every cluster has exactly two members; K-Means does not guarantee balanced clusters, and the paper does not report cluster sizes or adapt λ. Thus the reconstruction in Eq. (5) could deviate substantially from the original expert, and the reported downstream performance may reflect task robustness rather than preserved specialization. The appendix's comparison of cosine vs Euclidean vs Manhattan distance is not a functional test, and the 'similarity between merged experts' heatmap measures index overlap, not output behavior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Channel Merging, a method to merge multiple task-specific fine-tuned LLMs into a compact set of parameters. For each output channel, the delta vectors of the experts are clustered into K groups via K-Means based on cosine similarity; parameters within each cluster are merged with task arithmetic, and a small index per expert records the group assignment. At inference, the parameters of the activated expert are reconstructed by concatenating the appropriate group parameters. The paper reports that this approach matches unmerged experts on specialized reasoning, math, code, and Chinese benchmarks, and that with a task-specific router it reaches performance comparable to a full model ensemble while storing about 53% of the parameters. The method is evaluated against DARE, TIES, and BYOM on the Mistral-7B family.","tokens_in":14612,"tokens_out":6604,"duration_ms":62037,"significance":"The paper addresses a practically important problem: deploying multiple specialized LLMs without loading all of them into GPU memory. The channel-level clustering idea is simple and the storage-reduction argument is clear (K/N parameter ratio). The experiments cover four diverse domains and include useful ablations on granularity, clustering method, and number of groups. If the technical issues around the merging rule and the interpretation of the results are resolved, the method could be a practical contribution to model-merging and multi-expert deployment. However, the current support for the central claim that similarity-based channel averaging preserves expert specialization is incomplete, and one equation-level inconsistency needs clarification.","major_comments":[{"comment":"The merging rule Θk_i = P_i + λ Σ δt_i with a fixed λ=0.5 is not an averaging operation when clusters have sizes other than two. In particular, a singleton cluster would produce P_i + 0.5δ, i.e., a halved delta, yet Table 6 reports that with K=4 (where each cluster trivially contains one expert per channel) the results are exactly identical to the unmerged baseline. This is only possible if λ is effectively 1, or if pruning and scaling are not applied, for singleton clusters. The paper must clarify the exact formula used in this case, report cluster-size statistics for the K=2 setting, and either adapt λ to the cluster size (e.g., 1/|C|) or justify the fixed value empirically.","section":"Method, Eq. (3) and Table 6"},{"comment":"The load-bearing premise is that clustering channels by cosine similarity of delta vectors preserves expert function better than random grouping. Table 5 shows that Random grouping already achieves strong performance (e.g., Instruction avg 69.23 vs 69.88 for KMeans; Math avg 43.96 vs 44.95), so the benefit of similarity-based clustering over simply storing two mixed models is small on several tasks. The paper should report a more direct functional test, such as the agreement between the reconstructed expert's per-channel outputs and the original expert's outputs, or at least discuss why the similarity proxy matters beyond the storage reduction. Without this, the claim that 'preserving specialization' is the mechanism is not fully supported.","section":"Merging with Channel Similarity; Table 5"},{"comment":"The experimental support for the headline claims is weakened by the absence of error bars or significance tests. In Table 3 the difference between DARE-CM+router and the full ensemble is 0.01 on MMLU+CMMLU (54.42 vs 54.43) and 0.24 on AGIEval (38.01 vs 38.25), which is likely within run-to-run variation; the text calls these results 'comparable,' which is fair, but the paper also claims improvements over individual baselines with similar-sized gaps. In Table 2, DARE-CM beats the unmerged baseline on TriviaQA by 6.10 points (64.49 vs 58.39) with no mechanism explained. The authors should provide multiple seeds, error bars, or a statistical test, and explicitly discuss unexplained large gains.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"The text says 'where L denotes the concatenation operation' but the displayed equation uses a different symbol (a large operator that is not defined). Please align the notation and define the concatenation operator explicitly.","section":"Method, Eq. (5)"},{"comment":"The sentence 'the total number of parameters necessary is effectively diminished to KΨ/NΨ = K/N' is mathematically the ratio of new to old parameter counts, not the reduced count itself. Rephrase to avoid confusion, e.g., 'reduced from NΨ to KΨ, a factor of K/N.'","section":"Model size reduction analysis"},{"comment":"The terms 'Layer' and 'Model' granularities in Table 4 are not defined precisely. It should be stated clearly what is clustered at each granularity, e.g., whether the clustering is applied per layer to the entire weight matrix, or globally across layers. Without this, the ablation is difficult to interpret.","section":"Table 4 and surrounding text"},{"comment":"The heatmap in Figure 5 measures the fraction of channels where two experts share the same group index, which is an index-overlap measure, not a measure of functional or output similarity. Please clarify this in the caption and text to avoid overstating what the figure establishes.","section":"Appendix, 'The similarity between merged experts'"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution, but the λ inconsistency between Eq. (3) and the K=4 results in Table 6 is a technical correctness issue that must be fixed before the method can be trusted. I also recommend that the authors add a direct functional evaluation of the reconstruction or temper the 'preserving specialization' narrative, since the random-clustering ablation suggests a large part of the benefit may come from simply storing two mixed models rather than from the specific similarity-based grouping."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before the rush of merging papers hits: it takes partial merging down to the channel level. Instead of merging whole models or keeping whole layers, it clusters each output channel's delta vectors across experts via K-Means, merges within clusters, and stores an index of which cluster each expert's channel went to. At inference you reconstruct the activated expert by concatenating the right cluster slots. That is a real idea, and the storage math is clear: K groups per channel gives K/N of the ensemble footprint. The authors also do the right ablations: channel vs layer vs model granularity, KMeans vs random grouping, cosine vs Euclidean vs Manhattan, group count, and pruning ratio. The channel-level results beat model-level merging across the board, and the router results at 53% parameters are worth taking seriously. The limitations section is honest about needing a shared pretrained base and about parameter growth vs one-size-fits-all. This is a useful paper for anyone working on multi-expert serving or model merging. Now the soft spots. First, no error bars or significance tests anywhere, and some numbers are suspicious. DARE-CM jumps TriviaQA from 58.39 to 64.49, above the unmerged baseline, with no mechanistic story. That suggests either the method is doing something more than preserving specialization, or the evaluation is noisy. Second, K and lambda look selected on the test benchmarks. K=2 is used for N=4, lambda=0.5, and the pruning ratio is tuned to 30% using the same datasets. That is hyperparameter fitting by another name. Third, the paper never addresses the cluster-size problem. With K=2 and N=4, clusters can be 2-and-2, but they could also be 1-and-3, and lambda=0.5 then halves a singleton expert's delta instead of preserving it. The paper reports no cluster-size statistics and no variant with per-cluster lambda. The stress-test note is right that cosine similarity is only a proxy; the appendix compares distance metrics, not whether reconstructed outputs match the originals. The similarity heatmap measures index overlap, not function. Fourth, ME-Switch, an earlier paper by largely the same group, is cited but never positioned against; the router story overlaps heavily and the paper should say what is new beyond it. None of this kills the approach—the empirical pattern is consistent and the ablations show the clustering matters—but the paper currently argues from correlation. A serious referee should ask for error bars, a validation split for hyperparameters, a cluster-size report, and a functional test (logit or output similarity) of the reconstruction. I would send this to review rather than desk-reject; it deserves a response, but it needs a substantive revision before it is publishable as is.","headline":"A genuinely new channel-level merging idea with solid empirical gains, but the mechanism is under-validated and the paper needs a round of hardening.","tokens_in":752,"tokens_out":771,"would_cite":true,"duration_ms":28252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Channel Merging claims that merging fine-tuned LLM experts at the channel level rather than the whole-model level eliminates most parameter conflicts, so a zoo of N experts can be stored as K groups (K<N) and each expert reconstructed on…","keywords":["model merging","large language models","channel-wise merging","task arithmetic","parameter-efficient inference","expert routing","K-Means clustering","cosine similarity"],"falsifier":"Construct two fine-tuned experts on the same base model whose channel delta vectors are highly cosine-similar but functionally opposed (e.g., one channel encodes a feature that the other suppresses), then compute the distance between the merged layer's output and each expert's original output on that expert's own data; if the merged channel reproduces neither expert's behavior, the similarity proxy is refuted. Concretely, a reader could pick channels with cosine similarity above 0.9 and measure per-channel activation distances before and after merging as in Eq. (3).","tokens_in":1711,"feed_emoji":"🧩","tokens_out":1847,"duration_ms":49400,"temperature":0.7,"pith_summary":"The paper argues that the usual model-merging trade-off—performance loss from parameter conflicts versus storage cost from keeping every expert—can be avoided by merging at a finer granularity. Instead of averaging whole expert models, it clusters per-channel delta parameters across experts by cosine similarity, merges only within each cluster, and stores an index that lets each expert be reconstructed at inference time. It claims this reduces storage from N full experts to K full groups (K<N), with specialized-task performance matching unmerged experts. With a small task router, the merged model reportedly reaches ensemble-level accuracy on general tasks while using only 53% of ensemble parameters. This matters because deploying many specialized LLMs is memory-hungry, and prior merging methods degrade as the number of experts grows.","feed_headline":"Merging LLM experts by channel cuts storage to 53 percent","feed_subtitle":"Experts keep their specialized tasks while total parameters drop to 53 percent of an ensemble.","key_machinery":"The central object is the per-output-channel clustering of delta parameters. For the i-th output channel, the N experts' delta vectors $δ_i = {δ_i^{t_1}, ..., δ_i^{t_N}} ∈ R^{N × I}$ are grouped by K-Means into K clusters $C_i^1, ..., C_i^K$ using cosine similarity, and cluster members are merged with task arithmetic: $Θ_i^k = P_i + λ ∑_{δ_i^{t_n} ∈ C_i^k} δ_i^{t_n}$. Each expert stores an index set $S^{t_n}$ recording which cluster holds each channel; during inference the activated expert's parameters are reconstructed by concatenating the relevant group parameters. The storage analysis reduces the total from $NΨ$ to $KΨ$ parameters, a factor of $K/N$, because the index overhead scales only with the number of channels.","core_discovery":"The central claim is that channel-level merging preserves specialist performance that model-level merging loses. For each output channel, the delta vectors of all N experts are clustered by cosine similarity into K groups using K-Means; within each group the deltas are averaged onto the pretrained weights. Because only highly similar parameters are averaged, parameter conflicts largely disappear, and because each expert keeps an index of which group each of its channels came from, activating an expert is just a lookup-and-concatenate operation. The paper reports that with K=2 and N=4 experts, the total parameter count drops to 53% of a full ensemble (14.3B vs 26.8B) while matching or exceeding unmerged experts on English reasoning, math, code, and Chinese benchmarks, and matching an ensemble on general tasks when paired with a router.","pith_inferences":["Because the storage ratio K/N improves as N grows, the method's advantage should become more pronounced with larger expert zoos, provided the clustering assumption still holds; the paper's experiments only go up to six experts.","The paper motivates cosine similarity by citing a link to activation similarity, but clusters raw parameters rather than activations; a natural stronger variant would cluster channels by their functional effect on layer outputs, which the paper does not test.","The task router is trained with supervision from the experts' original fine-tuning datasets, so out-of-distribution queries are an untested regime; a misrouted query would reconstruct the wrong expert and could negate the accuracy benefit.","The layer-wise similarity analysis is performed on only two model families; whether the method transfers to other base models or to experts with non-overlapping fine-tuning distributions is an open question the paper does not address."],"forward_implications":["A zoo of N task-specialized LLMs can be deployed with roughly K/N of the original storage, and the ratio improves as N grows because K stays constant.","Channel-level merging can be layered on top of existing pruning-based merging methods: the paper shows DARE-CM and TIES-CM outperform their model-level counterparts on every reported task.","Adding a lightweight task router to select which expert to reconstruct preserves the specialized-task gains while improving general-task accuracy over any single expert.","The method's storage advantage holds even when the number of experts increases, unlike partial-merging approaches such as BYOM that must keep separate parameters for each expert.","The ablation across granularities (channel, layer, model) indicates that finer granularity is the reason for the reduced performance degradation, not the specific clustering algorithm."],"supporting_citations":[{"why":"Supplies task arithmetic, the merging rule used in Eq. (3) for combining cluster members.","marker":"Ilharco et al. 2023"},{"why":"Defines DARE, one of the two base merging algorithms the paper augments with channel merging.","marker":"Yu et al. 2024"},{"why":"Defines TIES, the other base algorithm whose pruning step is combined with channel merging.","marker":"Yadav et al. 2024"},{"why":"Provides BYOM, the partial-merging baseline that channel merging is compared against on storage efficiency.","marker":"Jiang et al. 2023b"},{"why":"Supplies the task-specific router paradigm that the paper adopts for general-task evaluation.","marker":"Liu et al. 2024"},{"why":"Cited to justify that high cosine similarity between parameters correlates with similar layer activations, motivating the similarity metric.","marker":"Mason-Williams and Dahlqvist 2024"}],"fun_headline_variants":["Channel merging keeps expert skills at 53% storage","Merge LLM experts by channel, retain 53% memory","Channel-wise expert merge: 53% size, full skills","Specialized LLM experts, merged by channel, 53% footprint"],"cache_read_input_tokens":17280,"weakest_assumption_plain":"The method assumes that two channels whose fine-tuning changes point in nearly the same direction can be averaged without losing either expert's behavior, even though the paper never checks the averaged channels against the original outputs.","fun_headline_variants_meta":{"raw":{"variants":["Channel merging keeps expert skills at 53% storage","Merge LLM experts by channel, retain 53% memory","Channel-wise expert merge: 53% size, full skills","Specialized LLM experts, merged by channel, 53% footprint"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1530,"prompt_tokens":966,"completion_tokens":564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":493}},"tokens_in":582,"tokens_out":564,"duration_ms":5529,"temperature":1.0,"reasoning_tokens":493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:34:45.581287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct two fine-tuned experts on the same base model whose channel delta vectors are highly cosine-similar but functionally opposed (e.g., one channel encodes a feature that the other suppresses), then compute the distance between the merged layer's output and each expert's original output on that expert's own data; if the merged channel reproduces neither expert's behavior, the similarity proxy is refuted. Concretely, a reader could pick channels with cosine similarity above 0.9 and measure per-channel activation distances before and after merging as in Eq. (3).","supporting_citations":[{"cited_title":"T.; Wortsman, M.; Gururangan, S.; Schmidt, L.; Hajishirzi, H.; and Farhadi, A","cited_arxiv_id":null,"evidence_quote":"Supplies task arithmetic, the merging rule used in Eq. (3) for combining cluster members."},{"cited_title":"A.; and Bansal, M","cited_arxiv_id":null,"evidence_quote":"Defines TIES, the other base algorithm whose pruning step is combined with channel merging."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited to justify that high cosine similarity between parameters correlates with similar layer activations, motivating the similarity metric."}],"review_version":1}