{"id":"a0e70eeb-6f8b-4f31-a13e-364bab68b662","arxiv_id":"2608.12842","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CABS+ prunes task vectors with a conflict-aware, balanced sparsification step and sets merging coefficients through CMA-ES search, improving efficiency and average merged-model performance.","lead":"CABS+ merges separately trained AI models into one multi-task model by pruning away conflicting parameters and using an evolutionary search to set each model's contribution. It reports higher average performance than AdaMerging and WUDIMerging across 27 datasets while using less GPU memory than AdaMerging.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AWA's fitness function (Eqs. 5–9) requires ground-truth labels on test data, which the paper never discloses; the 16.97% gain over AdaMerging may reflect a strictly stronger information regime rather than better merging.","rationale":"I agree with the reader's weakest_assumption: the AWA fitness function requires ground-truth labels, and this is the single most load-bearing concern about the central performance claim. Under the natural reading of Eqs. (5)–(9), L_t(θ(λ)) is a supervised loss on test samples; computing it requires ground-truth labels for every task. AdaMerging's objective is unsupervised entropy on test samples (Section II-B). The comparison in Fig. 3 is therefore not on equal terms: CABS+ receives strictly more information. If the fitness were switched to an unsupervised entropy objective, the 16.97% figure could weaken substantially; in the worst case the advantage disappears and the main contribution reduces to 'supervised test-time tuning helps,' not the claimed architectural improvement. This is the most load-bearing point because the performance gain is the first and primary element of the central claim.\n\nThe reader's CONDITIONAL verdict remains the right calibration. The concern is serious but addressable: disclose the supervision, run the entropy-only ablation, or reframe the contribution. The paper deserves credit elsewhere: the CABS+ vs CABS deltas are reported openly and are mostly small and consistent; the mergeability study (Section V) with RSS is largely independent of the AWA concern; and the efficiency measurements are concrete, although some abstract phrasing overclaims their scope (on RoBERTa, CABS+ uses 3.77/6.69 ≈ 56% of AdaMerging's memory, not <25%; on Mistral it uses 15.95GB vs WUDI's 12.04GB).\n\nSecondary weaknesses, also noted by the reader, reinforce the need for the ablation: the headline 16.97% is an average of per-scenario relative gains inflated by one scenario (GPT-2, 6 tasks; AdaMerging at 39.49) contributing +74.00%, and the code link is malformed ('CABS Plus-70C1' contains a space) with no commit hash. These do not change the verdict but support keeping the paper conditional on reproducibility and consistent reporting.","tokens_in":22722,"tokens_out":15521,"duration_ms":141535,"concrete_test":"Re-run the CABS+ pipeline (CABS pruning plus CMA-ES coefficient search with identical hyperparameters K=6, G=50, σ=0.05, bounds [0.1, 2]) using an unsupervised entropy fitness F(λ) = Σ_t H_t(θ(λ)) on unlabeled test samples, mirroring AdaMerging's objective. Recompute the eight per-scenario averages in Fig. 3 and the aggregate improvements over AdaMerging and WUDIMerging. If the aggregate gains drop substantially from 16.97% / 12.93% (for example, below 3–5%) or change sign, the label supervision is the source of the claimed advantage and the comparison is not on equal terms; if the gains persist, the label concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central performance claim is that CABS+ outperforms AdaMerging and WUDIMerging by 16.97% and 12.93% (Abstract, Fig. 3). The most load-bearing link in that claim is the AWA fitness function, Eqs. (5)–(9). The objective F(λ) = Σ_t f_t(λ) is built from L_t(θ(λ)), a task loss evaluated on test samples, with a relative change Δ_t normalized by the initial loss L_base,t (Eqs. 6–7). A task loss of this kind is computed against ground-truth labels (cross-entropy or equivalent); the paper nowhere defines L_t or states that labels are required. It does state that AdaMerging 'minimizes the unsupervised entropy on test samples' (Section II-B), so CABS+ and AdaMerging are optimized under different information regimes: supervised test labels versus unlabeled test entropy. The reported advantage could therefore be caused by the label signal alone. Because the headline improvement is an average of per-scenario relative gains (Fig. 3), with one scenario (GPT-2, 6 tasks, where AdaMerging drops to 39.49) contributing a +74.00% relative gain that dominates the average, the magnitude of the claimed advantage rests even more directly on an asymmetric comparison. If AWA used the same label-free objective as AdaMerging, the 16.97% gap could shrink or vanish; this is the assumption to test. Additionally, optimizing coefficients on test labels and then reporting accuracy on the same test set means the reported numbers are not unbiased estimates of generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CABS+, an extension of the prior CABS model-merging method. CABS+ replaces CABS's grid search for task-vector scaling coefficients with Adaptive Weight Allocation (AWA), a gradient-free CMA-ES-style search using boundary constraints and an asymmetric, baseline-normalized fitness function. The authors report that CABS+ outperforms AdaMerging and WUDIMerging by 16.97% and 12.93% on average across eight experimental scenarios, uses less than 25% of AdaMerging's GPU memory on a 7B-parameter model, and reduces merging time by roughly 4x relative to WUDIMerging. The paper also contributes an empirical study of factors affecting mergeability and proposes a Relative Synergy Score (RSS) for quantifying whether merging is synergistic or destructive.","tokens_in":23010,"tokens_out":3028,"duration_ms":33794,"significance":"If the empirical claims hold, CABS+ would be a practically useful contribution: it offers a training-free, low-memory coefficient search that scales to billion-parameter models, and it provides evidence on when merging is likely to succeed through the RSS analysis. The evaluation is broad, spanning 27 datasets, five model families, and both language and vision models, and the authors make code available. The asymmetric fitness function is a reasonable way to prevent high-loss tasks from dominating coefficient search. However, the central performance comparison is weakened by a disclosed-information asymmetry: AWA appears to optimize a supervised loss on test labels, while AdaMerging is unsupervised. The headline improvement is also computed as an average of per-scenario relative gains rather than an aggregate accuracy difference, and the coefficients are selected on the same test data used for final accuracy reporting. These issues are load-bearing for the paper's main claim, though they are addressable with additional experiments and more careful reporting.","major_comments":[{"comment":"The AWA fitness function is defined through L_t(theta(lambda)), a task loss evaluated on test samples, but the paper never defines L_t or states that it requires ground-truth labels. A task loss such as cross-entropy is computed against labels for every task. This is a strictly stronger information regime than AdaMerging, which the paper explicitly describes as minimizing 'unsupervised entropy on test samples' (Section II-B). The 16.97% improvement over AdaMerging may therefore reflect the availability of labeled test data rather than a better merging mechanism. The authors should disclose the label requirement, state exactly which loss is used, and, ideally, compare CABS+ with a label-free variant (e.g., entropy-based fitness) to place both methods on equal footing.","section":"Section III-C, Eqs. (5)-(9)"},{"comment":"The headline claim that CABS+ improves overall performance by 16.97% over AdaMerging is actually the unweighted average of eight per-scenario relative improvements, not an aggregate performance difference. One scenario (GPT-2, six tasks) contributes a +74.00% relative gain because AdaMerging drops to 39.49 while CABS+ reaches 68.71, and this single outlier dominates the average. The absolute differences are much smaller in other scenarios (for example, +1.07% on the Mistral LLM Leaderboard suite). The authors should report aggregate accuracies, absolute improvements, or both, and should state explicitly that 16.97% is an average of relative gains rather than an overall accuracy improvement.","section":"Figure 3, Section IV-F"},{"comment":"Because AWA selects lambda* by minimizing a loss on test samples (Eqs. 5-9) and the final accuracy is then reported on the same test set, the reported numbers are in-sample estimates rather than unbiased estimates of generalization. This is a methodological concern for all of the main tables, not just the efficiency comparison. The authors should use a validation split for coefficient selection and a separate test split for reporting, or add a sentence explaining why this in-sample selection does not materially affect the conclusions.","section":"Section IV-A and IV-C"}],"minor_comments":[{"comment":"The paper states that each configuration is evaluated three times and the average is reported, but no variance or standard deviation is given in any table or figure. Reporting error bars or standard deviations would strengthen the stability claim that is made repeatedly.","section":"Section IV-A"},{"comment":"The figure mixes performance bars and improvement percentages in one plot; the labels are dense and some values (e.g., the Overall Improvement row) appear to be computed as averages of the preceding columns without this being explained in the caption. The caption should define the computation.","section":"Figure 3"},{"comment":"The phrase 'improves overall performance by 16.97% and 12.93%' is ambiguous; it should say 'average relative improvement over eight scenarios' to avoid overstatement.","section":"Abstract and Section VI"},{"comment":"The anonymous source-code URL contains a space ('CABS Plus-70C1'); this is likely a typo and should be corrected.","section":"Source Code Availability"},{"comment":"The penalty coefficients alpha=100 and beta=1, the bounds l=0.1 and u=2, population size K=6, and generation count G=50 are chosen without sensitivity analysis or justification. Adding a small ablation or parameter-sensitivity table would help establish that the results are not sensitive to these choices.","section":"Section III-C, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core mechanism is plausible and the efficiency results are credible, but the main accuracy comparison is currently not on equal terms with AdaMerging because of the supervised-loss versus unsupervised-entropy asymmetry. The 16.97% headline also overstates the aggregate gain. Both issues can be fixed within the manuscript's scope, so I do not recommend rejection, but the revision needs to be substantive rather than cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a legitimate incremental extension of the authors' own CABS—replacing grid search with CMA-ES and an asymmetric loss penalty, plus an empirical mergeability study. The efficiency result is the strongest part: on Mistral, CABS+ uses 15.95 GB GPU versus 66.70 GB for AdaMerging and takes 1h versus 4h for WUDIMerging while staying within about 0.1 points of WUDI. That is a concrete practical gain for 7B-scale merging on a V100.\n\nWhat is new: AWA replaces exponential grid search with K×G evaluations, and the mergeability section with RSS is a useful pre-merge screening heuristic. The experiments span 27 datasets and 5 model families, which supports the core claim that CABS sparsification plus evolutionary weight allocation works. The citation pattern is fine; prior evolutionary optimization for merging is cited.\n\nThe soft spots are real but fixable. The AWA objective in Eqs. (5)–(9) is built from L_t, a task loss on test samples, which requires ground-truth labels. The paper never says that. AdaMerging, by contrast, minimizes unsupervised entropy on test samples. That is a different information regime, and the reported 16.97% gain over AdaMerging may be driven by the label signal rather than by better merging. The stress-test concern lands. The fix is to disclose the label requirement and ideally run a label-free variant of AWA.\n\nSecond, the headline number is an average of per-scenario relative improvements, not an average accuracy gain. The GPT-2 six-task scenario contributes a +74.00% relative gain because AdaMerging collapses there; without it the average looks much less dramatic. The per-scenario table is honest, but the abstract's single number overstates the typical improvement.\n\nThird, optimizing coefficients on test labels and then reporting accuracy on the same test set means the numbers are not clean estimates of generalization. This is common in the merging literature, but it should be stated.\n\nMinor: the code link is malformed and no commit hash is given, so the reproducibility claim is currently empty.\n\nThe central mechanism is not circular; the experiments are fresh. This paper deserves a serious referee, not a desk reject. I'd send it out and ask for label-regime disclosure, a label-free ablation, and a corrected headline metric.","headline":"Solid incremental extension of CABS with a real efficiency win, but the headline gain over AdaMerging is inflated by an undisclosed label-regime difference and by averaging relative improvements.","tokens_in":23598,"tokens_out":3136,"would_cite":true,"duration_ms":31020,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CABS+ claims that conflict-aware sparsification followed by a gradient-free covariance-adapted search for task scaling coefficients makes multi-task model merging more accurate and cheap enough for a single V100 GPU.","keywords":["model merging","task vectors","conflict-aware sparsification","adaptive weight allocation","CMA-ES","gradient-free optimization","model mergeability","Relative Synergy Score"],"falsifier":"Run CABS+ on a task suite while withholding labels from the fitness evaluation, using only unlabeled inputs as AdaMerging does, and compare average task accuracy; if the method depends on labels, it should degrade toward or below AdaMerging's level, contradicting the claimed 16.97% advantage. Separately, measure CABS+ GPU memory while doubling the number of task vectors on a 7B model; if memory usage grows linearly rather than staying flat, the claimed O(1) memory advantage fails.","tokens_in":22450,"feed_emoji":"🧩","tokens_out":5792,"duration_ms":52770,"temperature":0.7,"pith_summary":"The paper claims model merging can be both more effective and dramatically cheaper by pairing conflict-aware sparsification with a gradient-free, memory-light search for task scaling coefficients. It extends the earlier CABS method, replacing exponential grid search with a CMA-ES-style Adaptive Weight Allocation and an asymmetric fitness function that keeps losses from a few tasks from dominating the coefficient search. Across 27 datasets and 5 model families, it reports average gains of 16.97% over AdaMerging and 12.93% over WUDIMerging, using less than 25% of AdaMerging's GPU memory and achieving nearly a 4x time speedup over WUDIMerging. The paper also proposes the Relative Synergy Score as a pre-merge measure of how compatible two task models are, and identifies six factors—learning rate, training epochs, task and data heterogeneity, architecture, and scale—that predict mergeability. If these results hold, model merging becomes practical on modest GPUs and can be guided by measurable compatibility checks before any merging compute is spent.","feed_headline":"Merging models gets 4x faster and uses under 25% of the GPU memory","feed_subtitle":"A conflict-aware sparsifier plus covariance-adaptive search claims 16.97% higher average performance than AdaMerging.","key_machinery":"The central object is the merged model $$\\theta_{\\text{merged}} = \\theta_{\\text{base}} + \\sum_{t=1}^{T} \\lambda_t^* \\, \\tilde{\\tau}_t,$$ where $\\tilde{\\tau}_t$ are task vectors after CABS's sequential, block-wise $n{:}m$ pruning. Three pieces carry the argument. First, the sequential mask generated from task vector A and used to prune task vector B makes the pruned vectors orthogonal under the Frobenius inner product, removing the cross term from the merged update norm and enabling independent scaling of task contributions. Second, the AWA search distribution is a multivariate normal updated by CMA-ES, and its learned covariance can capture correlations between task coefficients that grid search cannot exploit. Third, the asymmetric fitness function $f_t(\\lambda)$, which penalizes a task's relative loss increase with $\\alpha=100$ and rewards a loss decrease with only $\\beta=1$, prevents any single task with a large loss scale from dominating the search.","core_discovery":"CABS+ argues that the main obstacles to task-vector model merging—parameter overlap, scale differences among task losses, and exponential coefficient search—can be removed in one pipeline. Conflict-aware sparsification performs sequential pruning: task vector A is pruned with block-wise n:m sparsification, its mask is used to remove overlapping positions in task vector B, and sign selection plus averaging handles any unavoidable residual overlap. The paper states this makes the pruned task vectors orthogonal in the Frobenius inner product, eliminating the cross term in the merged update norm and letting each task's contribution be scaled independently. Adaptive Weight Allocation then optimizes the scaling coefficients with a gradient-free covariance-matrix-adaptation evolution strategy, bounding coefficients to [0.1, 2] and scoring each candidate by a relative, asymmetric loss change: a task whose loss rises is penalized 100 times more than a task whose loss falls. The paper reports that the resulting method outperforms AdaMerging and WUDIMerging by 16.97% and 12.93% on average, stays competitive with or above CABS on every tested configuration, and in large-model experiments uses less than 25% of AdaMerging's GPU memory and roughly a quarter of WUDIMerging's merging time.","pith_inferences":["Because the AWA fitness function requires task-specific losses, the method as described needs labeled evaluation data for every task; an unsupervised analogue using batch entropy on unlabeled inputs would extend CABS+ to settings where labels are unavailable, and would make the comparison with AdaMerging fully apples-to-apples.","The orthogonality argument predicts that CABS+ should gain most when task vectors are highly conflicting; a targeted experiment merging deliberately homogeneous or overlapping task vectors would test whether the advantage shrinks as expected.","The RSS findings imply a practical decision rule: estimate RSS on a small validation subset before merging and decline merges with strongly negative scores, saving the compute that would otherwise be spent on destructive merges.","Since CMA-ES learns correlations among task coefficients, CABS+ may scale to task counts far beyond the six tested here, but population size and iteration count will likely need to grow with dimensionality; that scaling behavior is not demonstrated in the paper."],"forward_implications":["Merging seven-billion-parameter models becomes feasible on a 32GB V100 GPU: CABS+ keeps GPU memory at inference level, reported as O(1) in the number of tasks, while AdaMerging's gradient graph grows linearly with tasks, depth, and sequence length.","The number of coefficient evaluations drops from $O(S^T)$ for grid search to $O(KG)$, so merging many tasks no longer requires exhaustive search over a coefficient grid.","The asymmetric fitness function should yield more balanced per-task performance rather than merely a higher average, because any candidate that raises a task's loss is heavily penalized.","Relative Synergy Score computed before merging can guide model selection: the paper's empirical study predicts that similar task semantics, similar data distributions, moderately tuned learning rates, and larger model scales all improve mergeability.","If the efficiency and accuracy results generalize, model merging becomes a practical substitute for retraining in multi-task deployment, especially on memory-limited hardware."],"supporting_citations":[{"why":"Defines task vectors as the difference between finetuned and base weights, which is the object CABS+ prunes and merges.","marker":"[7]"},{"why":"Provides the AdaMerging baseline and the memory-heavy gradient-graph coefficient optimization that motivates AWA's gradient-free design.","marker":"[8]"},{"why":"Provides the WUDIMerging baseline, the main competing coefficient-optimization method and the reference for the reported 4x speedup.","marker":"[9]"},{"why":"Documents that magnitude pruning can underperform random sparsification in merging, motivating the balanced sparsification strategy in CABS.","marker":"[12]"},{"why":"Supplies the sign-selection and parameter-averaging treatment that CABS+ uses for unavoidable residual overlap between task vectors.","marker":"[13]"},{"why":"Supplies the CMA-ES algorithm that AWA adapts, providing the covariance-matrix evolution and step-size control used in the search.","marker":"[32]"}],"fun_headline_variants":["CABS+ merges models 4x faster with 75% less GPU memory","CABS+: 4x faster merging, 25% GPU memory, 17% better performance","Model merging upgrade: CABS+ runs 4x faster, uses 25% memory, beats AdaMerging by 17%","CABS+ accelerates model merging 4x, cuts GPU memory to 25%, improves by 17%","CABS+ cuts merging time and GPU memory by 75%, plus 17% performance gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The AWA search scores each candidate by the loss of the merged model on test samples, so it requires labeled evaluation data for every task; if such labels are unavailable, the coefficient search cannot be run as described.","fun_headline_variants_meta":{"raw":{"variants":["CABS+ merges models 4x faster with 75% less GPU memory","CABS+: 4x faster merging, 25% GPU memory, 17% better performance","Model merging upgrade: CABS+ runs 4x faster, uses 25% memory, beats AdaMerging by 17%","CABS+ accelerates model merging 4x, cuts GPU memory to 25%, improves by 17%","CABS+ cuts merging time and GPU memory by 75%, plus 17% performance gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001425,"raw_usage":{"total_tokens":5841,"prompt_tokens":1126,"completion_tokens":4715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":742,"completion_tokens_details":{"reasoning_tokens":4583}},"tokens_in":742,"tokens_out":4715,"duration_ms":31161,"temperature":1.0,"reasoning_tokens":4583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:09:07.555304+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CABS+ on a task suite while withholding labels from the fitness evaluation, using only unlabeled inputs as AdaMerging does, and compare average task accuracy; if the method depends on labels, it should degrade toward or below AdaMerging's level, contradicting the claimed 16.97% advantage. Separately, measure CABS+ GPU memory while doubling the number of task vectors on a 7B model; if memory usage grows linearly rather than staying flat, the claimed O(1) memory advantage fails.","supporting_citations":[{"cited_title":"Editing models with task arithmetic,","cited_arxiv_id":null,"evidence_quote":"Defines task vectors as the difference between finetuned and base weights, which is the object CABS+ prunes and merges."},{"cited_title":"Adamerging: Adaptive model merging for multi- task learning,","cited_arxiv_id":null,"evidence_quote":"Provides the AdaMerging baseline and the memory-heavy gradient-graph coefficient optimization that motivates AWA's gradient-free design."},{"cited_title":"Whoever started the interference should end it: Guiding data-free model merging via task vectors,","cited_arxiv_id":null,"evidence_quote":"Provides the WUDIMerging baseline, the main competing coefficient-optimization method and the reference for the reported 4x speedup."},{"cited_title":"Language models are super mario: Absorbing abilities from homologous models as a free lunch,","cited_arxiv_id":null,"evidence_quote":"Documents that magnitude pruning can underperform random sparsification in merging, motivating the balanced sparsification strategy in CABS."},{"cited_title":"TIES- Merging: Resolving Interference When Merging Models,","cited_arxiv_id":null,"evidence_quote":"Supplies the sign-selection and parameter-averaging treatment that CABS+ uses for unavoidable residual overlap between task vectors."}],"review_version":1}