{"id":"5f19b3d4-9241-49f4-9b64-eeb7f931945b","arxiv_id":"2506.18135","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A training-free, sample-wise model merging method that rescales task vectors based on representation similarity outperforms several static merging baselines on vision and language benchmarks.","lead":"This paper introduces SE-Merging, a training-free method that finds which task a test input belongs to by comparing internal representations of a merged model with those of the task experts, then re-weights the merge for that input. A generalist might read it to see whether self-identification by representation similarity can replace trained routers in multi-task model merging.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may hinge on undisclosed layer and coefficient selection on the test set; without a held-out hyperparameter choice or error bars the central empirical claim is not yet verifiable.","rationale":"The central claim is empirical: SE-Merging achieves significant performance improvements without additional training. The only component separating SE-Merging from static merging is the sample-wise rescaling driven by representation similarity at layer ℓ. Therefore, if ℓ and λ are chosen using test accuracy, the comparison against AdaMerging and TIES-Merging is not a fair out-of-sample evaluation. The reader's weakest assumption identifies Hypothesis 1 and the router accuracy as the key risk; my concern is a sharper formulation of the same risk: even a perfect router would not make the headline claim reproducible if the router's layer was tuned on the test set. Conversely, if the protocol is fixed and the gains persist on held-out data, the claim is solid. This is a checkable condition, which is why conditional acceptance is appropriate. I do not move the verdict because the reader already assigned CONDITIONAL and their requested additions—code, hyperparameters, error bars, and a missing language baseline—would resolve the concern. I mark agreement as partial because I place more weight on the test-set selection of ℓ and λ than on the abstract validity of Hypothesis 1, but the two are closely related and the practical fix is the same: disclose and validate the selection protocol.","tokens_in":14760,"tokens_out":4206,"duration_ms":44893,"concrete_test":"Re-run Algorithm 2 on ViT-B/32, ViT-L/14, and GPT-2 under a fixed protocol: (i) split each test set into a validation split and a held-out split; (ii) select ℓ and λ on the validation split only; (iii) report mean and standard deviation over at least three fine-tuning seeds on the held-out split. Additionally, sweep ℓ across all layers and λ in {0.2, 0.3, 0.4} and plot the resulting Table 1–3 averages. If the reported numbers correspond to the maximum of this sweep, or if some choices of ℓ underperform the relevant baseline, the headline improvements are not established as training-free, selection-free gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 2 takes as input a layer ℓ and a base coefficient λ (Algorithm 2, line 1), yet the paper never reports the values used for Tables 1–3 or how they were chosen. Figure 3, which validates Hypothesis 1, is computed on the same eight test sets used for evaluation, and the high acc@1 values are shown only for selected deep layers ('12-th layer for ViT-B/32 and 24-th layer for ViT-L/14'). If ℓ (and λ) were selected by maximizing router accuracy or final accuracy on these test sets, then SE-Merging is effectively using test labels to choose its routing layer, and the reported gains of +3.86, +0.57, and +6.86 over AdaMerging and TIES-Merging would partly reflect in-sample selection rather than a training-free advantage. This concern is load-bearing because the whole method reduces to the router: when Hypothesis 1 fails at the chosen layer, rescaling upweights the wrong task vector and can hurt accuracy. Section 5(a) concedes the hypothesis does not hold for all samples, but no analysis quantifies how often the router is wrong at the deployed ℓ or how final accuracy varies across layers. The absence of error bars further means the improvements on individual datasets cannot be distinguished from seed or run variance. Since the central claim is the reported performance improvement, this unreported selection step is the least secure link in the argument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies model merging from a representation perspective. It proposes the Representation Auto-Adaptation Hypothesis: for a sample from task T_i, the internal representation of a task-arithmetic merged model is closer to the representation of the expert fine-tuned on T_i than to other experts. Building on this, the authors introduce SE-Merging, a training-free, sample-wise method that computes per-sample merging coefficients by comparing inner representations of the merged model with those of each fine-tuned model, then rescaling the task vectors accordingly. The method is evaluated on eight vision tasks with ViT-B/32 and ViT-L/14 and on seven GLUE tasks with GPT-2, reporting average gains of +3.86, +0.57, and +6.86 points over AdaMerging or TIES-Merging. The paper also gives a short theoretical discussion connecting the hypothesis to the existing Weight Disentanglement condition.","tokens_in":15022,"tokens_out":4200,"duration_ms":42766,"significance":"If the empirical claims hold, SE-Merging is a simple and attractive contribution: it is training-free, compatible with static merging methods, and it turns a plausible representation-level observation into a concrete inference-time algorithm. The paper is also useful in that it makes a falsifiable hypothesis explicit and evaluates that hypothesis quantitatively via acc@1. However, the central performance claims currently lack essential experimental reporting: the main free inputs (layer ell and base coefficient lambda) are not disclosed for the main tables, no error bars or seeds are provided, and the language experiments omit a leading adaptive baseline. These gaps are load-bearing because the whole method reduces to the accuracy of the router, so the claimed gains cannot yet be separated from in-sample hyperparameter selection or seed noise. The theoretical section is also weaker than the paper's contribution statement implies, since Equation (4) restates the hypothesis at the output level rather than deriving it.","major_comments":[{"comment":"The two free inputs of the method, the representation layer ell and the base coefficient lambda, are never reported for the main results. Algorithm 2 (lines 1–2) takes ell and lambda as inputs, and Section 3's validation uses lambda=0.3 with ell=12 for ViT-B/32 and ell=24 for ViT-L/14, but Tables 1–3 do not state which values were used or how they were selected. Figure 3, which is computed on the same eight test sets used for Tables 1–2, shows acc@1 close to 1 only at selected deep layers. If ell (and lambda) were chosen by test-set accuracy, then the reported gains of +3.86, +0.57, and +6.86 partly reflect in-sample selection rather than a training-free advantage. Please report the chosen ell and lambda for every table, fix them a priori on a held-out split, or provide a sensitivity analysis showing that results are stable over a range of ell and lambda.","section":"Algorithm 2; Tables 1–3"},{"comment":"No seeds, runs, or error bars are reported for any result. Some of the claimed improvements are small on individual datasets (e.g., ViT-L/14 average +0.57, SVHN +0.90, DTD +1.28), and without variance estimates these differences cannot be distinguished from seed or run-to-run noise. Please report the mean and standard deviation over at least three independent runs, or explicitly state the single-run protocol used.","section":"Tables 1–3"},{"comment":"The language experiments omit AdaMerging, which is the strongest adaptive baseline in the vision tables, so the claim of a 'significant performance gain' in language tasks is only supported relative to static baselines. Please add AdaMerging (or a comparable adaptive training-based method) to Table 3, or restrict the claim to the static baselines actually compared. Additionally, Section 2 lists 'STS-2' as one of the seven GLUE tasks, but Table 3 shows SST-2 and no STS column; please clarify which tasks were used and fix the naming.","section":"Section 2; Table 3"},{"comment":"The theoretical support for Hypothesis 1 is circular. Equation (4) is exactly the output-level statement of Hypothesis 1 under the Weight Disentanglement definition, so it does not provide independent evidence for the hypothesis. The paper's contribution list claims a 'theoretical analysis' of the hypothesis, but the analysis assumes the conclusion. Please either remove the theoretical-analysis claim and present Section 3 as an interpretation, or provide a derivation that does not presuppose the hypothesis at the output level.","section":"Section 3; Equation (4)"}],"minor_comments":[{"comment":"There is a typo: 'fine-tnued models' should be 'fine-tuned models'.","section":"Algorithm 1, line 4"},{"comment":"The text says 'compare SE-Merging with ine-tuned individual models'; 'ine-tuned' should be 'fine-tuned'.","section":"Section 4, Performance"},{"comment":"The GLUE task list in the text is inconsistent with Table 3: the text names STS-2 but the table contains SST-2, and the citation [42] is for SST. Please choose a consistent task list and dataset name.","section":"Section 2; Table 3"},{"comment":"The symbol lambda is reused both as the initial scalar coefficient and as the per-sample rescaled coefficient lambda_t. Using a different symbol (e.g., alpha_t) for the per-sample coefficients would reduce confusion.","section":"Algorithm 2, lines 12–15"},{"comment":"The caption says 'ℓ1 distance' but the y-axis label is shown as '𝑙! Distance'; please fix the label to read 'ℓ1 Distance'.","section":"Figure 4"},{"comment":"Reference [36] is formatted inconsistently with the rest of the bibliography; the author names should follow the standard 'Last, First' style used elsewhere.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The omission of the concrete ell and lambda values for the main tables is the key obstacle. If the authors can provide those values together with a validation-based selection protocol and error bars, the paper may be publishable after a revision. I saw no indication of fabrication, but the current manuscript allows a reader to suspect test-set hyperparameter tuning, and that concern must be resolved before the empirical claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on SE-Merging. The paper has a new algorithm: a training-free, sample-wise merging method that uses representation similarity to rescale merging coefficients at inference. That much is real. It also offers a clean hypothesis—Representation Auto-Adaptation—backed by a nice quantitative study (acc@1 vs layer) and a useful representation-bias analysis. The empirical gains over AdaMerging on ViT-B/32 and TIES on GPT-2 are substantial, and the method is compatible with existing merging techniques. If the numbers hold, this is a useful contribution to the model-merging subfield.\n\nThe soft spots are in the verification, not the idea. The paper never reports the layer ℓ and base coefficient λ used in Tables 1–3. Algorithm 2 takes these as inputs, and Figure 3, which is supposed to justify the choice, is computed on the same test sets used for evaluation. That opens the door to test-set tuning: if ℓ (or λ) were picked to maximize router accuracy or final accuracy, the reported gains are partly in-sample. The authors do concede in Section 5(a) that Hypothesis 1 doesn't hold for all samples, but they don't quantify how often the router is wrong at the deployed layer or show how final accuracy varies across layers. There are also no error bars or seeds on any table, so on individual datasets the gains could be noise.\n\nThe theoretical section is explicitly conditional on Weight Disentanglement, which the paper itself says is not rigorously justified. So the \"mechanism unveiling\" claim is overstated; the theory mostly restates the known trivial decomposition. That part is weaker than the empirical contribution.\n\nNone of this is fatal. The central claim—that a representation-similarity router can improve dynamic merging without training—is plausible and worth testing. What's needed is a revised version with the hyperparameters reported, error bars, and ideally a held-out layer selection or a sensitivity analysis. The language experiments should also include AdaMerging or a similar adaptive baseline for completeness, though that's minor.\n\nWho is this for? People working on model merging, especially training-free methods, and anyone interested in representation-based routing. I'd send it to a competent referee, not desk reject. My own verdict would be conditional acceptance.\n\nRecommendation: engage with it, but require the missing details before trusting the numbers.","headline":"A solid, useful training-free dynamic merging method whose reported gains are plausible but currently unverifiable because the key hyperparameters are undisclosed.","tokens_in":15554,"tokens_out":1992,"would_cite":true,"duration_ms":17942,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A training-free, sample-wise rescaling scheme for merged models, built on the Representation Auto-Adaptation Hypothesis, delivers multi-task accuracy gains on vision and language benchmarks.","keywords":["model merging","multi-task learning","representation similarity","task arithmetic","training-free inference","task vectors","transformer models","weight disentanglement"],"falsifier":"Take the paper's reported layer settings (deep layers such as 12 of ViT-B/32) and a held-out set of tasks not used for evaluation; measure the acc@1 of the similarity-based task identification. If acc@1 is well below 1.0 on that new task set, SE-Merging's gains should shrink or reverse. Additionally, check whether the layer $\\ell$ was chosen by maximizing test accuracy: if so, part of the reported gain could be the result of test-set selection rather than the mechanism itself.","tokens_in":14563,"feed_emoji":"🧩","tokens_out":5907,"duration_ms":49836,"temperature":0.7,"pith_summary":"This paper sets out to explain why model merging works and to use that explanation to make merging better. Its claim is that a merged model succeeds because it separates samples by task and, at deep layers, produces internal representations that match the corresponding fine-tuned expert — the Representation Auto-Adaptation Hypothesis. On that basis the paper builds SE-Merging, a training-free procedure that, for each test sample, computes the $\\ell^2$ distance between the merged model's representation and each expert's, converts these distances into similarity scores, and rescales the merging coefficients to upweight the best-matching task vector before inference. The reported outcome is accuracy gains on vision and language benchmarks: +3.86 average points over AdaMerging on ViT-B/32, +0.57 on ViT-L/14, and +6.86 over TIES-Merging on GPT-2, without any additional training. If correct, this gives a simple, compatible plug-in that turns any task-vector-based merge into a per-sample dynamic router.","feed_headline":"A similarity check routes merged models per sample","feed_subtitle":"SE-Merging upweights the right task expert at inference, beating AdaMerging by 3.86 points on ViT-B/32.","key_machinery":"The load-bearing object is the Representation Auto-Adaptation Hypothesis (Hypothesis 1) together with the rescaling rule of Algorithm 2. Given a test sample, the mechanism computes $r_{\\mathrm{Merged}}=f^{(\\ell)}(x;\\theta_{\\mathrm{Merged}})$ and $r_t=f^{(\\ell)}(x;\\theta_{PT}+\\lambda\\tau_t)$ for each task $t$, records the $\\ell^2$ distances $d_t=\\|r_{\\mathrm{Merged}}-r_t\\|_2$, converts them into similarities via a min-max normalized reversal, and produces per-sample coefficients $\\lambda_t = \\frac{\\exp(s^{\\mathrm{norm}}_t)}{\\sum_j \\exp(s^{\\mathrm{norm}}_j)} T\\lambda$. The layer $\\ell$ is a hyperparameter; the paper's experiments show that task-specific representations concentrate at deep layers. This machinery turns the merged model itself into an implicit router, so no separate router network or training on test data is needed.","core_discovery":"The central claim is that model merging works because a merged model implicitly adapts to the correct expert for each input: for a sample $x_i$ from task $T_i$, the merged model's representation at layer $\\ell$ stays close to that of the fine-tuned model $\\theta_{PT}+\\lambda\\tau_i$ (Hypothesis 1). The paper supports the claim with t-SNE visualizations and with acc@1 measurements showing near-1.0 rates at deep layers (e.g., layer 12 of ViT-B/32 and layer 24 of ViT-L/14), and ties it to the weight-disentanglement condition. The discovery is that this hypothesis can be inverted into a method: instead of training a router, SE-Merging uses representation similarity of the merged model itself to identify the sample's task and then rescales the task-vector coefficients $\\lambda_i$ accordingly before classifying. The paper claims this makes static merging methods dynamic and stronger, achieving state-of-the-art results among training-free approaches and outperforming the training-based AdaMerging.","pith_inferences":["If the hypothesis holds at deep layers for many architectures, the layer index could be selected automatically per model family (e.g., by a validation proxy) rather than per task set, making the method more robust to hidden test-set tuning.","The similarity-based router is essentially a zero-shot surrogate for the 'perfect router' used in prior dynamic merging work; comparing SE-Merging against the perfect-router upper bound would quantify how much task confusion still costs.","A natural stress-test is out-of-distribution or cross-domain samples, where deep-layer task separation may fail; the paper's analysis does not yet report such cases.","Because the rescaling is a soft weight over task vectors, SE-Merging could be combined with sparsification (like DARE) or conflict-reduction methods to further isolate the right expert's signal."],"forward_implications":["Any task-vector merge (e.g., Task Arithmetic or TIES-Merging) can be upgraded to per-sample rescaling by adding a forward pass over the experts to compute similarities.","Training-free dynamic merging removes the need for routers or unlabeled test-set optimization (as used by AdaMerging), reducing computational cost.","The representation-bias measure drops after rescaling, indicating the merged model's inference process more closely mirrors the correct fine-tuned model.","The method extends to language models (GPT-2 on seven GLUE tasks) with gains beyond static baselines, suggesting general applicability beyond vision encoders."],"supporting_citations":[{"why":"Provides the task-vector formulation and the base merging setting that SE-Merging builds on and compares against.","marker":"[17]"},{"why":"Supplies the weight-disentanglement condition used to theoretically justify the Representation Auto-Adaptation Hypothesis.","marker":"[31]"},{"why":"The training-based adaptive merging method that SE-Merging claims to outperform without training.","marker":"[58]"},{"why":"A conflict-resolution baseline whose setup SE-Merging is compatible with and outperforms on language tasks.","marker":"[56]"},{"why":"Defines representation bias, which SE-Merging uses to demonstrate its alignment effect.","marker":"[57]"},{"why":"The benchmark supplying the language-task experimental setup (GPT-2 on GLUE tasks).","marker":"[44]"},{"why":"Supplies the vision encoders (ViT-B/32, ViT-L/14) used as pre-trained models.","marker":"[34]"}],"fun_headline_variants":["Model merging goes dynamic with per-sample rescaling","No-training model merging adapts per sample, boosts accuracy","SE-Merging: Dynamic task routing without training","Per-sample coefficient rescaling makes merging smarter","Merged models get a self-routing boost at inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands on the premise that, at the chosen (unreported) layer, the merged model's representation is closer to the correct expert than to any other expert for nearly every test sample; if that similarity ranking is often wrong, the rescaling upweights the wrong task vector and can hurt accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Model merging goes dynamic with per-sample rescaling","No-training model merging adapts per sample, boosts accuracy","SE-Merging: Dynamic task routing without training","Per-sample coefficient rescaling makes merging smarter","Merged models get a self-routing boost at inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000697,"raw_usage":{"total_tokens":3156,"prompt_tokens":955,"completion_tokens":2201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2125}},"tokens_in":571,"tokens_out":2201,"duration_ms":14286,"temperature":1.0,"reasoning_tokens":2125,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:54:41.586708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the paper's reported layer settings (deep layers such as 12 of ViT-B/32) and a held-out set of tasks not used for evaluation; measure the acc@1 of the similarity-based task identification. If acc@1 is well below 1.0 on that new task set, SE-Merging's gains should shrink or reverse. Additionally, check whether the layer $\\ell$ was chosen by maximizing test accuracy: if so, part of the reported gain could be the result of test-set selection rather than the mechanism itself.","supporting_citations":[{"cited_title":"Editing models with task arithmetic","cited_arxiv_id":null,"evidence_quote":"Provides the task-vector formulation and the base merging setting that SE-Merging builds on and compares against."},{"cited_title":"Task arithmetic in the tangent space: Improved editing of pre-trained models.Advances in Neural Information Processing Systems, 36, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the weight-disentanglement condition used to theoretically justify the Representation Auto-Adaptation Hypothesis."},{"cited_title":"Representation surgery for multi-task model merging","cited_arxiv_id":null,"evidence_quote":"Defines representation bias, which SE-Merging uses to demonstrate its alignment effect."},{"cited_title":"Fusionbench: A comprehensive benchmark of deep model fusion, 2024","cited_arxiv_id":null,"evidence_quote":"The benchmark supplying the language-task experimental setup (GPT-2 on GLUE tasks)."}],"review_version":2}