{"id":"e28934ab-0457-4f38-a9c6-6400c532fa49","arxiv_id":"2412.17023","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Applying ReFT-style interventions at every transformer block of a merged model improves multi-task accuracy beyond post-hoc single-layer repair, and slicing the representation keeps the parameter cost low.","lead":"IntervMerge adds small trainable interventions to every block of a merged vision model, reducing the representation mismatch that hurts multi-task accuracy. It reports consistent gains over the prior Surgery approach on eight image benchmarks while using fewer extra parameters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's headline comparison conflates intervention type with layer placement; without a distributed-Surgery control, the 1.46% advantage cannot be attributed to IntervMerge.","rationale":"The most load-bearing issue is not the data footprint, which is shared with Surgery, nor the absence of error bars, which would only weaken the numeric comparison; it is the unperformed control that distinguishes the proposed intervention module from the known benefit of distributing parameter-efficient modules across layers. The paper itself demonstrates a large placement effect (Table 3: 85.48% at one block vs. 88.83% at twelve), so without holding placement fixed across methods, the 1.46% margin in Table 1 cannot be attributed to the ReFT-style intervention. This is a standard confounding-variable concern: the treatment is a bundle (formula + distribution), and the claim is about the formula. A distributed-Surgery baseline is straightforward to implement and directly adjudicates the central claim. If the baseline matches IntervMerge, the paper's stated contribution—that its specific interventions are better—collapses to a weaker claim about distributing adapters generally; if it does not match, the strongest_claim is strongly supported. Given this missing control, the appropriate verdict remains conditional pending this experiment.","tokens_in":12889,"tokens_out":8663,"duration_ms":77558,"concrete_test":"Implement the Surgery adapter (h + W_up ReLU(W_down h)) after MHSA at every one of the 12 ViT-B/32 blocks, with rank 1 per block (≈147k params) and the same distillation loss, optimizer, and learnable lambdas used for IntervMerge. Evaluate on the 8 tasks and compare average accuracy with AdaMerging+IntervMerge (rank 1) from Table 1 and with final-layer Surgery (rank 64). If distributed Surgery reaches ≥88.96%, the paper's attribution of the gain to the intervention formula fails; if it stays near 86-87%, the distributed-intervention claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's key evidence (Table 1) contrasts IntervMerge with 12 rank-1 blocks (147k params, 88.96%) against Surgery with a single rank-64 final-layer adapter (524k params, 87.5%). This design varies two factors at once: the intervention formula (ReFT-style residual vs. Surgery's ReLU adapter) and the placement (every block vs. last block). The claimed 1.46% improvement is presented as evidence that distributed low-rank interventions are a superior repair strategy, but it could equally be explained by the simple fact that interventions are applied at multiple blocks rather than only the final layer. Tables 3 and 5 show that distributing IntervMerge across blocks improves accuracy (85.48% for one block vs. 88.83% for twelve), which supports a placement effect, but no analogous ablation exists for Surgery. Without matching Surgery's adapter to every block with a similar parameter budget and training protocol, the central claim that IntervMerge's intervention design—rather than its distribution—drives the gain is unsupported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IntervMerge, a post-merging repair method for multi-task ViT classifiers. Task-specific low-rank intervention modules, inspired by ReFT, are inserted after the MHSA sublayer in every transformer block and trained by distillation between the merged model and the individual task models, optionally using a small amount of data. A 'mini-intervention' variant edits only a contiguous slice of the representation to reduce the parameter overhead. The method is evaluated on eight image classification datasets with ViT-B/32 and ViT-L/14 (the latter in the supplement), comparing against base merging methods and the Surgery baseline. The headline result is that rank-1 IntervMerge with AdaMerging reaches 88.96% average accuracy with 147k extra parameters, surpassing rank-64 Surgery at 87.5% with 524k extra parameters.","tokens_in":13144,"tokens_out":12123,"duration_ms":110303,"significance":"If the comparisons are properly controlled, the contribution is practically useful: it would show that distributing low-rank representation interventions across blocks can repair representation bias more cheaply than a single final-layer adapter. The paper also provides a careful ablation of token choice, block placement, rank, and edit length, which is valuable design guidance for the merging community. The use of three-seed standard deviations in Tables 2, 4, and 7 is a strength. However, the central attribution of the improvement is currently under-supported because the headline comparison varies the intervention formula and the number of intervened layers simultaneously, and a few reported claims are not consistent with the tables. The method is simple enough to reproduce, but the paper does not state whether code will be released.","major_comments":[{"comment":"The claim that rank-1 IntervMerge outperforms rank-64 Surgery by 1.46% with 3.5x fewer parameters is not an apples-to-apples comparison of the intervention methods. The two models differ in the intervention function (residual ReFT-style map vs. the ReLU adapter) and in placement (all 12 blocks vs. only the final representation). Since Table 3 shows that the number of intervened blocks matters (85.48% for one middle block vs. 88.83% for all 12), the advantage may reflect placement rather than the proposed intervention design. Please add a controlled comparison, for example Surgery applied at every block with a comparable parameter budget and training protocol, or IntervMerge restricted to the last block with a matched budget.","section":"§5, Table 1"},{"comment":"The statement that the 64-part mini-intervention with the h + RT(b) formula 'achieves an average accuracy 3% higher than Surgery, using fewer parameters' is contradicted by Table 1, where AdaMerging w/IntervMerge (mini-interv.) has 83.45% with 3k parameters while AdaMerging w/Surgery has 86.1% with 131k parameters. Please clarify which Surgery configuration is being referenced (e.g., the rank-1 row in Table 6), and report the numbers for that configuration in Table 1 or adjust the claim.","section":"§5, paragraph after Table 1"},{"comment":"The objective in Eq. (3.1) is written as a loss on the test datasets, and Table 8 explicitly uses 1-10% of the available test set to fit the interventions. If the same test split is also used for hyperparameter selection (Tables 2-5) and final evaluation, the reported accuracies may be optimistically biased. Please state clearly whether the available subset is disjoint from the evaluation set; if it is not, add experiments with a separate validation split or a subset of the training set, and add a limitation note in Section 6.","section":"§3.1, Eq. (3.1); §5.5, Table 8"},{"comment":"The headline comparisons are single runs without error bars, while Tables 2, 4, and 7 report ± values over three seeds. Because several differences in the tables are small (e.g., Table 4 rows 87.59 vs. 87.54, Table 7 rows 89.53 vs. 89.56), the paper should report mean and standard deviation over at least three seeds for Tables 1 and 6, or at least for the key rank-1 IntervMerge vs. rank-16/64 Surgery comparisons.","section":"§5, Tables 1 and 6"}],"minor_comments":[{"comment":"The specified dimensions W1, W2 ∈ R^{k×r} are inconsistent with the product W2^T(W1 z_b + b - W2 z_b); W1 and W2 should be in R^{r×k} and b in R^r for the expression to be well-defined. Please correct the dimensions.","section":"§3.2.1, Eq. (3.4)"},{"comment":"The edited segment is written as z_b[j:p] in the equation but as z_b[j:k] in the prose; p is not defined. Please unify the notation.","section":"§3.2.2, Eq. (3.5)"},{"comment":"The range '0-786' appears to be a typo for '0-768', and the range '200-184' appears reversed; please fix these entries.","section":"Table 5"},{"comment":"The note that intervened blocks are 'marked in green' does not carry over to the typeset table, and the exact block indices are not provided; please list them explicitly.","section":"Table 3"},{"comment":"The notation RT(b) should be written as R^T b (transpose of R applied to b) to avoid ambiguity with a matrix product.","section":"Table 7"},{"comment":"The two tables report slightly different accuracies for the same AdaMerging w/IntervMerge rank-1 configuration (88.96 vs. 88.90); please clarify whether these come from different settings or different runs.","section":"Tables 1 and 6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is a plausible fit for a venue on model merging or parameter-efficient tuning. The core issue is not novelty but experimental attribution: the authors need to add a placement-controlled comparison, reconcile the mini-intervention text with Table 1, and address the test-set usage. These are fixable in a revision; I do not see a reason to reject outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth reading and worth refereeing. IntervMerge applies ReFT-style interventions to multi-task model merging, placing low-rank task-specific modules after MHSA in every ViT block, and it includes a 'mini-intervention' variant that edits only a slice of the representation. That combination is new, and the ablation work is genuinely thorough: token position, block count and position, part size, rank, and intervention function are all examined, with standard deviations on most secondary tables. The paper also does a fair thing by re-running Surgery with learnable lambdas and showing it doesn't close the gap.\n\nThe headline number holds up as a system-level comparison: rank-1 IntervMerge on AdaMerging gets 88.96 average versus 87.5 for Surgery at rank 64, with 147k extra parameters against 524k. That is a real, useful result for the post-merge repair stage. I would not call it a new phenomenon; it is a solid subfield improvement.\n\nThe soft spots are the following. First, the paper contains an internal inconsistency about mini-interventions. Section 5 claims the 64-part mini-intervention 'achieves an average accuracy 3% higher than Surgery,' but Table 1 shows the mini-intervention row at 83.45 versus 86.1 for Surgery at rank 16 and 87.5 at rank 64. Unless there is an unshown configuration in the supplement, the claim is the opposite of the table. That needs to be fixed or clarified.\n\nSecond, the stress-test confound is real but milder than stated. The main comparison varies both the intervention formula and the layer placement. The paper's own ablations (Table 3) show that distribution across blocks matters a lot for IntervMerge, so without a Surgery version placed on all blocks, you cannot attribute the gain specifically to the ReFT-style function rather than to the fact that you are editing everywhere. That is a fair referee request, but it does not undermine the system-level claim that the full IntervMerge recipe beats Surgery.\n\nThird, the main Table 1 is single-run for most rows; a few error bars would make the 1.5-point lead more convincing. Fourth, Table 8 explicitly uses 1-10% of the test set to train interventions. This matches the recent Surgery setup, but it is a data leak relative to the stated 'no training data' setup and worth an explicit discussion. Finally, no code is released.\n\nWho is this for? Anyone working on model merging, especially post-hoc repair methods, and people interested in parameter-efficient fine-tuning with interventions. The paper deserves a serious referee; I would send it to review with a request for the distributed-Surgery control, the mini-intervention clarification, error bars on the main table, and code release. It is not ready in its current form, but the core idea and the main result are solid.","headline":"Solid, useful post-merge repair paper with a genuine new application of interventions; the main system comparison holds, but the mini-intervention claim is internally inconsistent and a distributed-Surgery control is missing.","tokens_in":13652,"tokens_out":4582,"would_cite":true,"duration_ms":39375,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distributed low-rank edits beat one big adapter in model merging","keywords":["model merging","representation bias","parameter-efficient interventions","low-rank adaptation","multi-task learning","vision transformer","representation surgery","distillation"],"falsifier":"Train IntervMerge with zero calibration data (e.g., using only the merged model's own predictions as pseudo-targets) and show that accuracy falls below the base merge; or, with data available, draw the calibration sample from a different distribution than the test split (e.g., one class per task removed) and show that the gains reported in Table 1 vanish. Alternatively, compare a single rank-64 intervention at the final layer against rank-1 interventions at every layer under the same total parameter budget on a broader suite; if the former wins, the depth-distribution hypothesis fails.","tokens_in":12684,"feed_emoji":"🧩","tokens_out":6569,"duration_ms":53778,"temperature":0.7,"pith_summary":"The paper claims that the performance loss of merged multi-task vision models comes from representation bias that accumulates across all layers, not just at the output. It proposes IntervMerge, which attaches tiny task-specific low-rank intervention modules after the attention of every transformer block and trains them with a distillation loss against the original task models using a small sample of task data. It also introduces mini-interventions that edit only a slice of the representation, cutting extra parameters further. On eight image-classification tasks with a CLIP ViT-B/32 encoder, IntervMerge applied to AdaMerging reaches 88.96% average accuracy at rank 1, beating the previous state-of-the-art Surgery at rank 64 (87.5%) while using 3.5 times fewer extra parameters. If correct, this means widely-distributed, parameter-light representation repairs are more effective than a single high-capacity adapter at the final layer.","feed_headline":"Distributed low-rank edits beat one big adapter in model merging","feed_subtitle":"IntervMerge fixes merged vision models with tiny per-task modules, beating Surgery with 3.5x fewer extra parameters.","key_machinery":"The central object is the low-rank intervention module $\\Phi^t_b$, an additive residual edit to the representation produced by the self-attention block, parameterized by two rank-$r$ projection matrices plus a bias, applied to the class token (or a slice of it) in every block and for each task. It is trained with a distillation loss that pushes the merged model's per-task representations toward those of the task-specific models, distributing the correction across depth rather than concentrating it at the last layer, which stops early errors from propagating and stabilizes later representations. The mini-intervention variant $\\Phi^t_b(z_b[j:p])$ restricts the edit to a contiguous slice of the representation, and shifting that slice across blocks produces a parameter-efficient way to spread knowledge throughout the network.","core_discovery":"The central discovery is that representation bias in a merged model can be corrected more effectively by lightweight, task-specific interventions applied at every block of the network than by a single high-rank adapter at the end. IntervMerge inserts, after the multi-head self-attention of each ViT block, a module $\\Phi^t_b$ acting on the [CLS] token: $\\Phi^t_b(z_b) = z_b + W_2^T(W_1 z_b + b - W_2 z_b)$, with low-rank matrices $W_1, W_2$ of rank $r$ (default $r=1$). Mini-interventions generalize this to editing only a contiguous slice $z_b[j:p]$ of the representation, reducing parameters to as little as 3k while still outperforming Surgery. Using the same distillation loss as Surgery, the modules are trained on a tiny held-out sample (1–10% of the test set) and can be combined with any base merge method; with AdaMerging the rank-1 IntervMerge reaches 88.96% average accuracy versus 87.5% for Surgery at rank 64, and the method also improves weaker merges such as Weight Averaging and Ties-Merging.","pith_inferences":["If distributed low-rank corrections are the right model of representation bias, the same recipe should transfer to decoder-only language models, where the role of the class token would be played by a fixed summary position or a learned prompt; testing that would separate the vision-specific part of the claim from the general mechanism.","The shift heuristic in mini-interventions implies the bias has a layer-dependent, structured component; learning the slice positions per task (rather than fixing a shift) might recover most of the full-intervention accuracy at the mini-intervention parameter cost.","The data-visibility results suggest the method is useful precisely when calibration data is scarce; the extreme zero-data case is untested, and a plausible outcome is that without any task data IntervMerge falls back to the base merge and could underperform Surgery on some tasks.","A practical consequence implicit in the paper is a new serving configuration: one shared backbone plus several 3k-parameter per-task modules instead of full per-task models, which is attractive for memory-limited deployment on edge devices."],"forward_implications":["IntervMerge beats Surgery on all five base merging methods tested, with improvements of 2.56 points over Weight Averaging, 4.57 over Task Arithmetic, and 5.02 over Task-wise AdaMerging (Table 1).","When interventions are applied to every one of the 12 ViT blocks, average accuracy is highest (88.83%); halving the number of blocks to every second block costs less than a point, while concentrating all interventions in the last block drops accuracy by about 3 points.","Intervening on the class token is the best token choice for image classification, reaching 89.49% average accuracy versus 70.82% for patch tokens (Table 2).","With the same total parameter budget, shifting a 64-element mini-intervention across blocks (85.63%) slightly exceeds the best single-block full intervention (85.48%), so distributed small edits are at least as good as a carefully chosen single location.","With only 1% of the test set available for calibration, IntervMerge reaches 85.55% average accuracy versus 82.8% for Surgery, and the gap widens as more data is added (Table 8)."],"supporting_citations":[{"why":"Supplies the post-merging baseline (Surgery) and the distillation loss (Eq. 3.2) that IntervMerge shares.","marker":"[45]"},{"why":"Provides AdaMerging, the base merging method used in the headline result and the adaptive-coefficient framework IntervMerge builds on.","marker":"[46]"},{"why":"Contributes the low-rank intervention function that IntervMerge adapts from language models to merged vision models.","marker":"[40]"},{"why":"The task-vector merging baseline that IntervMerge also repairs in Tables 1 and 6.","marker":"[14]"},{"why":"The TIES-Merging baseline that IntervMerge also improves, showing generality across base merges.","marker":"[43]"},{"why":"Weight averaging baseline, the simplest merge IntervMerge is applied to.","marker":"[37]"}],"fun_headline_variants":["IntervMerge: tiny per-task fixes beat big adapters in merging","IntervMerge: less parameters, better merged models","Distributed low-rank edits beat Surgery with 3.5x fewer params","Mini-interventions fix merged vision models with 3k params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a small sample of each task's data (1–10% of the test set) is available at merge time and is representative enough to train the intervention modules through the distillation loss; if that data is missing or unrepresentative, the interventions will overfit and the merged model gains nothing over the base merge.","fun_headline_variants_meta":{"raw":{"variants":["IntervMerge: tiny per-task fixes beat big adapters in merging","IntervMerge: less parameters, better merged models","Distributed low-rank edits beat Surgery with 3.5x fewer params","Mini-interventions fix merged vision models with 3k params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001088,"raw_usage":{"total_tokens":4518,"prompt_tokens":891,"completion_tokens":3627,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":3553}},"tokens_in":507,"tokens_out":3627,"duration_ms":24273,"temperature":1.0,"reasoning_tokens":3553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:51:07.340665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train IntervMerge with zero calibration data (e.g., using only the merged model's own predictions as pseudo-targets) and show that accuracy falls below the base merge; or, with data available, draw the calibration sample from a different distribution than the test split (e.g., one class per task removed) and show that the gains reported in Table 1 vanish. Alternatively, compare a single rank-64 intervention at the final layer against rank-1 interventions at every layer under the same total parameter budget on a broader suite; if the former wins, the depth-distribution hypothesis fails.","supporting_citations":[{"cited_title":"Repre- sentation surgery for multi-task model merging","cited_arxiv_id":null,"evidence_quote":"Supplies the post-merging baseline (Surgery) and the distillation loss (Eq. 3.2) that IntervMerge shares."},{"cited_title":"Adamerg- ing: Adaptive model merging for multi-task learning","cited_arxiv_id":null,"evidence_quote":"Provides AdaMerging, the base merging method used in the headline result and the adaptive-coefficient framework IntervMerge builds on."},{"cited_title":"Editing models with task arithmetic","cited_arxiv_id":null,"evidence_quote":"The task-vector merging baseline that IntervMerge also repairs in Tables 1 and 6."},{"cited_title":"TIES-merging: Resolving interference when merging models","cited_arxiv_id":null,"evidence_quote":"The TIES-Merging baseline that IntervMerge also improves, showing generality across base merges."},{"cited_title":"Model soups: averaging weights of multiple fine-tuned models improves accuracy with- out increasing inference time","cited_arxiv_id":null,"evidence_quote":"Weight averaging baseline, the simplest merge IntervMerge is applied to."}],"review_version":1}