{"id":"2b083ef8-5852-412b-a5e6-8f59c027a8ec","arxiv_id":"2607.24465","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Dual-Tuned Experts (prompt learning then vision fine-tuning) shrink task-specific weight updates and consistently raise merged multi-task accuracy across CLIP merging methods.","lead":"Training CLIP experts in two stages—first learn text prompts, then fine-tune the vision encoder—makes those experts much easier to merge into one multi-task model. The method improves standard weight-merging recipes without inventing a new merger, and even simple averaging of these experts often beats advanced mergers of ordinary fine-tunes.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The DTE-vs-Vision-FT merged comparison changes two things at once — expert training and the inference-time text interface (learned prompts vs hand-crafted templates) — so the \"smaller task vectors → better mergeability\" mechanism is confounded by prompt advantage.","rationale":"The reader identified the task-aware protocol as the weakest assumption; I agree it is the correct frame, but within that protocol there is a sharper, uncontrolled confound the reader did not flag: the interface asymmetry between conditions. The reader's conditionality (code release, variance reporting) is appropriate, and my concern points in the same direction — a missing ablation, not a demonstrated error — so I do not move the verdict to REJECT. The paper deserves credit for the equal-compute control (§B.5), the LoRA extension, the heterogeneous-mix study, and the paired statistical tests on update norms; these show the authors anticipate fairness objections. However, every piece of mechanistic evidence except the Frobenius norm (interpolation profiles, representation drift comparisons) is collected under the same asymmetric prompting setup, so the \"higher merge compatibility\" mechanism currently rests substantially on one modest (~12%) norm difference plus outcomes that mix prompt and weight effects. The proposed control is cheap — it reuses already-trained Stage-1 prompts and already-trained Vision-FT experts, requiring only re-evaluation of merged checkpoints under swapped text interfaces — and would decisively settle whether the central claim lands. Until that decomposition is reported (or code enables others to run it), CONDITIONAL remains the right verdict, with the condition expanded to include the interface-controlled ablation.","tokens_in":18051,"tokens_out":2564,"duration_ms":99728,"concrete_test":"Run the interface-controlled ablation on ViT-B/32 and ViT-L/14, 8/14/20 tasks: (a) merge standard Vision-FT experts but evaluate the merged model using the Stage-1 learned prompts (trained on the frozen backbone) instead of hand-crafted templates; (b) merge DTE vision encoders and evaluate with hand-crafted templates only. Report WA and TSV-M for both. If (a) recovers most of the WA-DTE advantage (e.g., reaches ≥95% normalized accuracy), the mergeability story weakens and the contribution reduces to prompt selection; if (b) still beats the Vision-FT+templates baseline by margins comparable to Table 1, the mechanism claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that two-stage expert training produces vision encoders with \"higher merge compatibility\" (smaller updates, flatter interpolation, less drift), and that this is why merged DTEs beat merged Vision-FT under the same merging operators. But the evaluation is not interface-controlled: merged DTEs are scored with their task-specific learned prompts, while merged Vision-FT is scored with hand-crafted templates (§5, \"Task-aware evaluation\"). The paper's own numbers show how much the prompt interface alone carries: on ViT-L/14, prompt learning with a completely frozen backbone already achieves 87.28/87.95/87.92% (8/14/20 tasks), while WA-merged Vision-FT gets 79.21/76.52/71.39 and WA-merged DTEs get 89.59/89.07/88.69. So the prompt interface by itself, with zero weight merging, sits only ~1–2 points below the WA-DTE merged numbers and ~8–17 points above the WA-Vision-FT baseline the gains are measured against. The missing control is merged Vision-FT (or merged DTE) evaluated under an identical text interface — e.g., both conditions scored with the same Stage-1 learned prompts, or both scored with hand-crafted templates only. Without it, one cannot tell how much of the Table 1/Table 2 gains reflect genuinely more mergeable vision task vectors versus simply evaluating through a better, task-adapted text interface. Notably, the mechanistic evidence inherits the same confound: the interpolation analysis in §4.3 explicitly states \"for DTEs, we use the learned prompts associated with each task; for Vision-FT, hand-crafted templates,\" so the flatter L_mix profiles may partly reflect the prompt interface rather than weight-space geometry. The Frobenius-norm reduction (2.27 vs 2.58, ~12%) is real but modest, and is the only interface-independent evidence offered. The claim isn't shown to be false — LoRA-DTE results and the heterogeneous-mix experiment are suggestive — but the load-bearing decomposition (prompt effect vs weight-mergeability effect) is never run.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper argues that expert training — not just the merging operator — is a design axis for model merging. It first observes that per-task prompt learning on a frozen CLIP backbone, with prompts selected at inference, is already competitive with state-of-the-art weight-merging methods (Fig. 2), especially in the many-task regime. It then introduces Dual-Tuned Experts (DTEs): Stage 1 learns unified-context prompts with the backbone frozen; Stage 2 fine-tunes the vision encoder with prompts fixed. Across ViT-B/32 and ViT-L/14, 8/14/20-task CLIP benchmarks, full FT and LoRA experts, and five merging methods (WA, TA, TIES, TSV-M, Iso-C), merged DTEs outperform merged one-stage Vision-FT experts while separate-expert accuracy is matched (Tables 1-2). Mechanistic analyses report smaller representation drift (~64% relative cosine-similarity improvement to the pretrained backbone, Fig. 4), smaller Frobenius update norms (2.27 vs 2.58, paired tests p=0.025/0.001, §4.3), and flatter pairwise interpolation profiles (Fig. 5). A heterogeneous experiment shows monotonic improvement as Vision-FT experts are replaced by DTEs (Fig. 6), and a FLOP-matched control (Table 5) shows the gains do not require extra compute.","tokens_in":18436,"tokens_out":4766,"duration_ms":177444,"significance":"If the central claim holds under interface-controlled evaluation, the paper makes a useful and somewhat contrarian point for the merging literature: expert training is an orthogonal, under-explored design axis, and a trivially cheap first stage (prompt learning, 0.005% of parameters) materially changes the mergeability of the resulting task vectors. The finding that a frozen-backbone prompt baseline is competitive with SOTA merging at 20 tasks is independently valuable and likely to be cited on its own. The evaluation is broad (two backbones, 8/14/20 tasks, full FT and LoRA, five mergers, heterogeneous pools), the claims are falsifiable, and the needed control experiment is cheap. The mechanistic story (reduced representation drift, reduced update norms) is plausible and partially interface-independent, though not yet cleanly tied to the accuracy gains.","major_comments":[{"comment":"Tables 1-2, §5.1: the headline comparison conflates two changes — expert training and the inference-time text interface. Merged DTEs are scored with task-specific learned prompts, merged Vision-FT with hand-crafted templates, so absolute accuracy gains mix 'better mergeable vision vectors' with 'better text interface.' The paper's own numbers show the interface alone carries most of the gap: on ViT-L/14, prompt learning with a frozen backbone (no merging at all) reaches 87.28/87.95/87.92 on 8/14/20 tasks — within 1-2 points of WA-merged DTEs (89.59/89.07/88.69) and 8-17 points above the WA-Vision-FT baseline (79.21/76.52/71.39) against which gains are measured. Claims like 'simple weight averaging of DTEs already outperforms Vision-FT merged by any model merging method' are therefore not established as merging claims. Two mitigations are available within the paper's own framework, and I","section":"§5.1, Table 1"},{"comment":"Eq. (9)-(10), Fig. 5: the interpolation analysis, presented as mechanistic evidence for 'higher merge compatibility,' inherits the same interface confound. The text states that Vision-FT endpoints are evaluated with hand-crafted templates and DTE endpoints with learned prompts. Flatter mixture-loss profiles for DTEs may therefore partly reflect that the DTE encoders were co-adapted with their prompt interface, not intrinsically better linear mode connectivity of the vision parameters. The analysis should either use a matched interface across conditions or be explicitly caveated. Note that the other two mechanistic analyses are not confounded: representation drift (Fig. 4) and Frobenius update norms are vision-only measurements, and they do support the claim — the paper would be stronger leaning on these. Additionally, Fig. 5 shows only 'representative' task pairs; an aggregate statistic","section":"§4.3"},{"comment":"Fig. 6: in the heterogeneous replacement experiment, the evaluation interface is swapped together with the expert: tasks whose expert was replaced by a DTE are evaluated with learned prompts, remaining tasks with templates. The monotone improvement as DTEs are added could therefore be driven substantially by the interface swap per replaced task rather than by improved cross-expert compatibility — the confound is exactly correlated with the x-axis. An interface-controlled variant (all tasks evaluated with templates, or all with learned prompts where feasible) is needed for the 'DTEs are beneficial even in mixed pools' conclusion. Also, only three runs are used and no variance band is shown; given that replacement order is random, error bars matter for the monotonicity claim.","section":"§5.3, Fig. 6"}],"minor_comments":[{"comment":"Tables 1-2 report single-run numbers with no seed variance. Since expert training is stochastic, at least a subset (e.g., WA and one advanced merger on the 20-task benchmark) should report mean ± std over 2-3 seeds, or the paper should state that differences of the reported magnitude far exceed measured seed noise.","section":"Tables 1-2"},{"comment":"The relative-improvement annotations include +762%, +1809%, and +323%, implying near-zero Vision-FT baseline similarities on those tasks; absolute similarity values should be reported alongside. Two tasks (SUN397, STL10) show slightly negative changes (-3%, -1%) that go undiscussed. Also Fig. 4 reports ~64% relative improvement on ViT-L/14 while the norm analysis uses ViT-B/32 — the backbone for each analysis should be stated consistently in the main text.","section":"Fig. 10, §B.4"},{"comment":"The Frobenius-norm comparison (2.27±0.95 vs 2.58±0.81) has heavily overlapping distributions and a marginal paired t-test (p=0.025); the Wilcoxon result (p=0.001) is more convincing. Report effect size and consider a per-task scatter. A 12% norm reduction is modest relative to the ~64% representation-drift reduction, and the discrepancy deserves a sentence.","section":"§4.3, parameter-update analysis"},{"comment":"LoRA-DTE separate experts are slightly worse than Vision-LoRA experts (87.73 vs 88.59 on 8 tasks) yet merge far better in normalized terms — this is actually one of the cleaner pieces of evidence for the mergeability claim (worse experts, better merges) and deserves explicit discussion rather than being left implicit.","section":"Table 2"},{"comment":"Table 4: for SUN397 the remaining Stage-2 budget under the equal-budget protocol is only 0.046 PFLOPs, i.e., essentially no vision fine-tuning; clarify how Approach 2 handles such tasks and whether the SUN397 DTE in Table 5 is effectively prompt-only.","section":"§B.5, Table 4"},{"comment":"The caption of Table 5 says 'Subscripts report normalized accuracy' but the table uses brackets, not subscripts. Typos: 'setings' (§5.1); missing space in 'introduceDual-Tuned Experts' (§1 contributions); 'decompose' → 'decomposes' in §A.1 (Iso-C); inconsistent spacing in 'W A' throughout.","section":"Table 5 caption, various"},{"comment":"The task-aware evaluation assumption (task identity known at inference) is standard in this benchmark lineage, but the paper should state explicitly in the Limitations that reported numbers do not transfer to task-agnostic inference, since the learned prompts are per-task and no prompt-merging or task-inference mechanism is provided.","section":"§5, 'Task-aware evaluation'"},{"comment":"Fig. 9's ablation is useful; it would help to state whether M=16 was also used for ViT-L/14 (d=768) and whether the saturation point shifts with backbone width.","section":"§B.2"}],"recommendation":"major_revision","confidential_remarks":"The author team overlaps with several of the cited merging baselines (Iso-C [24], Core Space [30], TSV-M-adjacent work), so the benchmark protocol and baselines are partly in-house. This is not improper and the comparisons appear competently executed, but it does mean the \"standard task-aware evaluation protocol\" that licenses the per-task prompt selection is largely defined by this same research lineage; the editor may wish to weigh how general that assumption is considered in the broader merging community. The revision burden I recommend is modest — mainly one control evaluation and reframing — and the topic is well suited to the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful move here is simple: stop treating expert training as fixed and use CoOp-style prompts first, then fine-tune the vision encoder. That two-stage recipe (DTEs) lifts WA, TA, TIES, TSV-M, and Iso-C on 8/14/20-task CLIP setups, full FT and LoRA, B/32 and L/14, and still helps in a heterogeneous mix. Separate-expert accuracy stays matched; merged accuracy jumps. Prompt-only multi-task inference is also a strong baseline as task count grows—worth stating clearly, and they do.\n\nWhat they do well: the experimental surface is wide and directionally consistent. Equal-compute check, LoRA + core-space, progressive DTE replacement, smaller Frobenius updates with paired tests, higher cosine similarity to the pretrained backbone, flatter interpolations. Citations sit in the right neighborhood (Task Arithmetic, TIES, TSV-M, Iso-C, CoOp). No circular metric games.\n\nSoft spot, in proportion: the stress-test lands. Merged DTEs are scored with learned prompts; merged Vision-FT with hand-crafted templates. Prompt-only with a frozen backbone already sits near WA-DTE and well above WA-Vision-FT, so part of Table 1/2 is interface quality, not only “more mergeable task vectors.” Interpolation inherits the same split. The Frobenius drop (~12%) is the cleanest interface-independent signal and it is modest. A same-prompt control (both sides on Stage-1 prompts, or both on templates) would pin down the decomposition. Task-aware evaluation is standard in this benchmark culture but load-bearing. Missing multi-seed bars and no code in the manuscript are ordinary for the venue, not fatal.\n\nWho it is for: people who actually merge CLIP/VLM experts or ship multi-task adapters. Not a theory paper. Central effect looks real enough that a serious referee should see it; the interface confound is a revision item, not a desk-reject. I would engage, cite if I am writing on mergeability or PEFT composition, and bring it to reading group.","headline":"Real training-side lever for mergeability with broad tables, but the headline gains partly ride on a better text interface that the Vision-FT baseline never gets.","tokens_in":19494,"tokens_out":543,"would_cite":true,"duration_ms":19987,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Training experts in two stages—prompts first, then the vision encoder—makes them merge into stronger multi-task models without changing the merger.","keywords":["model merging","prompt learning","CLIP","task vectors","Dual-Tuned Experts","LoRA","multi-task learning","vision-language models"],"falsifier":"On the same 8/14/20-task CLIP suites, train matched DTE and Vision-FT experts to equal separate accuracy, merge with the same operator and lambda protocol, and check whether DTE merges no longer beat Vision-FT merges—or whether the cosine-similarity and interpolation-flatness gaps disappear.","tokens_in":19195,"feed_emoji":"🔀","tokens_out":855,"duration_ms":16932,"temperature":0.7,"pith_summary":"Model merging usually treats expert training as fixed and tries to fix interference only at merge time. This paper argues the opposite: how you train each expert strongly shapes how well those experts combine. It first shows that learned text prompts alone, with a frozen CLIP backbone, already give competitive multi-task accuracy because no backbone weights need to be merged. Building on that, Dual-Tuned Experts (DTEs) learn those prompts first, then fine-tune the vision encoder with prompts held fixed. The second stage still reaches full expert accuracy, but the parameter updates stay smaller and closer to the pretrained model, so standard mergers (weight averaging, task arithmetic, TIES, TSV-M, Iso-C) recover more per-task performance. The gains hold for full fine-tuning and LoRA, across CLIP sizes and 8/14/20-task suites, and even when DTE and ordinary experts are mixed in one merge.","feed_headline":"Train prompts first, then vision: merged models jump in accuracy","feed_subtitle":"Two-stage CLIP experts stay as strong alone but combine far better under ordinary mergers","key_machinery":"Dual-Tuned Experts (DTEs): a two-stage recipe that first absorbs task semantics into a small set of learned context tokens, then fine-tunes the vision encoder against those fixed prompts, shrinking task-vector magnitude and representation drift so linear and spectral mergers interfere less.","core_discovery":"Dual-Tuned Experts—Stage-1 unified-context prompt learning with a frozen backbone, then Stage-2 vision-encoder fine-tuning with those prompts fixed—match separate-expert accuracy of ordinary one-stage vision fine-tuning while consistently raising merged multi-task accuracy under the same merging operators on CLIP vision benchmarks.","pith_inferences":["If task identity were unknown, one would still need a cheap task router or a way to merge prompt sets; the paper’s multi-task numbers assume that router is free.","The pattern suggests other PEFT first stages (adapters, visual prompts) might similarly shrink backbone drift before a second fine-tune, not only text prompts on CLIP.","Platforms that ship only weight deltas may need a lightweight side-channel for per-task prompts if DTE-style experts become common."],"forward_implications":["Expert-training recipe becomes a first-class knob for merge quality, orthogonal to designing new merge operators.","Simple weight averaging of DTEs can beat advanced mergers of ordinary full fine-tunes when many tasks are combined.","The same two-stage idea transfers to LoRA experts and reduces the gap between naive full-space LoRA merges and core-space methods.","Partially replacing ordinary experts with DTEs in a mixed pool steadily raises merge accuracy, so partial adoption still helps."],"fun_headline_variants":["Prompts then vision: dual-tuned experts merge cleaner on CLIP","Stage-1 prompts, stage-2 vision: higher merge compatibility","Dual-tuned experts cut task drift, lift merged multi-task accuracy","Train prompts first on frozen backbone, then tune vision for merges","DTEs match solo strength, raise merged CLIP scores under standard ops"],"cache_read_input_tokens":128,"weakest_assumption_plain":"At test time the system already knows which task is running, so it can pick the right learned prompt set (or hand-crafted templates) without merging or guessing prompts.","fun_headline_variants_meta":{"raw":{"variants":["Prompts then vision: dual-tuned experts merge cleaner on CLIP","Stage-1 prompts, stage-2 vision: higher merge compatibility","Dual-tuned experts cut task drift, lift merged multi-task accuracy","Train prompts first on frozen backbone, then tune vision for merges","DTEs match solo strength, raise merged CLIP scores under standard ops"]},"model":"grok-4.5","effort":"low","cost_usd":0.003427,"raw_usage":{"total_tokens":1107,"prompt_tokens":695,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":34268000,"prompt_tokens_details":{"text_tokens":695,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":338,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":695,"tokens_out":74,"duration_ms":7452,"temperature":1.0,"reasoning_tokens":338,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T14:08:37.941953+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same 8/14/20-task CLIP suites, train matched DTE and Vision-FT experts to equal separate accuracy, merge with the same operator and lambda protocol, and check whether DTE merges no longer beat Vision-FT merges—or whether the cosine-similarity and interpolation-flatness gaps disappear.","supporting_citations":[],"review_version":1}