{"id":"7cd89b10-60a8-45f4-83dd-6ff799bf5aa7","arxiv_id":"2608.08107","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NeuPAT uses neuron-level probing to freeze language-critical neurons and regularize shared neurons during multimodal instruction tuning, recovering most lost language performance while keeping multimodal scores comparable.","lead":"This paper introduces NeuPAT, a training method that identifies which neurons in a language model are important for text skills and protects them during multimodal fine-tuning. The approach recovers most of the language performance lost when models are trained on image-and-text data, without sacrificing multimodal ability.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Eq. (1) importance proxy is still only correlationally validated; current ablations never vary the score formula, so the claimed text-vs-vision causal role distinction is the least secure link.","rationale":"The reader's weakest assumption is exactly the load-bearing point: Eq. (1) is a correlational proxy, and the paper explicitly disclaims causal status in Section 8. The existing ablations are better than typical because random-partition and fixed-ratio partition preserve role counts and layer proportions, so they control for the amount of freezing/regularization. However, they do not control for the composition of the score itself. A decisive test requires varying the score formula while holding role sizes fixed. If a simpler score (activation only, norm only, or even the vision score) yields the same language recovery, then the central conceptual claim about language-preserving vs. multimodal-adaptive neurons is not supported, even though the engineering result may still hold. The paper earns credit for multi-backbone generalization and for ablating role identities, but those experiments do not close this gap. A secondary issue is that the headline '94.5% recovery' is never formally defined: recomputing from Table 1 gives 101.4% for the default backbone, and 94.5% only emerges as a mean of per-backbone aggregate ratios across six backbones, with values exceeding 100% for some models. This should be clarified regardless, but it is secondary to the proxy-validity concern. Given these unresolved points, the reader's CONDITIONAL verdict remains appropriate; adding the proposed component ablations and a formal recovery metric would either vindicate or revise the central claim.","tokens_in":22314,"tokens_out":11903,"duration_ms":127883,"concrete_test":"On the default Qwen3-4B setup, run matched-size role-allocation ablations that vary only how the language-freeze set is chosen: (i) full Eq. (1) text score, (ii) activation-RMS only (drop the ||W_out|| factor), (iii) output-weight-norm only (drop the activation RMS factor), (iv) vision score V_l in place of T_l, and (v) lowest text-importance (reserve) neurons. Keep other role sets and training settings identical to the Table 4/6 ablations. If variants (ii)-(v) recover at least 90% of the language gain of variant (i) over Vanilla Tuning (approximately 4.6 of the 5.14-point gain in Table 1), the modality-specific product in Eq. (1) is not the causal ingredient and the central insight is unverified. If variant (i) substantially outperforms all alternates, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"NeuPAT's central claim rests on Eq. (1)'s modality-associated importance score identifying neurons whose freezing preserves language. Section 8 explicitly concedes that 'the proposed neuron allocation strategies are response-based functional approximations rather than causal explanations of model behavior.' Tables 4-6 show that the full NeuPAT allocation beats random and fixed-ratio partitions, but those baselines keep the same numbers of frozen/regularized neurons and only randomize identity; they never vary the score formula itself. In particular, no ablation separates the activation-RMS term from the output-weight-norm term, and no ablation freezes the vision-preferred set instead of the text-preferred set at matched size. If freezing the vision-preferred set or the highest-norm neurons recovers the same language performance, the claimed text-vs-vision role distinction is not causal, and the mechanism could simply be 'freeze a well-chosen subset of high-magnitude neurons.' Because all language gains are reported as single runs without error bars, a proxy failure in some layers or model families would not be visible in Tables 4-6.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NeuPAT, a neuron-level plasticity allocation method for multimodal instruction tuning that aims to preserve the language capabilities of a pretrained LLM while still adapting it to visual instruction data. NeuPAT first computes modality-associated importance scores for each neuron as the product of its output weight norm and its activation RMS on small text-only and vision probing sets, then partitions neurons into language-associated, multimodal-adaptive, shared, and reserve roles via cumulative importance thresholds. During tuning, language-associated neurons are frozen, multimodal-adaptive and reserve neurons receive full updates, and shared neurons receive input-side L2 and output-side cosine regularization. Experiments compare NeuPAT with vanilla tuning, LoRA, EWC, WINGS, TIES, Locate-then-Merge, and PlaM across six LLM backbones, eleven text benchmarks, and five multimodal benchmarks, reporting that NeuPAT recovers most of the language degradation caused by vanilla tuning while maintaining comparable multimodal performance. Appendices provide sensitivity analyses over the importance threshold, probing set size, and regularization coefficients, plus complete cross-backbone and ablation tables.","tokens_in":22568,"tokens_out":5761,"duration_ms":55523,"significance":"If the proposed mechanism is validated, NeuPAT would be a practical, architecture-agnostic alternative to text replay, architectural modification, and post-hoc merging for language-preserving multimodal expansion. The paper has notable strengths: broad empirical coverage across six model families and scales, role-wise ablation studies, sensitivity analyses over all free hyperparameters, and complete results in the appendix. The core limitation is that the central text-versus-vision role distinction rests on an importance proxy whose causal validity is not tested, and the headline recovery numbers are internally inconsistent. With additional ablations that vary the score formula itself and more careful reporting of run-to-run variability, the contribution would be substantially stronger.","major_comments":[{"comment":"The headline recovery statistic is internally inconsistent. The abstract states that NeuPAT recovers 94.5% of language capability degradation on 11 language benchmarks, the introduction reports 90.0% on 4 language reasoning benchmarks, and Table 1 yields 5.14/5.07 ≈ 101.4% recovery, with NeuPAT slightly surpassing the original LLM average (49.06 vs. 48.99). These numbers are not equivalent, and no recovery formula is defined. Please define the recovery metric explicitly, apply it consistently to the full benchmark set, and report the resulting value for all claims.","section":"Abstract; §1; §5.2.1, Table 1"},{"comment":"The central claim that the neurons identified by Eq. (1) play distinct text-preserving versus vision-adaptive roles is not yet supported by the ablations. Tables 4–6 vary the allocation strategy and the update constraint while keeping the importance score formula fixed; they never vary the score formula itself. I request ablations that (i) omit the output-weight-norm term, (ii) omit the activation-RMS term, (iii) swap or shuffle the text and vision scores at matched set sizes, and (iv) freeze the vision-preferred set instead of the text-preferred set. Without such ablations, the results are consistent with the weaker explanation that any well-chosen high-magnitude subset of neurons can be frozen to preserve language, rather than with the claimed modality-specific mechanism. The concession in Section 8 that the allocation is a response-based functional approximation should be reflected in the abstract and conclusions unless these ablations are added.","section":"§4.1–4.3; §5.2.3, Tables 4–6; §8"},{"comment":"All experimental results are reported as single runs without error bars or significance tests, and the default hyperparameters τ_a = 0.8 and λ_in = λ_out = 0.1 are selected from sensitivity sweeps on the same evaluation benchmarks used for the headline comparisons. Because one headline claim is a 0.07-point average 'surpass' of the original LLM, run-to-run noise could change the conclusion. Please report multiple seeds or at least per-benchmark variance for the main tables, and either select hyperparameters on a validation split or explicitly state that the sensitivity analysis doubles as test-set model selection.","section":"§5.2; Tables 1–3, 8–10"}],"minor_comments":[{"comment":"Notation is inconsistent: Algorithm 1 uses C^{text}, C^{vision}, C^{high}, and C^{low}, while Section 4.2 and Table 4 use C^{lang}, C^{multi}, C^{shared}, and C^{reserve}. Please unify the notation throughout.","section":"Algorithm 1; §4.2"},{"comment":"The definition of C^{reserve} is ambiguous in the inline typesetting: it should be the complement of T_l ∪ V_l with respect to all neurons in the layer, but the current rendering can be read as the union itself. Please rewrite the equation with an explicit complement operator.","section":"Eq. (6)"},{"comment":"The sentence 'the importance scores are normalized using layer-wise normalization using Eq. 2' should reference Eq. (3), because Eq. (2) defines only the raw scores s^V and s^T, not the normalized scores v and t.","section":"§4.1"},{"comment":"The multimodal benchmark is abbreviated as 'MMS' in Table 2 but as 'MMStar' in Table 3 and Section 5.1.3; please use a single abbreviation.","section":"Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the central direction is salvageable. I do not see a fatal flaw, but the inconsistent recovery numbers, the untested importance-score formula, and the absence of run-to-run variability all affect load-bearing claims and need to be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the framework is well-built and the experiments are impressively broad: six backbones, eleven text and five multimodal benchmarks, plus ablations that actually test the design choices. Second, the central mechanistic story — that the proxy in Eq. (1) identifies neurons with distinct causal roles — is softer than the abstract implies, and the paper's own Section 8 admits as much.\n\nThe genuinely new piece is combining modality-specific activation importance with role-based freezing/regularization during multimodal tuning. That is not in the cited literature, and it is a sensible, lightweight idea. Credit where due: the authors run a proper generalization check across model families, and the ablations support the overall allocation strategy over global or random schemes. The honest limitations section also helps.\n\nSoft spots, in order of softness. The recovery metric is reported inconsistently: the abstract says 94.5%, the introduction says 90.0%, and Table 1 actually has NeuPAT slightly surpassing the original LLM. That needs reconciliation. All results are single runs, so the +2–5 point gains over LoRA/EWC lack error bars. Hyperparameters (tau, lambdas) are chosen via sensitivity analysis on the same benchmarks used for the headline numbers, which is mild test-set fitting; a separate validation split would clean this up. There is no text-replay baseline, which is an obvious competitor. And the stress-test concern about Eq. (1) is fair: the ablations vary allocation identity but never the score formula, so we cannot tell if the mechanism is really 'text vs vision' or simply 'freeze some high-magnitude neurons.' The paper's own caveat in Section 8 is the right framing, but then the title and abstract should not lean so hard on the causal language.\n\nThese are fixable issues. The central claim — that neuron-aware allocation can recover most lost language performance while keeping multimodal scores flat — is supported by the tables as reported, conditional on the missing error bars. That claim is worth refereeing, not desk-rejecting.\n\nI would send this out for serious peer review. The right reader is someone working on MLLM forgetting or neuron-level interpretability, who will benefit from the method and can push on the proxy validation. I'd cite it once the reporting is tightened, and I'd bring it to our reading group as a maybe — there is real content but we should see the revision first.","headline":"A solid, broad, and useful method for language-preserving multimodal tuning; the causal neuron-role claim outruns the evidence, and the reporting needs tightening.","tokens_in":23041,"tokens_out":2609,"would_cite":true,"duration_ms":27187,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that freezing language-sensitive neurons while updating vision-adaptive ones during multimodal instruction tuning recovers 94.5% of lost language ability without sacrificing multimodal performance.","keywords":["multimodal large language models","language forgetting","neuron plasticity","neuron importance","modality preference","instruction tuning","update allocation","catastrophic forgetting"],"falsifier":"Take a tuned NeuPAT model and ablate the top-ranked language-associated neurons on reasoning benchmarks such as BBH and GSM8K; if those benchmarks stay roughly unchanged, the identified neurons are not causal for preservation. Alternatively, invert the importance ranking during training, freezing what the probe calls vision neurons and updating what it calls language neurons, and check whether the language average falls back toward the vanilla-tuned level; the ranking, not the overall freeze-and-update regimen, is what the preservation claim predicts.","tokens_in":22160,"feed_emoji":"🧠","tokens_out":9976,"duration_ms":91089,"temperature":0.7,"pith_summary":"Turning a pretrained language model into a multimodal model by visual instruction tuning usually costs pure-text reasoning ability: the paper measures a roughly five-point drop over 11 language benchmarks on its default backbone. NeuPAT is a training-time method that treats this degradation as a neuron-allocation problem rather than a data or architecture problem. Before tuning, it probes the model with text-only and visual samples and scores every neuron by how strongly it responds to each modality, then freezes language-sensitive neurons, fully updates vision-adaptive and low-importance neurons, and only regularizes neurons important to both. On the 11 language benchmarks the paper reports recovering the large majority of the lost capability, with the average climbing from 43.92 to 49.06 and slightly surpassing the original LLM's 48.99, while the multimodal average stays at 61.05 versus 60.98. The claim is that heterogeneous neuron plasticity is a general, architecture-agnostic resource that can be allocated explicitly.","feed_headline":"94.5% of lost language returns with NeuPAT","feed_subtitle":"Per-neuron update rules hold the 11-benchmark language average at LLM level while keeping vision scores.","key_machinery":"The load-bearing object is the modality-associated importance score $s_{l,u}(\\mathcal{D}) = \\|W_{l,u}^{\\mathrm{out}}\\|_F \\cdot \\mathrm{RMS}_{x\\in\\mathcal{D}, t\\in V(x)} \\|h_{l,u}(x_t)\\|_2$, an activation-aware estimate of how much a neuron writes to the residual stream under a given input modality. From its text and vision variants the paper forms a modality preference $P_{l,u}=v_{l,u}-t_{l,u}$ and an overall importance $I_{l,u}=(v_{l,u}+t_{l,u})/2$, then for each layer selects the smallest neuron sets whose cumulative importance reaches a coverage threshold of 0.8. The four role sets (language, multimodal, shared, reserve) determine the update rules, with shared neurons regularized by an input-side L2 penalty plus an output-side cosine penalty; this mapping from measured response to role assignment is what converts an observation about plasticity into a training algorithm.","core_discovery":"The paper's central discovery is that a pretrained LLM backbone does not adapt uniformly when visual instruction tuning is applied; neurons divide into adaptively distinct populations. A neuron's modality preference is estimated by an importance score that combines its output-side weight norm with its activation magnitude under probing inputs: neurons whose importance is dominated by text are language-associated, those dominated by vision are multimodal-adaptive, those important to both are shared, and those important to neither are reserve. NeuPAT freezes the language-associated neurons, fully trains the multimodal-adaptive and reserve neurons, and regularizes shared neurons with an input-side L2 penalty plus an output-side cosine constraint, so the residual-stream direction of shared neurons cannot drift far from pretrained behavior. The paper argues this role-aware allocation is why, across six LLMs of different families and sizes, the 11-benchmark language average is preserved (49.06 versus the original LLM's 48.99) while the 5-benchmark multimodal average holds at 61.05 versus vanilla tuning's 60.98.","pith_inferences":["Beyond the paper, the same probing-plus-allocation recipe could be re-targeted at other capability axes, such as coding, instruction following, or safety, by choosing probing sets that isolate those skills; the paper does not test this.","Beyond the paper, the main comparison set is dominated by methods that intervene after training or with extra supervision, so a head-to-head against text-replay at matched compute would show whether neuron allocation and data replay are substitutes, complements, or partly redundant.","Beyond the paper, a causal test of the mechanism would be to measure language benchmarks after ablating the top language-associated neurons in the tuned model; a sharp drop would show the identified neurons are load-bearing, while no drop would suggest the preservation comes from elsewhere in the procedure."],"forward_implications":["Multimodal instruction tuning can be made language-preserving without text replay, extra data, or architectural changes: the same recipe, probe, assign roles, and constrain updates, is the intervention.","The method is architecture-agnostic in the sense that the probing and allocation steps read only forward activations and weight slices, so they should transfer to new LLM families as long as the importance estimate is recomputed for each backbone.","Because the probing set is small and used only for forward passes, the overhead before training is a few hundred forward passes rather than a second optimization loop.","The ablations imply that the balance is load-bearing: freezing vision neurons or updating language neurons each erodes one side of the text-multimodal trade-off, so the four-role split, not just any freeze, is what preserves language.","Recovering the original LLM's language average while holding multimodal performance means practitioners could expand an LLM into an MLLM without budgeting for a separate language-preservation stage."],"supporting_citations":[{"why":"Supplies the activation-aware importance estimation that Eq. (1) adapts into separate text and vision scores.","marker":"Sun et al. 2024"},{"why":"Defines the LoRA baseline, a parameter-efficient adaptation method NeuPAT must match or beat on language preservation.","marker":"Hu et al. 2022"},{"why":"Defines the EWC baseline whose Fisher-based parameter protection is compared against NeuPAT's neuron allocation.","marker":"Kirkpatrick et al. 2017"},{"why":"Provides WINGS, the architecture-based forgetting-mitigation baseline that achieves the highest multimodal average in Table 2.","marker":"Zhang et al. 2024"},{"why":"Provides TIES, the task-vector merging baseline that NeuPAT is compared with as a post-hoc alternative.","marker":"Ratzlaff et al. 2024"},{"why":"Provides Locate-then-Merge, the neuron-level merging baseline that motivates NeuPAT's training-time alternative.","marker":"Yu and Ananiadou 2025"},{"why":"Provides PlaM, a training-free merging baseline for visual grounding used in the comparison table.","marker":"Wang et al. 2026"},{"why":"Provides the visual instruction data for Stage-2 tuning and the source of the vision probing set.","marker":"Liu et al. 2024b"},{"why":"Provides the evaluation harness that produces the 11 language benchmark scores.","marker":"Gao et al. 2024"}],"fun_headline_variants":["NeuPAT recovers 94.5% of lost language in multimodal tuning","Neuron-aware allocation tuning preserves language in MLLMs","94.5% language recovery via neuron-aware plasticity allocation","NeuPAT freezes language neurons, trains vision neurons"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the importance score from Eq. (1) identifies neurons whose preservation actually protects language ability; the paper's own Section 8 calls the allocation a response-based functional approximation rather than a causal explanation, so if that proxy identifies only which inputs a neuron tends to see, freezing the wrong neurons could hurt both text and vision performance.","fun_headline_variants_meta":{"raw":{"variants":["NeuPAT recovers 94.5% of lost language in multimodal tuning","Neuron-aware allocation tuning preserves language in MLLMs","94.5% language recovery via neuron-aware plasticity allocation","NeuPAT freezes language neurons, trains vision neurons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":3021,"prompt_tokens":923,"completion_tokens":2098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2028}},"tokens_in":539,"tokens_out":2098,"duration_ms":27813,"temperature":1.0,"reasoning_tokens":2028,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:24:26.774470+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a tuned NeuPAT model and ablate the top-ranked language-associated neurons on reasoning benchmarks such as BBH and GSM8K; if those benchmarks stay roughly unchanged, the identified neurons are not causal for preservation. Alternatively, invert the importance ranking during training, freezing what the probe calls vision neurons and updating what it calls language neurons, and check whether the language average falls back toward the vanilla-tuned level; the ranking, not the overall freeze-and-update regimen, is what the preservation claim predicts.","supporting_citations":[],"review_version":1}