{"id":"93d3833d-3e37-4533-8f8c-57c0d19a9ae5","arxiv_id":"2505.21191","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Activation-frequency analysis identifies sparse units in LLMs that respond to instructions; same-category instructions share more of these units than different-category ones, and fine-tuning measurably changes the sets.","lead":"This paper introduces SPARCOM, a method that picks out the neurons and experts in large language models that fire most strongly for specific instruction types, and applies it to LLaMA, Mistral, and Qwen models before and after fine-tuning. It reports that these sparse components are partly shared across instruction types and partly specialized, and that their layer distribution shifts after fine-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The identification step in Eqs. (6)–(7) confounds instruction-specific neurons with category-level lexical overlap; no token-matched control is run, so the same-type/cross-type separation and fine-tuning changes may be surface artifacts.","rationale":"The paper's descriptive apparatus is internally consistent, and the balanced six-category dataset plus the inclusion of dense and MoE model families provide useful groundwork. The identification step is nonetheless load-bearing: every downstream metric (Eqs. 10, 13-16) is computed on the sets S(I) produced by Eq. (7). The reader correctly identified the activation-frequency proxy and the unspecified epsilon; my concern is a more specific, testable consequence of that proxy, namely that it is sensitive to surface token identity and no control separates category-level lexical similarity from instruction-level functional similarity. That is why I mark agreement as partial rather than full. The paper's own Section 5.1 acknowledges vocabulary overlap is 'inevitable,' and the limitations section's deferral of actionable use matches the reader's point that 'critical role' is asserted rather than demonstrated. I do not see an internal inconsistency severe enough to reject; the appropriate disposition remains conditional pending the token-overlap control, an explicit epsilon with sensitivity analysis, and ideally an intervention (e.g., ablating or amplifying the identified components and measuring instruction-following accuracy). Because the reader's verdict is already CONDITIONAL, my recommendation is UNCHANGED.","tokens_in":13859,"tokens_out":5797,"duration_ms":63902,"concrete_test":"Run SPARCOM on a matched control set: for each of the 1,200 HexaInst instructions, randomly permute the tokens within the instruction (or resample words from the same category's unigram distribution) to destroy instruction semantics while preserving token identity, length, and category vocabulary. Using the same epsilon and the same pairwise metrics, recompute the same-type/cross-type similarity matrix behind Figure 3 and the before/after fine-tuning Jaccard values in Table 1. If the diagonal-vs-off-diagonal gap persists in the permuted controls, the reported uniqueness is attributable to token overlap rather than to instruction-specific neurons.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Eqs. (6)–(7): a neuron is called instruction-specific if its positive-activation frequency over the raw instruction tokens falls in the top epsilon percentile. Because the criterion is applied to surface tokens, any neuron that responds to category-specific vocabulary or shared template words will be selected. HexaInst categories are drawn from different benchmarks and generation prompts (Section 4.1), so same-category instructions share characteristic words ('classify', 'code', 'summarize', 'matrix', 'median'), while cross-category instructions share fewer. The same-type versus cross-type Jaccard separation in Figure 3 and the before/after fine-tuning changes in Table 1 could therefore reflect token overlap rather than instruction-specific computation. Section 5.1 concedes that 'overlap in certain vocabulary is inevitable' but no control is provided: there is no shuffled-token baseline, no matched non-instruction control, and no permutation test against random neuron sets. Since the identification step is the foundation of all three SPARCOM components, a lexical confound would undercut the generality, uniqueness, and claimed critical role of ISNs and ISEs. The limitations section also defers actionable use to future work, consistent with the absence of any causal intervention supporting the abstract's 'critical role' assertion.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HEXAINST, a small balanced instruction dataset spanning six categories (classification, code, general QA, generation, math, summarization), and SPARCOM, a three-step framework for identifying Instruction-Specific Neurons (ISNs) in dense LLMs and both ISNs and Instruction-Specific Experts (ISEs) in MoE models. ISNs are defined as neurons whose positive-activation frequency over instruction tokens falls in the top epsilon percentile (Eqs. 5-7); ISEs are defined as the top-k routed experts. The framework then measures same-type versus cross-type overlap via Jaccard similarity (Eq. 10) and expert activation correlation (Eqs. 13-14), and compares ISN/ISE distributions before and after fine-tuning (Eqs. 15-16). Based on experiments on LLaMA-2, Mistral-7B, and Qwen1.5-MoE-A2.7B, the authors claim functional generality and uniqueness of these sparse components and a 'critical role' in instruction execution, with fine-tuning refining existing and adding new ISNs while preserving overall layer-wise structure.","tokens_in":14103,"tokens_out":4028,"duration_ms":47818,"significance":"If the central claims hold, the paper would provide a sparse-component account of instruction-following behavior, complementing prior work on language-specific and skill-specific neurons and offering a practical target for future interpretability and fine-tuning interventions. The paper's strengths are its cross-model scope (four model families/sizes, dense and MoE), its explicit, computable formulas for activation frequency and overlap, and the construction of a balanced category-level instruction dataset with both natural and AI-generated instances. The framework is simple enough to be reproduced, which is a virtue. However, the empirical content is correlational: the identification step is a thresholded percentile selection with an unspecified epsilon, no baseline controls are run against lexical or template overlap, and no significance testing or error bars accompany the central figures. The dataset is small (1,200 instructions, 200 per category) and the paper states that code and data will be released only upon acceptance. I therefore see this as a promising but currently under-supported set of findings rather than a demonstrated mechanism.","major_comments":[{"comment":"The value of the percentile threshold epsilon in Eq. (7) is never specified in the paper, and no sensitivity analysis is provided. Since S(I) is the input to every downstream overlap and correlation metric (Eqs. 10, 14-16, Tables 1-2, Figure 5), the quantitative results are uninterpretable until epsilon is given a concrete value and shown not to drive the main conclusions. A sweep over a range of percentiles, with a report of the stability of Figures 3-4, is necessary.","section":"Section 3.1, Eq. (7)"},{"comment":"The ISN identification operates on raw instruction tokens. The six HEXAINST categories necessarily share category-level vocabulary and template words (e.g., 'classify', 'code', 'summarize', 'matrix', 'median'), so neurons selected for high positive-activation frequency may be responding to surface lexical features rather than to instruction-specific computation. The paper itself concedes in Section 5.1 that 'overlap in certain vocabulary is inevitable', but it provides no control: there is no token-shuffled baseline, no matched non-instruction text, and no permutation test over random neuron sets. Without such a control, the same-type/cross-type separation in Figure 3 and the fine-tuning shifts in Table 1 could be artifacts of lexical overlap.","section":"Section 3.1, Eqs. (5)-(7) and Section 4.1"},{"comment":"All conclusions about same-type versus cross-type separation are based on point estimates with no error bars, confidence intervals, or significance tests. Appendix D says the similarity cells use 300 randomly sampled instances, but the paper does not report the variance or the number of sampled neurons. A permutation or bootstrap test is required to substantiate claims that the diagonal is 'notably darker' than chance, especially given that the thresholding procedure itself guarantees a sparse set of ISNs for every instruction.","section":"Section 3.2, Eqs. (10)-(14) and Figures 3-4"},{"comment":"The ISE identification step is not a selection procedure: every expert in the top-k routing set is labeled an Instruction-Specific Expert. This makes the subsequent correlation analysis in Eqs. (13)-(14) a description of overall routing behavior rather than a property of instruction-specific sparse components. A criterion for selectivity across instruction types (e.g., an expert is more frequently routed for one type than others) is needed before the paper can claim unique or general ISEs.","section":"Section 3.1, 'ISEs Identification' and Eqs. (8)-(9)"},{"comment":"The abstract and conclusion assert that ISNs and ISEs play a 'critical role' in instruction execution, but the paper performs no causal intervention such as activation patching, ablations, or neuron editing. The Limitations section explicitly defers 'developing effective strategies to leverage these components' to future work, which is inconsistent with the strong causal claim. The evidence is correlational, so the language should be softened to 'associated with' or 'co-occur with' until causal evidence is provided.","section":"Section 5.3 and Conclusion"}],"minor_comments":[{"comment":"The text contains numerous typos and spacing errors (e.g., 'theun-derlying', 'posesspecific neuronscrucial', 'HEXAINST' vs. 'HexaInst' inconsistency), which should be corrected before publication.","section":"Abstract and Introduction"},{"comment":"Equation (13) is typeset incorrectly; the square root symbols and fraction structure make the formula difficult to read, and the placement of the indices in the numerator should be fixed.","section":"Section 3.2, Eq. (13)"},{"comment":"The figure caption says 'ISN Numbers' but the figure panels are labeled 'Finetuned Models' and 'Vanilla Models'; the caption should explain what is plotted on the y-axis and how the layer index is defined.","section":"Figure 5"},{"comment":"The statement that '300 randomly sampled instances' are used for the Jaccard and Pearson calculations should appear in the main text near Figures 3 and 4, because it affects the interpretation of those figures, and the precise sampling procedure (seeds, replacement) should be reported.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict aligns with my reading: the paper's central contribution is a plausible but under-controlled analytical framework. The unspecified epsilon is a simple but genuine reproducibility gap, and the absence of a lexical-overlap baseline is the most scientifically serious issue because it bears directly on the claimed uniqueness and generality of ISNs. However, these gaps are fixable within the manuscript's scope: a sensitivity analysis for epsilon, a token-matched or permuted control, and significance testing would substantially strengthen the central claims. The 'critical role' phrasing in the abstract and conclusion should be toned down unless causal evidence is added. I do not see a load-bearing error that would require rejection, but the manuscript is not yet ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for two things: the HexaInst dataset and the MoE expert analysis. The dataset is genuinely more careful than most instruction collections—six categories, 100 synthetic and 100 natural examples each, manual validation, and contamination checks. The comparison of expert activation across instruction types before and after fine-tuning is also new territory relative to the language-neuron literature the paper builds on. The same-type vs. cross-type overlap matrices and the layer-wise trends are internally consistent and give a plausible descriptive picture: stable overall structure, some refinement and addition of sparse components after fine-tuning.\n\nThe soft spots are real, and the stress-test note lands. The identification step in Eqs. (6)-(7) selects top-percentile neurons by activation frequency over surface tokens, so a sparse set exists for every instruction by construction. The interesting empirical content is the same-type vs. cross-type separation and the before/after shifts, but those can be driven by shared vocabulary or template words. The paper concedes that overlap in vocabulary is inevitable but runs no shuffled-token baseline, no matched non-instruction control, and no permutation test against random neuron sets. Epsilon is never given a value or a sensitivity analysis. There are no error bars or significance tests anywhere, and the \"critical role\" claim in the abstract has no causal support—the limitations section explicitly defers actionable use to future work. The ISE definition is also loose: calling every activated expert instruction-specific doesn't distinguish specialization, so the expert-level results mainly show routing stability.\n\nNone of this means the paper is worthless. The dataset is a real artifact, the MoE comparison is a useful extension, and the descriptive findings could survive proper controls. The citation pattern is fair: it builds on Tang et al. and related neuron-localization work and adds a balanced multi-category instruction set and expert-level analysis. The math is straightforward and the writing is honest about limitations. What's missing is verification: epsilon, baselines, error bars, and code/data release. The paper says code and data will be released upon acceptance, which makes the gaps harder to check now.\n\nWho is this for? Researchers working on mechanistic interpretability of instruction tuning, especially MoE routing. It deserves a serious referee and a revision request—send it to review, but ask for the missing implementation details and null controls. If those don't come, the central claims should not stand.","headline":"Useful descriptive work with a good balanced dataset and a novel MoE angle, but the 'critical role' claim outruns the evidence and the identification method needs stronger baselines before the core findings can be trusted.","tokens_in":14675,"tokens_out":1664,"would_cite":true,"duration_ms":24075,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that instruction-following in LLMs is carried by a sparse set of instruction-specific neurons and experts, and that fine-tuning improves it by refining these components and adding new ones while leaving the layer-wise…","keywords":["instruction-specific neurons","instruction-specific experts","mixture-of-experts","mechanistic interpretability","instruction tuning","sparse components","fine-tuning analysis","LLM interpretability"],"falsifier":"Ablate or amplify the union of instruction-specific neurons $S(I)$ for one category (say code) in LLaMA-2-Chat-7B and measure whether code-instruction performance drops (or rises) far more than performance on other categories; the paper's identification criterion predicts a selective effect. A cheaper partial check is to recompute the Jaccard similarity matrices under a grid of $\\epsilon$ values and see whether the same-type diagonal remains darker, since the reported patterns would be artifacts if they flip when $\\epsilon$ moves.","tokens_in":13644,"feed_emoji":"🧠","tokens_out":8740,"duration_ms":85185,"temperature":0.7,"pith_summary":"This paper asks where instruction-following lives inside a large language model. It tries to establish that the ability is carried by a small set of 'instruction-specific' components: individual neurons in dense models like LLaMA and Mistral, and both neurons and experts in mixture-of-experts models like Qwen-MoE. To make the case, the authors build a balanced instruction dataset across six categories and a framework that identifies the neurons and experts that activate most consistently for each instruction, then measure how those sets overlap across same-type and different-type instructions and how they shift after fine-tuning. Their conclusion is that fine-tuning works by refining existing instruction-specific components and adding new ones, while the overall layer-wise logic of instruction processing stays unchanged. If true, this means instruction-following can be localized, inspected, and potentially edited in sparse parts of the network.","feed_headline":"Fine-tuning edits sparse neurons, not the whole LLM","feed_subtitle":"A new framework locates the neurons and experts that carry each instruction type and tracks how chat models change them.","key_machinery":"The load-bearing object is the activation-frequency set $S(I)$: for an instruction $I$, each neuron's activation frequency $p_{ij}(I)$ is the fraction of tokens where that neuron's post-activation output is positive, and $S(I)$ selects the top $\\epsilon$ percentile of these frequencies across all layers. This set is the unit of analysis for everything else. ISN-level claims are computed as Jaccard similarities between $S(I)$ sets for pairs of instructions; ISE-level claims are computed as Pearson correlations between flattened expert-activation-frequency vectors $F(I)$. The comparison before and after fine-tuning uses the same metrics between a vanilla model and its chat version. The paper also introduces a three-stage mechanistic reading of the layer distribution: many ISNs in early layers (shallow encoding of instructions), few in middle layers (generalized understanding), and many again in late layers (decoding output tokens), with MoE models showing a longer early increase. The HEXAINST dataset, balanced across six categories and two sources, is the controlled stimulus that lets overlap patterns be attributed to instruction type rather than data source.","core_discovery":"The central claim is that instruction tuning reconfigures sparse components rather than the whole network. For each instruction, SPARCOM ranks every neuron by the fraction of instruction tokens on which it activates above zero, and labels the top $\\epsilon$ percentile as Instruction-Specific Neurons (ISNs); in MoE models, the experts selected most often are labelled Instruction-Specific Experts (ISEs). Using the balanced HEXAINST dataset (six categories, 100 human-curated and 100 AI-generated instructions each), the paper reports that Jaccard similarity between ISN sets is higher for instructions of the same type than for different types, especially for classification, code, and math, which it takes as evidence that these components are functionally unique, while cross-type overlap indicates general instruction components. Comparing vanilla and fine-tuned model pairs, the layer-wise distribution of ISNs is roughly preserved, but fine-tuned models have more specialized ISNs, concentrated in early parsing and late generation layers; the same instruction activates substantially different neurons after fine-tuning (Jaccard similarity roughly 0.4–0.6 across model families), whereas in MoE models expert-routing correlation stays high (0.91–0.94). The paper concludes that fine-tuning refines existing instruction-specific components, adds new ones, and does so without changing the model's fundamental layer-wise instruction-processing structure.","pith_inferences":["A causal test would be to ablate or amplify $S(I)$ for one instruction type and measure whether performance on that type changes selectively; the paper's identification criterion is correlational, so this is the natural next step.","If expert routing stays so stable under fine-tuning, one could fine-tune only the neurons inside experts while freezing the router, which would cut training cost if the claimed mechanism holds.","Because HEXAINST pairs AI-generated with human-curated instructions, the same framework could separate semantic instruction specificity from surface-form confounds by testing whether rephrased versions of the same instruction recruit the same ISNs."],"forward_implications":["Instruction-following behavior can be localized to a small fraction of neurons (and experts in MoE models), so auditing or interpreting a chat model could focus on these sparse components instead of the full network.","Fine-tuning improves instruction following mainly by refining existing instruction-specific components and adding new ones, without re-architecting the layer-wise processing logic, so supervised fine-tuning can be understood as a targeted edit of a stable substrate.","Same-type instructions share specialized components, which implies that an intervention on one instruction category's ISNs could affect an entire category of instructions rather than a single prompt.","In MoE models, expert routing is almost unchanged by fine-tuning (correlation around 0.92), so the improvement from fine-tuning is carried mostly by the neurons inside experts rather than by rerouting which experts are used.","The three-stage layer pattern (parse, generalize, decode) identifies where instruction-specific computation concentrates, suggesting layer-selective fine-tuning or editing strategies that preserve the stable structure while updating the specialized components."],"supporting_citations":[{"why":"Supplies the language-activation-probability method that SPARCOM adapts to identify instruction-specific neurons.","marker":"Tang et al., 2024"},{"why":"Contributes the neuron-level domain-specific interpretation approach and the layer-distribution pattern the three-stage mechanism builds on.","marker":"Huo et al., 2024"},{"why":"Provides the prior claim that fine-tuning enhances existing mechanisms, which the alteration comparison is designed to test at neuron and expert level.","marker":"Prakash et al., 2024"},{"why":"Inspires the three-phase framework the paper uses to interpret ISN counts across layers.","marker":"Zhao et al., 2024"},{"why":"Defines the LLaMA model family on which the dense-model findings are demonstrated.","marker":"Touvron et al., 2023"},{"why":"Defines the Mistral model family used as a second dense-model testbed.","marker":"Jiang et al., 2023"},{"why":"Defines the Qwen model family, including the MoE model used for expert-level analyses.","marker":"Bai et al., 2023"}],"fun_headline_variants":["Fine-tuning edits sparse neurons, not whole LLM","New framework finds instruction-specific neurons and experts","Sparse edits: how fine-tuning targets instruction neurons","Mapping instruction-specific components in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything downstream rests on the assumption that the neurons and experts that fire most often while an instruction is being processed are the components actually executing that instruction, rather than bystanders that happen to activate alongside it; the paper never fixes the percentile cutoff $\\epsilon$ or tests how sensitive its results are to that choice.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuning edits sparse neurons, not whole LLM","New framework finds instruction-specific neurons and experts","Sparse edits: how fine-tuning targets instruction neurons","Mapping instruction-specific components in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3066,"prompt_tokens":1008,"completion_tokens":2058,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":2000}},"tokens_in":624,"tokens_out":2058,"duration_ms":17932,"temperature":1.0,"reasoning_tokens":2000,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:33:05.763238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ablate or amplify the union of instruction-specific neurons $S(I)$ for one category (say code) in LLaMA-2-Chat-7B and measure whether code-instruction performance drops (or rises) far more than performance on other categories; the paper's identification criterion predicts a selective effect. A cheaper partial check is to recompute the Jaccard similarity matrices under a grid of $\\epsilon$ values and see whether the same-type diagonal remains darker, since the reported patterns would be artifacts if they flip when $\\epsilon$ moves.","supporting_citations":[],"review_version":1}