{"id":"385ebef7-3f90-4001-bdca-d201564fa639","arxiv_id":"2505.00509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A k-winner-takes-all training constraint on transformer activations improves interpretability metrics while increasing, not decreasing, a measure of overall activation and weight magnitude.","lead":"The paper introduces a training-time 'self-ablation' mechanism for small transformers that forces only a few neurons and attention heads to be active, and reports that this makes the models more interpretable by several metrics. It also claims that this improved interpretability comes with less overall sparsity, challenging a common assumption.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'Less Sparsity' headline rests entirely on an invalid L1-norm metric; Section 4.5 defines it as normalized weight sum, which is not a sparsity measure, so the paper's central surprise is unsubstantiated until a true sparsity metric is reported.","rationale":"The reader's weakest-assumption analysis correctly identifies the L1-norm metric as the linchpin of the paper's most surprising claim. The manuscript defines L1 norm inconsistently (weight sum in Section 4.5, activation norms in the abstract), and in neither case does it quantify sparsity. Since the central title claim 'Less Sparsity' rests on this single column, the paper cannot be accepted as is. The fix is concrete and easy given the public codebase: report a true sparsity metric (e.g., fraction of near-zero activations) with variance across seeds. The other reported interpretability improvements are plausible and independently supported by ACDC, SAE, and N2G metrics, though those metrics are also single-run measurements; the missing gating-weight equations in Section 3.1 and the lack of error bars are secondary concerns. If the proper sparsity metric confirms the direction, the claim survives; if not, the headline claim should be revised. Therefore the appropriate verdict is CONDITIONAL: accept only after the sparsity measurement is corrected and reported transparently.","tokens_in":13625,"tokens_out":4265,"duration_ms":45473,"concrete_test":"Recompute the sparsity comparison with a true sparsity measure: for the baseline and each self-ablated checkpoint, run 1,000 TinyStories validation samples and record (i) the fraction of MLP post-activation ReLU outputs exactly zero, (ii) the fraction of post-activation values with |x| < 1e-4, and (iii) the fraction of weight elements with |w| < 1e-4, computed per layer and averaged across at least three training seeds. If the ablated models do not show a lower zero-fraction than the baseline on (i) or (ii), the 'decreased overall sparsity' claim fails; additionally, check whether the reported L1 norm correlates with these zero-fractions before it is used as a sparsity proxy.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's most novel and title-bearing claim is that self-ablation decreases overall sparsity. That claim rests entirely on the 'Sparsity (L1)' column in Table 1. Section 4.5 defines L1 norm as 'the normalized sum of the model's weights,' while the abstract and introduction refer to 'L1 activation norms.' Neither definition measures sparsity in any standard sense: sparsity is the fraction of zero or near-zero components, not the sum of their absolute magnitudes. A dense vector can have a low L1 norm if its entries are small, and a sparse vector can have a high L1 norm if its nonzero entries are large. No methodology is given for computing activation sparsity, and the term is used inconsistently between 'weight sparsity' and 'activation sparsity.' The other reported metrics (ACDC circuit edges, SAE L0 norm, N2G graph density) all measure local sparsity or specialization, so they do not validate the global 'decreased sparsity' conclusion. Because the title, abstract, and discussion section 6 all emphasize this surprising result, the claim is load-bearing and currently unsupported. If the intended metric is weight L1 norm, it cannot distinguish fewer nonzero weights from larger-magnitude weights; if it is activation L1 norm, then higher activation norms could still coexist with a larger fraction of zeroed units. Without a proper sparsity metric, the central counterintuitive finding is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-ablation mechanism for transformers: learned gating weights select top-k MLP neurons and attention heads during training via a straight-through estimator, with an auxiliary loss on the ablated output, while the gates are removed at inference. Models are trained on TinyStories at roughly 3M scale. The authors report improved interpretability relative to a standard GPT-Neo baseline, including smaller ACDC IOI circuits, lower SAE L0 norms, higher neuron explanation scores, and sparser neuron graphs, at a modest perplexity cost. They also report higher values in the 'Sparsity (L1)' column of Table 1 under self-ablation, which they interpret as a surprising decrease in overall sparsity.","tokens_in":13935,"tokens_out":5716,"duration_ms":57760,"significance":"If the findings hold, the paper makes a useful contribution: an ante-hoc training intervention that improves several interpretability indicators without changing the inference-time architecture, together with a challenge to the common assumption that sparsity and interpretability always go hand in hand. The code release and the use of multiple standard interpretability tools are strengths. However, the central 'Less Sparsity' claim rests on a metric that is not a sparsity measure, and the quantitative comparisons lack repeated-run variance, so the significance of the paper is currently conditional on a corrected analysis.","major_comments":[{"comment":"The 'Sparsity (L1)' column is defined in Section 4.5 as 'the normalized sum of the model's weights.' The L1 norm of weights is a magnitude measure, not a sparsity measure: a dense weight matrix with small entries can have a lower L1 norm than a sparse matrix with a few large entries. Thus the observed increase in L1 from 0.44 to roughly 0.7 does not establish a decrease in sparsity, and the title and abstract's 'Less Sparsity' claim is unsupported. Please report a true sparsity metric (for example, the fraction of weights or activations that are exactly zero or below a threshold) for both weights and activations, and state exactly which parameters are included (in particular, whether the auxiliary gating weights are counted).","section":"§4.5, Table 1"},{"comment":"The conclusion that interpretability improves through increased specialization rather than through sparsity relies on the L1 result as the only global-sparsity evidence. The other reported metrics — ACDC edge count, SAE L0, and N2G density — are local circuit-level or representation-level measures, and lower values of these metrics do not imply that the model's overall weight or activation vector is more sparse. Consequently, even if all the interpretability metrics improve, the 'decreased global sparsity' conclusion does not follow from the data presented.","section":"§5.5, §6"},{"comment":"All comparisons are based on single training runs with no error bars or significance tests. For example, the ACDC edge counts vary from 30 to 54 across ablation configurations with no indication of run-to-run variability, so claims such as 'the local approach generally outperformed global ablation' are not supported. Please provide multiple seeds, at least for the baseline and one or two ablation configurations, and report means and standard deviations, or otherwise justify why single runs are sufficient for the comparative claims.","section":"§5, Tables 1 and 2"},{"comment":"The ACDC results use a single fixed threshold τ=0.03 for all models, with no sensitivity analysis. Circuit edge counts can depend substantially on the threshold and on the patching metric; a sweep over τ, or a demonstration that the edge-count ranking is stable across thresholds, is needed to support the claim that self-ablation produces smaller circuits rather than that the chosen threshold happens to favor the ablated models. Additionally, Section 4.1 and Appendix D make contradictory statements about whether lower τ yields sparser circuits or more edges; this must be corrected.","section":"§4.1, Appendix D"}],"minor_comments":[{"comment":"The term 'L1 norm' is used inconsistently: the abstract and introduction refer to 'L1 activation norms,' while Section 4.5 defines it as the normalized sum of weights and Table 1 labels it 'Sparsity (L1).' This inconsistency should be resolved and the exact definition should be stated in the table caption.","section":"Abstract, §4.5, Table 1"},{"comment":"The reference entry 'Curt Tigges Joseph Bloom and David Chanin. Saelens' is malformed; the author list should be corrected (Bloom, Joseph, Curt Tigges, and David Chanin).","section":"References"},{"comment":"The explanation-score histograms are shown for the baseline and one ablated model, but the text reports only the two means; please report the number of neurons, the standard deviation, and a significance test for the difference.","section":"Figure 2 (right), §5.3"},{"comment":"Section 5.3 analyzes only the global k=2 model; the conclusion that self-ablation improves neuron explainability should be qualified as limited to that configuration, or additional configurations should be analyzed.","section":"§5.3"},{"comment":"The appendix claims 'constant time complexity of self-ablation,' but the described sorting step is O(N log N); the intended claim appears to be that the overhead is independent of the input length, and this should be stated precisely.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the code release is a plus. The main concern for the editor is that the headline result, emphasized in the title, abstract, and discussion, is based on a metric that does not measure sparsity; if the authors supply a proper sparsity analysis, clarify which parameters are included, and add repeated-run variability, the paper could become suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the training-time self-ablation mechanism: applying kWTA to both MLP neurons and attention heads via learned gating weights, with local and global variants, and a dual residual stream that lets the model train on clean and ablated paths in one pass. That combination is new, and the small-scale empirical results are actually interesting. ACDC edge counts drop from 79 to around 30 in several configurations, SAE L0 norms fall from 7.22 to about 4-5, and the N2G graph analysis shows substantially lower density. Those are consistent, multi-tool observations pointing toward more localized circuits and more specialized neurons. Code is public, which is a real plus.\n\nThe soft spot is exactly where you said it is. Section 4.5 defines the L1 norm as the normalized sum of the model's weights, and claims sparser models have lower L1 norm. That is not a sparsity measure. Weight sum cannot distinguish few large-magnitude weights from many small ones. The abstract and introduction talk about \"L1 activation norms,\" which is a different quantity. So the title claim \"Less Sparsity\" is not supported. The other metrics (L0, ACDC edges, graph density) are local sparsity or specialization measures; they do not validate a global sparsity conclusion. This is a load-bearing flaw because the paper's least intuitive finding is built on it.\n\nThere are smaller issues. All results are single runs with no error bars, so the ACDC and SAE improvements could be within random variation. The ACDC threshold tau=0.03 is fixed without sensitivity analysis. The neuron explainability section analyzes only one model (global k=2). The self-ablation equations omit the learned gating weights entirely, despite the text saying the mechanism uses them. The paper's own limitations section is honest about TinyStories and small models, but it does not flag the metric inconsistency.\n\nIf the authors re-analyze with a real sparsity measure, like fraction of near-zero activations or weights, and report variance across seeds, the paper could be solid. As it stands, the interpretability gains are worth taking seriously, but the \"less sparsity\" claim should not be cited as a result until it is properly measured.\n\nFor peer review: send it to referees, but require revision. The mechanism is worth publishing, the metric problem is fixable, and the interpretability evidence is suggestive enough to merit scrutiny.","headline":"The self-ablation training mechanism is genuinely novel and the interpretability results are promising, but the headline 'less sparsity' claim rests on a metric that doesn't measure sparsity, so the central surprise is unsupported as written.","tokens_in":786,"tokens_out":705,"would_cite":false,"duration_ms":27266,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A k-winner-takes-all training constraint makes transformers more interpretable without making them sparser.","keywords":["self-ablation","k-winner-takes-all","transformer interpretability","mechanistic interpretability","sparsity","neuron specialization","circuit localization","TinyStories"],"falsifier":"Count the actual fraction of dead or near-zero-weight components in both baseline and self-ablated models; if the self-ablated models have fewer such components rather than more, the paper's claim of decreased overall sparsity is contradicted. The same test could be run with an exact zero-weight count or a per-layer activity census to settle whether the L1 result reflects true density or just larger magnitudes.","tokens_in":13428,"feed_emoji":"🧠","tokens_out":8492,"duration_ms":74812,"temperature":0.7,"pith_summary":"This paper tries to establish that interpretability can be built into a transformer during training rather than inferred after the fact. It introduces a self-ablation mechanism that enforces a $k$-winner-takes-all constraint, keeping only the top-$k$ neurons and attention heads active for each input while a straight-through estimator lets gradients flow through the hard selection. Trained on a small synthetic-story corpus, the resulting models show substantially smaller task circuits, sparser autoencoder features, and more explainable, specialized neurons at only a modest perplexity cost. The paper's headline result is that these gains come with lower global sparsity, so interpretability here is driven by specialization and localization rather than by widespread inactivity.","feed_headline":"Forcing transformers to pick winners makes them more interpretable","feed_subtitle":"A top-k activation rule yields localized circuits and specialized neurons even as global sparsity drops.","key_machinery":"The load-bearing object is the self-ablation gate built on a $k$-winner-takes-all constraint. For each set of gating activations, the model sorts the values, sets a threshold $\\gamma=(x_k+x_{k+1})/2$ and a temperature $T=x_k-x_{k+1}$, and derives softmax weights $w_i$ for gradient computation while a hard top-$k$ mask is used in the forward pass. These gates sit on MLP neurons and attention-head outputs, are trained jointly with the base transformer weights, and are switched off at inference, leaving a standard transformer architecture. A dual residual stream computes clean and ablated outputs in a single pass, and the total loss is the sum of the clean cross-entropy and the ablated cross-entropy. The gates are what force the model to specialize components from the start of training, and their removal at inference is what makes the interpretability gain free at deployment.","core_discovery":"The central claim is that self-ablation, applied during training, produces more interpretable transformers without sacrificing language-modelling quality, and that the improvement arises through local specialization rather than reduced overall activity. Concretely, the paper reports that a local variant with $k=4$ cuts the Indirect Object Identification circuit from 79 to 30 edges and reduces the average number of active sparse-autoencoder features from 7.22 to 4.01, while raising validation perplexity from 5.73 to 6.58. The same model family yields higher automated neuron explanation scores and sparser, more selective token-activation graphs than the baseline. The broader conclusion is that the common intuition tying sparsity to interpretability is incomplete: decreased global sparsity can coexist with increased local specialization, and the latter is what makes circuits and neurons more legible.","pith_inferences":["An implicit extension is that the same training-time constraint could be applied during fine-tuning of a larger pretrained model, testing whether localization benefits survive when the base weights already encode distributed circuits.","The 'less sparsity' result suggests that interpretability metrics and sparsity metrics are tracking different quantities; a natural next experiment is to vary $k$ continuously and watch whether circuit size and weight L1 norm move independently.","If the reported neuron specialization persists at scale, interventions like targeted unlearning or concept removal could become easier on self-ablated models, because the relevant computation is confined to fewer, more legible units."],"forward_implications":["Interpretability can be engineered into a model during training without changing its inference-time architecture, so existing deployment and analysis tooling continues to work.","Task circuits and sparse-autoencoder feature sets can shrink sharply under the constraint; the paper's runs show up to a 62% reduction in circuit edges and a 44% reduction in active features.","If higher weight L1 norms really indicate lower global sparsity, then global weight sparsity is the wrong target for interpretability; selective per-input activation with dense weights retained is what matters.","Local ablation generally beats global ablation on both circuit size and perplexity in the paper's comparisons, pointing to layer-local decisions as the preferred design."],"supporting_citations":[{"why":"introduces the k-winner-takes-all network that the self-ablation mechanism implements as a gating constraint","marker":"(Majani et al., 1988)"},{"why":"supplies the transformer architecture whose MLP and attention units are selectively ablated","marker":"(Vaswani et al., 2017)"},{"why":"provides the GPT-Neo base architecture extended with the gating parameters","marker":"(Black et al., 2021)"},{"why":"provides the TinyStories dataset and the 3M configuration used for training and evaluation","marker":"(Eldan & Li, 2023)"},{"why":"supplies the circuit-discovery method and the IOI edge-count metric used to measure localization","marker":"(Conmy et al., 2023)"},{"why":"supplies the sparse-autoencoder analysis whose L0 norm and cross-entropy score measure feature concentration","marker":"(Huben et al., 2024)"},{"why":"supplies the automated examiner-LLM procedure used to score neuron explainability","marker":"(Bills et al., 2023)"},{"why":"supplies the Neuron-to-Graph analysis used for neuron connectivity, density, and specialization metrics","marker":"(Foote et al., 2023a)"}],"fun_headline_variants":["Self-ablation makes transformers interpretable, not sparse","Less sparsity, more interpretability via self-ablation","Top-k ablation boosts interpretability, cuts sparsity","Self-ablation yields interpretable transformers, less sparsity","Self-ablation: interpretable networks, not sparse ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's surprising 'less sparsity' claim rests on treating the L1 norm of the model's weights as a measure of overall sparsity, and if weight magnitude is not a valid proxy for how many components are actually active, that headline result collapses.","fun_headline_variants_meta":{"raw":{"variants":["Self-ablation makes transformers interpretable, not sparse","Less sparsity, more interpretability via self-ablation","Top-k ablation boosts interpretability, cuts sparsity","Self-ablation yields interpretable transformers, less sparsity","Self-ablation: interpretable networks, not sparse ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001125,"raw_usage":{"total_tokens":4650,"prompt_tokens":890,"completion_tokens":3760,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":3691}},"tokens_in":506,"tokens_out":3760,"duration_ms":26893,"temperature":1.0,"reasoning_tokens":3691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:39:30.578533+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the actual fraction of dead or near-zero-weight components in both baseline and self-ablated models; if the self-ablated models have fewer such components rather than more, the paper's claim of decreased overall sparsity is contradicted. The same test could be run with an exact zero-weight count or a per-layer activity census to settle whether the L1 result reflects true density or just larger magnitudes.","supporting_citations":[{"cited_title":"Language Models Can Explain Neurons in Language Models , May 2023","cited_arxiv_id":null,"evidence_quote":"supplies the automated examiner-LLM procedure used to score neuron explainability"}],"review_version":1}