{"id":"d7f8c916-31a8-4990-96ed-08e198bda816","arxiv_id":"2607.19618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"Sparse dictionaries extracted from two genomic language models contain features whose ablation shifts masked-token predictions specifically at ChIP-seq bound versus unbound sites, after GC-composition controls.","lead":"This paper decomposes two genomic AI models into sparse, interpretable features, then removes single features during prediction to test whether the models actually use them to represent transcription-factor binding. It reports 7-14 of 15 tested features per condition are causally used across CTCF, GATA1 and REST, while negative controls stay silent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GC-only matching fails to control repeat-element composition, threatening the interpretation of binding-specific causal effects.","rationale":"We agree with the reader's identification of the weakest assumption: the GC-only matching between bound and unbound-motif windows does not control for repeat-element composition, which the paper itself demonstrates is a strong confound. This is the most load-bearing concern because it directly threatens the central causal claim. If unbound windows differ from bound windows in repeat content after GC matching, then the differential ablation effect could be caused by the feature's sensitivity to repeats rather than to TF binding. The negative controls, while well-designed, do not rule this out: they randomize labels, making the sets exchangeable, but do not test the adequacy of the matching. The paper's own observation that Alu repeats drive spurious CTCF enrichment underscores the plausibility of residual confounding. Moreover, the causal ablation test as implemented measures the effect of removing a feature that is more active on bound windows; if that activity is driven by a repeat element enriched in peaks, the test will produce a positive result regardless of any specific representation of binding. We therefore view this as a condition that must be satisfied before the central claim can be accepted. The reader's conditional verdict is appropriate; we recommend no change to the verdict.","tokens_in":10395,"tokens_out":5879,"duration_ms":57945,"concrete_test":"For each TF/model, compute the distribution of Alu-fraction (or RepeatMasker-annotated repeat content) in bound vs. GC-matched unbound-motif windows; a significant difference indicates residual confounding. Then re-run the binding-sensitivity and causal-ablation pipelines with unbound windows exact-matched to bound windows on repeat content (e.g., matching on Alu fraction within GC bins); if the number of causally validated features drops substantially (e.g., to near zero), the original results were driven by repeat composition rather than TF binding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The composition-matched test (Sec. 3.2) matches unbound-motif windows to bound windows only on GC content via rejection sampling. Yet Sec. 5.2 shows that Alu repeats, which are GC-rich, are a major confound for CTCF enrichment. If ChIP-seq peaks are enriched for Alu or other repeat classes beyond what GC matching captures, then a feature that detects such repeats would have higher activation on bound than unbound windows, leading to a binding-sensitive AUC > 0.55. In the causal ablation (Sec. 3.3), this feature would be more active on bound windows, so ablating it would produce a larger KL shift there, yielding a significant binding-specific causal effect—even though the feature represents repeat content, not TF binding. The negative controls (SCRAMBLE, GATA1-SCRAM) do not address this because they randomize labels, making the two sets exchangeable; they do not test whether the unbound set is compositionally matched to the bound set. Thus, without repeat-matching, the finding that 7–14 features are causally validated per factor may simply reflect repeat-element differences, and the central claim that the model uses these features to represent cell-type-specific TF binding is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for interpreting genomic language models by training top-k sparse autoencoders on hidden activations of two models (Nucleotide Transformer and DNABERT-2), then validating and causally testing the resulting dictionary features. The validation pipeline first uses a composition-matched test that compares feature activation on ChIP-seq bound windows versus GC-matched unbound motif windows, with a GC-correlation filter and a binding-sensitivity AUC threshold. The causal test ablates individual dictionary directions and measures the KL divergence in masked-token predictions between bound and unbound sites. The authors report that across CTCF, GATA1, and REST, and across both architectures, 7–14 of 15 tested features per condition show significant binding-specific causal effects, while scrambled-label and random-feature controls return nulls. They also provide Proposition 1 to formalize composition confounds and Proposition 2 to argue that the causal test separates general feature importance from binding specificity.","tokens_in":10727,"tokens_out":6308,"duration_ms":62956,"significance":"If the results hold, the paper makes a valuable contribution by importing sparse-dictionary and causal-intervention methodology from NLP interpretability into regulatory genomics, and by explicitly exposing compositional confounds in naive PWM-based feature validation. The experimental design has real strengths: cross-architecture replication, three functionally distinct transcription factors, two negative-control classes, and a theoretical account of why magnitude-based feature importance is insufficient. The observation that GATA1 binding features are causally validated despite weak motif selectivity is biologically interesting and supports the case for intervention-based validation. The paper also benefits from using only public data. However, the central claim that the causally validated features represent cell-type-specific transcription-factor binding, rather than compositionally correlated repeat elements, is not yet fully established because the matching protocol controls only GC content.","major_comments":[{"comment":"The composition-matched test matches only GC content via rejection sampling, yet §5.2 identifies Alu repeats as a major GC-rich confound for CTCF enrichment. The manuscript does not report the repeat-element composition of the bound versus unbound-motif windows after GC matching. If bound ChIP-seq peaks are enriched for Alu or other repeat families beyond what GC matching captures, a feature detecting such repeats will appear binding-sensitive (AUC>0.55) and, because it is more active on bound windows, will also pass the causal ablation test — without representing TF binding. The SCRAMBLE and GATA1-SCRAM controls do not rule this out, since they randomize labels and therefore make the two sets exchangeable; they do not test whether the unbound set matches the bound set in repeat content. Please report repeat composition (e.g., Alu fraction, LINE/SINE content) of bound and unbound sets, a","section":"§3.2, §5.2"},{"comment":"Key thresholds and selection criteria are unspecified: the 'fixed fraction of the matrix maximum' for strong-motif calling, the ranking criterion for the 'top 15 binding-sensitive, GC-robust features' (AUC? p-value? activation?), and the number of windows per class after GC matching. These choices determine the unbound set and which features are causally tested, so without them the population numbers in Table 1 are not reproducible. Specify all constants and include the selection rule.","section":"§3.2, §3.3"},{"comment":"Equation (5) subtracts f_j d_j from the hidden state. Since SAE dictionary directions are not orthogonal and reconstruction is approximate, this does not exactly remove feature j's contribution: it also changes the effective contribution of any feature correlated with d_j. Proposition 2 additionally assumes the geometric factor γ(S) is independent of B (or does not anti-correlate with binding), but γ(S) depends on S through the post-layer Jacobian and is not shown to be binding-independent. As stated, the proposition is a heuristic rather than a proof that the test isolates binding-specific use; please either provide conditions/empirical checks (e.g., orthogonality of validated features, distribution of γ across bound/unbound) or soften the claim.","section":"§3.3, Proposition 2"}],"minor_comments":[{"comment":"No code or data availability statement is provided despite the claim of a 'fully reproducible procedure'; please include a statement or link to an implementation, or specify exactly which public resources and scripts are needed to reproduce the pipeline.","section":"Throughout"},{"comment":"Please specify the activation normalization/scale s in Eq. (5) and the exact SAE training details (number of tokens, batch size, layer selection criterion) in the main text rather than in passing.","section":"§3.1, Eq. (5)"},{"comment":"The '—' entries for random-feature control rows could be confusing; state explicitly that motif-selectivity is undefined for randomly chosen features, and that negative controls have no binding-sensitive features so the causal test is not applicable.","section":"Table 1"},{"comment":"The caption says (a) shows 'two exemplar binding features' and (b) shows 'three NT features'; please clarify the relationship between the panels, e.g., whether the third NT feature in (b) is one of the two in (a).","section":"Figure 3 caption"},{"comment":"Define 'GC-matched' precisely: the bin width used for the GC histogram and the rejection-sampling acceptance criterion should be given, as the result can be sensitive to these choices.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and the empirical design has genuine strengths, especially the negative controls and the cross-architecture comparison. The main technical risk is the repeat-composition confound in the GC-matched unbound set, which I believe is addressable with additional analysis. The unspecified thresholds and lack of code/data availability are also important for reproducibility. I would not accept in current form, but I do not see a fundamental flaw that requires rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this paper is the first to bring top-k sparse autoencoders plus causal ablation to genomic language models, and it shows convincingly that naive PWM-based validation is a disaster—hundreds of spurious CTCF features that are really GC/Alu detectors. The composition-matched test is a genuine contribution, and the negative controls are the right idea. But the central causal claim is not yet established, because the unbound windows are matched only on GC content, not on repeat composition, and the paper's own analysis says Alu repeats are a major confound. That gap is load-bearing, not cosmetic.\n\nWhat's actually new: the cross-architecture transfer (NT and DNABERT-2), the binding-resolved protocol, and the observation that a motif-only feature can have a large absolute ablation effect but no binding-specificity—that's a useful caution. The GATA1 result is interesting: the model encodes binding without a strong motif correlation.\n\nWeaknesses, in order: (1) GC-only matching. If bound peaks are enriched in Alu beyond what GC matching captures, then an Alu-detecting feature will score higher on bound windows and its ablation will shift the model's predictions more there, producing a 'causal' result that has nothing to do with TF binding. The scrambled-label controls don't address this because they randomize labels, not sequence composition. (2) No code or data release, which makes the 7–14/15 numbers hard to audit. (3) The top-15 features are selected on the same windows used for the causal test, so some selection bias is likely. (4) The motif-score threshold is unspecified. (5) Proposition 2 assumes the geometric factor γ is independent of binding without proof; that's an assumption, not a theorem, and the proof sketch doesn't cover it.\n\nWho this is for: anyone doing interpretability of genomic models, and anyone validating motif discoveries against ChIP-seq. It deserves a serious referee. I'd send it out with a request for repeat-matched windows, artifact release, and a more careful treatment of the selection/testing overlap.","headline":"Useful framework and a real warning about GC/Alu confounds, but the causal claim needs repeat-matched controls and artifact release.","tokens_in":11150,"tokens_out":2678,"would_cite":true,"duration_ms":25733,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a causal dictionary-learning framework to prove that specific intermediate features in genomic language models are used by the model to represent transcription-factor binding, not just to correlate with motifs.","keywords":["causal dictionary learning","sparse autoencoders","genomic language models","transcription-factor binding","causal ablation","composition confounds","ChIP-seq validation","interpretability"],"falsifier":"Stratify both bound and unbound-motif windows by repeat-element content (e.g., Alu-containing vs. non-Alu) after GC matching and recompute the causal ablation AUC within each stratum: if the bound-vs-unbound effect disappears or reverses inside Alu-matched strata, the features encode repeat composition rather than binding.","tokens_in":10225,"feed_emoji":"🧬","tokens_out":6606,"duration_ms":55900,"temperature":0.7,"pith_summary":"The paper's aim is to establish that the features a genomic language model uses can be extracted and verified, not just assumed: intermediate activations in DNA transformer models can be decomposed into sparse, nearly monosemantic dictionary directions, and a subset of those directions are causally used by the model to represent cell-type-specific transcription-factor binding. It shows that the standard validation of such features by motif enrichment is badly confounded by GC content and repetitive elements, and replaces it with a composition-matched, binding-resolved test plus a causal ablation that measures how much masking a single dictionary direction shifts the model's own predictions at bound versus unbound motif sites. Across three transcription factors and two architecturally different models, 7–14 of 15 tested binding-sensitive features pass the causal test, while scrambled-label and random-feature controls produce clean nulls. The framework matters because it gives the field a way to tell whether an apparent 'concept' inside a genomic model is actually used by the model, rather than being a statistical artifact of sequence composition.","feed_headline":"Causal test proves DNA models encode binding, not just motifs","feed_subtitle":"Ablating one learned feature shifts predictions more at bound sites; random and scrambled controls stay silent.","key_machinery":"The load-bearing mechanism is the causal dictionary: a top-k sparse autoencoder trained on hidden activations with a large overcomplete dictionary (16x hidden width, k=32 active features per token), whose columns d_j are candidate features. Each candidate is first filtered by a composition-matched binding test—bound windows (ChIP-seq peak centers with strong motif) versus unbound-motif windows matched by GC content via rejection sampling, scored by one-sided Mann–Whitney AUC with |corr(f_j, GC)| < 0.2—and then tested by causal ablation, where the feature's contribution s^{-1} f_j(x) d_j is subtracted from the hidden state and the model's output is re-run; the readout is the average KL diverg","core_discovery":"The central claim, stated on the paper's own terms, is that features recovered from a genomic language model's hidden activations by top-k sparse dictionary learning can be causally validated: deleting a single dictionary direction during the forward pass produces a larger shift in the model's masked-token prediction distribution at ChIP-seq-bound windows than at GC-matched unbound-motif windows, so the feature is not merely correlating with a position weight matrix but is part of the model's representation of binding. The evidence is a population-level effect: across CTCF, GATA1, and REST and across both a 6-mer-tokenized and a byte-pair-encoded transformer, 7–14 of 15 tested binding-sensit","pith_inferences":["Editorial inference: The causal effect is measured by a single-feature ablation; if several dictionary directions redundantly encode the same binding information, ablating any one of them will understate the feature's true role, so the reported counts likely represent a lower bound on how many directions the model uses.","Editorial inference: The fact that GATA1 binding is causally encoded without motif selectivity suggests testable flanking-context experiments: mutating sequence context around GATA1 motifs while preserving the WGATAR core should change the causal effect if the model's binding representation is context-dependent.","Editorial inference: The KL-shift readout could be inverted into a variant-prioritization tool; for a candidate SNP inside a feature's receptive window, the bound-versus-unbound difference in ablation shift could flag variants that the model treats as binding-relevant without retraining or cell-type labels."],"forward_implications":["Any genomic-model interpretability claim should now be expected to include a composition-matched control and a causal ablation; motif enrichment alone is demonstrably insufficient.","The framework returns clean negatives on scrambled labels, so it can be used as a screening pipeline to test which transcription factors and cell types a model genuinely encodes.","Model representations of binding can exist even when no canonical motif is enriched (GATA1), so correlation-based motif scans understate what genomic language models learn.","Because the causal effect replicated in two tokenization schemes (6-mer and byte-pair encoding), the result is not an accident of one vocabulary.","The set of causally validated dictionary directions gives a concrete, prioritized list of features for downstream hypothesis generation about regulatory grammar."],"fun_headline_variants":["Causal test confirms DNA models encode binding, not just motifs","Ablating one feature proves genomic AI sees binding, not only motifs","Causal dictionary learning validates TF binding in genomic transformers","Genomic language models' binding features pass causal ablation test"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that GC-matched unbound-motif windows are compositionally equivalent to bound windows except for the binding label; if unbound windows still differ in repeat-element content or higher-order k-mer statistics, the binding-specific ablation effects could be driven by those differences rather than by transcription-factor binding.","fun_headline_variants_meta":{"raw":{"variants":["Causal test confirms DNA models encode binding, not just motifs","Ablating one feature proves genomic AI sees binding, not only motifs","Causal dictionary learning validates TF binding in genomic transformers","Genomic language models' binding features pass causal ablation test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000644,"raw_usage":{"total_tokens":2847,"prompt_tokens":844,"completion_tokens":2003,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1933}},"tokens_in":588,"tokens_out":2003,"duration_ms":12955,"temperature":1.0,"reasoning_tokens":1933,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T12:12:10.281529+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Stratify both bound and unbound-motif windows by repeat-element content (e.g., Alu-containing vs. non-Alu) after GC matching and recompute the causal ablation AUC within each stratum: if the bound-vs-unbound effect disappears or reverses inside Alu-matched strata, the features encode repeat composition rather than binding.","supporting_citations":[],"review_version":1}