{"id":"5abff088-059a-4cb1-8b1e-65cb164cdfc8","arxiv_id":"2412.12101","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Sparse autoencoders trained on ESM-2 recover thousands of interpretable features that align with Swiss-Prot concepts and can influence sequence generation.","lead":"The authors trained sparse autoencoders on ESM-2 protein language model embeddings and extracted thousands of readable features that align with known protein biology. These features can flag missing database annotations and steer sequence generation, offering a new window into what the model knows.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline metric may be inflated by validation-set threshold/feature selection; a permutation-style control or a precision/recall-matched neuron comparison is needed before the 2,548-vs-46 claim is accepted.","rationale":"The reader correctly identifies the per-domain recall and threshold-selection choices as the weakest methodological link. My stress-test agrees, and I have made the attack more concrete by pointing to the double use of the validation set (threshold selection + best-feature selection) and the asymmetry between SAEs and neurons in how thresholds are applied. The central claim needs this control; without it, the headline 2,548-vs-46 gap is not yet established as evidence for superposition or interpretability. The paper has real strengths (released code/visualizer, randomized-weight control, LLM-description validation), but those do not resolve this specific concern. Therefore the reader's CONDITIONAL verdict is appropriate.","tokens_in":18539,"tokens_out":1319,"duration_ms":11834,"concrete_test":"Run a permutation control that preserves the SAE architecture and feature sparsity: compute features on the same ESM-2 embeddings but break the alignment to Swiss-Prot annotations (e.g., permute concept labels across proteins in the validation set), then repeat the full validation-set threshold and best-feature selection and report the number of features with F1>0.5 on both validation and held-out sets. If the permutation control yields a similar count (or a substantial fraction of the 2,548), the headline gap is an artifact of metric/threshold selection. Additionally, rerun the neuron baseline with the same number of thresholds and an equalized per-domain recall computation to confirm the 46-neuron count is not depressed by using a coarser activation scale.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The core quantitative claim is the F1-based comparison in §3.3/§5.2.2. The modified metric computes per-amino-acid precision but per-domain recall, and for each feature-concept pair the activation threshold is chosen on the validation set to maximize F1. The same validation set is then used to select the best feature per concept before reporting held-out F1. This double-selection (threshold + best feature per concept) can inflate the reported SAE counts even without genuine concept specificity, because the validation set is used for model selection and then again for feature/threshold selection. The neuron baseline is only scaled to [0,1] and passed through an identity SAE; it is not matched for the number of candidate thresholds or for the granularity of the per-domain recall. Since SAE features are much sparser than neuron activations, they benefit far more from threshold selection under a precision/recall trade-off. The gap 2,548 vs 46 could therefore be partly an artifact of the evaluation protocol rather than evidence that SAEs discover more biology. The paper's own Appendix Figure 9a shows under 20% of features are concept-associated even at F1>0.5, so the count is sensitive to the threshold and should be validated against a matched control.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains sparse autoencoders (SAEs) on ESM-2-8M embeddings across all six layers and evaluates the interpretability of the resulting features against Swiss-Prot annotations. It reports that up to 2,548 SAE features per layer achieve F1 > 0.5 with 143 biological concepts on held-out data, versus 46 neurons and 15 concepts for raw ESM-2 neurons. Additional contributions include an LLM-based pipeline for describing features, examples of using feature activations to suggest missing database annotations, and steering experiments in which activating periodic-glycine features increases glycine probability at masked positions. The authors release code and an interactive dashboard (InterPLM).","tokens_in":18824,"tokens_out":6764,"duration_ms":64356,"significance":"If the main quantitative claim holds, the paper provides a substantial toolkit for interpreting protein language models and a new evaluation protocol for SAE features. The held-out split, the randomized-weight control, and the LLM description validation are credible design choices, and the public code and dashboard are valuable community resources. However, the headline SAE-versus-neuron comparison rests on a non-standard F1 metric and on validation-set threshold and feature selection that may systematically favor the much larger and sparser SAE dictionary, so the 2,548-versus-46 gap should be treated as provisional until additional controls are supplied.","major_comments":[{"comment":"The modified F1 metric is asymmetric: precision is computed per amino acid while recall is computed per domain (Eq. 2). This can produce F1 = 1 for a feature that activates on exactly one residue inside each annotated domain, regardless of whether the feature encodes the concept. Because SAE features are trained to be sparse, they are far more likely than raw neuron activations to achieve this 'one hit per domain' pattern, so the gap between SAE features and neurons may be inflated by sparsity rather than by conceptual specificity. I request a label-permutation or activation-permutation control that reports the distribution of F1 > 0.5 counts under the null, and a secondary analysis using standard per-residue precision and recall so readers can see the effect of the domain-level recall adjustment.","section":"§3.3, Eqs. (1)-(3)"},{"comment":"The validation set is used both to select the best feature per concept and to select the activation threshold for each feature-concept pair before held-out F1 is reported. Because the SAE dictionary has 10,240 features per layer versus 320 neurons for the identity-SAE baseline, the validation maximum over features is subject to a much larger winner's curse for SAEs. Although held-out F1 provides some protection, the validation and test sets are drawn from the same Swiss-Prot distribution and share systematic biases (for example, amino acid composition inside domains), so spurious validation winners may persist on test. The paper should report results with a fixed threshold (for example, 0.5) without per-pair threshold selection, or with a multiple-testing correction, or with a null distribution from permuted concept labels.","section":"§5.2.3"},{"comment":"The conclusion that PLMs store concepts in superposition is not directly supported by the current comparison, because the SAE dictionary is 32 times larger than the neuron set and the features are much sparser. The 2,548-versus-46 count is therefore consistent with a trivial 'more features, more hits' explanation. To support the superposition claim, the paper should demonstrate that individual neurons are polysemantic (for example, one neuron has high F1 for multiple unrelated concepts) and that SAE features are more monosemantic, ideally after matching the neuron baseline for dictionary size or sparsity. This analysis is load-bearing for the paper's central interpretability claim.","section":"§4.1 and §3.3"},{"comment":"The steering experiments select features because they activate on periodic glycine patterns and then show that activating these features increases glycine probability; this is partly circular, since the selection criterion already encodes the periodic glycine property. The comparison against non-periodic glycine features is informative but those features were also selected for glycine association, so they do not form a matched control. A stronger control would include randomly selected features matched for activation frequency and sparsity, with steering effects reported on all amino acid positions, not only glycine. The paper's own limitation statement acknowledges the narrow scope, but the abstract's claim of 'targeted steering of protein sequence generation' needs this additional control.","section":"§3.7 and Figure 7"}],"minor_comments":[{"comment":"Steps 3 and 4 of the sequential/structural analysis are identical ('Generate null distributions through averaging 5 random permutations per protein'); one of them should be removed or the intended difference stated.","section":"§5.4.2"},{"comment":"The caption uses 'Steer %' while the text describes clamping values where '1x is the maximum observed activation value'; these units should be made consistent and precisely defined.","section":"Figure 7 caption"},{"comment":"The phrase 'These feature's characteristic activation modes' should be 'These features' characteristic activation modes'.","section":"§3.2"},{"comment":"The headings 'Swiss-Prot Metadata Categories' appear twice in immediate succession; renumber or merge these appendices.","section":"Appendix C.1 and C.2"},{"comment":"It would be helpful to state whether the 5M UniRef50 proteins used for SAE training overlap with the 50,000 Swiss-Prot proteins used for evaluation, and how any overlap was handled.","section":"§5.1.1 and §5.2.1"},{"comment":"The caption should explicitly state that the F1 score shown is the held-out F1 for the validation-selected best feature per concept, rather than a validation-set F1.","section":"Figure 3a"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal and the resource contributions are real. My main concern is the evaluation metric behind the headline claim: the asymmetric F1, combined with validation-set threshold and feature selection over a 32-times-larger dictionary, could inflate the SAE counts. I would like to see a permutation control, a fixed-threshold analysis, and a sparsity-matched neuron baseline before the 2,548-versus-46 claim is accepted. The 'superposition' language also overstates what the current results establish. These issues are fixable with additional analysis, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a serious and useful application of sparse autoencoders to ESM-2, with a real evaluation framework and community resources. The headline numbers (2,548 vs 46 features) are probably in the right direction, but I wouldn't quote them without a matched control.\n\nWhat's new: SAEs are standard in language-model interpretability, but nobody had done a systematic sweep across a protein LM's layers with concept-level validation. The paper ships code, an interactive dashboard, and a randomized-weight baseline, which is more than most interpretability papers offer. The LLM autointerpretability pipeline is clever: generating descriptions with Claude and validating them on held-out proteins with Pearson r (median 0.72) is a nice, falsifiable check. The annotation-filling examples (Nudix box, peptidase S1, glycosyltransferase binding sites) are backed by InterPro hits, so those are genuinely interesting.\n\nWhere I'm cautious: the quantitative comparison rests on a modified F1 where precision is per amino acid but recall is per domain. That's a reasonable choice for coarse annotations, but it can flatter features that hit one conserved residue per domain, and the paper doesn't show the neuron baseline behaves the same way. On top of that, the activation threshold is chosen per feature-concept pair on the validation set, and then the best feature per concept is also chosen on that set. The test set is genuinely held out, so it's not circular, but the selection can inflate the count of features that clear the bar. The stress-test note's suggestion of a permutation control or a threshold-matched neuron comparison is the right ask. The steering experiments are also small: only three periodic-glycine features, a few sequences, and modest correlations. They're fine as demonstrations, but not systematic.\n\nWho this is for: anyone working on protein language model interpretability or trying to extract understandable features from ESM-2. The dashboard alone is a nice resource.\n\nMy recommendation: send it to review. A good referee will ask for the matched control and more systematic application evaluations, but the core methods and resources are solid, and the paper is honest about its limitations. It deserves a serious round.","headline":"A solid transfer of SAE interpretability to protein language models with a useful evaluation framework, but the headline feature counts need a matched control before I'd trust the exact gap.","tokens_in":19292,"tokens_out":2912,"would_cite":true,"duration_ms":28005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sparse autoencoders find thousands of interpretable features in protein language models.","keywords":["protein language models","sparse autoencoders","mechanistic interpretability","superposition","ESM-2","biological concept annotations","feature steering","sequence generation"],"falsifier":"Recompute the feature-concept F1 scores with recall counted per amino acid instead of per domain; if the number of sparse autoencoder features above 0.5 F1 falls to the level of raw neurons, the paper's headline gap is metric-driven. A second check is experimental: take a protein with a predicted but unannotated binding site from feature activation, assay whether the proposed residues actually bind the predicted ligand, and see whether the 'missing annotation' calls survive.","tokens_in":18348,"feed_emoji":"🧬","tokens_out":8346,"duration_ms":74315,"temperature":0.7,"pith_summary":"The paper asks what a protein language model actually knows about biology and argues that most of that knowledge is stored in a form that individual neurons do not expose. By training sparse autoencoders on the hidden embeddings of the protein language model ESM-2, the authors claim to recover up to 2,548 interpretable features per layer that match 143 known biological concepts such as binding sites, structural motifs, and functional domains. Raw neurons, by contrast, yield at most 46 matching features across 15 concepts, which the paper reads as evidence that ESM-2 represents biological concepts in superposition rather than in single neurons. The paper also shows that these features can flag missing database annotations and that activating a feature can steer sequence generation in interpretable ways. If correct, this turns the model into a source of both known and potentially novel protein biology.","feed_headline":"Sparse autoencoders find 2,548 interpretable protein features per layer","feed_subtitle":"Raw neurons match 15 concepts; SAE features match 143 and steer generation.","key_machinery":"The central object is the sparse autoencoder dictionary: for each of ESM-2's six layers, the 320-dimensional embedding of every amino acid is reconstructed as a sparse combination of 10,240 feature vectors, with sparsity enforced by an L1 penalty during training. The features are normalized so activation values lie between 0 and 1. Concept matching uses a modified F1 metric: precision is scored per amino acid, while recall is scored per annotated domain, and the best activation threshold for each feature-concept pair is chosen on a validation split and then evaluated on a held-out split. The other load-bearing pieces are the large-language-model pipeline that converts activation examples into feature descriptions, and the steering procedure that clamps a feature's activation in the reconstructed embedding, lets the model continue processing, and measures changes in output probabilities.","core_discovery":"On its own terms, the paper's central claim is that sparse autoencoders trained on ESM-2's per-layer amino acid embeddings decompose the model's representations into a large dictionary of sparse, human-interpretable features, and that these features are the right unit for reading biology out of the model. The headline evidence is quantitative: using a modified F1 score that counts precision per amino acid but recall per annotated domain, up to 2,548 features in a layer exceed 0.5 F1 against one of 143 curated protein concepts on held-out data, while the raw neurons of the same embeddings reach only 46 such matches across 15 concepts. The same analysis on a model with randomly shuffled weights finds no biological concepts, indicating the features reflect learned weights rather than trivial sequence statistics. The paper further claims that features can identify missing database annotations, that a large language model can generate descriptions of features that predict activation on held-out proteins with a median Pearson correlation of 0.72, and that clamping a periodic-glycine feature on one residue raises the model's probability of glycine at a masked position three residues away. Together these results are offered as evidence that ESM-2 encodes rich, superpositioned biological knowledge that sparse autoencoder features can extract, interpret, and intervene on.","pith_inferences":["Beyond the paper's direct evidence, the same pipeline could be run on larger ESM variants to test whether the feature-count gap grows with model scale; the paper only analyzes the 8-million-parameter ESM-2.","The per-domain recall metric is the load-bearing choice; if a stricter amino-acid-level recall were applied, the 2,548-versus-46 gap might shrink, so an independent reanalysis with that metric would sharpen the claim.","Features that activate consistently but match no curated concept could be treated as predictions of novel motifs and tested experimentally, for example by mutating the activated residues and assaying binding or activity.","The steering result suggests feature-level control could become a protein engineering tool, but the paper tests only a few short sequences; broader sequence contexts are needed to know when the control holds."],"forward_implications":["Individual-neuron probes understate what protein language models know; most biological concepts in ESM-2 appear to be stored in superposition and only become visible through dictionary learning.","Sparse autoencoder features can propose edits to protein databases: activations on unannotated proteins point to missing motif, domain, and binding-site annotations that external databases corroborate.","Feature steering gives a handle on generation: activating a periodic-glycine feature at a single residue raises glycine probability at later masked positions, so features can act at a distance.","Large-language-model-generated descriptions extend interpretability beyond existing annotations, covering features that no curated concept labels, with activation predictions that correlate with measured values.","The release of an interactive platform makes these per-layer features explorable across sequence, structure, and annotation views."],"supporting_citations":[{"why":"Supplies ESM-2, the protein language model whose embeddings are decomposed into features.","marker":"[1]"},{"why":"Provides the sparse autoencoder architecture and training procedure used throughout.","marker":"[14]"},{"why":"Supplies the feature-steering method that the intervention experiments adapt.","marker":"[20]"},{"why":"Provides the curated protein concept annotations used for feature evaluation.","marker":"[23]"},{"why":"Is the external database used to confirm that feature-suggested missing annotations are real.","marker":"[24]"},{"why":"Provides the language model used to generate and validate feature descriptions.","marker":"[4]"},{"why":"Frames the coevolutionary-statistics view of protein language models that the feature analysis engages.","marker":"[3]"},{"why":"Provides the model-internals manipulation tool used to run the steering experiments.","marker":"[26]"}],"fun_headline_variants":["SAEs crack ESM-2: 2,548 features beat 46 neurons","From 15 to 143 concepts: SAEs unveil ESM-2's hidden biology","SAE features steer protein generation—raw neurons don't","InterPLM: 2,548 interpretable features per layer expose protein biology","SAEs reveal 143 concepts in ESM-2; neurons only 15"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a feature matches a concept when it hits at least one amino acid in each annotated domain, so a feature covering two residues of a long domain counts as full recall; if that per-domain recall choice is unfair, the large advantage of sparse autoencoder features over neurons could be an artifact of the metric rather than a real difference in interpretability.","fun_headline_variants_meta":{"raw":{"variants":["SAEs crack ESM-2: 2,548 features beat 46 neurons","From 15 to 143 concepts: SAEs unveil ESM-2's hidden biology","SAE features steer protein generation—raw neurons don't","InterPLM: 2,548 interpretable features per layer expose protein biology","SAEs reveal 143 concepts in ESM-2; neurons only 15"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000687,"raw_usage":{"total_tokens":3185,"prompt_tokens":1084,"completion_tokens":2101,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":2010}},"tokens_in":700,"tokens_out":2101,"duration_ms":13670,"temperature":1.0,"reasoning_tokens":2010,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:13:15.726533+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the feature-concept F1 scores with recall counted per amino acid instead of per domain; if the number of sparse autoencoder features above 0.5 F1 falls to the level of raw neurons, the paper's headline gap is metric-driven. A second check is experimental: take a protein with a predicted but unannotated binding site from feature activation, assay whether the proposed residues actually bind the predicted ligand, and see whether the 'missing annotation' calls survive.","supporting_citations":[{"cited_title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","cited_arxiv_id":null,"evidence_quote":"Supplies ESM-2, the protein language model whose embeddings are decomposed into features."},{"cited_title":"Towards Monosemanticity: Decomposing Language Models With Dictionary Learning","cited_arxiv_id":null,"evidence_quote":"Provides the sparse autoencoder architecture and training procedure used throughout."},{"cited_title":"Daniel Freeman, Theodore R","cited_arxiv_id":null,"evidence_quote":"Supplies the feature-steering method that the intervention experiments adapt."},{"cited_title":"On expert curation and scalability: UniProtKB/Swiss-Prot as a case study","cited_arxiv_id":null,"evidence_quote":"Provides the curated protein concept annotations used for feature evaluation."},{"cited_title":"InterPro in 2022","cited_arxiv_id":null,"evidence_quote":"Is the external database used to confirm that feature-suggested missing annotations are real."},{"cited_title":"The Claude 3 Model Family: Opus, Sonnet, Haiku, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the language model used to generate and validate feature descriptions."},{"cited_title":"Wayment-Steele, Garyk Brixi, Haobo Wang, Dorothee Kern, and Sergey Ovchinnikov","cited_arxiv_id":null,"evidence_quote":"Frames the coevolutionary-statistics view of protein language models that the feature analysis engages."}],"review_version":1}