Position: Use Sparse Autoencoders to Discover Unknowns

Emma Pierson; Jon Kleinberg; Kenny Peng; Nikhil Garg; Rajiv Movva

REVIEW 3 major objections 5 minor 2 cited by

Position: Use Sparse Autoencoders to Discover Unknowns

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that sparse autoencoders fail when acting on predefined concepts but excel when discovering unknown ones, a distinction that reconciles negative and positive results.

desk verdict A useful conceptual reframing of SAE results, but the discovery claim is stronger than the evidence: no recall/coverage analysis. read the letter →

arxiv 2506.23845 v2 pith:LTYGAJAS submitted 2025-06-30 cs.LG cs.AIcs.CLcs.CY

Kenny Peng , Rajiv Movva , Jon Kleinberg , Emma Pierson , Nikhil Garg This is my paper · ORCID

classification cs.LGcs.AIcs.CLcs.CY

keywords sparseautoencodersinterpretabilityconceptdiscoveryhypothesisgenerationmechanisticmodelsteeringdetectiontext-as-data

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This position paper argues that the conflicting results on sparse autoencoders (SAEs) resolve once tasks are separated by whether the concept is known in advance. On detection, steering, and unlearning tasks, where the concept is an input, SAEs underperform simple baselines, and this is where the negative evaluations concentrate. On discovery tasks such as hypothesis generation and explaining a language model's internal computations, the concept is an output to be found, and SAEs produce a tractable, precise list of interpretable concepts that outperforms non-SAE pipelines. The authors claim that SAEs are therefore not a general tool for acting on concepts but are especially well-suited to discovering unknown ones, and they trace applications in ML fairness, auditing, interpretability, safety, and the social and health sciences. A sympathetic reader would come away with a clearer allocation of where SAE research is most likely to pay off.

What carries the argument

The load-bearing object is the known-versus-unknown concept taxonomy, a two-way split of tasks by whether the concept is an input (prespecified) or an output (to be discovered). The mechanism that carries the positive results is the SAE trained on model activations or dense text embeddings, whose sparse, roughly monosemantic neurons each fire on a single natural-language concept, together with autointerpretation, in which a language model names each neuron from its most-activating texts. By enumerating a tractable and expressive set of precise concepts and then checking which ones satisfy a target property, the SAE turns uninterpretable embeddings into interpretable ones, and this enumeration-plus-validation pipeline is what the paper says the acting-on-known-concepts baselines cannot do.

What would settle it

Run a large-scale evaluation of an explicitly open-ended discovery task, for example enumerating concepts that predict engagement or partisanship from a held-out corpus with the target concepts unknown to all methods at selection time, and compare SAE features against prompt-generated candidate concepts and n-gram rankings; if SAEs do not yield more validated hypotheses than the baselines, the discovery claim fails. Alternatively, reformulate a steering benchmark so the concept must be discovered rather than given; if SAEs still underperform prompting, then the taxonomy is not the operative variable.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a single distinction, whether the concept is an input or an output of the task, cleanly separates the failures of sparse autoencoders from their successes. In concept detection, model steering, and concept unlearning, the concept is prespecified; here SAEs lose to prompting, finetuning, and logistic regression on raw representations, and the paper does not contest those results. In hypothesis generation and the 'biology of LLMs', the concept is unknown in advance; here SAEs win because they enumerate a set of precise monosemantic features that express variation in the data, letting the researcher select and statistically validate a shortlist. The paper's explanation is that reconstruction encodes strictly less information than the original representation, so raw representations are better for acting, while the ability to convert uninterpretable text embeddings into interpretable ones makes SAEs a discovery tool, with downstream validation making feature unreliability tolerable.

Load-bearing premise

The paper assumes that the known-versus-unknown concept distinction, rather than differences in evaluation protocol, feature selection, or model size, is what makes SAEs succeed or fail on a given task.

Editorial extensions

If this is right

The negative evaluations from concept detection and steering should not be read as a general indictment of SAEs; they diagnose a task class, and deprioritizing all SAE research on that basis would skip the discovery use cases.
For hypothesis generation, SAE-based pipelines should be expected to outperform n-grams, topic models, and direct prompting in producing statistically significant and human-validated hypotheses, as they did on news engagement and partisan speech.
In ML fairness, auditing, and safety, SAEs offer a way to surface unknown concepts that bias model outputs, including input features beyond demographic variables and undesirable output behaviors.
In the social and health sciences, SAEs convert uninterpretable text embeddings into interpretable ones, allowing established text-as-data applications to be revisited with predictive power that keyword and topic-model features lack.
Because discovered concepts can be validated downstream, unreliability of individual SAE features is survivable in discovery tasks, whereas it is directly harmful in concept detection and steering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The taxonomy implies a testable workflow not run by the paper: a benchmark where target concepts are hidden from all methods at selection time should favor SAE enumeration, and failure there would narrow the thesis.
The paper's own speculation that SAE reconstructions lose information predicts a boundary: discovery tasks whose target property depends on fine-grained details lost in reconstruction would fail despite being discovery tasks.
In medical or social-science settings, SAE-generated concept lists could serve as a bounded, preregistered enumeration step, turning the prediction-explanation gap into a checklist of candidate mechanisms for confirmatory testing.
If the distinction holds, the right division of labor is to use raw representations for prediction, SAEs for hypothesis generation, and probes or steering only to validate a small set of discovered concepts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, what this paper gets right. The known-vs-unknown concept distinction is a genuinely useful way to organize the scattered SAE literature. It cleanly separates the negative results on concept detection and steering (where concepts are prespecified) from the positive results on hypothesis generation and "biology of LLMs" (where concepts are the output). That framing is new, it explains the apparent contradiction, and it points to concrete use cases in interpretability, fairness, and social/health science. The writing is clear and the survey of prior work is fair, including the negative results.

The soft spots are real but not fatal. The taxonomy is post hoc: tasks are classified by whether concepts were known ahead of time, and the paper asserts rather than tests that task type is the causal factor. There's also a circularity burden: one of the two positive-result pillars (Movva et al.) comes from the same research group, though Lindsey et al. is independent and strengthens the case.

The more substantive concern is the one about false negatives. The paper's defense of discovery—that downstream validation catches bad features—only handles false positives. If an SAE never encodes a concept, no validation step recovers it. The positive results show SAEs found some useful concepts on specific tasks, but neither measures recall nor compares against raw-representation baselines. So "powerful tools for discovering unknown concepts" is not established; the evidence shows SAEs can be useful in discovery, and the conceptual asymmetry is plausible, but coverage remains an open empirical question.

That said, the paper is a position paper, not an empirical claim. As a conceptual contribution it largely holds up: it correctly identifies that the negative results don't transfer to discovery tasks, and it gives the field a productive way to think about where SAEs belong. The missing recall evidence is a limitation to flag, not a reason to reject.

For whom: anyone working on interpretability, SAEs, or text-as-data in social science. It deserves a serious peer review, ideally with demands for a more careful discussion of coverage and a clearer separation of the authors' own prior work from independent evidence.

Referee Report

3 major / 5 minor

Summary. The paper proposes a conceptual distinction between two uses of sparse autoencoders: acting on known concepts, where recent large-scale evaluations find SAEs underperform simple baselines, and discovering unknown concepts, where positive results have been reported in hypothesis generation and in the biology of LLMs. It argues that this distinction reconciles the apparently conflicting SAE literature and then outlines applications in ML interpretability, fairness, auditing, safety, and the social and health sciences. The paper is a position paper rather than a new empirical study; its central evidence is a reinterpretation of cited results, chiefly Kantamneni et al. (2025), Wu et al. (2025), Movva et al. (2025), and Lindsey et al. (2025).

Significance. If the proposed distinction holds, it gives the field a clear way to separate domains where SAEs are currently underperforming from domains where they are genuinely promising, and it would redirect research effort toward discovery-oriented applications. The paper is clearly written, engages honestly with negative results, and offers a useful agenda for applying SAEs beyond mechanistic interpretability. However, the central claim is stated more strongly than the evidence supports: the known-versus-unknown taxonomy is constructed post hoc from the very results it is meant to explain, and the two positive-result papers provide evidence of precision but not of recall or coverage. The paper would be substantially stronger if it specified what empirical evidence could falsify the taxonomy, and if it either softened the 'powerful tools' claim or supported it with coverage/recall analysis. As written, it is a valuable organizing framework but not yet an established reconciliation.

major comments (3)

[§3–§4 and Table 2] The central taxonomy is constructed post hoc: tasks are assigned to 'acting on known concepts' or 'discovering unknown concepts' based on whether the concept was prespecified, and the paper then claims this distinction 'cleanly separates' the cited positive and negative results. The paper provides no controlled comparison or analysis showing that task category, rather than evaluation protocol, model size, feature selection, or baseline choice, is the causal variable behind SAE success or failure. For instance, the negative results use logistic-regression probes and prompting baselines, while the positive results compare against topic models, n-grams, and LLM-prompting pipelines; these differences in baselines and evaluation metrics are confounded with the known-versus-unknown distinction. The paper should acknowledge this limitation explicitly and state what evidence would falsify or support the taxonomy, otherwise the reconciliation remains an assertion rather than a demonstrated explanation.
[§4, 'Also note that after identifying concepts...'] The downstream-validation defense addresses only false positives: if an SAE feature fires on unrelated texts, a downstream check can catch that. It does not address false negatives: if a concept is not captured by any SAE feature, it will never be proposed as a hypothesis, and no downstream validation can recover it. This matters because the same lossy reconstruction argument used in §3 to explain SAE failures on concept detection also constrains the set of concepts available to discovery. The two positive results (Movva et al. 2025 and Lindsey et al. 2025) show that SAEs found some useful concepts on particular tasks, but neither measures how many relevant concepts were missed, nor compares the recall of SAE-based discovery against a raw-representation baseline. Without such coverage/recall evidence, the claim that SAEs are 'powerful tools for discovering unknown concepts' is not established; the observed successes could be confined to concepts that happen to be well-represented in SAE features.
[§4, 'What explains these positive results?'] The explanatory mechanism offered for the positive results—that SAEs produce a set of concepts that is 'both tractable and expressive'—is speculative and is not tested against alternative explanations. For example, the success of Movva et al. (2025) could be due to the particular density of the embedding space, the target variable selection procedure, or the autointerpretation prompt, rather than to the discovery of unknown concepts per se. Moreover, one of the two positive-result papers (Movva et al. 2025) is authored by four of the current paper's six authors, which increases the need for independent evidence. The paper should either present additional empirical support for the mechanism or frame the mechanism as a hypothesis to be tested in future work.

minor comments (5)

[§4, first paragraph of 'Hypothesis generation'] The sentence 'In both works, the goal is use a dataset as input and surface unknown, task-relevant concepts as output' appears to contain a typo; it should read 'the goal is to use a dataset as input.'
[§3, final paragraph] The transition from 'Though, methodological innovations may make SAEs more competitive at these tasks as well' to 'However, these baselines are less equipped to perform another simple task' is confusing: it is not clear whether 'these baselines' refers to non-SAE baselines or to the alternative SAE-based methods from Arad et al. (2025). Please clarify.
[§3, 'What explains these negative results?'] The statement that 'a reconstruction encodes strictly less information about a token than the original LM representation' is too general: an overcomplete sparse code can in principle preserve information on the data manifold, and whether information is lost is an empirical property of the trained SAE. The argument should be qualified to the SAE configurations evaluated in the cited negative results.
[§5, 'Health and social sciences'] The claim that SAEs enable the same applications as keyword or topic-model methods 'but potentially with significantly higher quality' is an unsupported assertion; providing citations or a brief argument for why SAE features would yield higher-quality concepts would strengthen this section.
[Abstract and §1] The abstract says the paper 'establish[es] a conceptual distinction,' while the body repeatedly uses hedged language such as 'we claim' and 'we speculate' (e.g., §3 and §4). Aligning the abstract with the evidential status of the argument would better represent the paper's contribution.

Circularity Check

1 steps flagged · score 4.0 of 10

The central known/unknown taxonomy is a post-hoc redescription of which SAE tasks succeeded or failed; the thesis is partly self-definitional, though it retains independent content in its use-case program.

renaming known result [Section 1 (Introduction); the taxonomy is applied again in Section 3 and Section 4.]
"Now consider the tasks where positive results have been shown. Hypothesis generation involves finding concepts that predict a target variable... In both tasks, concepts are outputs—unknown beforehand. So while SAEs have been shown to underperform baselines when acting on known concepts, SAEs remain promising and underexplored tools for discovering unknown concepts."

The negative-result tasks are all labeled 'acting on known concepts' and the positive-result tasks are all labeled 'discovering unknown concepts'; the labels are assigned after the outcomes are known. The thesis that SAEs are 'powerful tools for discovering unknown concepts' is then supported by the same examples used to populate the 'discovery' category. No independent criterion (e.g., a pre-registered task taxonomy, a negative discovery result, or a positive known-concept result) is used to test the distinction, so the central claim reduces to a redescription of the observed success/failure pattern rather than a derivation.

full rationale

The paper does not contain a formal derivation, so no equation-level circularity is present. The main circularity risk is the known/unknown taxonomy: the paper constructs the categories after observing which SAE tasks produced positive or negative results, and then uses those same examples to assert that SAEs are powerful for discovering unknown concepts. That is a renaming of the empirical pattern (SAEs fail at probing/steering but succeed in hypothesis generation and mechanistic explanation), and as stated the distinction is unfalsifiable. I do not count the self-citation to Movva et al. (2025) as an independent circular step: it is a load-bearing citation from the present authors, but it is an externally checkable empirical study and is accompanied by the independent Lindsey et al. (2025) result. The paper also offers substantive extrapolations and use cases that go beyond the observed results, which is why the circularity is only partial (score 4) rather than total.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It rests on domain assumptions about how concepts are operationalized and how SAE features are interpreted, plus the paper's own taxonomy. No fitted values appear anywhere.

assumptions (3)

domain assumption A concept is a qualitative characteristic that may or may not be present in a given input, operationalized via natural language descriptions.
Defined in Section 2 under Concepts and used to frame all tasks; this operationalization is not validated.
domain assumption SAE features are monosemantic and interpretable via autointerpretation.
Assumed from cited prior work (Cunningham et al., Templeton et al.); the paper does not re-evaluate this and relies on it for the discovery claim.
ad hoc to paper The known-versus-unknown concept distinction is a clean binary that cleanly separates the cited results.
Introduced by the authors in Sections 1 and 3; it is the paper's central framing assumption and is asserted rather than derived.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Position: Use Sparse Autoencoders to Discover Unknowns." pith.science (2026). https://pith.science/paper/LTYGAJAS

@misc{pith2026250623845,
  author       = {Pith},
  title        = {Pith review of: Position: Use Sparse Autoencoders to Discover Unknowns},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTYGAJAS}},
  note         = {Machine review of arXiv:2506.23845}
}

read the original abstract

While sparse autoencoders (SAEs) have generated significant excitement, a series of negative results have added to skepticism about their usefulness. Here, we establish a conceptual distinction that reconciles competing narratives surrounding SAEs. We argue that even if SAEs may be less effective for \textit{acting on known concepts}, SAEs are especially powerful tools for \textit{discovering unknown concepts}. This distinction separates existing negative results from positive results, and suggests several classes of SAE applications. Specifically, we outline use cases for SAEs in (i) ML interpretability, explainability, fairness, auditing, and safety, and (ii) social and health sciences.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach
econ.EM 2025-11 unverdicted novelty 7.0 of 10

A new framework combines AI-derived concept embeddings with high-dimensional selective inference to enable statistically principled, interpretable discovery from unstructured data in empirical economics.
Practicing with Language Models Cultivates Human Empathic Communication
cs.CL 2026-03 conditional novelty 6.5 of 10

Personalized LLM feedback after practice conversations with AI partners significantly improves human empathic communication on six preregistered dimensions without homogenizing responses, while trait empathy fails to ...

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages · cited by 2 Pith papers

[4]

Comment: Also available at https://transformer- circuits.pub/2022/toy_model/index.html. E. Farrell, Y .-T. Lau, and A. Conmy. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint arXiv:2410.19278,

arXiv 2022
[6]

Kantamneni, J

S. Kantamneni, J. Engels, S. Rajamanoharan, M. Tegmark, and N. Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint arXiv:2502.16681,

arXiv
[9]

doi: 10.18653/v1/2021.nuse-1.5

Association for Computational Linguistics. doi: 10.18653/v1/2021.nuse-1.5. URL https://aclanthology.org /2021.nuse-1.5/. J. Ludwig and S. Mullainathan. Machine learning as a tool for hypothesis generation. The Quarterly Journal of Economics, 139(2):751–827,

doi:10.18653/v1/2021.nuse-1.5 2021
[12]

Z. Wu, A. Arora, A. Geiger, Z. Wang, J. Huang, D. Jurafsky, C. D. Manning, and C. Potts. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. arXiv preprint arXiv:2501.17148,

arXiv
[2010]

URL https://doi.org/10.1214/10-STS330

doi: 10.1214/10-S TS330. URL https://doi.org/10.1214/10-STS330. L. Smith, S. Rajamanoharan, A. Conmy, C. McDougall, J. Kramar, T. Lieberum, R. Shah, and N. Nanda. Negative results for sparse autoencoders on downstream tasks and deprioritising sae research (mechanistic interpretability team progress update),

work page doi:10.1214/10-s
[2019]

Sharkey, B

11 L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimer- sheim, A. Ortega, J. Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496,

arXiv
[2020]

Lakkaraju, E

H. Lakkaraju, E. Kamar, R. Caruana, and J. Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 131–138,

work page 2019
[2021]

Huang, J

K. Huang, J. Altosaar, and R. Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342,

arXiv 1904

Show all 12 references

[2022]

ISSN 0891-2017. S. Bills, N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models. https://openaipubl ic.blob.core.windows.net/neuron-explainer/paper/index.html,

2017
[2023]

J. Dai, I. D. Raji, B. Recht, and I. Y . Chen. Aggregated individual reporting for post-deployment evaluation. arXiv preprint arXiv:2506.18133,

arXiv
[2024]

URL https://www.anthropic.com/news/golden-gat e-claude. D. Arad, A. Mueller, and Y . Belinkov. SAEs are good for steering–if you select the right features. arXiv preprint arXiv:2505.20063,
[2025]

URL https://transformer-circuits.pub/2025/attribution-graphs/biol ogy.html. N. F. Liu, M. Gardner, Y . Belinkov, M. E. Peters, and N. A. Smith. Linguistic Knowledge and Transferability of Contextual Representations, Apr

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [4]

Comment: Also available at https://transformer- circuits.pub/2022/toy_model/index.html. E. Farrell, Y .-T. Lau, and A. Conmy. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint arXiv:2410.19278,

arXiv 2022

[2] [6]

Kantamneni, J

S. Kantamneni, J. Engels, S. Rajamanoharan, M. Tegmark, and N. Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint arXiv:2502.16681,

arXiv

[3] [9]

doi: 10.18653/v1/2021.nuse-1.5

Association for Computational Linguistics. doi: 10.18653/v1/2021.nuse-1.5. URL https://aclanthology.org /2021.nuse-1.5/. J. Ludwig and S. Mullainathan. Machine learning as a tool for hypothesis generation. The Quarterly Journal of Economics, 139(2):751–827,

doi:10.18653/v1/2021.nuse-1.5 2021

[4] [12]

Z. Wu, A. Arora, A. Geiger, Z. Wang, J. Huang, D. Jurafsky, C. D. Manning, and C. Potts. Axbench: Steering llms? even simple baselines outperform sparse autoencoders. arXiv preprint arXiv:2501.17148,

arXiv

[5] [2010]

URL https://doi.org/10.1214/10-STS330

doi: 10.1214/10-S TS330. URL https://doi.org/10.1214/10-STS330. L. Smith, S. Rajamanoharan, A. Conmy, C. McDougall, J. Kramar, T. Lieberum, R. Shah, and N. Nanda. Negative results for sparse autoencoders on downstream tasks and deprioritising sae research (mechanistic interpretability team progress update),

work page doi:10.1214/10-s

[6] [2019]

Sharkey, B

11 L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimer- sheim, A. Ortega, J. Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501.16496,

arXiv

[7] [2020]

Lakkaraju, E

H. Lakkaraju, E. Kamar, R. Caruana, and J. Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 131–138,

work page 2019

[8] [2021]

Huang, J

K. Huang, J. Altosaar, and R. Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342,

arXiv 1904