How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

Aske Plaat; Niki van Stein; Xi Chen

REVIEW 4 major objections 5 minor 1 cited by

How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that chain-of-thought prompting changes a large language model's internal features into sparser, more interpretable, causally effective units—but only past a capacity threshold, and it tests this by swapping features…

desk verdict Descriptive results are worth a look, but the causal claim is built on unaligned SAE dictionaries and a CoT-vs-few-shot confound. read the letter →

arxiv 2507.22928 v1 pith:ZP34TZYD submitted 2025-07-24 cs.CL cs.AI

Xi Chen , Aske Plaat , Niki van Stein This is my paper · ORCID

classification cs.CLcs.AI

keywords chain-of-thoughtpromptingmechanisticinterpretabilitysparseautoencodersactivationpatchingfaithfulnesslanguagemodelreasoningscalethresholdGSM8K

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether chain-of-thought prompting changes how a language model actually reasons, not just what it outputs. To test this, it trains sparse autoencoders on the hidden activations of a 70-million-parameter and a 2.8-billion-parameter model solving grade-school math problems under chain-of-thought and plain prompting, then patches the extracted features between the two conditions. The central finding is that swapping chain-of-thought features into a plain run substantially raises the log-probability of the correct answer in the larger model but has no reliable effect in the smaller one, a scale threshold. Chain-of-thought also makes activations sparser and its features more interpretable in the larger model. The paper reads this as evidence that chain-of-thought can induce genuinely faithful internal structure when the model has enough capacity.

What carries the argument

The work is carried by a three-part mechanism: sparse autoencoders (SAEs) that compress the residual-stream activation at the final token into a sparse set of interpretable feature directions; activation patching that constructs a hybrid feature vector $h_{patch}[S] = h_{CoT}[S]$, $h_{patch}[\bar{S}] = h_{NoCoT}[\bar{S}]$, decodes it back into activation space, and measures the change in log-probability of the correct answer; and patch-curve analysis that varies K for top-K and random-K feature subsets to see whether causal signal is concentrated or distributed. The sparsity analysis adds a structural correlate: under CoT, most neurons are near zero and each SAE feature engages few neurons, with high variance across features in the larger model, which the paper calls structured sparsity.

What would settle it

Train a single SAE on pooled CoT and noCoT activations so the dictionary is shared, then repeat the patching; if the log-probability gains disappear, the original effect depends on coincidental index alignment rather than on CoT features. A complementary check is token-level path patching: if the final-token patch still helps when all earlier CoT tokens are masked, the effect is not a trace of the reasoning process itself.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a scale-dependent signature of chain-of-thought faithfulness. In the 2.8-billion-parameter model, replacing a selected set of feature values in a noCoT forward pass with values taken from a CoT forward pass raises the answer log-probability from about 1.2 to 4.3, with the top-K patch curves jumping by more than 3.2 log-prob at K=2 under a sparser dictionary; the reverse direction has little effect. In the 70-million-parameter model, the same intervention produces unstable, often negative log-probability changes. A second finding is that random-K patching of CoT features often beats top-K patching in the larger model, indicating that CoT's causal information is spread across many moderately activated features rather than concentrated in the most activated directions. The paper also reports that CoT prompts yield significantly sparser residual activations and higher feature explanation scores in the larger model. Together, these results support the conclusion that CoT reshapes internal computation into sparser, more modular, causally effective features—but only above a capacity threshold.

Load-bearing premise

The load-bearing premise is that feature index i in the CoT sparse-autoencoder dictionary means the same thing as feature index i in the separately trained noCoT dictionary; if the two dictionaries are not aligned, the patch does not swap CoT features but writes arbitrary coordinates into the noCoT representation.

Editorial extensions

If this is right

Above the observed capacity threshold, CoT-elicited features act as transferable causal units: inserting a small set of them into a noCoT run shifts the model's output toward the correct answer, and the direction is asymmetric—CoT-to-noCoT helps while noCoT-to-CoT does not.
Because random feature subsets outperform top-K subsets in the larger model, selection by activation magnitude is not a reliable way to find causally important CoT features; a distributed, coverage-based selection policy should be considered.
CoT's benefits are not purely superficial: in large models it measurably increases residual-stream sparsity and feature interpretability, so CoT can be described as a structured prompting method rather than only a surface scaffold.
The 70M result acts as a lower bound for the approach: in low-capacity models, CoT features do not transfer causally and can even degrade performance, so future feature-level studies of CoT should expect and report a scale threshold rather than assuming uniform effects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the scale threshold generalizes beyond this model family, a practical consequence is that CoT faithfulness can be screened cheaply by measuring activation sparsity or random-patch transfer before doing full mechanistic mapping.
The random-K-over-top-K result suggests a testable hypothesis for feature steering generally: ranking features by activation or by $|h_{CoT} - h_{NoCoT}|$ may select outliers, while a diverse sample of the feature vector is a better causal proxy; this could be checked on other tasks and models.
The alignment caveat is where I would place my own concern: because the CoT and noCoT SAE dictionaries are trained separately, the patching result implicitly assumes the same index in the two dictionaries corresponds to the same semantic feature; an alignment-free formulation of the test would harden the causal conclusion.
A sharper estimate of the threshold would come from repeating the patch-curve comparison at intermediate model sizes (around 1B parameters) and on multi-step non-math reasoning, which the paper does not do.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper has a real descriptive finding but the causal headline doesn't survive contact with the method. CoT and NoCoT SAE dictionaries are trained separately and never aligned, so the patching operation swaps arbitrary coordinate indices between two different bases. The stress-test note is right: hpatch[S]=hCoT[S], hpatch[not S]=hNoCoT[not S] assumes feature i means the same thing in both dictionaries, and no permutation, shared dictionary, or alignment check is reported. That makes the log-probability increases in 2.8B and the absence in 70M uninterpretable as evidence about CoT features.

What's actually good: the paper ships code, uses two Pythia sizes, and gives a clear workflow. The random-K baseline and patch curves are a sensible way to ask whether useful information is concentrated or distributed, and the descriptive sparsity and interpretability results are worth reporting even without the causal overlay. The finding that CoT leads to sparser activations, especially at 2.8B, is worth taking seriously.

The soft spots are in proportion: the confounding of CoT with the presence of three few-shot examples is serious, because noCoT gets no examples at all. Patching only layer 2 at the final token is a shallow snapshot; the paper acknowledges this, but it limits the scope of any causal statement. The uniform "p<0.001" across many K values and two dictionary ratios looks like multiplicity is not being handled. None of these rescue the causal claim; the alignment problem alone is enough. The citation pattern is fine—they cite Makelov et al. and Wu et al. on interpretability illusions—but they never connect that concern to their own unaligned patch.

Who should read it: interpretability researchers working on CoT faithfulness, and people who want a clean example of why SAE-based intervention needs a common dictionary. I would send it to peer review—not because the current causal claims are acceptable, but because the question is important and a competent referee can force a fix or a reframing. As is, the conclusion should not be cited.

Referee Report

4 major / 5 minor

Summary. The paper proposes a feature-level causal analysis of whether chain-of-thought (CoT) prompting induces faithful internal reasoning. For Pythia-70M and Pythia-2.8B on GSM8K, the authors train separate sparse autoencoders (SAEs) on CoT and NoCoT residual activations at layer 2, then perform activation patching by replacing selected feature coordinates of the NoCoT sparse code with those of the CoT sparse code. They report that patching a small set of CoT features into NoCoT runs increases answer log-probabilities in the 2.8B model but not in the 70M model, that interpretation scores for SAE features are higher under CoT in the larger model, and that CoT activations are sparser. They interpret these results as evidence of a scale threshold for CoT-induced faithful, distributed internal features. The paper also introduces Top-K and Random-K patch curves and provides code and configuration files.

Significance. If the causal claims were supported, the paper would be a notable step in mechanistic interpretability of CoT reasoning: it would provide feature-level, intervention-based evidence about faithfulness, introduce a patch-curve methodology, and report a scale-dependent effect. The paper is transparent about several limitations, states that code is available, and uses established tools (SAEs, activation patching). However, the central causal claim depends on an unverified assumption that feature indices in two separately trained SAE dictionaries are semantically aligned. Because that assumption is not tested or established, the main experimental result cannot currently be interpreted as measuring CoT features, and the scale-threshold conclusion is not supported by the described procedure. The descriptive observations about activation sparsity remain potentially useful but do not carry the causal conclusion.

major comments (4)

[Methodology, Causal Intervention] The patching operation hpatch[S] = hCoT[S], hpatch[not S] = hNoCoT[not S] presumes that feature index i in DCoT and index i in DNoCoT refer to the same semantic feature. The paper states that two separate SAE models are trained to obtain 'distinct feature dictionaries' and provides no alignment, permutation, or shared-dictionary procedure. Consequently, the Top-K selection based on |hCoT − hNoCoT| subtracts sparse codes written in different bases, and decoding the resulting mixed vector requires choosing one of the two dictionaries, which will misread the other component. The observed log-probability increases in Pythia-2.8B and the absence of effect in Pythia-70M therefore cannot be attributed to CoT-reasoning features; the causal claim and the scale-threshold claim are unsupported as described. A shared dictionary for both conditions, a validated feature-alignment step, or a reformulated intervention is required.
[Experiment Setup and Results (Random-K)] The Random-K baseline is described as 'uniformly sampled from the full dictionary.' In a sparse code, most features are zero for a given input, so random sampling from the full dictionary should mostly select inactive features whose values are unchanged by patching, diluting any effect relative to Top-K. The reported result that Random-K outperforms Top-K in Pythia-2.8B is therefore surprising and unexplained. The paper must clarify whether sampling was restricted to active CoT features, and should report the fraction of zero activations in the dictionary, error bars on the patch curves, and confidence intervals for the random baseline.
[Results and Appendix (statistical significance)] Multiple statements report p-values for the patching effects (e.g., Figure 5 and Figure 6 captions: 'p < 0.001'; Appendix: 'All patching effects achieved statistical significance (p < 0.001)') without describing the statistical test. It is not stated whether the test is paired across the same set of problems, how the 'up to 1000 problem pairs' are used, or whether any multiple-comparison correction is applied across K values, dictionary ratios, and model sizes. The test procedure, effect sizes, and confidence intervals should be reported.
[Abstract and Conclusion (scale threshold)] The abstract and conclusion claim a 'clear scale threshold' based on two model sizes, Pythia-70M and Pythia-2.8B, evaluated at a single layer (layer 2) and a single token position (final token). A threshold claim requires more than two scale points; the evidence supports at most a scale-dependent effect. The language should be softened, or the claim should be supported with intermediate model sizes and additional layers and token positions.

minor comments (5)

[Appendix, cross-reference] The appendix contains an incomplete cross-reference: 'We will further explain this phenomenon through an analysis of feature sparsity structure in Section .' The section number is missing.
[Reproducibility link] The GitHub URL in the paper is given as 'https://github.com/sekirodie1000/cot faithfulness' with a space; this should be URL-encoded or replaced with a working repository link.
[Related Work and Appendix, typos] There are several typographical errors, including 'reverse-engineere' in Related Work and the repeated phrase 'combining explanation scores with results from explanation scores' in the Semantic Interpretation section.
[Semantic Interpretation, terminology] The terms 'explanation score' and 'interpretation score' are used interchangeably; the definitions should be unified, and the near-zero mean scores (e.g., 0.056 vs -0.013 for 2.8B) should be interpreted with appropriate caution rather than described as evidence of 'semantically coherent' features.
[Related Work, novelty claim] The claim that the paper is 'the first to apply SAE-based feature extraction in the context of CoT prompting' is too strong given that Dutta et al. (2024) is cited as examining CoT reasoning mechanistically; the novelty statement should be qualified.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: the causal claim is an empirical comparison, not an equation-level reduction; only a minor non-load-bearing self-citation prevents a score of 0.

full rationale

The paper's central chain is empirical rather than definitional: activations are collected under CoT and NoCoT prompting, separate SAEs are trained to produce dictionaries D_CoT and D_NoCoT, features are selected by absolute activation differences or by random sampling, and the measured quantity is the change in log-probability of the correct answer after patching. No constant is fitted to the target answers, and no equation is present in which a claimed prediction is identical to an input by construction. The scale-threshold result is an observed difference between Pythia-70M and Pythia-2.8B, not a consequence of the SAE reconstruction objective or of the patching formula. The separately trained dictionaries do raise a genuine validity concern: because D_CoT and D_NoCoT are learned independently, feature index i need not denote the same semantic feature in both dictionaries, so hpatch[S] = hCoT[S], hpatch[not S] = hNoCoT[not S] may mix incompatible coordinate systems. However, this is a methodological threat to causal interpretation, not circularity: the reported log-probability changes are not analytically forced to be positive by the construction, and the paper does not define the outcome in terms of the intervention. The only self-citation, Plaat et al. 2024, appears as background support for CoT effectiveness alongside Wei et al. 2022 and is not load-bearing for the central claims. The paper also candidly lists limitations that further separate its claims from any pre-supposed conclusion. Overall, the analysis is self-contained against an external benchmark, and no load-bearing circular step is identifiable.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several unvalidated assumptions: dictionary alignment, snapshot sufficiency, patching validity, and LLM-based interpretability scoring. These are not machine-checked and are not independently benchmarked.

free parameters (3)

SAE dictionary ratio = 4 and 8
Selected by authors to represent lower and higher sparsity; the central comparisons are repeated under both, but the choice is not justified by a principled criterion.
K for distributional patching = 20
The main log-probability distributions patch the top or random 20 features; the paper fixes this without showing sensitivity to the exact value.
Layer and token for activation extraction = layer 2, final token
All SAEs and patching use only the layer 2 residual at the final token; the paper acknowledges this snapshot limits causal tracing.

assumptions (4)

ad hoc to paper Feature indices in separately trained CoT and NoCoT SAE dictionaries correspond to the same semantic features.
Used by the patching equation without any alignment or validation; this is the weakest assumption.
domain assumption The layer 2 final-token residual stream carries the causally relevant features for multi-step GSM8K reasoning.
The method extracts and patches only this snapshot; the authors note this is a static, snapshot-based limitation.
domain assumption Activation patching in SAE feature space is a valid causal intervention.
The paper cites the interpretability-illusion debate but proceeds on the assumption that feature-level patching measures causal influence.
domain assumption GPT-3.5-based explanation scores measure feature interpretability.
The semantic interpretation module relies on an LLM scorer and is not validated against human labels or causal tests.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding." pith.science (2026). https://pith.science/paper/ZP34TZYD

@misc{pith2026250722928,
  author       = {Pith},
  title        = {Pith review of: How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZP34TZYD}},
  note         = {Machine review of arXiv:2507.22928}
}

read the original abstract

Chain-of-thought (CoT) prompting boosts Large Language Models accuracy on multi-step tasks, yet whether the generated "thoughts" reflect the true internal reasoning process is unresolved. We present the first feature-level causal study of CoT faithfulness. Combining sparse autoencoders with activation patching, we extract monosemantic features from Pythia-70M and Pythia-2.8B while they tackle GSM8K math problems under CoT and plain (noCoT) prompting. Swapping a small set of CoT-reasoning features into a noCoT run raises answer log-probabilities significantly in the 2.8B model, but has no reliable effect in 70M, revealing a clear scale threshold. CoT also leads to significantly higher activation sparsity and feature interpretability scores in the larger model, signalling more modular internal computation. For example, the model's confidence in generating correct answers improves from 1.2 to 4.3. We introduce patch-curves and random-feature patching baselines, showing that useful CoT information is not only present in the top-K patches but widely distributed. Overall, our results indicate that CoT can induce more interpretable internal structures in high-capacity LLMs, validating its role as a structured prompting method.

Figures

Figures reproduced from arXiv: 2507.22928 by the authors.

**Figure 3.** Distribution of log-probability changes after patching the top 20 CoT features into NoCoT runs under dictionary ratio 4. Left: Pythia-70M; Right: Pythia-2.8B. While 2.8B shows a strong positive shift indicating consistent benefit from CoT features, 70M shows highly variable effects, including large performance drops, suggesting unstable or less effective feature transfer. Causal Effects of CoT Features via Activatio… view at source ↗

**Figure 2.** Comparison of feature explanation scores under CoT and NoCoT prompts. Left: Pythia-70M; Right: Pythia-2.8B. The 2.8B model shows higher explanation scores under CoT, indicating stronger causal features are learned in the larger model when CoT prompting is applied. Each plot is based on 50 features per condition. In summary, while CoT is not sufficient for logically faithful reasoning chains in LLMs, it serves as a… view at source ↗

Figures from the paper (11 more)

**Figure 5.** Figure 5: Top-K and Random-K patching performance under dictionary ratio 4. Left: Pythia-70M; Right: Pythia-2.8B. CoT→NoCoT patching shows the effect of patching CoT features into NoCoT, while NoCoT→CoT patching shows the reverse. In 2.8B, patching CoT features yields consisten…

**Figure 7.** Figure 7: shows that CoT prompts lead to significantly sparser residual activations compared to NoCoT. In the NoCoT condition, more neurons exhibit moderate to high activation; under CoT, most neurons are near zero, with only a few strongly activated. This effect is markedly m…

**Figure 8.** Figure 8: Activated neuron counts per SAE feature under NoCoT prompting, across thresholds from 0.0 to 1.0. Left: Pythia-70M; Right: Pythia-2.8B. The large model (2.8B) activates significantly more neurons per feature at each threshold, indicating denser feature composition com…

**Figure 9.** Figure 9: Activated neuron counts per SAE feature under CoT prompting. Left: Pythia-70M; Right: Pythia-2.8B. Compared to NoCoT, CoT prompts yield substantially sparser activations in both models, with 2.8B showing stronger sparsity and higher interfeature variance. As shown ear…

**Figure 10.** Figure 10: Distribution of log-probability changes after patching the top 20 CoT features into NoCoT runs under dictionary [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]

**Figure 11.** Figure 11: Distribution of log-probability changes after patching the top 20 CoT features into NoCoT runs under dictionary ratio [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

**Figure 12.** Figure 12: Top-K and Random-K patching performance under dictionary ratio 4. Left: Pythia-70M; Right: Pythia-2.8B. CoT→NoCoT patching shows the effect of patching CoT features into NoCoT, while NoCoT→CoT patching shows the reverse. In 2.8B, patching CoT features yields consiste…

**Figure 13.** Figure 13: Top-K and Random-K patching performance under dictionary ratio 8. Left: Pythia-70M; Right: Pythia-2.8B. For 2.8B, CoT→NoCoT patching consistently improves performance, with diminishing returns as K increases. NoCoT→CoT patching gradually degrades the CoT run, suggesti…

**Figure 14.** Figure 14: Sparsity comparison of residual activations under CoT and NoCoT prompts. In both models, CoT leads to signif [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]

**Figure 15.** Figure 15: Activated neuron counts per SAE feature under NoCoT prompting, across thresholds from 0.0 to 1.0. The large model [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]

**Figure 16.** Figure 16: Activated neuron counts per SAE feature under CoT prompting. Compared to NoCoT, CoT prompts yield substan [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Thinking vs. NoThinking: Towards Interpreting Reasoning Mechanisms of Large Language Models via Sparse Autoencoders
cs.CL 2026-08 conditional novelty 6.0 of 10

Chain-of-thought reasoning in DeepSeek-R1-Distill-Qwen-7B depends on a small set of high-intensity sparse features; suppressing them breaks both reasoning and LaTeX formatting.

Reference graph

Works this paper leans on

42 extracted references · 8 canonical work pages · cited by 1 Pith paper

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

H.; and Lakkaraju, H

Agarwal, C.; Tanneru, S. H.; and Lakkaraju, H. 2024. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614

arXiv 2024
[4]

G.; and Augenstein, I

Atanasova, P.; Camburu, O.-M.; Lioma, C.; Lukasiewicz, T.; Simonsen, J. G.; and Augenstein, I. 2023. Faithfulness tests for natural language explanations. arXiv preprint arXiv:2305.18029

arXiv 2023
[5]

Belinkov, Y. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1): 207--219

2022
[6]

Bereska, L.; and Gavves, E. 2024. Mechanistic Interpretability for AI Safety--A Review. arXiv preprint arXiv:2404.14082

arXiv 2024
[7]

Bills, S.; Cammarata, N.; Mossing, D.; Tillman, H.; Gao, L.; Goh, G.; Sutskever, I.; Leike, J.; Wu, J.; and Saunders, W. 2023. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html

2023
[8]

Braun, D.; Taylor, J.; Goldowsky-Dill, N.; and Sharkey, L. 2024. Identifying functionally important features with end-to-end sparse dictionary learning. Advances in Neural Information Processing Systems, 37: 107286--107325

work page 2024

Show all 42 references

[9]

Bricken, T.; Templeton, A.; Batson, J.; Chen, B.; Jermyn, A.; Conerly, T.; Turner, N.; Anil, C.; Denison, C.; Askell, A.; et al. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2

2023
[10]

Chuang, Y.-N.; Wang, G.; Chang, C.-Y.; Tang, R.; Zhong, S.; Yang, F.; Du, M.; Cai, X.; and Hu, X. 2024. FaithLM: Towards faithful explanations for large language models. arXiv preprint arXiv:2402.04678

2024
[11]

Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168

2021 arXiv
[12]

Cunningham, H.; Ewart, A.; Riggs, L.; Huben, R.; and Sharkey, L. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600

2023 arXiv
[13]

K.; Binz, M.; and Schulz, E

Demircan, C.; Saanum, T.; Jagadish, A. K.; Binz, M.; and Schulz, E. 2024. Sparse autoencoders reveal temporal difference learning in large language models. arXiv preprint arXiv:2410.01280

2024 arXiv
[14]

Dooms, T.; and Wilhelm, D. 2025. Tokenized SAEs: Disentangling SAE Reconstructions. arXiv preprint arXiv:2502.17332

2025 arXiv
[15]

Dutta, S.; Singh, J.; Chakrabarti, S.; and Chakraborty, T. 2024. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. arXiv preprint arXiv:2402.18312

2024 arXiv
[16]

Elhage, N.; Hume, T.; Olsson, C.; Schiefer, N.; Henighan, T.; Kravec, S.; Hatfield-Dodds, Z.; Lasenby, R.; Drain, D.; Chen, C.; et al. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652

2022 arXiv
[17]

Geiger, A.; Ibeling, D.; Zur, A.; Chaudhary, M.; Chauhan, S.; Huang, J.; Arora, A.; Wu, Z.; Goodman, N.; Potts, C.; et al. 2023. Causal abstraction: A theoretical foundation for mechanistic interpretability. arXiv preprint arXiv:2301.04709

2023 arXiv
[18]

Geiger, A.; Lu, H.; Icard, T.; and Potts, C. 2021. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34: 9574--9586

2021
[19]

Geiger, A.; Wu, Z.; Potts, C.; Icard, T.; and Goodman, N. 2024. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning, 160--187. PMLR

2024
[20]

Goldowsky-Dill, N.; MacLeod, C.; Sato, L.; and Arora, A. 2023. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969

2023 arXiv
[21]

Hanna, M.; Liu, O.; and Variengien, A. 2023. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36: 76033--76060

2023
[22]

Heimersheim, S.; and Nanda, N. 2024. How to use and interpret activation patching. arXiv preprint arXiv:2404.15255

2024 arXiv
[23]

Karvonen, A.; Rager, C.; Marks, S.; and Nanda, N. 2024. Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks. arXiv preprint arXiv:2411.18895

2024 arXiv
[24]

S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199--22213

2022
[25]

Li, Q.; Li, J.; Liu, T.; Zeng, Y.; Cheng, M.; Huang, W.; and Liu, Q. 2024. Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach. arXiv preprint arXiv:2410.21779

2024 arXiv
[26]

Makelov, A.; Lange, G.; and Nanda, N. 2023. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching. arXiv preprint arXiv:2311.17030

2023 arXiv
[27]

J.; Belinkov, Y.; Bau, D.; and Mueller, A

Marks, S.; Rager, C.; Michaud, E. J.; Belinkov, Y.; Bau, D.; and Mueller, A. 2024. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647

2024 arXiv
[28]

O.; Guttag, J.; and K c man, E

Matton, K.; Ness, R. O.; Guttag, J.; and K c man, E. 2025. Walk the talk? Measuring the faithfulness of large language model explanations. arXiv preprint arXiv:2504.14150

2025 arXiv
[29]

Meng, K.; Bau, D.; Andonian, A.; and Belinkov, Y. 2022. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35: 17359--17372

2022
[30]

Menon, A.; Shrivastava, M.; Krueger, D.; and Lubana, E. S. 2024. Analyzing (In) Abilities of SAEs via Formal Languages. arXiv preprint arXiv:2410.11767

2024 arXiv
[31]

Nanda, N.; Chan, L.; Lieberum, T.; Smith, J.; and Steinhardt, J. 2023. Progress measures for grokking via mechanistic interpretability, 2023. URL https://arxiv. org/abs/2301.05217

2023 arXiv
[32]

Paul, D.; West, R.; Bosselut, A.; and Faltings, B. 2024. Making reasoning matter: Measuring and improving faithfulness of chain-of-thought reasoning. arXiv preprint arXiv:2402.13950

2024 arXiv
[33]

Plaat, A.; Wong, A.; Verberne, S.; Broekens, J.; van Stein, N.; and Back, T. 2024. Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511

2024
[34]

Shojaee, P.; Mirzadeh, I.; Alizadeh, K.; Horton, M.; Bengio, S.; and Farajtabar, M. 2025. The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

2025
[35]

Y.; Camburu, O.-M.; Heess, N.; and Perez-Ortiz, M

Siegel, N. Y.; Camburu, O.-M.; Heess, N.; and Perez-Ortiz, M. 2024. The probabilities also matter: A more faithful metric for faithfulness of free-text explanations in large language models. arXiv preprint arXiv:2404.03189

2024 arXiv
[36]

Tighidet, Z.; Mogini, A.; Mei, J.; Piwowarski, B.; and Gallinari, P. 2024. Probing Language Models on Their Knowledge Source. arXiv preprint arXiv:2410.05817

2024 arXiv
[37]

V.; Zhou, D.; et al

Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824--24837

2022
[38]

interpretability illusion

Wu, Z.; Geiger, A.; Huang, J.; Arora, A.; Icard, T.; Potts, C.; and Goodman, N. D. 2024. A reply to makelov et al.(2023)'s" interpretability illusion" arguments. arXiv preprint arXiv:2401.12631

2023 arXiv
[39]

Xu, J.; Fei, H.; Pan, L.; Liu, Q.; Lee, M.-L.; and Hsu, W. 2024. Faithful logical reasoning via symbolic chain-of-thought. arXiv preprint arXiv:2405.18357

2024 arXiv
[40]

H.; Paturi, R.; and Bergen, L

Yee, E.; Li, A.; Tang, C.; Jung, Y. H.; Paturi, R.; and Bergen, L. 2024. Dissociation of faithful and unfaithful reasoning in llms. arXiv preprint arXiv:2405.15092

2024 arXiv
[41]

J.; Satapathy, R.; and Cambria, E

Yeo, W. J.; Satapathy, R.; and Cambria, E. 2024. Towards faithful natural language explanations: A study using activation patching in large language models. arXiv preprint arXiv:2410.14155

2024 arXiv
[42]

Zhang, F.; and Nanda, N. 2023. Towards best practices of activation patching in language models: Metrics and methods. arXiv preprint arXiv:2309.16042

2023 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

H.; and Lakkaraju, H

Agarwal, C.; Tanneru, S. H.; and Lakkaraju, H. 2024. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv preprint arXiv:2402.04614

arXiv 2024

[4] [4]

G.; and Augenstein, I

Atanasova, P.; Camburu, O.-M.; Lioma, C.; Lukasiewicz, T.; Simonsen, J. G.; and Augenstein, I. 2023. Faithfulness tests for natural language explanations. arXiv preprint arXiv:2305.18029

arXiv 2023

[5] [5]

Belinkov, Y. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1): 207--219

2022

[6] [6]

Bereska, L.; and Gavves, E. 2024. Mechanistic Interpretability for AI Safety--A Review. arXiv preprint arXiv:2404.14082

arXiv 2024

[7] [7]

Bills, S.; Cammarata, N.; Mossing, D.; Tillman, H.; Gao, L.; Goh, G.; Sutskever, I.; Leike, J.; Wu, J.; and Saunders, W. 2023. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html

2023

[8] [8]

Braun, D.; Taylor, J.; Goldowsky-Dill, N.; and Sharkey, L. 2024. Identifying functionally important features with end-to-end sparse dictionary learning. Advances in Neural Information Processing Systems, 37: 107286--107325

work page 2024