Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Scaling sparse feature circuit finding for in-context learning

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In a 2-billion-parameter language model, in-context learning is carried by two causally linked families of sparse features: detectors that recognize the task from completed examples and executors that apply it at the arrow.

desk verdict A serious scaling of SAE-based circuit analysis to a 2B model with a new decomposition algorithm; the ICL detection-to-execution claim is real but rests on attribution-patching quality that is only moderate. read the letter →

arxiv 2504.13756 v1 pith:SK6XNCC3 submitted 2025-04-18 cs.LG cs.AIcs.CL

Dmitrii Kharlapenko , Stepan Shabalin , Fazl Barez , Arthur Conmy , Neel Nanda This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords in-contextlearningsparseautoencodersmechanisticinterpretabilitytaskvectorsfeaturecircuitscausalablationattributionpatchinglargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that in-context learning in a large language model is a localized, inspectable computation: a small set of sparse autoencoder features arranged in a detection-then-execution chain carries the demonstrated task through the model. It identifies two feature families: task-execution features that fire on arrow tokens and drive the production of correct answers, and task-detection features that fire on the output tokens of completed demonstrations and encode which task is being performed. Ablating a detection feature measurably suppresses its matching execution feature, showing the families are causally linked through attention and MLP layers. The paper also claims that task vectors, directions extracted from the residual stream that were already known to encode whole tasks, are well approximated by a sparse sum of these SAE latents, via a new cleaning algorithm that cuts active features by 50-80% without losing the vector's effect on task loss. A sympathetic reader would care because this turns an opaque capability into a short, editable list of feature directions, and it demonstrates that circuit-level analysis can reach a 2-billion-parameter model.

What carries the argument

The central object is the sparse feature circuit: a causal graph whose nodes are SAE latents and whose edges are estimated indirect effects, built from a suite of sparse autoencoders trained on residual streams, attention outputs, and MLP inputs. Two pieces of new machinery carry the argument. Task Vector Cleaning (TVC) starts from the SAE encoding of a task vector and optimizes the reconstruction weights to minimize zero-shot task loss plus an $\ell^1$ sparsity penalty, a procedure that keeps most of the vector's effect on loss while cutting the number of active features by 50-80% and exposing the task-execution features. The adapted sparse-feature-circuit search computes each node's indirect effect, $IE(m;a) = m(x \mid do(a=a')) - m(x)$, via attribution patching in the SAE basis, aggregates effects per token position type (prompt, input, arrow, output, newline), and restricts itself to layers 10-17 because indirect-effect approximations before layer 10 are too inaccurate. The object it produces is a graph in which detection features sit upstream of execution features, passing through attention-output SAE nodes and transcoder (MLP) nodes, with faithfulness of about 0.6 reached at roughly 500 nodes.

What would settle it

Recompute the indirect effects of the top-scoring task-detection features in layers 10-17 with exact activation patching, zero-ablating one node at a time, instead of gradient approximations, and compare the rankings; the paper's own Appendix E.2 reports predicted-versus-actual correlations below 0.7 in most middle layers, so if the correlation on the specific detection-to-execution edges is also below that level, the claimed causal chain is not supported by the evidence.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that in-context learning in Gemma-1 2B is mediated by two causally connected families of SAE features, ordered as detect-then-execute. Task-detection features activate almost exclusively on the output tokens of earlier demonstrations (about 97% of their activation mass in the paper's measurements), and steering with them induces the corresponding task zero-shot; task-execution features activate overwhelmingly on arrow tokens (about 90% of their mass), and steering with them also induces the task. When the strongest detection feature for a task is ablated, the activation of the matching execution feature drops substantially, establishing the causal chain through attention-output and transcoder (MLP) nodes. The paper further claims that the task vectors of prior work are well approximated by a sparse sum of SAE latents, including these task-execution features, and that the extracted circuits are task-specific, since ablating the top nodes of one task's circuit degrades other tasks' faithfulness only in closely related cases.

Load-bearing premise

The load-bearing premise is that the gradient-based estimates of each feature's causal effect are accurate enough to pick out the true circuit edges, since the paper's own checks find these estimates only moderately correlated with actual effects in the middle layers and poor before layer 6, and the same inaccuracy in layers 10-17 would make the detection-to-execution links artifacts.

Editorial extensions

If this is right

  • If the two-family circuit is right, then a handful of SAE feature directions, not diffuse activity, carries in-context learning, so switching a task on or off reduces to steering or ablating those few directions without retraining.
  • Task vectors become sparse, interpretable objects rather than opaque residual-stream directions: a few execution features, not the usual dozen or more noisy latents, preserve most of a task vector's effect on loss.
  • Because detection features sit upstream of execution features, a model should fail to perform a task it cannot detect, so audits and interventions on ICL behavior should target the detection stage first.
  • The pipeline ran end-to-end on a 2-billion-parameter model with 23 tasks, which makes full circuit analysis, rather than single-feature inspection, accessible for models of this scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The two-family dichotomy is shown for discrete token-to-token tasks; for graded ICL settings such as in-context regression or classification with unseen labels, the analog of a detection feature may be a continuous axis rather than a discrete SAE latent, and looking for one would test whether detect-then-execute is an architectural constant.
  • The paper notes that translation-to-English tasks share a generic executor while language identity lives in a separate feature; taken further, this suggests ICL circuits factor into a reusable task kernel plus a per-instance modifier, which predicts that perturbing the modifier alone should switch tasks within a family.
  • TVC is validated on other architectures, but the full detection-to-execution circuit is only recovered on Gemma-1 2B; re-running the adapted circuit search on a second architecture is the direct test of whether the discovered structure is a general transformer property rather than a quirk of one model.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper scales sparse feature circuit (SFC) discovery, previously applied to GPT-2-small, to Gemma-1 2B for the study of in-context learning (ICL) on a set of 23 tasks. The authors introduce Task Vector Cleaning (TVC), an optimization algorithm that decomposes task vectors into a small number of SAE latents, which they call task-execution features. Steering experiments show that these features improve zero-shot task performance. They then adapt SFC to the larger model, introducing token-type aggregation and a modified loss, and claim to discover task-detection features that activate on output tokens and causally drive task-execution features through attention and MLP sublayers. The paper includes multiple validation experiments: held-out steering evaluations, cross-task ablation specificity (Figure 8), and a direct ablation of detection directions (Figure 11).

Significance. If the central causal circuit claim holds, this would be a significant step forward: it extends automated SAE circuit discovery to a 2B-parameter model, unifies the task-vector literature with SAE interpretability, and identifies two concrete, editable feature families (detection and execution) behind ICL. The paper's strengths include careful held-out evaluation of steering, extensive cross-model sweeps for TVC (Gemma-2 2B/9B and Phi-3), qualitative max-activating examples that support feature interpretability, and a commitment to release SAEs and code. However, the claimed detection-to-execution pathway is not verified at the edge level, and the steering evidence carries a selection-on-metric circularity. These issues are fixable with additional experiments, so the work is promising but not yet conclusive as a mechanistic account.

major comments (4)
  1. [Section 3.1, Algorithm 1; Section 3.2] The claim that task-execution features "causally induce the task zero-shot" is weakened by a selection-on-metric circularity: TVC optimizes weights to minimize the zero-shot negative log-likelihood loss (L = LNLL + λ·L1), and the same loss is then used in Section 3.2 to measure the steering effect of those features. Held-out prompt splits reduce overfitting but do not break the loop of selecting features specifically to optimize the evaluation metric. I recommend providing a non-circular validation, for example by testing features selected purely from max-activating examples or by comparing against a random-subspace baseline with matched L0.
  2. [Section 4.1.3, Appendix E.2, Figure 7] The specific claim that task-detection features drive task-execution features through attention-output and transcoder nodes rests entirely on attribution-patching indirect effects, yet the paper's own validation (Appendix E.2, Figures 20-21) shows predicted-vs-actual IE correlations below 0.7 in most layers, and the authors exclude layers 1-9 for this reason. No layer-specific or edge-specific accuracy breakdown is given for the exact detector-to-executor edges. Moreover, Appendix E.1 states that the authors "do not perform full ablation of circuit edges," and Figure 7 measures faithfulness on node sets, not edges. Therefore the main new mechanistic contribution (detection → attention/MLP → execution) is not verified at the claimed granularity. Please provide edge-level validation (e.g., path patching or ablation on a sample of top edges) or temper the causal-connectivity claim to what node-level evidence supports.
  3. [Section 4.2, Figure 11] The direct ablation experiment in Figure 11 may not isolate the claimed pathway. SAE feature directions are generally not orthogonal in the residual stream, so ablating a detector direction could reduce executor SAE activations through direct projection overlap rather than through the named attention/MLP sublayers, even when attention patterns are fixed. The paper does not report cosine similarities or a control ablation of a random equal-norm direction. I request such a control, or an orthogonalized ablation, to support the pathway-specific interpretation.
  4. [Appendix F.1, Figure 23] The negative steering results in Figure 23 show substantially lower task specificity than the positive steering heatmaps, and the authors themselves note that "steering experiments alone may not suffice for a comprehensive analysis of the ICL mechanism." This is in tension with the strength of the causal language used for positive steering in Sections 1 and 3.2. Please clarify how the positive steering results should be interpreted in light of these negative results, and specify what evidential weight each experiment family carries for the central claim.
minor comments (5)
  1. [Algorithm 1] The pseudocode in Algorithm 1 omits the L1 coefficient λ, the early-stopping patience n, and the learning rate α; the text in Appendix D mentions some values (e.g., lr=0.15) but not all. Please report the exact hyperparameters used for each model.
  2. [Figure 24 caption] The caption of Figure 24 says "Full version of the heatmap in Figure 10" but it appears to describe the unfiltered executor-feature heatmap, which should reference Figure 6.
  3. [Appendix I] The max-activating examples are presented qualitatively without quantitative interpretability or specificity scores; consider using automated interpretability metrics or at least reporting activation statistics to support the claim that these features are task-specific.
  4. [Section 2.2, Eq. (3)] The definition of indirect effect in Eq. (3) uses do(a=a′) but the paper does not define a′; since the implementation uses zero ablation, this should be stated explicitly to avoid ambiguity.
  5. [Section 6] The conclusion says the authors "will also share SAE training codebase" but does not explicitly promise release of the TVC and SFC implementation; please clarify the intended code release scope.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'causal' steering validation of task-execution features reuses the exact NLL objective used to fit them, making that prediction partly circular; the detection-to-execution circuit claim retains independent support.

  1. fitted input called prediction [Section 3.1 (Algorithm 1) and Section 3.2 / Appendix D]
    "3. Optimizes θ to minimize L = LNLL(θ) + λ∥θ∥1 ... We first extracted relevant task features using our cleaning algorithm. Then steered the zero-shot prompt using them and calculated relative loss improvement, normalizing and clipping it after that."

    The task-execution features are produced by Algorithm 1, which explicitly minimizes negative log-likelihood loss on zero-shot prompts. The steering experiments then validate these features by measuring the same relative log-likelihood improvement on zero-shot prompts. Thus the headline claim that these latent vectors 'causally induce the task zero-shot' is a report of the fitted objective on a held-out slice of the same distribution, not an independent behavioral prediction. Held-out batches and max-activating examples provide some independent content, but the causal claim about executor features is not statistically forced only in the weak sense of out-of-sample generalization of the fit.

full rationale

The paper's central mechanistic claim — that task-detection features on output tokens drive task-execution features on arrow tokens through attention and MLP sublayers — is not circular. It is supported by SFC attribution patching, faithfulness measurements on held-out batches, and a direct ablation (Figure 11) in which ablating detection directions reduces execution activations. The low IE approximation quality in early layers (Appendix E.2) is an empirical validity concern, not a circularity. However, the TVC-based 'prediction' that task-execution features causally induce the task zero-shot does reduce, in part, to the fitting objective: Algorithm 1 optimizes the feature weights to minimize NLL on zero-shot prompts, and Section 3.2 evaluates them with the same NLL-based steering metric. The held-out split reduces overfitting but does not make the evaluation independent of the fitted quantity. No load-bearing self-citation chain or imported uniqueness theorem is present. Overall, partial circularity in the executor-feature causal claim, while the detection-to-execution circuit discovery is independently grounded.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The paper's central claims rest on several domain assumptions rather than on a first-principles derivation: SAE features faithfully capture causally relevant directions; task vectors from prior work are valid and causally active; attribution patching approximates true indirect effects well enough in the layers searched; and ICL works by discrete task recognition (stated in Appendix H). The TVC algorithm introduces fitted scalars (L1 coefficient, steering scale, target layers) that directly shape the main steering results. The two new feature families are empirical discoveries with independent evidence, so they are not hat-pulled entities, though their causal role is partly established through the same loss that selected them.

free parameters (4)
  • TVC L1 coefficient lambda = Main results use lambda between 0.001 and 0.025; sweeps over 1e-5 to 0.1
    Controls the sparsity/performance tradeoff in task vector cleaning; chosen by hand and swept in Figure 3 and Appendix D.1, with strong sensitivity to performance.
  • Steering scale for executor features (Gemma-1) = 15
    Selected via preliminary experiments (Appendix F); directly scales the added feature direction and affects all steering heatmaps.
  • Per-feature steering scale for Gemma-2 executors = Per-feature optimal scale between 30 and 300
    Because no common optimal scale existed, a per-feature scale maximizing loss decrease on any task was chosen, then reused across tasks (Appendix F.2); this is a per-feature fit to task loss.
  • Target layers for task vectors and detection = Layer 12 (execution), layer 11 (detection)
    Selected as the optimal single target layer from layer-wise loss curves (Figure 2 and Appendix G); a free modeling choice that shapes the circuit location.
assumptions (5)
  • domain assumption SAE features faithfully decompose model activations into interpretable, causally relevant directions
    The entire analysis operates in the SAE latent basis (Sections 2.1, 4). The paper validates indirectly via reconstruction and faithfulness metrics but does not prove the SAE basis captures all causally relevant directions.
  • domain assumption Task vectors exist, are extractable, and steering with them induces zero-shot task behavior
    Taken from Hendel et al. (2023) and Todd et al. (2024); the paper builds on this rather than re-deriving it, though its own steering experiments partly support it.
  • domain assumption Gradient-based attribution patching approximates true indirect effects for circuit edge discovery
    Used in the SFC adaptation (Section 4.1, Appendix E.2); the paper measures only moderate correlations (Figures 20-21) between predicted and actual indirect effects.
  • domain assumption In-context learning is mediated by discrete task recognition rather than task learning from scratch
    Explicitly flagged in Appendix H: 'The existence of discrete task detection and execution features hinges on the assumption that in-context learning works by classifying the task to perform and not by learning a task.'
  • domain assumption The task-relevant directions lie in the span of the SAE decoders
    TVC optimizes in the SAE decoder span (Section 3.1); if the true task direction has an out-of-span component, the sparse decomposition misses it. TVC parity with the full task vector on held-out loss is the only evidence for sufficiency.
invented entities (2)
  • Task-execution (executor) features independent evidence
    purpose: Sparse SAE latents that, when added to the residual stream, induce the corresponding task zero-shot; form the output side of the ICL circuit.
    Identified via TVC, but have independent evidence: max-activating examples on natural text (Appendix I), task-specific steering heatmaps on held-out batches (Figure 6), and negative steering effects (Figure 23).
  • Task-detection (detector) features independent evidence
    purpose: SAE latents that activate on output tokens and detect that a task has been completed; claimed to be causally upstream of executor features via attention and MLP sublayers.
    Discovered through SFC and TVC on detection prompts; independent evidence includes activation masses on output tokens (Table 2), steering effects (Figure 10), and ablation of detector directions reducing executor activations (Figure 11).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling sparse feature circuit finding for in-context learning." pith.science (2026). https://pith.science/paper/SK6XNCC3

@misc{pith2026250413756,
  author       = {Pith},
  title        = {Pith review of: Scaling sparse feature circuit finding for in-context learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SK6XNCC3}},
  note         = {Machine review of arXiv:2504.13756}
}
read the original abstract

Sparse autoencoders (SAEs) are a popular tool for interpreting large language model activations, but their utility in addressing open questions in interpretability remains unclear. In this work, we demonstrate their effectiveness by using SAEs to deepen our understanding of the mechanism behind in-context learning (ICL). We identify abstract SAE features that (i) encode the model's knowledge of which task to execute and (ii) whose latent vectors causally induce the task zero-shot. This aligns with prior work showing that ICL is mediated by task vectors. We further demonstrate that these task vectors are well approximated by a sparse sum of SAE latents, including these task-execution features. To explore the ICL mechanism, we adapt the sparse feature circuits methodology of Marks et al. (2024) to work for the much larger Gemma-1 2B model, with 30 times as many parameters, and to the more complex task of ICL. Through circuit finding, we discover task-detecting features with corresponding SAE latents that activate earlier in the prompt, that detect when tasks have been performed. They are causally linked with task-execution features through the attention and MLP sublayers.

Figures

Figures reproduced from arXiv: 2504.13756 by the authors.

Figure 1
Figure 1. A diagram of the in-context learning circuit, showing task detection features (yellow) causing task execution features (blue) which cause the model to output the antonym (left → right). A more concrete circuit, along with texts these features activate on, can be seen in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The effect on the Gemma 1 2B’s task losses by steering with different kinds of reconstructed task vectors, at each layer. We see that cleaning performs similarly to the original task vector until layer 14. Average relative loss change measured as a post-steering relative loss change compared to 0-shot, averaged across all tasks. 0 0.2 0.4 0.6 0.8 10μ 2 5 100μ 2 5 0.001 2 5 0.01 2 5 0.1 0 0.2 0.4 0.6 L1 coefficient R… view at source ↗
Figure 3
Figure 3. Evaluation of task vectors after applying our TVC al￾gorithm across different L1 coefficient λ values. Top: relative decrease in loss after steering (higher → better); bottom: fraction of retained active features (lower → better). Transparent lines represent different model and SAE combinations; solid lines show means across all of them; the results are averaged across tasks; (x-axis: L1 coefficient λ). Further deta… view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Overview of the task vector cleaning algorithm; TV stands for task vector. Given these observations, we developed a novel method called task vector cleaning ( [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: A subset of max activating examples for executor features from Appendix I. To analyze the activation patterns of executor features, we split all ICL prompt tokens into several types (highlighted in Example 1 and discussed later in Section 4.1.1). For each executor feat…
Figure 6
Figure 6. Figure 6: shows a heatmap of steering results for each pair of tasks and task-relevant features. Higher values indicate greater improvement in the loss after steering. It can be seen that most features that have a high effect on a single task generally do not significantly affec…
Figure 7
Figure 7. Figure 7: Faithfulness for task circuits C (a) and their complements M \ C (b). The ideal score for C is 1, and 0 for M \ C. lowing Marks et al. (2024), we define faithfulness as: Faithfulness(C) = m(C) − m(∅) m(M) − m(∅) (4) where m(C) is the metric with circuit C, m(∅) repre￾s…
Figure 9
Figure 9. Figure 9: A subset of max activating examples for detector features from Appendix I. Token Type Mass (%) output 96.76 input 3.22 newline 0.01 arrow 0.0 prompt 0.0 [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: Heatmap showing the effect of steering with the task￾detection feature most relevant to each task, on every task. We see that task detection features are typically specific to the task, with exceptions for similar tasks. To evaluate the causal connection between task-…
Figure 11
Figure 11. Figure 11: Heatmap showing the causal effect of the top task￾detection features of each task, on the activation of the top task￾execution features for every task. Averaged across all initial non￾zero activations in all tasks. tasks diverged from this pattern: person profession a…
Figure 12
Figure 12. Figure 12: An example of a circuit found using our SFC variant. We focused on a subcircuit with high indirect effects. Maximum activating examples from the SAE training distribution are included. BOS Follow the pattern : \n tall → short \n · · · old → young \n hot → cold Example…
Figure 13
Figure 13. Figure 13: An overview of our Task Vector Cleaning algorithm. TV stands for Task Vector. 12 optimizer = adam(weights, lr=0.15) 13 last_l0, without_change = 0, 0 # early stopping 14 for _ in range(1000): 15 grad = jax.grad(tvc_loss)(weights, tokens) 16 weights = optimizer.step(gr…
Figure 14
Figure 14. Figure 14: Performance of ITO and TVC across different tasks and optimization parameters compared to task vectors for Gemma 1 2B. The Y-axis shows relative improvement over task vector loss, while the X-axis shows the fraction of active TV features used. Metric calculation detai…
Figure 15
Figure 15. Figure 15: Performance of ITO and TVC across different tasks and optimization parameters compared to task vectors for Phi-3. The Y-axis shows relative improvement over task vector loss, while the X-axis shows the fraction of active TV features used. Metric calculation details ar…
Figure 16
Figure 16. Figure 16: Performance of ITO and TVC across different tasks and optimization parameters compared to task vectors for Gemma 2 Gemma Scope SAEs. The Y-axis shows the relative improvement over the loss from steering with a task vector, while the X-axis shows the fraction of active…
Figure 17
Figure 17. Figure 17: L1 coefficient sweeps across different models and SAEs. All metrics are averaged across all tasks. Error bars show the standard deviation of the average for each case. Metric calculation details are available in D.1. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_17.png]
Figure 18
Figure 18. Figure 18: L1 coefficient sweeps across different target SAE sparsities and widths for Gemma 2 2B. All metrics are averaged across all tasks. Error bars show the standard deviation of the average for each case. Metric calculation details are available in Appendix D.1. 21 [PITH_…
Figure 19
Figure 19. Figure 19: L1 coefficient sweeps across two SAE widths for Gemma 2 9B. All metrics are averaged across all tasks. Error bars show the standard deviation of the average for each case. Metric calculation details are available in D.1. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Average correlation of predicted and actual IEs across tasks for “input”, “output” and “arrow” non-zero nodes. Overall, we observe that the approximation quality remains relatively low before layer 6, which is much deeper in the model than layer 2, as reported by the …
Figure 21
Figure 21. Figure 21: Average correlation of predicted and actual IEs across tasks for “input”, “output” and “arrow” non-zero nodes for different SAE types. F. Steering with task-execution features To evaluate the causal relevance of our identified ICL features, we conducted a series of st…
Figure 22
Figure 22. Figure 22: Full version of the heatmap in [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]
Figure 23
Figure 23. Figure 23: displays the resulting heatmap. While we observe some degree of task specificity — and even note that some executing features from [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]
Figure 24
Figure 24. Figure 24: Unfiltered version of the heatmap in [PITH_FULL_IMAGE:figures/full_fig_p026_24.png]
Figure 25
Figure 25. Figure 25: Unfiltered positive steering heatmap for Gemma 2 2B SAEs showing the effect of steering with individual task-execution features for each task. Steering scales were optimized for each feature. The features present in the task vector of the corresponding task are marked…
Figure 26
Figure 26. Figure 26: Max activating examples for executor features from [PITH_FULL_IMAGE:figures/full_fig_p029_26.png]
Figure 27
Figure 27. Figure 27: Max activating examples for detector features from [PITH_FULL_IMAGE:figures/full_fig_p030_27.png]
Figure 28
Figure 28. Figure 28: Max activating examples for Gemma 2 2B 16k executor features from Figure 25a. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_28.png]
Figure 29
Figure 29. Figure 29: Max activating examples for Gemma 2 2B 65k executor features from Figure Figure 25b. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_29.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 21 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Cai, Q., Chaudhary, V., Chen, D., Chen, D., Chen, W., Chen, Y.-C., Chen, Y.-L., Cheng, H., Chopra, P., Dai, X., Dixon, M., Eldan, R., Fragoso, V., Gao, J., Gao, M., Gao, M., Garg...

    arXiv 2024

  3. [3]

    Many-shot jailbreaking

    Anil, C., Durmus, E., Sharma, M., Benton, J., Kundu, S., Batson, J., Rimsky, N., Tong, M., Mu, J., Ford, D., et al. Many-shot jailbreaking. 2024

    work page 2024

  4. [4]

    Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    Bai, Y., Chen, F., Wang, H., Xiong, C., and Mei, S. Transformers as statisticians: Provable in-context learning with in-context algorithm selection

    work page

  5. [5]

    Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale

    Bansal, H., Gopalakrishnan, K., Dingliwal, S., Bodapati, S., Kirchhoff, K., and Roth, D. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. URL http://arxiv.org/abs/2212.09095

    arXiv

  6. [6]

    Finding transformer circuits with edge pruning, 2024

    Bhaskar, A., Wettig, A., Friedman, D., and Chen, D. Finding transformer circuits with edge pruning, 2024. URL https://arxiv.org/abs/2406.16778

    arXiv 2024

  7. [7]

    Open source sparse autoencoders for all residual stream layers of gpt2-small, 2024

    Bloom, J. Open source sparse autoencoders for all residual stream layers of gpt2-small, 2024. URL https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD

    work page 2024

  8. [8]

    J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q

    Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., Vander P las, J., Wanderman- M ilne, S., and Zhang, Q. JAX : composable transformations of P ython+ N um P y programs, 2018. URL http://github.com/jax-ml/jax

    2018

Show all 70 references
  1. [9]

    Identifying functionally important features with end-to-end sparse dictionary learning, 2024

    Braun, D., Taylor, J., Goldowsky-Dill, N., and Sharkey, L. Identifying functionally important features with end-to-end sparse dictionary learning, 2024. URL https://arxiv.org/abs/2405.12241

    2024 arXiv

  2. [10]

    E., Hume, T., Carter, S., Henighan, T., and Olah, C

    Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Car...

    2023

  3. [11]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., L...

    1901

  4. [12]

    Batchtopk: A simple improvement for topk-saes, 2024

    Bussmann, B., Leask, P., and Nanda, N. Batchtopk: A simple improvement for topk-saes, 2024. URL https://www.alignmentforum.org/posts/Nkx6yWZNbAsfvic98/batchtopk-a-simple-improvement-for-topk-saes

    2024

  5. [13]

    Thread: Circuits

    Cammarata, N., Carter, S., Goh, G., Olah, C., et al. Thread: Circuits. Distill, 2020. doi:10.23915/distill.00024. https://distill.pub/2020/circuits

    2020 doi

  6. [14]

    Causal scrubbing: A method for rigorously testing interpretability hypotheses, 2022

    Chan, L., Garriga-Alonso, A., Goldowsky-Dill, N., Greenblatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris, B., and Thomas, N. Causal scrubbing: A method for rigorously testing interpretability hypotheses, 2022. URL https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN...

    2022

  7. [15]

    Unveiling induction heads: Provable training dynamics and feature learning in transformers

    Chen, S., Sheen, H., Wang, T., and Yang, Z. Unveiling induction heads: Provable training dynamics and feature learning in transformers. URL http://arxiv.org/abs/2409.10559

    arXiv

  8. [16]

    Update on how we train saes, 2024

    Conerly, T., Templeton, A., Bricken, T., Marcus, J., and Henighan, T. Update on how we train saes, 2024. URL https://transformer-circuits.pub/2024/april-update/index.html#training-saes

    2024

  9. [17]

    N., Lynch, A., et al

    Conmy, A., Mavor-Parker, A. N., Lynch, A., et al. Towards automated circuit discovery for mechanistic interpretability. In Proceedings of NeurIPS, 2023

    2023

  10. [18]

    Sparse autoencoders find highly interpretable features in language models, 2023

    Cunningham, H., Ewart, A., Riggs, L., et al. Sparse autoencoders find highly interpretable features in language models, 2023. URL https://arxiv.org/abs/2309.08600

    2023 arXiv

  11. [19]

    Why can GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers

    Dai, D., Sun, Y., Dong, L., Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can GPT learn in-context? language models implicitly perform gradient descent as meta-optimizers. URL http://arxiv.org/abs/2212.10559

    arXiv

  12. [20]

    A survey on in-context learning, 2024

    Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., Chang, B., Sun, X., Li, L., and Sui, Z. A survey on in-context learning, 2024. URL https://arxiv.org/abs/2301.00234

    2024 arXiv

  13. [21]

    The llama 3 herd of models, 2024

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  14. [22]

    Transcoders find interpretable llm feature circuits, 2024

    Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable llm feature circuits, 2024. URL https://arxiv.org/abs/2406.11944

    2024 arXiv

  15. [23]

    Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing

    Elad, M. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. Springer, New York, 2010. ISBN 978-1-4419-7010-7. doi:10.1007/978-1-4419-7011-4

    2010 doi

  16. [24]

    A mathematical framework for transformer circuits

    Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan...

    2021

  17. [25]

    Toy Models of Superposition

    Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. Toy Models of Superposition . arXiv preprint arXiv:2209.10652, 2022

    2022 arXiv

  18. [26]

    Experiments with an alternative method to promote sparsity in sparse autoencoders, 2024

    Farrell, E. Experiments with an alternative method to promote sparsity in sparse autoencoders, 2024. URL https://www.lesswrong.com/posts/cYA3ePxy8JQ8ajo8B/experiments-with-an-alternative-method-to-promote-sparsity

    2024

  19. [27]

    What can transformers learn in-context? a case study of simple function classes

    Garg, S., Tsipras, D., Liang, P., and Valiant, G. What can transformers learn in-context? a case study of simple function classes

  20. [28]

    Localizing model behavior with path patching, 2023

    Goldowsky-Dill, N., MacLeod, C., Sato, L., and Arora, A. Localizing model behavior with path patching, 2023. URL https://arxiv.org/abs/2304.05969

    2023 arXiv

  21. [29]

    Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., Hubinger, E., Lukošiūtė, K., Nguyen, K., Joseph, N., McCandlish, S., Kaplan, J., and Bowman, S. R. Studying large language model generalization with influence functi...

    2023 arXiv

  22. [30]

    Understanding in-context learning via supportive pretraining data

    Han, X., Simig, D., Mihaylov, T., Tsvetkov, Y., Celikyilmaz, A., and Wang, T. Understanding in-context learning via supportive pretraining data. URL http://arxiv.org/abs/2306.15091

    arXiv

  23. [31]

    How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023

    Hanna, M., Liu, O., and Variengien, A. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, 2023. URL https://arxiv.org/abs/2305.00586

    2023 arXiv

  24. [32]

    In-context learning creates task vectors, 2023

    Hendel, R., Geva, M., and Globerson, A. In-context learning creates task vectors, 2023. URL https://arxiv.org/abs/2310.15916

    2023 arXiv

  25. [33]

    Johnson, D. D. Penzai + treescope: A toolkit for interpreting, visualizing, and editing models as data, 2024. URL https://arxiv.org/abs/2408.00211

    2024 arXiv

  26. [34]

    and Garcia, C

    Kidger, P. and Garcia, C. Equinox: neural networks in jax via callable pytrees and filtered transformations, 2021. URL https://arxiv.org/abs/2111.00254

    2021 arXiv

  27. [35]

    Attention output saes improve circuit analysis, 2024

    Kissane, C., Krzyzanowski, R., Conmy, A., and Nanda, N. Attention output saes improve circuit analysis, 2024. URL https://www.alignmentforum.org/posts/EGvtgB7ctifzxZg6v/attention-output-saes-improve-circuit-analysis

    2024

  28. [36]

    Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023

    Lieberum, T., Rahtz, M., Kramár, J., et al. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023. URL https://arxiv.org/abs/2307.09458

    2023 arXiv

  29. [37]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URL https://arxiv.org/abs/2408.05147

    2024 arXiv

  30. [38]

    Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023

    Lin, J. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. URL https://www.neuronpedia.org. Software available from neuronpedia.org

    2023

  31. [39]

    B., and Ma, T

    Mahankali, A., Hashimoto, T. B., and Ma, T. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. URL http://arxiv.org/abs/2307.03576

    arXiv

  32. [40]

    J., et al

    Marks, S., Rager, C., Michaud, E. J., et al. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Computing Research Repository, arXiv:2403.19647, 2024. URL https://arxiv.org/abs/2403.19647

    2024 arXiv

  33. [41]

    Transformer circuit faithfulness metrics are not robust, 2024

    Miller, J., Chughtai, B., and Saunders, W. Transformer circuit faithfulness metrics are not robust, 2024. URL https://arxiv.org/abs/2407.08734

    2024 arXiv

  34. [42]

    Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022

    Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022. URL https://arxiv.org/abs/2202.12837

    2022 arXiv

  35. [43]

    Sparse autoencoder

    Ng, A. Sparse autoencoder. CS294A Lecture Notes, 2011. Unpublished lecture notes

    2011

  36. [44]

    Zoom in: An introduction to circuits

    Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduction to circuits. Distill, 2020. doi:10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

    2020 doi

  37. [45]

    In-context learning and induction heads, 2022

    Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Cla...

    2022 arXiv

  38. [46]

    v., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M

    Oswald, J. v., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vladymyrov, M. Transformers learn in-context by gradient descent. URL http://arxiv.org/abs/2212.07677

    arXiv

  39. [47]

    What in-context learning “learns” in-context: Disentangling task recognition and task learning

    Pan, J., Gao, T., Chen, H., and Chen, D. What in-context learning “learns” in-context: Disentangling task recognition and task learning. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023 , pp.\ 8298--831...

    2023 doi

  40. [48]

    F., Lee, A., Lubana, E

    Park, C. F., Lee, A., Lubana, E. S., Yang, Y., Okawa, M., Nishi, K., Wattenberg, M., and Tanaka, H. Iclr: In-context learning of representations, 2024. URL https://arxiv.org/abs/2501.00070

    2024 arXiv

  41. [49]

    Direct and indirect effects

    Pearl, J. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI'01, pp.\ 411–420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001

    2001

  42. [50]

    Causality: Models, Reasoning and Inference

    Pearl, J. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X

    2009

  43. [51]

    B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L

    Penedo, G., Kydlíček, H., allal, L. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https://arxiv.org/abs/2406.17557

    2024 arXiv

  44. [52]

    When does in-context learning fall short and why? a study on specification-heavy tasks, 2023

    Peng, H., Wang, X., Chen, J., Li, W., Qi, Y., Wang, Z., Wu, Z., Zeng, K., Xu, B., Hou, L., and Li, J. When does in-context learning fall short and why? a study on specification-heavy tasks, 2023. URL https://arxiv.org/abs/2311.08993

    2023 arXiv

  45. [53]

    Improving ghost grads, 2024

    Rajamanoharan, S. Improving ghost grads, 2024. URL https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/progress-update-1-from-the-gdm-mech-interp-team-full-update#Improving_ghost_grads

    2024

  46. [54]

    Improving dictionary learning with gated sparse autoencoders, 2024 a

    Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving dictionary learning with gated sparse autoencoders, 2024 a . URL https://arxiv.org/abs/2404.16014

    2024 arXiv

  47. [55]

    Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024 b

    Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024 b . URL https://arxiv.org/abs/2407.14435

    2024 arXiv

  48. [56]

    Pretraining task diversity and the emergence of non-bayesian in-context learning for regression

    Raventós, A., Paul, M., Chen, F., and Ganguli, S. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. URL http://arxiv.org/abs/2306.15063

    arXiv

  49. [57]

    and Brinkman, J

    Riggs, L. and Brinkman, J. Improving sae's by sqrt()-ing l1 and removing lowest activating features, 2024. URL https://www.lesswrong.com/posts/YiGs8qJ8aNBgwt2YN/improving-sae-s-by-sqrt-ing-l1-and-removing-lowest

    2024

  50. [58]

    Do pretrained transformers learn in-context by gradient descent? URL http://arxiv.org/abs/2310.08540

    Shen, L., Mishra, A., and Khashabi, D. Do pretrained transformers learn in-context by gradient descent? URL http://arxiv.org/abs/2310.08540

    arXiv

  51. [59]

    Measuring inductive biases of in-context learning with underspecified demonstrations

    Si, C., Friedman, D., Joshi, N., Feng, S., Chen, D., and He, H. Measuring inductive biases of in-context learning with underspecified demonstrations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 11289-...

    doi

  52. [60]

    Replacing sae encoders with inference-time optimisation, 2024

    Smith, L. Replacing sae encoders with inference-time optimisation, 2024. URL https://www.alignmentforum.org/posts/C5KAZQib3bzzpeyrg/full-post-progress-update-1-from-the-gdm-mech-interp-team#Replacing_SAE_Encoders_with_Inference_Time_Optimisation

    2024

  53. [61]

    Attribution patching outperforms automated circuit discovery, 2023

    Syed, A., Rager, C., and Conmy, A. Attribution patching outperforms automated circuit discovery, 2023. URL https://arxiv.org/abs/2310.10348

    2023 arXiv

  54. [62]

    S., Love, J., Tafti, P., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., Tafti, P., et al. Gemma: Open models based on gemini research and technology, 2024. URL https://arxiv.org/abs/2403.08295

    2024 arXiv

  55. [63]

    L., McDougall, C., MacDiarmid, M., Freeman, C

    Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and H...

    2024

  56. [64]

    L., Sen Sharma, A., et al

    Todd, E., Li, M. L., Sen Sharma, A., et al. Function vectors in large language models. In Proceedings of the 2024 International Conference on Learning Representations, 2024

    2024

  57. [65]

    Interpretability in the wild: A circuit for indirect object identification in GPT -2 small, 2022

    Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the wild: A circuit for indirect object identification in GPT -2 small, 2022. URL https://arxiv.org/abs/2211.00593

    2022 arXiv

  58. [66]

    Label words are anchors: An information flow perspective for understanding in-context learning

    Wang, L., Li, L., Dai, D., Chen, D., Zhou, H., Meng, F., Zhou, J., and Sun, X. Label words are anchors: An information flow perspective for understanding in-context learning. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Methods...

    2023 doi

  59. [67]

    Wang, X., Zhu, W., Saxon, M., Steyvers, M., and Wang, W. Y. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning, 2024. URL https://arxiv.org/abs/2301.11916

    2024 arXiv

  60. [68]

    M., Raghunathan, A., Liang, P., and Ma, T

    Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. An explanation of in-context learning as implicit bayesian inference. URL http://arxiv.org/abs/2111.02080

    arXiv

  61. [69]

    Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing, 2024. URL https://arxiv.org/abs/2406.08464

    2024 arXiv

  62. [70]

    Pretraining data mixtures enable narrow model selection capabilities in transformer models

    Yadlowsky, S., Doshi, L., and Tripuraneni, N. Pretraining data mixtures enable narrow model selection capabilities in transformer models. URL http://arxiv.org/abs/2311.00871

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.