Pith. sign in

REVIEW 1 major objections 7 minor 56 references

Editing 0.01% of neurons activates task behavior in language models

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 19:19 UTC pith:GFF6SE6U

load-bearing objection DSI: sparse neuron-level steering that works, but the ablation design conflates two improvements at once the 1 major comments →

arxiv 2607.07128 v1 pith:GFF6SE6U submitted 2026-07-08 cs.LG

Distributed Sparse Interventions in Language Models

Maximilian S. Ernst (1 , 2 , 3) , Lorenz Linhardt (3 , 4) , Aaron Peikert (2) , Oliver Eberle (3 , 4) ((1) Max Planck School of Cognition
show 4 more authors
(2) Center for Lifespan Psychology Max Planck Institute for Human Development (3) Machine Learning Group Technische Universit\"at Berlin (4) Berlin Institute for the Foundations of Learning Data (BIFOLD))
This is my paper
classification cs.LG
keywords language modelsmodel steeringneuron-level interventionssparse representationsinterpretabilitytask compositionnonlinear interactionsin-context learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces Distributed Sparse Interventions (DSI), a method that identifies tiny sets of neurons—as few as 8 out of roughly 100,000—distributed across layers and attention heads, whose additive modification activates a specific task in an instruction-tuned language model. The central claim is that task-relevant computation lives in a sparse, nonlinear, neuron-specific basis rather than in dense global directions in activation space, and that by iteratively re-evaluating intervention effects (rather than relying on a single first-order gradient estimate) one can find these small neuron sets and steer model behavior with high fidelity. The paper demonstrates this across 12 tasks and three model families, showing that 8–64 neurons often match or exceed 10-shot in-context-learning accuracy, and that both gradient robustification and iterative refinement are necessary because neuron interactions produce substantial nonlinear effects that linear steering methods miss. A set-based perspective on the identified neurons further allows decomposing tasks into shared subcomponents (e.g., input-copying neurons shared between translation and tense-switching) and task-specific transformation neurons, offering a compositional view of how tasks are structured at the neuron level.

Core claim

The paper's central discovery is that task behavior in large language models can be activated by intervening on a remarkably sparse set of individual neurons—distributed across layers and attention heads—provided that nonlinear interactions between neurons are explicitly modeled through iterative refinement of intervention effects, rather than approximated by a single first-order gradient. This finding challenges the prevailing assumption that effective steering requires modifying global directions across all neurons in a layer, and shows that the native neuron basis contains sufficiently sparse task-relevant structure to support fine-grained behavioral control.

What carries the argument

DSI works in four steps: (1) compute average activation differences between 10-shot and 0-shot prompts for every neuron; (2) estimate each neuron's expected effect on the correct output logit using a first-order Taylor approximation with LRP-modified (robustified) gradients; (3) select the top-n neurons by expected effect as a starting point and iteratively refine both which neurons are selected and their per-neuron intervention scaling using ZeroFPR, a nonconvex sparse optimization algorithm that re-evaluates expected effects at each step to capture nonlinear interactions; (4) apply the refined per-neuron interventions additively during the forward pass on 0-shot inputs. The activation-diff

Load-bearing premise

The method depends on a first-order gradient approximation (even if robustified) to provide a good enough starting point for the iterative optimizer to find effective sparse neuron sets; if this initialization were poor, the 50-step optimization with 10 restarts might not escape local optima, and the paper does not systematically test how sensitive the results are to initialization quality or optimization settings.

What would settle it

If one could show that randomly chosen neuron sets of the same size, when paired with the same iterative optimization procedure, achieve comparable task accuracy to DSI-selected sets, then the contribution of the neuron-selection mechanism (as opposed to the optimization procedure itself) would be undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If sparse neuron sets can reliably activate tasks, then targeted suppression of those same sets could selectively disable capabilities—such as translation or sentiment analysis—without retraining, offering a surgical alternative to coarse methods like fine-tuning or safety filters.
  • The decomposition of tasks into shared neuron sets (e.g., copy neurons) and task-specific neuron sets suggests a natural hierarchy: if higher-level reasoning tasks decompose similarly, one could map compositional structure of model capabilities by analyzing overlaps and differences in neuron sets across tasks.
  • The finding that interventions distribute across many layers and heads, rather than concentrating in predefined layers, implies that task representations are genuinely distributed—challenging the common practice of steering at a single chosen layer.
  • If the iterative refinement is essential because first-order approximations miss nonlinear interactions, then other interpretability methods that rely on single-pass gradient estimates (e.g., attribution patching) may systematically misidentify the most causally important components.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper tests single-token prediction only; if sparse neuron sets identified for single-token tasks do not transfer to multi-token generation (where the model must maintain task behavior across positions), the practical scope of DSI would be narrower than the results suggest.
  • The set-based decomposition into copy and transform neurons is demonstrated on two tasks with rigid input-output structure; whether richer, open-ended tasks exhibit similarly decomposable neuron-set structure remains an open question that the paper's case studies only begin to address.
  • If the LRP-modified gradient initialization were replaced with a fundamentally different attribution method, the iterative optimizer might converge to different neuron sets—raising the question of whether the identified neurons are stable features of the model or artifacts of the search procedure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The paper introduces Distributed Sparse Interventions (DSI), a method for identifying sparse sets of neurons distributed across layers and attention heads that can activate task-relevant behavior in instruction-tuned language models. DSI proceeds by (1) computing activation differences between k-shot and 0-shot prompts, (2) selecting an initial neuron set via a robustified (LRP-modified) gradient estimate, and (3) iteratively refining both the neuron set and per-neuron scaling factors using the ZeroFPR optimizer. The method is evaluated on 12 tasks across three models (Qwen 3 8B, Gemma 3 4B, Llama 3.2 3B), showing that intervening on as few as 8–64 neurons can substantially improve 0-shot task accuracy. Two case studies investigate task composition by decomposing neuron sets into shared (copying) and task-specific components via set operations.

Significance. The paper addresses a genuine gap in the model steering literature: most prior work operates on global directions in activation space, assuming approximate linearity and additivity. The demonstration in Figure 2 that neuron-level interventions exhibit nonlinear, neuron-specific effects—including interaction effects between neuron pairs—is a valuable empirical contribution. The finding that tasks can be activated by intervening on 0.01–0.04% of neurons, distributed across layers and heads, is striking and well-supported by the experimental design (train/test splits, 3 models × 12 tasks, 95% confidence intervals). The set-based perspective on task composition (Section 5) is a novel conceptual angle. Code is reportedly available on GitHub, supporting reproducibility.

major comments (1)
  1. The ablation design in Section 4.1 conflates two methodological changes, undermining the paper's attribution of performance gains to iterative refinement. The 'no iter.' baseline (Table 1) differs from full DSI in two ways simultaneously: (1) it removes iterative refinement of the neuron set, and (2) it replaces per-neuron scaling factors with a single joint scalar s ∈ [0.01, 50]. The paper states (Section 4.1) that 'both gradient robustification and iterative refinement contribute consistently to improved intervention performance,' but this claim cannot be evaluated because the contribution of iterative set refinement is entangled with the contribution of per-neuron scaling. Figure 2a already shows that optimal intervention strength varies substantially across neurons, so a uniform-scaling baseline is expected to perform poorly regardless of whether iterative set refinement helps. An中间基
minor comments (7)
  1. Section 3.2, Algorithm 1: The notation s₀ ← 1(topₙ(e)) is introduced without explicit definition of the indicator function. Readers unfamiliar with this convention may need to infer it from context.
  2. Table 1: The ranges (e.g., '0.18–0.97') are described as 'minimum and maximum accuracy across tasks' but it is unclear whether these are per-task accuracies or per-condition accuracies. Clarifying this would help interpretation.
  3. Figure 3: The x-axis uses a log-like spacing for neuron set sizes (8, 16, 32, ..., 2048) but is labeled linearly. This could mislead readers about the rate of performance change with set size.
  4. Section 5.1: The case study reports that the 'specific' neuron set (9 neurons for present-past) achieves 0.37 accuracy on the present-past task, compared to 0.85 for the full 32-neuron set. This is a substantial drop, and the text acknowledges that the specific set 'induce[s] less copy behaviour' but does not fully discuss why the task-specific neurons alone are insufficient. This weakens the decomposition narrative.
  5. Appendix F: The optimization uses 10 perturbed restarts with noise scale 0.3 and a step size multiplier of 1.5. No sensitivity analysis to these hyperparameters is provided. A brief note on robustness would strengthen the methodological contribution.
  6. The paper does not compare DSI against existing steering baselines (e.g., function vectors [49], contrastive activation addition [43], LoFiT [54]) in terms of task accuracy. While the ablations against simplified versions of DSI are informative, a direct comparison to established methods would better contextualize the reported gains.
  7. Section 4, footnote 2: The compute time (~108h on MI300A APUs) is noted, but the per-task or per-intervention-set-size compute cost is not broken down. This would help readers assess practical scalability.

Circularity Check

0 steps flagged

No significant circularity found; DSI's derivation is self-contained with proper train/test separation and LRP rules used as gradient tools, not load-bearing claims.

full rationale

The paper's derivation chain is self-contained. DSI identifies sparse neuron interventions using activation differences between 10-shot and 0-shot prompts (Eq. 3), a robustified gradient for first-order effect estimation (Eq. 2), and iterative refinement via ZeroFPR optimization (Algorithm 1). The LRP-modified gradient rules (Appendix D, Table 4) are cited from prior work by overlapping author groups [1, 2, 4, 42], but these serve as a computational tool for gradient estimation—they are not the central claim and do not define the intervention in terms of its own output. The method is trained on 100 0-shot and 100 10-shot prompts from a training split and evaluated on 200 held-out 0-shot prompts from a separate test split, so the 'prediction' (task activation on test data) is not fitted to the evaluation data by construction. The ablation baselines ('no iter.' and 'no rob.') differ from full DSI in conflated ways (per-neuron scaling vs. joint scaling), which is a correctness/experimental design concern rather than a circularity issue. The skeptic's attack targets ablation isolation, not circular reasoning. No step in the derivation reduces to its own inputs by definition or self-citation.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

No new entities, particles, forces, or dimensions are postulated. DSI operates entirely within the existing architecture of transformer language models.

free parameters (6)
  • n (neuron set size) = 8 to 2048 (swept)
    The number of neurons to intervene on is a hyperparameter swept across values; results are reported as a function of n.
  • t (number of optimization steps) = 50
    Maximum number of ZeroFPR iterations, chosen by the authors (Appendix F).
  • Number of perturbed restarts = 10
    Number of random perturbation runs for escaping local minima (Appendix F).
  • Perturbation noise scale = 0.3
    Standard deviation of Gaussian noise for perturbed starting values (Appendix F).
  • Step size multiplier for exploration = 1.5
    Factor applied to initial step size to encourage exploration beyond first step (Appendix F).
  • k (number of ICL examples for activation differences) = 10
    Number of in-context examples used to compute activation differences; fixed at 10 throughout.
axioms (4)
  • domain assumption Task-relevant behavior can be activated by additive interventions on neuron activations (a ← a + δ) rather than requiring structural replacement.
    Section 3, notation paragraph. The paper intervenes by adding δ to neuron activations during the forward pass, assuming additive modification is sufficient to elicit task behavior.
  • domain assumption The last token position is the relevant position for capturing task-relevant computations.
    Section 4, Experimental setup: 'we consider the outputs of all attention heads in each layer at the last token position.' This assumes task information is concentrated at the last token.
  • domain assumption LRP-modified gradients provide a more robust estimate of intervention effects than raw gradients in transformer models.
    Section 3.1, citing [23]. The identity rule, half rule, and LayerNorm rule (Appendix D, Table 4) are assumed to improve effect estimation without independent verification in this paper.
  • domain assumption Neurons in the native basis (before projection to residual stream) carry task-relevant structure accessible for intervention.
    Section 4, Experimental setup. The paper intervenes on attention head outputs before residual stream projection, assuming this basis is sufficiently informative despite potential superposition [13].

pith-pipeline@v1.1.0-glm · 41321 in / 3848 out tokens · 302122 ms · 2026-07-09T19:19:43.015667+00:00 · methodology

0 comments
read the original abstract

Language models perform a wide range of tasks at varying levels of abstraction with the capacity to flexibly infer tasks from context, execute multiple tasks simultaneously, and select among competing tasks. To study the role of model components in task behaviour, their causal influence can be investigated through interventions. Prior work on model steering has largely focused on interventions along global directions in activation space, modeling task representations as approximately linear and additive. By studying interventions at the neuron level, we find substantial, neuron-specific nonlinear effects on model outputs that are not captured by current steering approaches. We introduce Distributed Sparse Interventions (DSI), an intervention approach that considers nonlinearities and interactions between neurons across layers to identify sparse sets of neurons that elicit task-relevant computations. Across a range of tasks, we demonstrate that DSI can activate task behaviour in instruction-tuned language models by localising and intervening on as few as 0.01% of neurons, highlighting the effectiveness of sparse, distributed interventions in the neuron basis. Additionally, adopting a set-based perspective enables computations over the identified neuron sets, offering insights into the roles of individual neurons by analysing their effects across tasks. Through sparse interventions, DSI enables fine-grained control over model behaviour, localisation of task-relevant neuron sets, and furthers our understanding of task composition.

Figures

Figures reproduced from arXiv: 2607.07128 by 2, (2) Center for Lifespan Psychology Max Planck Institute for Human Development, 3), (3) Machine Learning Group Technische Universit\"at Berlin, 4), 4) ((1) Max Planck School of Cognition, (4) Berlin Institute for the Foundations of Learning, Aaron Peikert (2), Data (BIFOLD)), Lorenz Linhardt (3, Maximilian S. Ernst (1, Oliver Eberle (3.

Figure 1
Figure 1. Figure 1: Schematic overview of how distributed sparse interventions are identified and applied [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Value of the correct output logit of Llama 3.2 (3B) for the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: 0-shot test set accuracy for interventions across set sizes for extractive (top row) and [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: b contains the resulting accuracies and a qualitative example for the set size of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 4
Figure 4. Figure 4: Task decomposition into neuron sets shared with a [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Example system with causal interventions [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: 0-shot test set accuracy for interventions across set sizes for extractive (top row) and [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: 0-shot test set accuracy for interventions across set sizes for extractive (top row) and [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Absolute intervention strength for the first-order and iterative interventions, summed over [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Overlap of neuron sets identified by first-order approximation and the iterative algorithm. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Number of intervened-on model components (attention heads), identified by DSI, per [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Number of intervened-on neurons, identified by DSI, within layers and heads for Qwen 3 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 13
Figure 13. Figure 13: Number of intervened-on neurons, identified by DSI, within layers and heads for Gemma [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 12
Figure 12. Figure 12: Number of intervened-on neurons, identified by DSI, within layers and heads for Llama [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages · 7 internal anchors

  1. [1]

    Achtibat, S

    R. Achtibat, S. M. V . Hatefi, M. Dreyer, A. Jain, T. Wiegand, S. Lapuschkin, and W. Samek. AttnLRP: Attention-aware layer-wise relevance propagation for transformers. InProceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 135–168. PMLR, 2024. URL https://proceedings.mlr.pres...

    work page 2024

  2. [2]

    A. Ali, T. Schnake, O. Eberle, G. Montavon, K.-R. Müller, and L. Wolf. XAI for transformers: Better explanations through conservative propagation. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 435–451. PMLR,

    work page

  3. [3]

    URL https://proceedings.mlr.press/v162/ali22a.html

    work page

  4. [4]

    Language Model Circuits Are Sparse in the Neuron Basis

    A. Arora, Z. Wu, J. Steinhardt, and S. Schwettmann. Language model circuits are sparse in the neuron basis. arXiv preprint arXiv:2601.22594, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  5. [5]

    Arras, J

    L. Arras, J. Arjona-Medina, M. Widrich, G. Montavon, M. Gillhofer, K.-R. Müller, S. Hochreiter, and W. Samek. Explaining and Interpreting LSTMs, page 211–238. Springer-Verlag, Berlin, Heidelberg, 2022. ISBN 978-3-030-28953-9. URL https://doi.org/10.1007/978-3-030-28954-6_11

    work page doi:10.1007/978-3-030-28954-6_11 2022

  6. [6]

    S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015

    work page 2015

  7. [7]

    Balduzzi, M

    D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W.-D. Ma, and B. McWilliams. The shattered gradients problem: if resnets are the answer, then what is the question? In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 342–350. JMLR.org, 2017

    work page 2017

  8. [8]

    D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017

    work page 2017

  9. [9]

    Beaglehole, A

    D. Beaglehole, A. Radhakrishnan, E. Boix-Adsera, and M. Belkin. Toward universal steering and monitoring of AI models. Science, 391(6787):787–792, 2026

    work page 2026

  10. [10]

    Bricken, A

    T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2(5):6, 2023

    work page 2023

  11. [11]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei...

    work page 1901

  12. [12]

    D. Dai, L. Dong, Y . Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8493–8502, Dublin, Ireland, 2022. Association for Computational Linguistics. do...

    work page doi:10.18653/v1/2022.acl-long.581 2022

  13. [13]

    Eberhardt and R

    F. Eberhardt and R. Scheines. Interventions and causal inference. Philosophy of science, 74(5):981–995, 2007

    work page 2007

  14. [14]

    Toy Models of Superposition

    N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  15. [15]

    U. Evci, T. Gale, J. Menick, P. S. C. Rivadeneira, and E. Elsen. Rigging the lottery: Making all tickets winners. In International Conference of Machine Learning, 2020. 10

    work page 2020

  16. [16]

    Frankle and M

    J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum? id=rJl-b3RcF7

    work page 2019

  17. [17]

    Gemma 3 technical report

    Gemma Team. Gemma 3 technical report. 2025. URL https://goo.gle/Gemma3Report

    work page 2025

  18. [18]

    The Llama 3 Herd of Models

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  19. [19]

    Hendel, M

    R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318–9333, 2023

    work page 2023

  20. [20]

    Linearity of Relation Decoding in Transformer Language Models

    E. Hernandez, A. S. Sharma, T. Haklay, K. Meng, M. Wattenberg, J. Andreas, Y . Belinkov, and D. Bau. Linearity of relation decoding in transformer language models. arXiv preprint arXiv:2308.09124, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    Fine-grained prediction of reading comprehension from eye movements,

    O. Honovich, U. Shaham, S. R. Bowman, and O. Levy. Instruction induction: From few examples to natural language task descriptions. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1935–1952, Toronto, Canada, July 2023. Association f...

    work page doi:10.18653/v1/ 1935

  22. [22]

    Huben, H

    R. Huben, H. Cunningham, L. R. Smith, A. Ewart, and L. Sharkey. Sparse autoencoders find highly inter- pretable features in language models. InThe Twelfth International Conference on Learning Representations, 2023

    work page 2023

  23. [23]

    Editing Models with Task Arithmetic

    G. Ilharco, M. T. Ribeiro, M. Wortsman, S. Gururangan, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  24. [24]

    F. R. Jafari, O. Eberle, A. Khakzar, and N. Nanda. Relp: Faithful and efficient circuit discovery in language models via relevance patching. arXiv preprint arXiv:2508.21258, 2025

    work page arXiv 2025

  25. [25]

    A. K. Lampinen, Y . Li, E. Hosseini, S. Bhardwaj, and M. Shanahan. Linear representations in language models can change dramatically over a conversation. arXiv preprint arXiv:2601.20834, 2026

    work page arXiv 2026

  26. [26]

    Lample, A

    G. Lample, A. Conneau, M. Ranzato, L. Denoyer, and H. Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum? id=H196sainb

    work page 2018

  27. [27]

    J. Li, Y . Hou, M. Sachan, and R. Cotterell. What do language models learn in context? the structured task hypothesis. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12365–12379, 2024

    work page 2024

  28. [28]

    L. Li, P. Dong, Z. Tang, X. Liu, Q. Wang, W. Luo, W. Xue, Q. Liu, X. Chu, and Y . Guo. Discovering sparsity allocation for layer-wise pruning of large language models. InThe Thirty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL https://openreview.net/forum?id=rgtrYVC9n4

    work page 2024

  29. [29]

    Y . Li, D. I. Campbell, S. C. Chan, and A. K. Lampinen. Just-in-time and distributed task representations in language models. In Mechanistic Interpretability Workshop at NeurIPS 2025 , 2025. URL https: //openreview.net/forum?id=KC13tBPbQg

    work page 2025

  30. [30]

    Lindsey, A

    J. Lindsey, A. Templeton, J. Marcus, T. Conerly, J. Batson, and C. Olah. Sparse crosscoders for cross- layer features and model diffing.https://transformer-circuits.pub/2024/crosscoders/index. html, 2024. Transformer Circuits Thread

    work page 2024

  31. [31]

    Lippl, T

    S. Lippl, T. McGee, K. Lopez, Z. Pan, P. Zhang, S. Ziadi, O. Eberle, and I. Momennejad. Algorithmic primitives and compositional geometry of reasoning in language models, 2026. URL https://arxiv. org/abs/2510.15987

    work page arXiv 2026

  32. [32]

    D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization.Mathematical programming, 45(1):503–528, 1989

    work page 1989

  33. [33]

    S. Liu, H. Ye, L. Xing, and J. Zou. In-context vectors: making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    work page 2024

  34. [34]

    Marks, C

    S. Marks, C. Rager, E. J. Michaud, Y . Belinkov, D. Bau, and A. Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Representations, 2025. 11

    work page 2025

  35. [35]

    Montavon, A

    G. Montavon, A. Binder, S. Lapuschkin, W. Samek, and K.-R. Müller. Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning, pages 193–209, 2019

    work page 2019

  36. [36]

    N. Nanda. Attribution patching: Activation patching at industrial scale. URL: https://www. neelnanda. io/mechanistic-interpretability/attribution-patching, 15:19, 2023

    work page 2023

  37. [37]

    Nanda and J

    N. Nanda and J. Bloom. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens, 2022

    work page 2022

  38. [38]

    Nikankin, A

    Y . Nikankin, A. Reusch, A. Mueller, and Y . Belinkov. Arithmetic without algorithms: Language models solve math with a bag of heuristics. In The Thirteenth International Conference on Learning Representa- tions, 2024

    work page 2024

  39. [39]

    B. A. Olshausen and D. J. Field. Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14(4):481–487, 2004. ISSN 0959-4388. doi: https://doi.org/10.1016/j.conb.2004.07.007. URL https: //www.sciencedirect.com/science/article/pii/S0959438804001035

    work page doi:10.1016/j.conb.2004.07.007 2004

  40. [40]

    B. A. Olshausen et al. Principles of image representation in visual cortex. The visual neurosciences, 2: 1603–1615, 2003

    work page 2003

  41. [41]

    Panigrahi, N

    A. Panigrahi, N. Saunshi, H. Zhao, and S. Arora. Task-specific skill localization in fine-tuned language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    work page 2023

  42. [42]

    Qwen3 Technical Report

    Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  43. [43]

    Rezaei Jafari, G

    F. Rezaei Jafari, G. Montavon, K.-R. Müller, and O. Eberle. MambaLRP: Explaining selective state space sequence models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 118540–118570. Curran As- sociates, Inc., 2024. doi: 10.52202/079017-3764....

    work page doi:10.52202/079017-3764 2024

  44. [44]

    Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suárez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr

    N. Rimsky, N. Gabrieli, J. Schulz, M. Tong, E. Hubinger, and A. Turner. Steering llama 2 via contrastive activation addition. In L.-W. Ku, A. Martins, and V . Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522, Bangkok, Thailand, Aug. 2024. Association for Co...

    work page doi:10.18653/v1/2024 2024

  45. [45]

    SmoothGrad: removing noise by adding noise

    D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  46. [46]

    Socher, A

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In D. Yarowsky, T. Baldwin, A. Korhonen, K. Livescu, and S. Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washing...

    work page 2013

  47. [47]

    L. Stella. ProximalAlgorithms.jl: Proximal algorithms for nonsmooth optimization in Julia. URL https://github.com/JuliaFirstOrder/ProximalAlgorithms.jl

    work page

  48. [48]

    Sundararajan, A

    M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research, pages 3319–3328. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/ sundararajan17a.html

    work page 2017

  49. [49]

    Themelis, L

    A. Themelis, L. Stella, and P. Patrinos. Forward-backward envelope for the sum of two nonconvex functions: Further properties and nonmonotone linesearch algorithms. SIAM Journal on Optimization, 28 (3):2274–2303, 2018. doi: 10.1137/16M1080240

    work page doi:10.1137/16m1080240 2018

  50. [50]

    E. Todd, M. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=AwyxtyMwaG

    work page 2024

  51. [51]

    Vargas-Barroso, J

    V . Vargas-Barroso, J. F. Watson, A. Navas-Olive, A. Schlögl, and P. Jonas. Developmental emergence of sparse and structured synaptic connectivity in the hippocampal ca3 memory circuit. Nature Communica- tions, 2026. 12

    work page 2026

  52. [52]

    X. Wang, K. Wen, Z. Zhang, L. Hou, Z. Liu, and J. Li. Finding skill neurons in pre-trained transformer- based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11132–11152, 2022

    work page 2022

  53. [53]

    E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927

    work page 1927

  54. [54]

    Xiong, Z

    Z. Xiong, Z. Cai, J. Cooper, A. Ge, V . Papageorgiou, Z. Sifakis, A. Giannou, Z. Lin, L. Yang, S. Agarwal, G. Chrysos, S. Oymak, K. Lee, and D. Papailiopoulos. Everything everywhere all at once: LLMs can in-context learn multiple tasks in superposition. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id...

    work page 2025

  55. [55]

    Q: input 1\nA: output1\n\nQ: input2\nA: output2\n\n[...]Q: inputk\nA: outputk\n\nQ: input\nA:

    F. Yin, X. Ye, and G. Durrett. Lofit: Localized fine-tuning on LLM representations. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 9474–9506. Curran Associates, Inc., 2024. doi: 10. 52202/079017-0301. URL https://proceedings.neurips.cc/paper...

    work page 2024

  56. [56]

    How are you feeling today?

    and a maximum of 200 generated tokens. During the autoregressive answering process, we intervene on the model on each forward pass. Some special characters (such as emotes) that could not be correctly displayed were stripped. Prompt 1: How are you feeling today? Baseline: I’m just a virtual assistant, so I don’t have feelings, but I’m here and ready to he...

    work page 2014