Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Instruction-following in LLMs is driven by localized instruction vectors that are linearly separable by task yet causally superadditive, and that gate later-layer information pathways.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 03:25 UTC pith:FM6NUQAA

load-bearing objection Solid localization evidence for instruction vectors, but the superadditivity claim rests on a statistically fragile test and the path-tracing method is unvalidated. the 4 major comments →

arxiv 2602.07930 v2 pith:FM6NUQAA submitted 2026-02-08 cs.CL

Patches of Nonlinearity: Instruction Vectors in Large Language Models

Irina Bigoulaeva , Jonas Rohweder , Subhabrata Dutta , Iryna Gurevych This is my paper
classification cs.CL
keywords instruction vectorsmechanistic interpretabilityactivation patchingsuperadditivitycircuit selectionlinear representation hypothesisinstruction tuningcausal mediation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that instruction-tuned language models do not process instructions just-in-time with the query; instead, they eagerly build a compact, localized digest of the instruction at the final instructional token, called an Instruction Vector (IV). These IVs are linearly separable by task, yet their causal effects are superadditive: patching two layers together improves the target answer's rank more than the sum of patching each layer alone. Because this inequality holds across tasks and model variants, the authors conclude that additive component-attribution and causal-graph circuit discovery are inadequate for instruction processing. To explain the mechanism, the paper decomposes the forward pass into locally linear token-to-token maps and finds that early layers construct the task digest with nearly task-agnostic circuits, while later layers switch to task-specific pathways conditioned on that digest. If correct, this reframes mechanistic interpretability: linear representation can coexist with nonlinear causal interaction, so single-component attributions will undervalue otherwise load-bearing variables.

Core claim

The central object is the Instruction Vector (IV), the model's residual-stream state at the final token of the instruction segment. The paper finds that IVs are formed eagerly before the query is processed, they are localized to a small set of layers, and their task identity is linearly recoverable. The key causal discovery is superadditivity: for high-contributing layer pairs, the causal effect of patching both together satisfies f(x_i ∪ x_j) ≥ f(x_i) + f(x_j), where f measures improvement in reciprocal rank or logit of the target token. Because the inequality holds with very small p-values, the authors conclude that IVs at different layers interact nonlinearly rather than additively. Using

What carries the argument

The central object is the Instruction Vector (IV), defined as the residual-stream state at the final instructional token. The identity doing the causal work is Eq. (2), the superadditivity inequality f(x_i ∪ x_j) ≥ f(x_i) + f(x_j) for the causal effect of patched layers; the paper uses this inequality to argue that IVs interact nonlinearly. To avoid additive assumptions, the machinery is a locally linear decomposition of the Transformer into token-to-token maps: normalization and MLP nonlinearities are replaced by input-dependent affine surrogates, which rewrites the forward pass as a sum of linear paths from each input token to the output logits. Tracing only maximal-attention paths that ra

Load-bearing premise

The load-bearing premise is that the reciprocal-rank (or logit) patching score faithfully measures causal effect, and that testing the inequality on the top-10 layer pairs chosen from the same heatmaps does not inflate apparent superadditivity; if reciprocal-rank saturation or selection-on-maximum artifacts produce f(x_i ∪ x_j) ≥ f(x_i) + f(x_j) even without true nonlinear interaction, the central claim fails even though localization is real.

What would settle it

Run 1- and 2-layer patching on all layer pairs rather than only the top-10, using a non-saturating metric such as the log-probability of the target token, and check whether the superadditivity inequality holds on held-out pairs; if the excess of pair effect over the sum of singles vanishes or becomes negative once saturation and selection bias are removed, the superadditivity claim is refuted.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Activation-patching and circuit-discovery methods that assume additive component contributions will systematically misattribute instruction-following, because a layer can have low effect alone and high effect in combination.
  • Instruction representations are formed eagerly, before the query arrives, so an instruction-following model contains a localized bottleneck at the final instructional token that could be monitored or defended.
  • The linear representation hypothesis can hold while causal interactions are nonlinear, so linear probing and causal attribution measure different properties and should not be conflated.
  • Pretrained models can already form task digests, but instruction tuning contributes the ability to use those digests to select task-specific circuits, giving a measurable mechanistic signature of what SFT and DPO change.
  • The locally linear path-tracing method offers an intervention-free way to localize information processing for behaviors where patching is unreliable, such as synergistic interactions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If IVs are circuit selectors, steering instruction-following could be more effective by modulating the early-layer digest rather than patching individual downstream heads; a direct test would compare interventions at the final instructional token with interventions on selected later-layer heads.
  • One would predict that instruction vectors transfer across queries and across paraphrases of the same instruction; this is testable by patching the representation at the final instructional token from one paraphrase into a new query and checking whether task performance is restored.
  • If the instruction bottleneck is real, adversarial perturbations at the final instructional token should be disproportionately disruptive compared with perturbations elsewhere in the prompt; token-level adversarial attacks could test this.
  • Similar superadditivity may appear for other task digests, such as in-context learning and prompt-level demos, which would make nonlinear interaction a general property of task digests rather than a quirk of instructions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper asks how instruction-tuned language models construct and use task instructions. Using activation patching on OLMo-2 base, SFT, and DPO models (1B and 7B), the authors identify localized 'instruction vectors' at the final instructional token, report that these vectors are linearly separable by task (linear-probe accuracy >98%), and claim that they are causally superadditive: two-layer patching yields a larger reciprocal-rank recovery than the sum of the individual layers (Eq. 2, §5.1). To explain this nonlinear interaction, the paper proposes an intervention-free path-tracing method built from locally linear token-to-token maps, and concludes that instruction vectors act as 'circuit selectors': early-layer task representations condition later-layer information pathways. The paper also compares base and post-trained models, arguing that instruction tuning is necessary for the circuit-selection ability.

Significance. If the superadditivity and circuit-selector claims hold, the paper makes a substantive contribution by challenging additive, component-wise circuit discovery and the linear representation hypothesis in the instruction-following setting. The study is transparent: code is released, models have clean base/SFT/DPO separation, and the appendix contains extensive heatmaps and numerical tables. The localization result and the linear-separability result are credible and useful on their own. However, the load-bearing superadditivity claim rests on a statistical test with clear selection bias and a bounded, nonlinear metric, and the path-tracing method relies on strong pruning assumptions that are not validated. These issues are fixable with additional experiments, so the paper is promising but not yet definitive.

major comments (4)
  1. [§5.1, Eq. (2), Appendix B (Tables 3–6)] The superadditivity test is subject to selection bias. The top-10 layer combinations are selected from the same heatmaps/score tables that define the joint effect f(xi ∪ xj). With roughly 120 pairs for 1B models and ~500 for 7B models, the maximum joint effect is inflated by sampling noise even if the true causal effects are additive. The near-zero p-values on these selected pairs do not establish a model-level nonlinear interaction. Please test Eq. 2 on all pairs (or a pre-specified holdout subset) with an appropriate multiple-comparison or permutation null.
  2. [§5.1, Eq. (2), Tables 3–6] The metric f is improvement in reciprocal rank, a bounded [0,1] nonlinear function of logits. Saturation or threshold effects can produce f(xi ∪ xj) ≥ f(xi) + f(xj) even when logit effects are additive. The T-test is described on rank-contribution scores, while logit heatmaps are shown only visually. Please re-run Eq. 2 on logit differences for the full set of pairs, or justify that reciprocal-rank superadditivity is the intended causal quantity. Additionally, the reported t-statistics are uniformly negative, e.g., Table 3, (3,3): t = -1.003e+03, which appears inconsistent with a positive mean difference f(i,j) - f(i) - f(j); please clarify the sign convention and the exact null hypothesis.
  3. [§6.1, Eq. (3), 'Determining the Mechanism'] The path-tracing method prunes to argmax attention edges and retains only paths with answer-token rank <100. This is a strong assumption: non-argmax attention heads/edges and individually low-ranked paths may collectively carry a substantial part of the mechanism. Without sensitivity analyses (e.g., top-k attention for k>1, varying the rank threshold, or comparison with the full linearized forward pass), the circuit-selector conclusion is not robust. The forward pass is still decomposed as a sum of linear maps, so the claim that the method is 'free from additive assumptions' is not validated at the level of the pruned path set.
  4. [§6.1, 'Attention Head Activity' and 'Circuit Selectors'] The conclusion that IVs act as circuit selectors is derived from paths already defined by leading to the answer token with rank <100. Finding that Tinst emits the most high-ranking paths and that later-layer head activity differs by task is partly by construction, since the path set excludes alternatives. Please provide baseline comparisons—random paths, paths emitted from other tokens, or ablation/perturbation along selected paths—to demonstrate that the identified pathways are specifically and causally relevant. Without such a control, the distinction between 'paths that happen to lead to the answer token' and 'instruction-conditioned circuit selection' is not established.
minor comments (5)
  1. [§5.1, last paragraph] The sentence 'we take their unnormalized patching scores and evaluate Eq. 2 (as a boolean truth value 1 or 0)' is confusing. A T-test on boolean indicators is not a standard procedure; please report the continuous differences and the exact test statistic.
  2. [Figure 2, Figures 10–11] The Figure 2 caption says 'logit improvement' but the heatmaps display reciprocal rank. Figures 10 and 11 have similar inconsistencies between captions and plotted quantities. Please align captions with the plotted metric.
  3. [Appendix A, Table 2] Table 2 lists an 'OLMo-3 7B' column, but OLMo-3 results do not appear in the paper. Clarify whether this model was used or remove the column.
  4. [Appendix B, Tables 3–6] Many entries have t = -inf or p = 0.000e+00 due to floating-point underflow. Please state how infinite values were handled and report finite precision or sufficient statistics.
  5. [§6.1, path tracing] The threshold 'answer rank <100' appears arbitrary. Please justify the choice and include a sensitivity analysis or explain why it does not affect the conclusions.

Circularity Check

0 steps flagged

No circular derivation: localization, linear-separability, and path-tracing claims are measured against independent baselines; the superadditivity test's selection-on-maximum is a statistical robustness concern, not a circular reduction.

full rationale

The paper's derivation chain is self-contained. Localization of instruction vectors is established by activation patching against a no-instruction baseline (filler token), producing full-grid heatmaps over all layer pairs; the location of the instruction vector is not definitionally tied to the conclusion. Linear separability is tested by training a linear probe on 200 instruction rephrasings and evaluating on held-out samples, a standard supervised separability test. The superadditivity claim (Eq. 2) is an empirical inequality on patching scores. The main concern is that the T-test is applied only to the top-10 layer combinations selected from the same heatmaps, and the reciprocal-rank metric is bounded/nonlinear, so selection-on-maximum and metric saturation could produce superadditivity-like numbers without true synergistic causal effects. However, this is a statistical-validity and multiple-comparisons issue, not a circular reduction: the inequality is not guaranteed by the selection criterion, and no fitted parameter is repackaged as a prediction. The path-tracing method in Section 6 is a locally-linear decomposition of the forward pass; defining high-ranking paths as those contributing to the answer token is an operational target, and the observation that Tinst emits many such paths and that attention-head activity differs across tasks is empirical rather than forced by the path definition. No load-bearing self-citation or imported uniqueness theorem appears; references to prior work (Meng et al., Todd et al., Davidson et al., Elhage et al.) are external. The Limitations section acknowledges that mechanism experiments are limited to 1B models and that human-defined contrastive tasks may not match the model's own task similarity; these are generalizability limitations, not evidence of circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 2 invented entities

The paper introduces one central new construct (IVs) and one functional role (circuit selector), both without external falsifiable handles, and the quantitative claims rely on hand-chosen thresholds and post-hoc selections. No continuous free parameter is fitted to produce a predicted constant; the main parameters are discrete method choices.

free parameters (3)
  • High-ranking path threshold = rank < 100
    Defines which traced paths count in §6.1; changing this threshold changes token-path statistics and the 'Tinst emits most paths' result.
  • Top-10 layer-combination selection = 10 per task
    The superadditivity T-tests are computed only on the ten best patching combinations per task selected from the same heatmaps (§5.1, Tables 3-6), so the near-zero p-values partly reflect selection on the dependent variable.
  • Attention pruning rule = argmax edge only
    Path tracing keeps only the single highest-attention edge per head (§6.1), discarding remaining attention mass; no ablation shows this preserves causal structure.
axioms (4)
  • domain assumption Activation patching on residual-stream layer states is a valid causal mediation tool.
    Section 5.1: the method follows Meng et al. (2022) and assumes clean interchange interventions; this is an established practice but remains an assumption.
  • domain assumption LayerNorm and MLP nonlinearities can be replaced by local affine/diagonal surrogates valid in an ε-neighborhood of the input.
    Section 6, Eq. 3: the locally-linear map decomposition depends on this linearization being accurate for the traced inputs.
  • ad hoc to paper The forward pass decomposes into a sum of token-to-token locally-linear maps whose effects are additive.
    Section 6.1: 'the effects of these paths remain additive'—this additivity is needed for the method but is in tension with the paper's own superadditivity finding.
  • ad hoc to paper Tracing only argmax-attention edges and answer-rank<100 paths preserves the mechanism.
    Section 6.1: computational pruning without a validation experiment; conclusions about circuit selection depend on it.
invented entities (2)
  • Instruction Vectors (IVs) no independent evidence
    purpose: Explanatory construct naming localized instruction digests at Tinst.
    Operationally defined through the paper's patching and probing setup; no falsifiable prediction outside the paper.
  • Circuit selector role no independent evidence
    purpose: Attributed function of IVs selecting downstream task-specific pathways.
    Inferred from head-activity differences along self-selected paths; not causally tested.

pith-pipeline@v1.3.0-alltime-deepseek · 38884 in / 11980 out tokens · 128806 ms · 2026-08-03T03:25:53.852805+00:00 · methodology

0 comments
read the original abstract

Despite the recent success of instruction-tuned language models and their ubiquitous usage, very little is known of how models process instructions internally. In this work, we address this gap from a mechanistic point of view by investigating how instruction-specific representations are constructed and utilized in different stages of post-training: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO). Via causal mediation, we identify that instruction representation is fairly localized in models. These representations, which we call Instruction Vectors (IVs), demonstrate a curious juxtaposition of linear separability along with non-linear causal interaction, broadly questioning the scope of the linear representation hypothesis commonplace in mechanistic interpretability. To disentangle the non-linear causal interaction, we propose a novel method to localize information processing in language models that is free from the implicit linear assumptions of patching-based techniques. We find that, conditioned on the task representations formed in the early layers, different information pathways are selected in the later layers to solve that task, i.e., IVs act as circuit selectors.

Figures

Figures reproduced from arXiv: 2602.07930 by Irina Bigoulaeva, Iryna Gurevych, Jonas Rohweder, Subhabrata Dutta.

Figure 1
Figure 1. Figure 1: We locate instruction vectors (IVs) in the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Effects of 1- and 2-layer patching configurations on the reciprocal rank of the target token. Each square of [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Conceptual decomposition of the Transformer [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Path contribution by token position for 1B models. For each task, we examine a subset of token positions [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Attention head activity for OLMo2-1B across [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effects of 1- and 2-layer patching configurations on the logit of the target token. Each square of the x- and [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Effects of 1- and 2-layer patching configurations on the rank of the target token for OLMo-2 7B models. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Effects of 1- and 2-layer patching configurations on the logit of the target token for BigBench tasks. Each [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effects of 1- and 2-layer patching configurations on the rank of the target token for BigBench tasks. Each [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Effects of 1- and 2-layer patching configurations on the logit of the target token. Each square of the x- and [PITH_FULL_IMAGE:figures/full_fig_p027_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Effects of 1- and 2-layer patching configurations on the reciprocal rank of the target token for OLMo-2 [PITH_FULL_IMAGE:figures/full_fig_p027_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: EMA scores (in color) for OLMo-1B and OLMo-7B models. Instruction Accuracy scores (gray) show that instruction-following abilities are present for most models and tasks (IA>50%) [PITH_FULL_IMAGE:figures/full_fig_p029_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Our LDA analysis of 200 instruction rephrasals on OLMo-2 models shows that tasks form organized [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L Turner, et al. 2025. https://transformer-circuits.pub/2025/attribution-graphs/methods.html Circuit tracing: Revealing computational graphs in language models . Transformer Circuits Thread

    2025

  2. [2]

    Nooshin Bahador. 2025. https://doi.org/10.48550/ARXIV.2504.02976 Localized definitions and distributed reasoning: A proof-of-concept mechanistic interpretability study via activation patching . CoRR, abs/2504.02976

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.02976 2025

  3. [3]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, et al. 2024. https://doi.org/10.48550/ARXIV.2401.02954 Deepseek LLM: scaling open-source language models with longtermism . CoRR, abs/2401.02954

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.02954 2024

  4. [4]

    Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, et al. 2022. https://arxiv.org/abs/2211.12312 Interpreting neural networks through the polytope lens . Preprint, arXiv:2211.12312

    Pith/arXiv arXiv 2022

  5. [5]

    Anwoy Chatterjee, H. S. V. N. S. Kowndinya Renduchintala, Sumit Bhatia, and Tanmoy Chakraborty. 2025. https://doi.org/10.1162/TACL.A.42 On the effect of instruction tuning loss on generalization . Trans. Assoc. Comput. Linguistics, 13:1360--1380

    doi:10.1162/tacl.a.42 2025

  6. [6]

    Mavor - Parker, Aengus Lynch, Stefan Heimersheim, and Adri \` a Garriga - Alonso

    Arthur Conmy, Augustine N. Mavor - Parker, Aengus Lynch, Stefan Heimersheim, and Adri \` a Garriga - Alonso. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Abstract-Conference.html Towards automated circuit discovery for mechanistic interpretability . In Advances in Neural Information Processing Systems 36: Annua...

    2023

  7. [7]

    Gureckis, Brenden M

    Guy Davidson, Todd M. Gureckis, Brenden M. Lake, and Adina Williams. 2025. https://doi.org/10.48550/ARXIV.2505.12075 Do different prompting methods yield a common task representation in language models? CoRR, abs/2505.12075

    doi:10.48550/arxiv.2505.12075 2025

  8. [8]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, et al. 2021. https://transformer-circuits.pub/2021/framework/index.html A mathematical framework for transformer circuits . Transformer Circuits Thread

    2021

  9. [9]

    Jaden Fried Fiotto - Kaufman, Alexander Russell Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal, et al. 2025. https://openreview.net/forum?id=MxbEiFRf39 Nnsight and NDIF: democratizing access to open-weight foundation model internals . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

    2025

  10. [10]

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramaneswaran S., Deepali Aneja, et al. 2024. https://openreview.net/forum?id=XkHJo8iXGQ A closer look at the limitations of instruction tuning . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net

    2024

  11. [11]

    Wes Gurnee and Max Tegmark. 2024. https://openreview.net/forum?id=jE8xbmvFin Language models represent space and time . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  12. [12]

    Boris Hanin and David Rolnick. 2019. https://proceedings.neurips.cc/paper/2019/hash/9766527f2b5d3e95d4a733fcfb77bd7e-Abstract.html Deep relu networks have surprisingly few activation patterns . In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancou...

    arXiv 2019

  13. [13]

    Zirui He, Haiyan Zhao, Yiran Qiao, Fan Yang, Ali Payani, et al. 2025. https://doi.org/10.48550/ARXIV.2502.11356 SAIF: A sparse autoencoder framework for interpreting and steering instruction following of language models . CoRR, abs/2502.11356

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.11356 2025

  14. [14]

    Roee Hendel, Mor Geva, and Amir Globerson. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.624 In-context learning creates task vectors . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 . Association for Computational Linguistics

    doi:10.18653/v1/2023.findings-emnlp.624 2023

  15. [15]

    Shih - Cheng Huang, Pin - Zu Li, Yu - Chi Hsu, Kuang - Ming Chen, Yu - Tung Lin, et al. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.590 Chat vector: A simple approach to equip llms with instruction following and model alignment in new languages . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, ACL 2024, Bang...

    doi:10.18653/v1/2024.acl-long.590 2024

  16. [16]

    Dick, Hidenori Tanaka, et al

    Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, et al. 2024. https://openreview.net/forum?id=A0HKeKl4Nl Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  17. [17]

    Yuxuan Li, Declan Campbell, Stephanie C. Y. Chan, and Andrew Kyle Lampinen. 2025. https://doi.org/10.48550/ARXIV.2509.04466 Just-in-time and distributed task representations in language models . CoRR, abs/2509.04466

    doi:10.48550/arxiv.2509.04466 2025

  18. [18]

    Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Yanchao Hao, et al. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/50ea4dbd1cff6bd3daef939eff10c092-Abstract-Conference.html From instance training to instruction learning: Task adapters generation from instructions . In Advances in Neural Information Processing Systems 38: Annual Conference on Ne...

    2024

  19. [19]

    Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, et al. 2024. Sparse crosscoders for cross-layer features and model diffing. https://transformer-circuits.pub/2024/crosscoders/index.html. Anthropic interpretability research note

    2024

  20. [20]

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/6f1d43d5a82a37e89b0665b33bf3a182-Abstract-Conference.html Locating and editing factual associations in GPT . In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2...

    2022

  21. [21]

    Trung Nguyen and Yan Leng. 2025. https://doi.org/10.48550/ARXIV.2502.16385 Toward a flexible framework for linear representation hypothesis using maximum likelihood estimation . CoRR, abs/2502.16385

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.16385 2025

  22. [22]

    Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, et al. 2025. https://doi.org/10.48550/ARXIV.2501.00656 2 olmo 2 furious . CoRR, abs/2501.00656

    Pith/arXiv arXiv doi:10.48550/arxiv.2501.00656 2025

  23. [23]

    OpenAI. 2025. https://platform.openai.com/docs/models/gpt-5-nano gpt-5-nano-2025-08-07

    2025

  24. [24]

    Wainwright, et al

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, et al. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html Training language models to follow instructions with human feedback . In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Informa...

    2022

  25. [25]

    Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. https://openreview.net/forum?id=UGpGkLzwpP The linear representation hypothesis and the geometry of large language models . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net

    2024

  26. [26]

    Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeffrey Wu, et al. 2025. https://openreview.net/forum?id=91H76m9Z94 Open problems in mechanistic interpretability . Trans. Mach. Learn. Res., 2025

    2025

  27. [27]

    Yang, Bin Wu, Laurence Aitchison, Emine Yilmaz, et al

    Zhengxiang Shi, Adam X. Yang, Bin Wu, Laurence Aitchison, Emine Yilmaz, et al. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/7ffb43adf37b3eeaba559098bc084cc6-Abstract-Conference.html Instruction tuning with loss over instructions . In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing System...

    2024

  28. [28]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, et al. 2023. https://openreview.net/forum?id=uyTL5Bvosj Beyond the imitation game: Quantifying and extrapolating the capabilities of language models . Trans. Mach. Learn. Res., 2023

    2023

  29. [29]

    Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. 2025. https://openreview.net/forum?id=wozhdnRCtw Improving instruction-following in language models through activation steering . In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

    2025

  30. [30]

    Denis Sutter, Julian Minder, Thomas Hofmann, and Tiago Pimentel. 2025. https://doi.org/10.48550/ARXIV.2507.08802 The non-linear representation dilemma: Is causal abstraction enough for mechanistic interpretability? CoRR, abs/2507.08802

    doi:10.48550/arxiv.2507.08802 2025

  31. [31]

    Naftali Tishby, Fernando C. N. Pereira, and William Bialek. 2000. http://arxiv.org/abs/physics/0004057 The information bottleneck method . CoRR, physics/0004057

    Pith/arXiv arXiv 2000

  32. [32]

    Li, Arnab Sen Sharma, Aaron Mueller, Byron C

    Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, et al. 2024. https://openreview.net/forum?id=AwyxtyMwaG Function vectors in large language models . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  33. [33]

    Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2023. https://openreview.net/forum?id=NpsVSN6o4ul Interpretability in the wild: a circuit for indirect object identification in GPT-2 small . In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

    2023

  34. [34]

    Zhao, Kelvin Guu, Adams Wei Yu, et al

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, et al. 2022. https://openreview.net/forum?id=gEZrGCozdqR Finetuned language models are zero-shot learners . In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net

    2022

  35. [35]

    Fred Zhang and Neel Nanda. 2024. https://openreview.net/forum?id=Hf17y6u9BC Towards best practices of activation patching in language models: Metrics and methods . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net

    2024

  36. [36]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, et al. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/ac662d74829e4407ce1d126477f4a03a-Abstract-Conference.html LIMA: less is more for alignment . In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023,...

    2023

  37. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  38. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...