pith. sign in

arxiv: 2606.11722 · v1 · pith:NIDQTUPCnew · submitted 2026-06-10 · 💻 cs.LG · cs.AI· cs.CL

ICA Lens: Interpreting Language Models Without Training Another Dictionary

Sida Liu , Feijiang Han This is my paper

Pith reviewed 2026-06-27 10:19 UTC · model grok-4.3

classification 💻 cs.LG cs.AIcs.CL
keywords independent component analysissparse autoencoderslanguage model interpretabilityactivation directionsnon-Gaussian structureFastICAmodel representationsprobing benchmarks
0
0 comments X

The pith

Independent component analysis recovers interpretable directions in language model activations without training sparse autoencoders.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that many token-selective directions in LLM activations are already visible as non-Gaussian structure, so classical independent component analysis can locate them directly. This avoids the storage and training overhead of building a new overcomplete dictionary for every layer and model. With an optimized FastICA pipeline plus stability recipes specific to LLM data, the recovered directions remain stable across layers and support audit. On SAEBench the method matches public SAEs on sparse probing and exceeds them on targeted probe perturbation when compute budgets are small to medium. The central suggestion is that ICA supplies a compact, reusable first lens rather than a weak baseline.

Core claim

Independent component analysis, applied through a GPU-parallel FastICA pipeline equipped with LLM-specific stability recipes and fitting diagnostics, recovers compact human-interpretable directions from activations of GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base; these directions prove competitive with public sparse autoencoders on sparse probing and superior on targeted probe perturbation under modest budgets, showing that substantial interpretable structure already exists in activation geometry before any neural dictionary is trained.

What carries the argument

ICALens workflow: an optimized GPU-parallel FastICA implementation augmented with LLM-specific stability recipes and diagnostics that produces auditable, layer-wise non-Gaussian directions.

If this is right

  • Layer-wise analysis of multiple models becomes feasible without per-layer dictionary training.
  • ICA directions can serve as an initial set of features before any SAE is trained.
  • Targeted interventions on model behavior can be tested at lower cost under small-to-medium budgets.
  • The same stabilized pipeline can be rerun on new checkpoints or architectures without retraining overhead.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach might be combined with SAEs by using ICA directions to initialize or constrain dictionary learning.
  • If non-Gaussianity reliably signals selectivity, the same test could be applied to attention heads or MLP neurons directly.
  • Rapid iteration across many models could change the default workflow from "train SAE first" to "run ICA first, train SAE only where needed."

Load-bearing premise

Non-Gaussian directions recovered by ICA correspond to human-interpretable, token-selective features in the activations.

What would settle it

A new language model in which ICA directions produce no better than random accuracy on sparse probing tasks would falsify the competitiveness claim.

Figures

Figures reproduced from arXiv: 2606.11722 by Feijiang Han, Sida Liu.

Figure 1
Figure 1. Figure 1: Layer-3 convergence diagnostics for GPT￾2 Small. Blue curves use row-normalized activations and orange curves use raw activations. Solid curves report max-LIM, while dashed curves report p95- LIM. Full diagnostics are provided in Appendix A [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Final ICA component counts selected by the adaptive refit procedure. For each layer, we initially attempt to fit the full hidden dimension and reduce the target component count when convergence fails. Dashed horizontal lines indicate the hidden dimension of each model. 4. What Does ICA Recover? Non-Gaussianity and Context Dependence The previous section made ICA stable and efficient enough to fit on LLM ac… view at source ↗
Figure 3
Figure 3. Figure 3: ICA recovers highly non-Gaussian directions in LLM activation space. For each model and layer, we project row-normalized residual-stream activations onto random unit directions, public SAE decoder directions, and fitted ICA score directions, then compute the excess kurtosis of each projection distribution. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, random projections remain close to Gaussian, SA… view at source ↗
Figure 4
Figure 4. Figure 4: Layerwise distribution of effective receptive field (ERF). For each ICA component, we average the sample-level ERF over its evidence examples and bucket components by the resulting mean ERF. Each stacked bar shows the fraction of components in one layer. The final bin includes components not recovered within the 11-token window, reported as 11+ . ERF reveals a local-to-contextual spectrum [PITH_FULL_IMAGE… view at source ↗
Figure 5
Figure 5. Figure 5: Relationship between effective receptive field (ERF) and excess kurtosis. Each cell counts ICA components with a given excess-kurtosis range and mean ERF. Across all three models, components with larger excess kurtosis tend to have smaller ERFs, with Spearman correlations ranging from −0.41 to −0.50. This indicates that high-kurtosis components are typically more local, while broad-context components tend … view at source ↗
Figure 6
Figure 6. Figure 6: Interactive inspection view for GPT-2 Small layer 6. For each token, the explorer displays the strongest ICA components, signed scores, working labels, top examples, ERF, kurtosis, and annotation metadata used during component annotation. Additional screenshots for Gemma 2 2B and Qwen 3.5 2B Base are provided in Appendix F. To support component interpretation, we built an interactive explorer that keeps th… view at source ↗
Figure 7
Figure 7. Figure 7: Contextual decomposition of a polysemous word in GPT-2 Small. We probe four occurrences of bank in one paragraph, including two financial-bank uses (F1, F2) and two river-edge uses (R1, R2). Each cell lists the five largest-absolute-score ICA components at the target token in a given layer. Bar lengths show relative score magnitude within the cell, and the legend decodes annotation type and confidence. Acr… view at source ↗
Figure 8
Figure 8. Figure 8: Sentence-level traces for selected GPT-2 Small ICA components. We plot raw absolute ICA scores across two sentences for layer-6 components C67 and C273. C67 tracks a financial-bank context and becomes active across several related tokens in Sentence 2. C273 tracks an arrival- or purpose-related construction and responds strongly to “arrived at the library to study” in Sentence 1, while also responding to r… view at source ↗
Figure 9
Figure 9. Figure 9: Embedding-layer ICA components for familiar analogy word sets in Qwen 3.5 2B Base. Each box corresponds to one token embedding and lists its five largest-absolute-score ICA components. Columns pair familiar masculine-coded tokens with their feminine-coded counterparts. Bar lengths show relative score magnitude within each box, and the legend decodes annotation type and confidence. The figure shows that fam… view at source ↗
Figure 10
Figure 10. Figure 10: SAEBench sparse-probing performance for ICA, public SAE, ITDA, and PCA representations. For each representation, SAEBench ranks features by class contrast on the training set and trains supervised probes using only the top-k ranked feature activations. Each curve reports test accuracy averaged over the eight default sparse-probing datasets and two evaluated layers for that model. SAE features are produced… view at source ↗
Figure 11
Figure 11. Figure 11: SAEBench sparse-probing comparison for Gemma 2 2B layer 12. We compare PCA, ICA, ITDA, prefix-restricted Matryoshka SAE variants, and the full Gemma Scope 16k SAE [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: SAEBench Targeted Probe Perturbation (TPP) for GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base. The x-axis is the number of top-ranked features ablated for each target class. The top row reports the overall TPP score. The bottom row decomposes the same interventions into intended effects on the target-class probe and unintended effects on non-target probes. Scores are averaged over evaluated layers and the … view at source ↗
Figure 13
Figure 13. Figure 13: Nearest-SAE overlap for ICA components. For each ICA component, we report the maximum absolute cosine with any public SAE decoder direction in the same layer. Across all three models, most components lie in a moderate-overlap range, with both weakly matched and strongly matched tails. The distributions show partial agreement between ICA and SAE rather than a one-to-one correspondence. 0 1 2 3 4 5 6 7 8 9 … view at source ↗
Figure 14
Figure 14. Figure 14: Layer-wise nearest-SAE overlap. Each point shows the median maximum absolute cosine between ICA components and public SAE decoder directions in one layer. Shaded bands show interquartile ranges. The median overlap remains moderate across depth, showing that partial ICA-SAE alignment is a consistent property of the decompositions rather than an artifact of a few layers. 26 [PITH_FULL_IMAGE:figures/full_fi… view at source ↗
Figure 15
Figure 15. Figure 15: Token-wise ICA and SAE responses on the same GPT-2 Small sentence. At each token, we select the strongest ICA component by absolute score and the strongest SAE feature by activation, then plot the union of selected directions across the sentence. Each row is normalized by its maximum value to show within-direction response shape. SAE features mostly appear as localized activations, while ICA directions of… view at source ↗
Figure 16
Figure 16. Figure 16: Layer-wise convergence diagnostics for FastICA on GPT-2 Small using 1k fitting rows. 0 50 100 150 200 250 300 10 −8 10 −7 10 −6 10 −5 10 −4 10 −3 10 −2 10 −1 10 0 Limit statistic lim Row-normalized Raw max p95 (a) Layer 0 0 50 100 150 200 250 300 10 −8 10 −7 10 −6 10 −5 10 −4 10 −3 10 −2 10 −1 10 0 (b) Layer 1 0 50 100 150 200 250 300 10 −8 10 −7 10 −6 10 −5 10 −4 10 −3 10 −2 10 −1 10 0 (c) Layer 2 0 50 1… view at source ↗
Figure 17
Figure 17. Figure 17: Layer-wise convergence diagnostics for FastICA on GPT-2 Small using 100k fitting rows. 0 50 100 150 200 250 300 10 −8 10 −7 10 −6 10 −5 10 −4 10 −3 10 −2 10 −1 10 0 Limit statistic lim Row-normalized Raw max p95 (a) Layer 0 0 50 100 150 200 250 300 10 −8 10 −7 10 −6 10 −5 10 −4 10 −3 10 −2 10 −1 10 0 (b) Layer 1 0 50 100 150 200 250 300 10 −8 10 −7 10 −6 10 −5 10 −4 10 −3 10 −2 10 −1 10 0 (c) Layer 2 0 50… view at source ↗
Figure 18
Figure 18. Figure 18: Layer-wise convergence diagnostics for FastICA on GPT-2 Small using 1M fitting rows. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Representative FastICA convergence curves across model families. Each panel shows component-wise FastICA limit values over iterations for one fitted layer. Solid colored lines show medians, shaded regions show the 5th–95th percentile interval, dashed black lines show the maximum, and dotted horizontal lines mark the 10−4 convergence threshold. B. Additional Human Interpretation Results 35 [PITH_FULL_IMAG… view at source ↗
Figure 20
Figure 20. Figure 20: Contextual decomposition of a polysemous word in Gemma 2 2B. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Contextual decomposition of a polysemous word in Qwen 3.5 2B Base. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Embedding-layer ICA components for familiar analogy word sets in GPT-2 Small. king W King, Royal F Letter K W Common Word W Leader, Head ? man W Man, Human W Male ? W Common Word W People, Person father W Parent, Father,... W Male W Common Word W Son, Daughter W Brother, Sister brother W Brother, Sister W Friend W Son, Daughter W Parent, Father,... W Male actor W Act W Researcher, Jou... W Player W Theate… view at source ↗
Figure 23
Figure 23. Figure 23: Embedding-layer ICA components for familiar analogy word sets in Gemma 2 2B. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Token-wise ICA and SAE patterns for the same Gemma 2 2B sentence. For each token in the bank sentence, we select the top-2 ICA components by absolute score and the top-2 SAE features by activation, then plot the union of selected directions across all token positions. May a stopped at the bank before the trip , waiting in line to deposit a check and withdraw enough cash for the weekend . C305 C1778 C504 C… view at source ↗
Figure 25
Figure 25. Figure 25: Token-wise ICA and SAE patterns for the same Qwen 3.5 2B Base sentence. For each token in the bank sentence, we select the top-2 ICA components by absolute score and the top-2 SAE features by activation, then plot the union of selected directions across all token positions. 45 [PITH_FULL_IMAGE:figures/full_fig_p045_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Explorer interface screenshots for Gemma 2 2B layer 12. Two screenshots are ICA Explorer and SAE Explorer. 46 [PITH_FULL_IMAGE:figures/full_fig_p046_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Explorer interface screenshots for Qwen 3.5 2B Base layer 12. 47 [PITH_FULL_IMAGE:figures/full_fig_p047_27.png] view at source ↗
read the original abstract

Finding interpretable directions in language-model representations is critical for understanding and controlling model behavior. Sparse autoencoders (SAEs) have become the standard tool for this purpose, but using them as the default first lens often requires training, storing, and evaluating large overcomplete dictionaries. This bottleneck limits rapid exploration and raises a fundamental question: how much interpretable structure is already visible from activation geometry before training another neural dictionary? Our intuition is simple: many interpretable directions are selective on tokens, and these directions should look less Gaussian than random directions. We therefore revisit independent component analysis (ICA), a classical method for finding non-Gaussian directions, as a compact lens for language-model interpretability. We find that ICA has been underestimated for LLM interpretability, because prior uses often relied on off-the-shelf ICA implementations that are brittle on LLM activations and lacked systematic tools for inspecting and evaluating the recovered directions. To bridge these gaps, we introduce ICALens, the first practical workflow for stable, efficient, and auditable ICA analysis of LLM representations. It combines an optimized GPU-parallel FastICA pipeline with LLM-specific stability recipes and better fitting diagnostics, enabling efficient and reliable layer-wise analysis. Across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, ICALens efficiently recovers compact, human-interpretable directions without per-layer gradient-based dictionary training. On SAEBench, ICA is competitive with public SAEs in sparse probing and outperforms them in targeted probe perturbation under small-to-medium budgets. These results suggest that ICA should not be viewed as a weak baseline, but as an efficient and complementary first lens for exploring language-model representations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript introduces ICALens, a workflow combining an optimized GPU-parallel FastICA pipeline with LLM-specific stability recipes and fitting diagnostics to recover non-Gaussian, token-selective directions from language-model activations. It argues that many interpretable features are already visible from activation geometry and that ICA, when properly stabilized, serves as an efficient alternative to training sparse autoencoders. The central empirical claim is that, across GPT-2 Small, Gemma 2 2B, and Qwen 3.5 2B Base, ICA directions are competitive with public SAEs on SAEBench sparse probing and outperform them on targeted probe perturbation under small-to-medium budgets, positioning ICA as a complementary first lens rather than a weak baseline.

Significance. If the benchmark results hold with full protocol details, the work would demonstrate that classical ICA, augmented with domain-specific stability techniques, can recover compact and auditable directions without per-layer gradient-based dictionary training. This would reduce computational barriers to rapid layer-wise exploration and provide a reproducible, parameter-light baseline that complements SAE-based methods. The explicit comparison to independently published public SAEs and the focus on stability recipes are strengths that support falsifiable evaluation.

major comments (1)
  1. [Abstract and experimental evaluation section] Abstract and experimental evaluation section: the claim that 'ICA is competitive with public SAEs in sparse probing and outperforms them in targeted probe perturbation under small-to-medium budgets' is unsupported by any quantitative results, error bars, dataset details, exact experimental protocol, or description of how the recovered ICA directions were evaluated on SAEBench. This absence is load-bearing for the central empirical claim and prevents verification of whether the data support the stated competitiveness.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for highlighting the need for explicit quantitative support of our central empirical claims. We agree that the experimental evaluation section must provide sufficient detail for verification and will revise accordingly.

read point-by-point responses

  1. Referee: [Abstract and experimental evaluation section] Abstract and experimental evaluation section: the claim that 'ICA is competitive with public SAEs in sparse probing and outperforms them in targeted probe perturbation under small-to-medium budgets' is unsupported by any quantitative results, error bars, dataset details, exact experimental protocol, or description of how the recovered ICA directions were evaluated on SAEBench. This absence is load-bearing for the central empirical claim and prevents verification of whether the data support the stated competitiveness.

    Authors: We acknowledge this gap in the current draft. The revised manuscript will expand the experimental evaluation section to include: (1) full SAEBench metric tables with numerical values for sparse probing (e.g., F1 scores) and targeted probe perturbation (e.g., accuracy drops) across the three models; (2) error bars computed over multiple random seeds and activation samples; (3) precise dataset splits and token counts used for fitting and evaluation; (4) the exact protocol for mapping ICA directions to SAEBench probes, including thresholding and normalization steps; and (5) direct side-by-side comparisons with the cited public SAEs under matched compute budgets. These additions will make the competitiveness claim directly verifiable. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper applies classical ICA (a pre-existing statistical method) to LLM activations and evaluates recovered directions via external benchmarks (SAEBench) against independently published public SAEs. No equations or claims define core quantities such as non-Gaussian directions or stability metrics from the paper's own fitted outputs; the central empirical claims rest on direct comparison rather than self-referential prediction or self-citation chains. The derivation chain is self-contained against external data and classical methods.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based on abstract only; the approach rests on the standard statistical assumptions of ICA and on unspecified LLM-specific stability recipes whose concrete form is not given.

axioms (1)
  • standard math The latent sources are statistically independent and at most one is Gaussian.
    Core modeling assumption of independent component analysis invoked to recover directions from LLM activations.

pith-pipeline@v0.9.1-grok · 5836 in / 1311 out tokens · 42728 ms · 2026-06-27T10:19:34.034443+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 5 canonical work pages

  1. [1]

    Anthony J

    doi: 10.1109/TSP.2018.2844203. Anthony J. Bell and Terrence J. Sejnowski. An information-maximization approach to blind separation and blind deconvolution.Neural Computation, 7(6):1129–1159,

    work page doi:10.1109/tsp.2018.2844203 2018

  2. [2]

    independent components

    doi: 10.1162/neco.1995.7.6.1129. Anthony J Bell and Terrence J Sejnowski. The “independent components” of natural scenes are edge filters. Vision research, 37(23):3327–3338,

    work page doi:10.1162/neco.1995.7.6.1129 1995

  3. [3]

    Bart Bussmann, Patrick Leask, and Neel Nanda

    https://transformer-circuits.pub/2023/monosemantic-features/index.html. Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders.arXiv preprint arXiv:2412.06410,

    arXiv 2023

  4. [4]

    Learning multi-level features with matryoshka sparse autoencoders.arXiv preprint arXiv:2503.17547,

    Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders.arXiv preprint arXiv:2503.17547,

    arXiv

  5. [5]

    David Chanin

    doi: 10.1049/ip-f-2.1993.0054. David Chanin. Are sparse autoencoder benchmarks reliable?arXiv preprint arXiv:2605.18229,

    work page doi:10.1049/ip-f-2.1993.0054 1993

  6. [6]

    Qwen-scope: Turning sparse features into development tools for large language models

    Boyi Deng, Xu Wang, Yaoning Wang, Yu Wan, Yubo Ma, Baosong Yang, Haoran Wei, Jialong Tang, Huan Lin, Ruize Gao, et al. Qwen-scope: Turning sparse features into development tools for large language models. arXiv preprint arXiv:2605.11887,

    Pith/arXiv arXiv

  7. [7]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. InInternational Conference on Learning Representations, volume 2025, pages 26721–26754,

    2025

  8. [8]

    Finding neurons in a haystack: Case studies with sparse probing.arXiv preprint arXiv:2305.01610,

    30 ICA Lens: Interpreting Language Models Without Training Another Dictionary (Ongoing) Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing.arXiv preprint arXiv:2305.01610,

    arXiv

  9. [9]

    Zerotuning: Unlocking the initial token’s power to enhance large language models without training

    Feijiang Han, Xiaodong Yu, Jianheng Tang, Qingyun Zeng, Licheng Guo, and Lyle Ungar. Zerotuning: Unlocking the initial token’s power to enhance large language models without training. InICML 2025 Workshop on Methods and Opportunities at Small Scale,

    2025

  10. [10]

    Robert Huben, Hoagy Cunningham, Logan Smith, Aidan Ewart, and Lee Sharkey

    URLhttps://openreview.net/forum? id=THSbsRWy9v. Robert Huben, Hoagy Cunningham, Logan Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. InInternational Conference on Learning Representations, volume 2024, pages 7827–7845,

    2024

  11. [11]

    Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, et al

    URL https://arxiv.org/abs/2502.16681. Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, et al. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability.arXiv preprint arXiv:2503.09532,

    arXiv

  12. [12]

    Inference-time decomposition of activations (itda): A scalable approach to interpreting large language models.arXiv preprint arXiv:2505.17769,

    Patrick Leask, Neel Nanda, and Noura Al Moubayed. Inference-time decomposition of activations (itda): A scalable approach to interpreting large language models.arXiv preprint arXiv:2505.17769,

    arXiv

  13. [13]

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda

    doi: 10.1162/089976699300016719. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma

    work page doi:10.1162/089976699300016719

  14. [14]

    Towards principled evaluations of sparse autoencoders for interpretability and control

    Aleksandar Makelov, Georg Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. InInternational Conference on Learning Representations, volume 2025, pages 33588–33636,

    2025

  15. [15]

    Anish Mudide, Josh Engels, Eric Michaud, Max Tegmark, and Christian Schroeder de Witt

    doi: 10.32614/RJ-2018-046. Anish Mudide, Josh Engels, Eric Michaud, Max Tegmark, and Christian Schroeder de Witt. Efficient dictionary learning with switch sparse autoencoders. InInternational Conference on Learning Representations, volume 2025, pages 101830–101844,

    work page doi:10.32614/rj-2018-046 2018

  16. [16]

    Exploring interpretability of independent components of word embeddings with automated word intruder test

    31 ICA Lens: Interpreting Language Models Without Training Another Dictionary (Ongoing) Tomáš Musil and David Mareček. Exploring interpretability of independent components of word embeddings with automated word intruder test. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors,Proceedings of the...

    2024

  17. [17]

    URLhttps://aclanthology.org/2024.lrec-main.605/

    ELRA and ICCL. URLhttps://aclanthology.org/2024.lrec-main.605/. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 5(3):e00024–001,

    2024

  18. [18]

    The linear representation hypothesis and the geometry of large language models.arXiv preprint arXiv:2311.03658,

    Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models.arXiv preprint arXiv:2311.03658,

    Pith/arXiv arXiv

  19. [19]

    Automatically interpreting millions of features in large language models.arXiv preprint arXiv:2410.13928,

    Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models.arXiv preprint arXiv:2410.13928,

    arXiv

  20. [20]

    Improving dictionary learning with gated sparse autoencoders.arXiv preprint arXiv:2404.16014, 2024a

    Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders.arXiv preprint arXiv:2404.16014, 2024a. Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: ...

    Pith/arXiv arXiv

  21. [21]

    Massive activations in large language models.arXiv preprint arXiv:2402.17762,

    Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models.arXiv preprint arXiv:2402.17762,

    Pith/arXiv arXiv

  22. [22]

    pub/2024/scaling-monosemanticity/index.html

    URL https://transformer-circuits. pub/2024/scaling-monosemanticity/index.html. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, andChristopherPotts. AxBench: SteeringLLMs? evensimplebaselinesoutperformsparseautoen- coders. InProceedings of the 42nd International Conference on Machine Learning, volu...

    2024

  23. [23]

    Discovering universal geometry in embed- dings with ica

    32 ICA Lens: Interpreting Language Models Without Training Another Dictionary (Ongoing) Hiroaki Yamagiwa, Momose Oyama, and Hidetoshi Shimodaira. Discovering universal geometry in embed- dings with ica. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4647–4675,

    2023

  24. [24]

    Axis tour: Word tour determines the order of axes in ica-transformed embeddings

    Hiroaki Yamagiwa, Yusuke Takase, and Hidetoshi Shimodaira. Axis tour: Word tour determines the order of axes in ica-transformed embeddings. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 477–506,

    2024

  25. [25]

    Spherical steering: Geometry-aware activation rotation for language models.arXiv preprint arXiv:2602.08169,

    Zejia You, Chunyuan Deng, and Hanjie Chen. Spherical steering: Geometry-aware activation rotation for language models.arXiv preprint arXiv:2602.08169,

    Pith/arXiv arXiv

  26. [26]

    a" W Any F Letter B ? unlabeled P Follows

    33 ICA Lens: Interpreting Language Models Without Training Another Dictionary (Ongoing) A. Additional FastICA Fitting Diagnostics 0 50 100 150 200 250 300 10−8 10−7 10−6 10−5 10−4 10−3 10−2 10−1 100 Limit statistic lim Row-normalized Raw max p95 (a)Layer 0 0 50 100 150 200 250 300 10−8 10−7 10−6 10−5 10−4 10−3 10−2 10−1 100 (b)Layer 1 0 50 100 150 200 250...

    2048