Reference Feature Atlases for Mechanistic Auditing of Language Models

Rui Wu; Tong Che

REVIEW 4 major objections 5 minor 30 references

The paper claims that auditing a new language model can be reduced to a linear decoder fit on a pre-trained shared feature atlas, with a separate residual dictionary making 'outside the reference panel' an explicit architectural signal; inj

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:30 UTC pith:XFBTLFWC

load-bearing objection The atlas-channel idea is a real contribution, but the residual-channel headline results are in-sample: the residual SAE is trained on the same transcripts used to rank and steer, so the paper needs major revisions before the central claims can be trusted. the 4 major comments →

arxiv 2607.22570 v1 pith:XFBTLFWC submitted 2026-06-01 cs.AI

Reference Feature Atlases for Mechanistic Auditing of Language Models

Rui Wu , Tong Che This is my paper

classification cs.AI

keywords reference feature atlasmechanistic auditingsparse autoencoderscrosscodersresidual streamhidden objectivesmodel diffingactivation steering

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's central claim is that mechanistic auditing of a new language model does not need to start from scratch. The authors propose a reference feature atlas: a sparse dictionary of features trained jointly on a panel of already-studied models, to which a new target model attaches by fitting only a linear decoder. The target is read through two complementary channels: the atlas channel expresses it on already-interpreted panel-shared coordinates, while the residual channel trains a separate sparse dictionary on exactly what the atlas fails to reconstruct, making 'outside the reference panel' an explicit audit signal rather than a post-hoc threshold choice. The claimed payoff is that hidden objectives injected via LoRA into held-out models are recovered as the top-ranked residual latent in all six paired runs, and runtime suppression drives their expression from 1.000 to 0.000 while controls stay untouched. A sympathetic reader should care because this promises cheaper, reusable, and directly comparable auditing across model releases.

Core claim

The paper's central discovery is that interpretable feature dictionaries can be amortized across models: a sparse feature atlas is trained once over a reference panel, and any new target is onboarded by fitting only a linear decoder against the frozen atlas code. The target's activations are then decomposed into two channels: h* ≈ D* zA + r*, where D* zA is the atlas-channel reconstruction on panel-shared coordinates and r* is the residual. A second sparse dictionary is trained on r*, so its features are panel-uncovered by construction—they live in a dedicated residual dictionary rather than being selected by post-hoc thresholding inside a shared dictionary. The authors report that this resi

What carries the argument

The central object is the two-channel decomposition of a target's activations: h* ≈ D* zA + D*_B zB, where zA is a shared 65,536-coordinate BatchTopK sparse code (a dictionary that keeps the largest k activations per token) trained over the reference panel, D* is the linear target decoder fit during attachment, and the residual r* = h* − D* zA is fed into a second sparse autoencoder whose features are by construction not reconstructed by the panel dictionary. The atlas code provides a stable coordinate system in which every attached model's decoder support can be compared one-vs-N; the residual dictionary converts the scalar attachment gap (FVU/KL) into a named feature inventory. The load-be

Load-bearing premise

The residual channel treats everything the atlas cannot reconstruct as 'outside the reference panel', but the atlas's reconstruction is incomplete (FVU 0.31–0.73), so shared panel structure that is merely hard to reconstruct also lands in the residual; a top-ranked residual feature need not be a target-only mechanism.

What would settle it

Attach one of the panel models as if it were a held-out target and compute its residual dictionary on the same neutral corpus. If any residual latent shows clear panel-shared semantics—such as a refusal feature, an instruction-following feature, or the same political-framing cluster found for Qwen—then residual membership is not equivalent to 'panel-uncovered target structure', and the interpretation of every residual-channel finding weakens.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Auditing a new model costs a ridge-regularized decoder fit plus a small residual SAE (about 52,622 tokens and a few hundred transcripts in the paper's setup), instead of a full per-target SAE or pairwise crosscoder retraining (about 400M tokens in the comparison).
Hidden objectives that are target-only can be recovered from residuals alone: the injected mechanism appears as the top-ranked latent in all six paired runs across two held-out model families, with the planted feature cleanly separated from panel-shared directions.
Runtime suppression of the top residual directions drives planted-objective expression monotonically from 1.000 to 0.000 while matched controls stay at 0.000; a prefill-only intervention is ineffective, locating the mechanism in generation rather than prompt processing.
Attachment quality itself—for example, Qwen's held-out KL gap of roughly 6–7× the panel mean at more than 15σ across seeds—acts as a panel-relative cross-lineage signature that single-model SAEs cannot produce and pairwise crosscoders only express implicitly.
The same frozen atlas gives an N-way coordinate system for model diffing: every attached target's decoder support lives on the same K coordinates as all panel models, generalizing pairwise crosscoder comparisons to one-vs-N comparison.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the atlas's own reconstruction is far from perfect (FVU 0.31–0.73), the residual channel will contain not only target-specific mechanisms but also panel-shared structure that a linear sparse decoder finds hard to reconstruct. A top-ranked residual latent is therefore not guaranteed to be target-only; a careful audit should cross-check by attaching a panel member in the same way before decl
The method's 'outside-the-panel' verdicts are relative to the panel's composition. A panel with closer training lineages would absorb the Qwen political-framing cluster into shared coordinates, so atlas audits produce panel-relative findings rather than intrinsic properties of a model—a scope the paper itself acknowledges.
A natural infrastructure follow-up: if atlas training can be amortized across model families, one could maintain a continuously updated public reference atlas, making each new model release auditable at the cost of a linear fit plus per-audit residual explanation. The residual explanations, however, would still require per-audit validation.
A testable extension: train residual SAEs for panel members themselves, not just held-out targets. If a panel member's residual dictionary contains semantically meaningful shared features, then the residual channel is not a pure 'target-only' signal and the architectural-separation claim would need a correction term.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The atlas-channel idea is a real contribution, but the residual-channel headline results are in-sample: the residual SAE is trained on the same transcripts used to rank and steer, so the paper needs major revisions before the central claims can be trusted.

read the letter

You should know two things about this paper. The atlas-channel architecture—a shared sparse dictionary trained once on a reference panel, new targets attached by fitting a linear decoder—is a genuinely useful idea that addresses a real bottleneck. And the residual-channel headline results (rank-1.0 recovery, 1.000-to-0.000 suppression) are not supported by the evidence as presented, because the residual SAE is trained on the same transcripts used to evaluate it.

What's new here is the structural separation of atlas and residual channels, which improves on pairwise crosscoders that require retraining per pair and use post-hoc decoder-norm thresholding. The LoRA-injection protocol for controlled audit ground truth is also a nice practical contribution. The atlas-channel evidence is solid: reconstruction quality holds up (downstream KL 0.17–0.44, FVU 0.31–0.73), the N-way diffing in shared coordinates is a clean idea, and refusal steering via atlas coordinates works on held-out Mistral and Qwen targets.

The main problem is circularity. App. E shows the residual SAE for each (LoRA, target) pair is trained on the objective-audit and control residual positions, and the recovery rank is computed by ranking latents on the same objective-vs-control contrast from those same transcripts. That makes rank 1.0 a memorization result, not a discovery. The Qwen political-framing finding has the same issue: the residual SAE in Sec. 7.5/App. F is trained on the 96+96 politics/control transcripts used to rank latents. Second, the 'panel-uncovered by construction' claim is an overstatement. With reconstruction FVU of 0.31–0.73, the residual contains a large mass of panel-shared structure that a linear sparse decoder cannot reconstruct (massive activations, rare-token directions, tokenizer artifacts). Membership in r* = h* - D*zA does not mean the feature is semantically outside the panel. Third, the baseline comparison is run only on Mistral, not Qwen; the paper acknowledges this, but it weakens the cross-target headline.

Who this is for: mechanistic interpretability and audit-tooling researchers. The atlas-channel notion is worth engaging with; the residual-channel claims need to be redone or heavily caveated. The paper deserves a serious referee: the idea is timely and the evaluation is extensive, but the central evidence for the residual channel is in-sample. My recommendation: conditional acceptance with major revisions—retrain residual SAEs on neutral text only, evaluate recovery and interventions on held-out audits, extend the baseline to Qwen or soften the cross-target claim, and release code/data.

Referee Report

4 major / 5 minor

Summary. The paper proposes a 'Reference Feature Atlas' for mechanistic auditing: a sparse feature dictionary (BatchTopK) trained jointly over a reference panel of models, with per-model linear decoders, so that a new target model can be attached by fitting only a linear decoder. The atlas yields two audit channels: an atlas channel that reads the target in pre-interpreted panel coordinates, and a residual channel that trains a separate SAE on the target residual r* = h* - D* z_A, claimed to capture 'panel-uncovered' structure. Experiments use leave-one-out atlases over five 7--9B instruction-tuned models, with Mistral and Qwen as held-out targets. On three injected LoRA hidden objectives, the residual dictionary is reported to recover the planted objective as rank-1 latent in all six paired runs and to suppress the objective from 1.000 to 0.000 via steering. A separate Qwen political-framing case study reports a residual cluster whose suppression shifts audited framing metrics while out-of-domain controls remain unchanged.

Significance. If the central claims held, the atlas would be a genuinely useful amortized auditing tool: it offers a reusable coordinate system, N-way comparison, and a low-cost attachment procedure. The paper is also commendably explicit about several limitations: the head-to-head baseline comparison is restricted to Mistral, the Qwen political-framing result is uncontrolled and panel-relative, and the authors acknowledge that code is only planned for release. The controlled LoRA setup is a good idea for ground-truth auditing, and the reconstruction/attachment tables are useful empirical documentation. However, the two load-bearing pillars of the residual-channel argument---'panel-uncovered by construction' and the rank-1.0/suppression results---are currently undermined by (i) training the hidden-objective residual SAEs on the same transcripts later used for ranking, and (ii) conflating 'not reconstructed by a linear sparse decoder' with 'semantically outside the reference panel.' These issues affect the paper's main claims, not just presentation.

major comments (4)

[Sec. 7.4 and App. E] The rank-1.0 hidden-objective recovery is in-sample. App. E states that the residual SAE is 'trained on the corresponding objective-audit and control residual-token positions,' and Sec. 7.4 first collects residuals on the full transcripts and then ranks latents by the objective-vs-control contrast computed on those same residuals. A rank-1.0 result under this protocol can reflect memorization of the training contrast rather than discovery of a panel-relative mechanism. To support the claimed 'recovery,' the residual SAE should be trained on neutral text (as is done for the Sec. 7.5 dictionaries) or on a held-out split, with the objective-vs-control ranking evaluated on unseen audit/control transcripts. Please report such out-of-sample ranks.
[Sec. 5.2, Eq. (6)-(7); Sec. 7.2, Table 2] The phrase 'panel-uncovered by construction' is an interpretive claim, not a mathematical guarantee. Membership in D_B* means only that the atlas decoder D* z_A did not reconstruct that part of h*; Table 2 reports reconstruction FVU of 0.31--0.73, and Sec. 7.2 notes that massive-activation dimensions are hardest to reconstruct. The residual therefore contains a large amount of panel-shared structure that is merely difficult for a linear sparse decoder to encode, not structure that is semantically outside the panel. The top-ranked residual latent in the controlled audits need not be target-only. A concrete test would be to train residual SAEs on each panel model's residual and check whether the injected-objective latent also appears there, or to verify that the residual cluster is absent when the LoRA is not injected.
[Sec. 7.5 and Sec. 7.6, App. F] The Qwen political-framing finding has the same circularity: App. F says residual-SAE latents are 'ranked by their politics-vs-control contrast on the same 96+96 training set used in Sec. 7.5,' and the intervention direction is formed from those same ranked latents before scoring M1/M2. Since the residual SAE itself is trained on those 192 transcripts, the discovery step is in-sample. The paper already labels the Qwen result as uncontrolled and panel-relative, but that does not address the in-sample training. Please either train the residual SAE on independent neutral text or evaluate on a held-out set of politics/control prompts and report whether the same cluster is recovered.
[App. J, Table 5; Sec. 7.4] The head-to-head comparison is not apples-to-apples for the stated purpose of 'recovery.' The per-target SAE and pairwise crosscoder are retrained on general activations, while the residual SAE in the atlas row is trained on the objective-audit and control transcripts that define the ranking metric. The 52,622-token ridge fit plus 360-transcript residual SAE is not comparable to the 400M-token baseline training if the residual SAE is allowed to see the test contrast. Moreover, the baseline comparison is run only on Mistral; the abstract and main text are careful about this, but the figure 'both baselines fail' should be scoped to the in-sample protocol. At minimum, re-run the baselines under the same in-sample training condition (or re-run the atlas under the baselines' out-of-sample condition) and report both.

minor comments (5)

[Sec. 5.2, Sec. 7.4, App. E] Terminology is inconsistent: the paper says 'panel-uncovered by construction' but also 'panel-unreconstructed' and 'not reconstructed by the atlas.' I suggest using 'atlas-unreconstructed' or 'panel-unreconstructed' and explicitly stating that this is a property of the atlas decoder, not a semantic claim.
[App. E vs. Table 5] Table 5 lists '360 transcripts' as audit-time training for the residual SAE, while App. E reports 17,056 residual-token positions (9,991 audit + 7,065 control) for the Mistral Lunaria instance. Please reconcile the units or state that the numbers are token positions.
[Sec. 7.6 and App. F] The M1/M2 metrics are scored on 20 curated audit prompts with 50 samples each, but the discovery contrast uses 96+96 generated transcripts. It would help to clarify whether the 20 prompts are a subset of the 96 or a separate held-out set, and whether any of the 96 were used to choose the direction.
[Sec. 7.5] The Mistral and Qwen residual dictionaries have very different liveness (2,939 vs 6,723 live latents) and activation scales. A brief discussion of why this is expected, and whether liveness thresholds were matched, would improve interpretability of the residual-channel comparison.
[App. F] The paper says 'we plan to release the code and reproduction scripts.' Since reproducibility is a stated goal, please provide an anonymity-compliant artifact or a clear commitment to release upon publication.

Circularity Check

1 steps flagged

The controlled hidden-objective 'recovery rank 1.0' and suppression-to-zero results are in-sample: the residual SAE is trained on the same objective-audit/control residuals later used for ranking and steering, so the headline audit signal is a fit on the evaluation data rather than an independent discovery.

specific steps

fitted input called prediction [Sec. 7.4; App. E]
"Sec. 7.4: 'We collect residuals r⋆ = h⋆ − D⋆zA on the full transcripts ... and train a residual SAE for each objective (App. E). Ranking latents by the objective-vs-control contrast surfaces features whose top contexts directly name the injected objective ... The residual dictionary has mean recovery rank 1.0 for the injected-objective feature in all six paired runs.' App. E: 'the residual SAE is ... trained on the corresponding objective-audit and control residual-token positions.'"

The residual SAE is trained on the same objective-audit and control residual positions that are later used to compute the objective-vs-control contrast and recovery rank; the intervention direction is also formed from the top-six contrastive latents on that same audit set. Recovery rank 1.0 is therefore the in-sample maximum of the ranking criterion, and the suppression sweep measures the effect of a direction selected on the very corpus used to score it. The headline 'recovers the planted mechanism' and 'perfectly controllable' are fit statistics on the evaluation data, not held-out discoveries; a per-target SAE trained and ranked on the same data would face the same in-sample advantage, so the comparison does not remove the circularity.

full rationale

The central controlled finding—that the residual channel recovers the injected LoRA objective as the top-ranked latent and suppresses it from 1.000 to 0.000—is scored in-sample. App. E states that each residual SAE is trained on the objective-audit and control residual-token positions, and Sec. 7.4 ranks latents by the objective-vs-control contrast on the same transcripts and forms the intervention direction from the top-six contrastive latents on the same audit set. Thus the rank-1.0 result and the causal sweep are not independent tests of discovery or controllability; they are selections and evaluations on one corpus. This is a genuine fitted-input-called-prediction issue for the paper's most emphasized empirical claim. The Qwen political-framing residual SAE is trained on neutral FineWeb text (App. F) and the M1/M2 prompts are withheld, so that finding is less exposed to this particular circularity, although it is explicitly panel-relative and uncontrolled. Separately, Sec. 5.2's 'panel-uncovered by construction' conflates 'not reconstructed by a linear sparse decoder' with 'semantically outside the reference panel'; because Table 2 reports reconstruction FVU of 0.31–0.73, the residual contains substantial panel-shared structure that is merely hard to reconstruct. I treat this as an interpretive overclaim rather than a separate circular step, but it amplifies the in-sample concern. The atlas-channel evidence (support profiles, N-way diffing, refusal steering) is trained and evaluated on disjoint data and is therefore not circular; my score of 6 reflects the central residual-channel result, not the whole paper.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The method introduces no new physical entities. The main free parameters are architecture hyperparameters. The core axioms are the linear representation hypothesis and two assumptions that are partially contradicted by the paper's own measurements (perfect reconstruction for the residual claim, and tokenizer alignment fidelity).

free parameters (5)

Atlas dictionary size K = 65,536
Chosen by hand; defines capacity of the shared coordinate system and influences how much content remains in the residual.
BatchTopK active budget = 100–200 per token
Chosen sparsity; controls feature granularity and the amount of unreconstructed structure.
Residual SAE dictionary size K' and active k = 4096/k=16 for objective audits; 8192/k=32 for politics audits
Chosen per experiment; determines which latent features appear in the residual and thus the ranking outcome.
Target decoder ridge strength λ = 1000
Selected by validation FVU; affects reconstruction quality and hence residual content.
Intervention strength α = -16 (hidden objectives); range -8..-15 for sweeps
Swept intervention knob; the reported 'perfect control' depends on the chosen strength and is not a fixed parameter.

axioms (4)

domain assumption Linear representation hypothesis: target activations can be linearly re-expressed in atlas coordinates
Used in Sec 3 and 5; the target decoder D* is a linear fit. Cites Park et al. 2024 and Huh et al. 2024.
ad hoc to paper Residual channel is 'panel-uncovered by construction'
Sec 5.2 claims residual features are panel-uncovered because the atlas absorbs shared structure. This assumes near-perfect reconstruction, which is contradicted by FVU 0.31–0.73 in Table 2.
domain assumption Tokenizer alignment via character-offset intersection preserves meaningful token correspondences
Sec 7.1; needed to align activations across models with different tokenizers. Unstated failure modes could distort cross-model comparisons.
domain assumption BatchTopK hard sparsity avoids crosscoder artifacts
Sec 4 and App C; relies on Minder et al. 2025. Justifies the choice of BatchTopK over norm-weighted L1, but the cited analysis is about crosscoders, not this atlas setup.

pith-pipeline@v1.3.0-alltime-deepseek · 19451 in / 10360 out tokens · 96664 ms · 2026-08-02T12:30:17.816500+00:00 · methodology

0 comments

read the original abstract

Auditing a new language model usually means relearning and reinterpreting its internal features from scratch. We propose a reference feature atlas: a sparse feature library trained once on a reference panel and reused for new targets, which attach by fitting only a linear decoder. This yields two complementary views. The atlas channel reads the target on already interpreted panel features, providing a stable coordinate system across models. The residual channel learns features only from what the atlas fails to reconstruct, making "outside the reference panel" an explicit audit signal. We train leave-one-out atlases over five 7-9B instruction-tuned models and audit held-out Mistral and Qwen targets. On three controlled LoRA hidden objectives injected into both targets, the residual channel makes the planted mechanism perfectly controllable at runtime while matched controls stay unaffected and recovers the planted objective as the top-ranked latent across both targets; on Mistral, where the per-target SAE and pairwise crosscoder baselines are retrained for a head-to-head benchmark, both baselines fail to do so. On Qwen-2.5, the same channel additionally reveals a panel-relative political-framing cluster; steering it shifts the audited framing metrics while out-of-domain controls remain unchanged.

Figures

Figures reproduced from arXiv: 2607.22570 by Rui Wu, Tong Che.

**Figure 1.** Figure 1: Reference feature atlas training and target attachment. Stage 1 trains a shared sparse coordinate system over a reference panel using a multi-input encoder and per-model decoders. Stage 2 freezes the atlas, computes zA from the reference panel, and attaches a target by fitting only the target decoder D⋆ against h⋆. The attachment outputs are the target decoder, its support profile over atlas coordinates, a… view at source ↗

**Figure 2.** Figure 2: Hidden-objective causal intervention sweeps for the three controlled LoRA objectives across both held-out targets. Each panel reports the planted-objective rate on the corresponding audit corpus (left to right: Lunaria brand promotion, Ptolemaic-epicycle advocacy, Flying Spaghetti Monster advocacy) as a function of intervention strength α ∈ {none, −8, . . . , −15}, overlaying the LoRA-injected Mistral targ… view at source ↗

**Figure 3.** Figure 3: Atlas training loss. Self-reconstruction loss Lself (sum of per-model FVU across the four panel members) over 150M training tokens for the two leaveone-out atlases. Both runs follow a fast initial drop and a gradual plateau, converging from ≈ 4.0 to 0.60 (Atlas A, hold Mistral) and 0.75 (Atlas B, hold Qwen) at ≈ 73k optimisation steps (batch 2,048). the support strength is am,j = log ∥dm,j∥2 + ϵ τm , τm =… view at source ↗

**Figure 4.** Figure 4: Atlas-coordinate steering on two held-out targets. Greedy refusal rates on AdvBench-100 harmful prompts: pale = unpatched baseline, hatched = additional refusal from steering (top: Mistral on Atlas A; bottom: Qwen on Atlas B). Benign controls stay at 0.000 across all settings. preted, named coordinate, providing a coordinatetraceable explanation for the behavioural change. H Reference-Atlas Feature Reco… view at source ↗

**Figure 5.** Figure 5: N-way model diffing. Cosine similarity between each attached target’s per-coordinate decodersupport profile and each panel model’s trained decodersupport profile in the shared atlas coordinate system (K = 65,536). Atlas A’s in-panel attachments remain high and the Mistral hold-out drops uniformly; Atlas B shows that the Qwen hold-out is much farther from its reference panel. tached to Atlas A averages 0… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

30 extracted references · 19 linked inside Pith

[1]

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717

Pith/arXiv arXiv 2024
[2]

Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. OpenAI Research

2023
[3]

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zachary Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Heni...

2023
[4]

Trenton Bricken, Rowan Wang, Sam Bowman, Euan Ong, Johannes Treutlein, Jeff Wu, Evan Hubinger, and Samuel Marks. 2025. Building and evaluating alignment auditing agents. Anthropic Alignment Science

2025
[5]

Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. BatchTopK sparse autoencoders. arXiv preprint arXiv:2412.06410

Pith/arXiv arXiv 2024
[6]

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600

Pith/arXiv arXiv 2023
[7]

Leo Gao, Tom Dupr\' e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093

Pith/arXiv arXiv 2024
[8]

Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

Pith/arXiv arXiv 2024
[9]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

Pith/arXiv arXiv 2024
[10]

Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Karina Nguyen, Nathan Bailey, Nicholas Schiefer, Nicholas Joseph, Catherine Olsson, So...

Pith/arXiv arXiv 2024
[11]

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987

Pith/arXiv arXiv 2024
[12]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

Pith/arXiv arXiv 2023
[13]

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J\' a nos Kram\' a r, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2. arXiv preprint arXiv:2408.05147

Pith/arXiv arXiv 2024
[14]

Jack Lindsey, Tom Conerly, and the Anthropic Interpretability Team. 2024. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread

2024
[15]

Jack Lindsey, Tom Conerly, and the Anthropic Interpretability Team. 2025. Insights on crosscoder model diffing. Anthropic Research

2025
[16]

Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, and collaborators

Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, and collaborators. 2025. Auditing language models for hidden objectives. arXiv preprint arXiv:2503.10965

Pith/arXiv arXiv 2025
[17]

Tobias Minder, Daniel Filan, and collaborators. 2025. On sparsity artifacts in crosscoder model diffing. arXiv preprint

2025
[18]

OLMo Team. 2025. 2 OLMo 2 Furious. arXiv preprint arXiv:2501.00656

Pith/arXiv arXiv 2025
[19]

Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681

Pith/arXiv arXiv 2024
[20]

Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, PMLR

2024
[21]

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ : A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022

2022
[22]

Guilherme Penedo, Hynek Kydlicek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track

2024
[23]

Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115

Pith/arXiv arXiv 2024
[24]

Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J\' a nos Kram\' a r, and Neel Nanda. 2024. Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders. arXiv preprint arXiv:2407.14435

Pith/arXiv arXiv 2024
[25]

Paul R\" o ttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest : A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics

2024
[26]

Zico Kolter, and Zhuang Liu

Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762

Pith/arXiv arXiv 2024
[27]

Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689

Pith/arXiv arXiv 2023
[28]

Turner, Callum McDougall, Monte MacDiarmid, C

Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling m...

2024
[29]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023a. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

Pith/arXiv arXiv
[30]

Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023b. Representation engineering: A top-down approa...

Pith/arXiv arXiv

[1] [1]

Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717

Pith/arXiv arXiv 2024

[2] [2]

Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. OpenAI Research

2023

[3] [3]

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zachary Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Heni...

2023

[4] [4]

Trenton Bricken, Rowan Wang, Sam Bowman, Euan Ong, Johannes Treutlein, Jeff Wu, Evan Hubinger, and Samuel Marks. 2025. Building and evaluating alignment auditing agents. Anthropic Alignment Science

2025

[5] [5]

Bart Bussmann, Patrick Leask, and Neel Nanda. 2024. BatchTopK sparse autoencoders. arXiv preprint arXiv:2412.06410

Pith/arXiv arXiv 2024

[6] [6]

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. 2023. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600

Pith/arXiv arXiv 2023

[7] [7]

Leo Gao, Tom Dupr\' e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093

Pith/arXiv arXiv 2024

[8] [8]

Gemma Team. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

Pith/arXiv arXiv 2024

[9] [9]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783

Pith/arXiv arXiv 2024

[10] [10]

Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Karina Nguyen, Nathan Bailey, Nicholas Schiefer, Nicholas Joseph, Catherine Olsson, So...

Pith/arXiv arXiv 2024

[11] [11]

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. 2024. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987

Pith/arXiv arXiv 2024

[12] [12]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

Pith/arXiv arXiv 2023

[13] [13]

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J\' a nos Kram\' a r, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma Scope: Open sparse autoencoders everywhere all at once on Gemma 2. arXiv preprint arXiv:2408.05147

Pith/arXiv arXiv 2024

[14] [14]

Jack Lindsey, Tom Conerly, and the Anthropic Interpretability Team. 2024. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread

2024

[15] [15]

Jack Lindsey, Tom Conerly, and the Anthropic Interpretability Team. 2025. Insights on crosscoder model diffing. Anthropic Research

2025

[16] [16]

Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, and collaborators

Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, and collaborators. 2025. Auditing language models for hidden objectives. arXiv preprint arXiv:2503.10965

Pith/arXiv arXiv 2025

[17] [17]

Tobias Minder, Daniel Filan, and collaborators. 2025. On sparsity artifacts in crosscoder model diffing. arXiv preprint

2025

[18] [18]

OLMo Team. 2025. 2 OLMo 2 Furious. arXiv preprint arXiv:2501.00656

Pith/arXiv arXiv 2025

[19] [19]

Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. Steering Llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681

Pith/arXiv arXiv 2024

[20] [20]

Kiho Park, Yo Joong Choe, and Victor Veitch. 2024. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, PMLR

2024

[21] [21]

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2022. BBQ : A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022

2022

[22] [22]

Guilherme Penedo, Hynek Kydlicek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro von Werra, and Thomas Wolf. 2024. The FineWeb datasets: Decanting the web for the finest text data at scale. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track

2024

[23] [23]

Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115

Pith/arXiv arXiv 2024

[24] [24]

Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, J\' a nos Kram\' a r, and Neel Nanda. 2024. Jumping ahead: Improving reconstruction fidelity with JumpReLU sparse autoencoders. arXiv preprint arXiv:2407.14435

Pith/arXiv arXiv 2024

[25] [25]

Paul R\" o ttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest : A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics

2024

[26] [26]

Zico Kolter, and Zhuang Liu

Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. arXiv preprint arXiv:2402.17762

Pith/arXiv arXiv 2024

[27] [27]

Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689

Pith/arXiv arXiv 2023

[28] [28]

Turner, Callum McDougall, Monte MacDiarmid, C

Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling m...

2024

[29] [29]

Zico Kolter, and Matt Fredrikson

Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023a. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043

Pith/arXiv arXiv

[30] [30]

Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023b. Representation engineering: A top-down approa...

Pith/arXiv arXiv