Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

This paper shows that sparse autoencoder features can score high on disentanglement metrics yet still steer multiple unrelated concepts when manipulated, arguing that correlational and spatial-separation evidence cannot establish causal sel

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:50 UTC pith:ASOSH5O4

load-bearing objection Useful empirical point on steering vs. correlational metrics, but the 'disjointness' result is not established by the reported additivity test. the 3 major comments →

arxiv 2512.15134 v2 pith:ASOSH5O4 submitted 2025-12-17 cs.LG cs.AIcs.CL

From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?

Aaron Mueller , Andrew Lee , Shruti Joshi , Ekdeep Singh Lubana , Dhanya Sridhar , Patrik Reizinger This is my paper
classification cs.LG cs.AIcs.CL
keywords interpretabilitysparse autoencodersdisentanglementsteeringcausal representation learningfeature independencemulti-concept evaluationlanguage models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the sparse features that interpretability methods extract from language models can actually isolate one concept at a time, rather than merely look separated on correlational metrics. Using a synthetic dataset with four known concepts—sentiment, domain, voice, and tense—with controllable correlations, the authors train several sparse autoencoder variants and probes, then do the decisive test: steer a feature and watch what happens to every concept. They find that features score well on standard disentanglement metrics, and even affect disjoint subspaces, yet steering one feature still changes the probabilities of unrelated concepts. The paper concludes that correlational and spatial-separation evidence cannot establish causal selectivity, and that multi-concept, interventional evaluations are necessary.

Core claim

Even when trained on uniform (uncorrelated) concept distributions, SAE features generally affect many concepts when steered, indicating they are neither selective nor independent; nonetheless, features act on disjoint subspaces. High scores on correlational metrics (MCC, DCI-ES) and near-perfect additivity of joint steering effects do not predict whether a feature can be steered without changing other concepts. The paper introduces steering-specific metrics—steering independence and steering selectivity—and shows that the gap between disjointness and independence is the rule rather than the exception.

What carries the argument

The causal independence test is feature steering: setting a feature to α times its maximum activation, decoding, and adding the reconstruction residual back to the hidden state. The paper measures cross-concept log-odds changes, and formalizes (1) steering independence and selectivity via ratios of target to off-target probability changes, and (2) disjointness as exact additivity of effect when two features are steered together. The load-bearing distinction is that disjoint subspaces are not independent mechanisms.

Load-bearing premise

The paper assumes that the steering operation it uses is a faithful causal intervention on the underlying concept, so that the observed cross-concept probability changes reflect real entanglement rather than an artifact of the editing mechanism.

What would settle it

Repeating the steering experiments with a different intervention style (for instance, direct activation replacement instead of residual addition) and finding that cross-concept effects disappear or change dramatically would show the results are mechanism artifacts. Conversely, a dataset where concepts are perfectly independent by construction and all SAE features are still cross-effective would confirm the non-selectivity claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Interpretability evaluations that report only correlational metrics such as MCC or DCI-ES can overstate how selectively a concept is controlled.
  • Demonstrating that two features live in non-overlapping subspaces does not demonstrate that they act independently on model outputs.
  • Steering-based multi-concept evaluations should become the default before claiming a feature 'controls' a single concept.
  • Feature splitting (low completeness) is widespread; at least ten features are needed before a sparse probe saturates, so single-feature steering is often insufficient.
  • Supervised probes remain the strongest baselines for concept identification, but even they degrade at high concept correlations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to measure steering selectivity across richer, more realistic text distributions, since the findings here use grammar-generated data; the prevalence of cross-concept effects might grow or shrink with natural language.
  • The disjointness result implies the model's internal computation may be closer to a linear overlay of feature subspaces; if so, additive counterfactuals could be combined with sparse circuit tracing to predict composite behaviors.
  • The gap between disjointness and independence suggests a research target: featurizers optimized for causal (interventional) disentanglement objectives rather than reconstruction or correlation objectives.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a multi-concept evaluation framework for interpretability methods, using a PCFG-generated dataset with four known concepts (voice, tense, sentiment, domain) and controlled correlations between concept values. The authors train several featurizers (ReLU/Top-K/SpADE SAEs, SSAEs, sparse probes) on activations of Pythia-70M and Gemma-2-2B, and evaluate them with correlational metrics (MCC, DCI-ES) and with causal steering interventions. They report three main findings: (1) featurizers achieve high correlational disentanglement but poor steering selectivity, (2) features are often sensitive to one concept yet concepts are distributed across many features, and (3) steering effects are additive (features are 'disjoint') even though steering a feature changes many concepts. They propose two new metrics, IS and SS, to quantify steering independence and selectivity, and argue that correlational metrics are insufficient for causal claims and that disjointness does not imply independence.

Significance. If the findings hold, this is a well-targeted and useful contribution to interpretability evaluation. The paper moves beyond single-concept tests by explicitly varying concept correlations and by contrasting correlational (MCC/DCI-ES) and interventional (steering) evidence. The empirical observation that high DCI-ES scores coexist with widespread cross-concept steering effects is important and could redirect evaluation practice. The PCFG dataset with exact ground-truth concepts and the systematic comparison of multiple SAE architectures on two model families is a concrete asset. The paper also transparently implements known causal-representation-learning ideas in a mechanistic-interpretability setting. However, the 'disjointness' claim—a headline contribution—requires stronger empirical support, because the proposed additivity test is partially confounded by the affine decoder structure.

major comments (3)
  1. [§4.2, Eq. (3), Fig. 5, Table 2] The additivity test in Eq. (3) is not a direct test of subspace disjointness. For each tested SAE the decoder is affine in f, and the intervention in Eq. (1) changes exactly one coordinate of f. Thus the change in the middle-layer activation hℓ from steering two features is exactly additive by construction, regardless of whether the decoder columns overlap. While the output-level log-odds are not necessarily additive (because the downstream transformer is nonlinear), the near-perfect R² values in Table 2 are consistent with the building-block artifact. The abstract's stronger claim that 'features affect disjoint subspaces' is not established without a direct measurement of subspace overlap (e.g., cosine similarity between W·,i and W·,j or between the effect vectors). Please either provide such a direct measurement, or reframe the claim as 'additive effects' and temper the Discussion acco
  2. [§4.1, Eq. (2), Table 1] The newly introduced steering independence (IS) and selectivity (SS) metrics are reported only as point means and maxima, without error bars or significance tests across the three training seeds. Without variance estimates it is impossible to know whether the differences between architectures (e.g., SSAE vs. Top-K) or between ρ=0.1 and ρ=0.9 are meaningful. In addition, the definition is degenerate when the numerator is zero (e.g., if the selected feature has no effect on the target concept), which may arise in practice; the paper does not state how such cases are handled. Please provide confidence intervals and a clear policy for degenerate cases, and ideally validate the metrics on synthetic data where the ground-truth causal structure is known.
  3. [App. B.2, §3.1] The SSAE is trained on the final layer, while all other SAEs are trained on the middle layer (layer 3 for Pythia, layer 13 for Gemma). The paper acknowledges this confound in a footnote, but the headline comparison 'SSAEs perform best among unsupervised methods' (Fig. 2, Fig. 3) is partly a comparison of layers, not architectures. A middle-layer SSAE or a final-layer ReLU/Top-K/SpADE should be reported as a robustness check. Without that, the architectural advantage claim is not fully supported.
minor comments (5)
  1. [§1 (typo)] In the Introduction: 'by by manipulating' — duplicate word.
  2. [§3.1, Fig. 3] The caption says 'DCI-ES scores' but the figure has four panels (Disentanglement, Completeness, Informativeness, Explicitness). Consider labeling which DCI-ES component each panel shows, and note that 'size' is omitted.
  3. [§4.2] The text calls R² the measure 'for each SAE' but Table 2 reports no error bars; also, the sentence 'we compute the R2... this measures how' is incomplete — likely a missing clause.
  4. [§4.1, Eq. (2)] The notation is dense: p(z_i | h̃(hat f_j)) is used for the effect of steering with feature of concept j on concept i, but the reader must infer that within-concept pairs are excluded. Consider clarifying the indexing, perhaps with an explicit definition of the feature-selection function.
  5. [References] The reference 'Zuheng, Xu, Moksh Jain, ...' appears malformed (the first author's surname is likely 'Xu, Zuheng'). Please fix formatting.

Circularity Check

0 steps flagged

No significant circularity: central steering/disjointness results are empirical interventions, not fitted predictions.

full rationale

The paper's main claims are empirical: MCC/DCI-ES are measured on held-out data, steering is an actual intervention (Eq. 1), and the IS/SS metrics (Eq. 2) are defined and then reported as measurements rather than derived from fitted parameters. The §4.2 disjointness test is an operationalization of additivity (Eq. 3): the paper measures whether the downstream log-odds effect of two steering operations equals the sum of individual effects. While affine SAE decoders make activation-level edits additive by construction, the paper evaluates downstream probe log-odds through the nonlinear model, and it explicitly notes ReLU deviations (App. I), stating that this 'provides some evidence that disjointness is not tautologically expected.' Thus the disjointness finding is not equivalent to the input by construction. Self-citations such as Joshi et al. (2025), Arad et al. (2025), Marks et al. (2025), and Mueller et al. (2025b) are used as baselines, design guidance, or related work, not as the derivation of the headline result; none is an unverified uniqueness theorem that forces the conclusion. The acknowledged different-layer confound (App. B.2) is a validity limitation, not a circular step. Overall, the core empirical contrast between correlational disentanglement and steering non-independence is self-contained and externally testable.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The paper introduces two new metrics (steering independence IS and steering selectivity SS) but no new physical or model entities. The central empirical claims rely mainly on the validity of the synthetic data generation, the assumption that the steering intervention is a faithful causal operation, and the linear sufficiency of concept encodings.

free parameters (3)
  • steering coefficient alpha = 5
    Chosen by hand in Section 4.1 to set feature value to alpha*max(f_i). Some robustness to alpha is shown in Figure 14, but no principled selection criterion is given.
  • Top-K active features k = 128
    Architecture hyperparameter for Top-K SAEs (App. B.2). No ablation is reported for this choice.
  • SAE sparsity regularizers (lambda / beta) = not explicitly stated for all architectures
    App. B.2 gives training details but does not list the exact sparsity coefficients for ReLU/Top-K/SpADE losses; these would need to be tuned by a re-implementer.
axioms (4)
  • domain assumption Linear sufficiency: for each ground-truth concept z_k there exists a linear invertible transformation T such that z_k = T h_l.
    Assumption 1 in Section 3.1. The paper validates this empirically with probe accuracies (Fig. 9), but it is still an assumption about how concepts are encoded in the model.
  • domain assumption One feature dimension is sufficient for concept detection and control.
    Assumption 2 in App. E. The paper actually finds this assumption often false, but it motivates the single-feature steering protocol used throughout Section 4.
  • domain assumption PCFG-generated sentences with four categorical concepts are a faithful testbed for how SAEs represent concepts in natural language.
    The entire experiment uses synthetic English text. The authors acknowledge this limitation in the Discussion, but the transfer of the negative result to natural-language distributions is not established.
  • domain assumption The unobserved common cause in the DGP induces the desired correlation between concept values without introducing other unintended correlations.
    Section 2 and App. A describe the data generation. If unintended correlations are introduced, the identification and steering results could be confounded.

pith-pipeline@v1.3.0-alltime-deepseek · 32422 in / 7939 out tokens · 90103 ms · 2026-08-03T15:50:27.524770+00:00 · methodology

0 comments
read the original abstract

A goal of interpretability is to recover disentangled representations of latent concepts (features) from the activations of neural networks. The quality of features is typically evaluated in isolation, and under implicit independence assumptions that may not hold in practice. Thus, it is unclear to what extent common featurization methods such as sparse autoencoders (SAEs) and probes disentangle one concept from another. We propose a multi-concept evaluation setting using concepts including sentiment, domain, voice, and tense. We evaluate how well featurizers produce disentangled representations of each concept, observing that features are typically sensitive to only one concept, but also that concepts are distributed across many features. Then, we steer these features, measuring whether each concept is independently manipulable, and whether features interact. Even in idealized settings, steering a feature often affects many concepts, despite a near absence of interaction effects. These results suggest that correlational metrics are insufficient to establish steering selectivity, and that demonstrating that two features operate in separate spaces is insufficient to claim that they will be selective for one concept. These results underscore the importance of multi-concept evaluations in interpretability research.

Figures

Figures reproduced from arXiv: 2512.15134 by Aaron Mueller, Andrew Lee, Dhanya Sridhar, Ekdeep Singh Lubana, Patrik Reizinger, Shruti Joshi.

Figure 1
Figure 1. Figure 1: Causal graph of our experimental setup. The values of 4 known ground-truth con￾cepts {zi} 4 i=1 are used to generate an example x. We train a featurizer F to generate vectors f given activation vectors h ℓ from the output of layer ℓ of language model M. When training F on exam￾ples with increasing correlations between pairs of concepts ρ(zi , zj ), we observe whether F learns the true latents or the correl… view at source ↗
Figure 2
Figure 2. Figure 2: Maximum correlation coefficient for domain=science (left), sentiment=positive (mid￾dle), and MCC (right) under varying correlational conditions. Shaded regions represent 1 std. dev. across 3 training seeds. Ideal performance looks like a flat line at a high MCC. Probes (su￾pervised featurizers, in black) perform best. SSAEs perform best among unsupervised featurizers. SAEs trained on large-scale natural da… view at source ↗
Figure 3
Figure 3. Figure 3: DCI-ES scores under varying correlational conditions. Shaded regions represent 1 std. dev. across 3 training seeds. Ideal performance looks like a flat line at high values for all metrics. All methods achieve high disentanglement, informativeness, and explicitness, but relatively low completeness. This suggests that most features capture only one concept, but also that concepts are generally distributed ac… view at source ↗
Figure 4
Figure 4. Figure 4: The effect of steering a given concept (row) on the log-odds of another (column), as measured by a probe. Results for Pythia-70M shown here; see App. I for Gemma-2-2B. If concept representations are causally independent, we expect a heatmap that resembles the ground￾truth: ∆LOGODDS should be high on the diagonal, negative for within-concept pairs, and close to 0.0 for across-concept pairs. All SAEs demonst… view at source ↗
Figure 5
Figure 5. Figure 5: Predicted ∆LOGODDS(zi) under disjointness assumptions vs. actual ∆LOGODDS(zi) when steering relevant feature ˆfi and unrelated feature ˆfj . Predicted ∆LOGODDS are obtained by adding the ∆LOGODDS(zi) when steering with either ˆfi or ˆfj separately. Actual ∆LOGODDS values are obtained by steering both simultaneously. ˆfi and ˆfj are typically disjoint, as indicated by the predicted change almost perfectly m… view at source ↗
Figure 6
Figure 6. Figure 6: Excerpts from the context-free grammar we use to generate our SAE training and evaluation [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The difference between feature disjointness and independence: (Left) Two concepts zi and zj with feature representations ˆfi and ˆfj , respectively, are disjoint if the left diagram commutes. (Right) If they are independent then there is no commutative relationship, as steering with ˆfi should not affect p(zj ). Intuitively, disjointness implies that two feature representations exist in non￾overlapping sub… view at source ↗
Figure 8
Figure 8. Figure 8: Correlation coefficients between probe logits and concept labels for domain=science (left), sentiment=positive (middle), and MCC (right). Results for Gemma-2-2B shown here; results for Pythia-70M are in App. H. We vary the number of dimensions k that the probe is allowed to have non-zero weights from. k-sparse probes trained on SAEs begin to converge around 10 dimensions for Top-K, SpADE, SSAE, and Natural… view at source ↗
Figure 9
Figure 9. Figure 9: Accuracy of binary probes (rows) on all concept value classification tasks (columns). We [PITH_FULL_IMAGE:figures/full_fig_p025_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Accuracy of multinomial probes on all concept value classification tasks (columns). We [PITH_FULL_IMAGE:figures/full_fig_p026_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: MCC for the two most performant sparse probing methods from Gurnee et al. (2023) at various k. Results for Pythia-70M shown here. The LR method achieves higher MCC at lower k, but MD overtakes LR at higher k. In §4.1 and §4.2, we instead use multinomial linear probes trained on the final layers of Pythia-70M and Gemma-2-2B. We find ( [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Correlation coefficients between probe logits and concept labels for domain=science (left), sentiment=positive (middle), and MCC (right). Results for Pythia-70M. We vary the number of dimensions k that the probe is allowed to have non-zero weights from. As with Gemma-2-2B, correlation coefficients tend to converge at around 10 dimensions. However, the neuron baseline is far more performant; perhaps this i… view at source ↗
Figure 13
Figure 13. Figure 13: The effect of steering a given concept (row) on the logit of another (column), as measured by a probe. Results for Gemma-2-2B. If concept representations are causally independent, we expect a heatmap that resembles the ground-truth: ∆LOGODDS should be high on the diagonal, negative for within-concept pairs, and close to 0.0 for across-concept pairs. All SAEs demonstrate the expected diagonals, but also si… view at source ↗
Figure 14
Figure 14. Figure 14: ∆LOGODDS(zi) under various steering coefficients α for steering feature ˆfi , and ˆfj for a different concept zj . Here, concept zi is domain=science, and zj is sentiment=positive. Results for Gemma-2-2B. ˆfi and ˆfj are more often independent here than for Pythia-70M, as indicated by flat red lines. ˆfi and ˆfj are typically nearly disjoint, as indicated by the dotted purple line and solid purple lines a… view at source ↗
Figure 15
Figure 15. Figure 15: Maximum correlation coefficient for tense=past (left), voice=passive (middle), and MCC (right) under varying correlational conditions. Shaded regions represent 1 std. dev. across 3 training seeds. Ideal performance looks like a flat line at a high MCC. Probes perform best, and Top-K SAEs are best among unsupervised methods. domain-fantasy domain-science domain-news domain-other sentiment-positive sentimen… view at source ↗
Figure 16
Figure 16. Figure 16: The effect of steering a given concept (row) and the logit of another (column), as measured by a probe. Results for Pythia-70M (a) and Gemma-2-2B (b). As when training on the previous set of correlated variables, diagonals are largely present, as are within-concept effects. There are significant across-concept effects, however, indicating non-independence. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Disentanglement Beyond Generative Models with Riemannian ICA

    cs.LG 2026-05 unverdicted novelty 8.0

    RICA replaces ICA's global generative model with local Riemannian geometry, introducing a disentanglement tensor based on the Hessian of the log-likelihood and Ricci curvature to measure pointwise disentanglement, whi...

  2. Tensor Product Representation Probes Reveal Shared Structure Across Linear Directions

    cs.LG 2026-05 unverdicted novelty 7.0

    Linear probes for Othello board states factor into tensor-product structure with square and color embeddings composed by a binding matrix, from which the linear probes can be directly recovered.

  3. Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space

    cs.CL 2026-04 unverdicted novelty 6.0

    PAM, a complex-valued associative memory model, exhibits steeper power-law scaling in loss and perplexity than a matched real-valued baseline when trained on WikiText-103 from 5M to 100M parameters.

Reference graph

Works this paper leans on

5 extracted references · 3 linked inside Pith · cited by 3 Pith papers

  1. [2]

    do- main=science

    Hindupur et al. (2025) show that this architecture can capture more irregular concept geometries, whereas ReLU SAEs assume linear separability, and Top-K SAEs assume angular separability. Sparse shift autoencoders.Sparse shift autoencoders (SSAEs) (Joshi et al., 2025) are trained using paired observations(x, ˜x)assumed to be sampled from the following gen...

    2025

  2. [19]

    Phillip Lippe, Sara Magliacane, Sindy L¨owe, Yuki M

    URLhttps://aclanthology.org/2024.blackboxnlp-1.19/. Phillip Lippe, Sara Magliacane, Sindy L¨owe, Yuki M. Asano, Taco Cohen, and Efstratios Gavves. CITRIS: Causal Identifiability from Temporal Intervened Sequences, June 2022. URL http: //arxiv.org/abs/2202.03169. Number: arXiv:2202.03169 arXiv:2202.03169 [cs, stat]. Phillip Lippe, Sara Magliacane, Sindy L¨...

    Pith/arXiv arXiv 2024

  3. [2024]

    arXiv:2402.04875 [cs, stat]

    URLhttp://arxiv.org/abs/2402.04875. arXiv:2402.04875 [cs, stat]. Kartik Ahuja, Jason Hartford, and Yoshua Bengio. Weakly Supervised Representation Learning with Sparse Perturbations. October 2022a. URL https://openreview.net/forum?id= 6ZI4iF_T7t. Kartik Ahuja, Yixin Wang, Divyat Mahajan, and Yoshua Bengio. Interventional Causal Rep- resentation Learning, ...

    Pith/arXiv arXiv 2023

  4. [2025]

    Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler

    URLhttps://openreview.net/forum?id=K2CckZjNy0. Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability Guarantees for Causal Disentanglement from Soft Interventions, July 2023. URL http://arxiv.org/abs/2307.06250. arXiv:2307.06250 [cs, math, stat]. Zuheng, Xu, Moksh Jain, Ali Denton,...

    Pith/arXiv arXiv 2023

  5. [4687]

    URL https://www.nature.com/articles/ s41586-023-06668-3

    doi: 10.1038/s41586-023-06668-3. URL https://www.nature.com/articles/ s41586-023-06668-3. Publisher: Nature Publishing Group. Elliot Layne, Jason Hartford, S´ebastien Lachapelle, Mathieu Blanchette, and Dhanya Sridhar. Sparsity regularization via tree-structured environments for disentangled representations.Transactions on Machine Learning Research, 2025....

    doi:10.1038/s41586-023-06668-3 2025