Causal Interpretation of Sparse Autoencoder Features in Vision

Nojun Kwak; Sangyu Han; Yearim Kim

REVIEW 4 major objections 4 minor 12 references

Causal Interpretation of Sparse Autoencoder Features in Vision

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ERF maps, not activation peaks, reveal what truly drives vision SAE features

desk verdict Useful observation about non-local SAE features and a plausible attribution-based explanation, but the causal claim outruns the evidence: only an insertion test, no suppression results, no error bars. read the letter →

arxiv 2509.00749 v1 pith:DEORSDNW submitted 2025-08-31 cs.CV cs.AI

Sangyu Han , Yearim Kim , Nojun Kwak This is my paper · ORCID

classification cs.CVcs.AI

keywords sparseautoencodersinterpretabilityeffectivereceptivefieldcausalattributionvisiontransformersCLIPinsertiontestfeatureexplanation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the usual way of interpreting sparse autoencoder (SAE) features in vision transformers—looking at the image patches where a feature activates most strongly—can be misleading, because self-attention mixes information across the whole image. It introduces Causal Feature Explanation (CaFE), which instead computes an effective receptive field (ERF) for each feature activation using input-attribution methods, identifying the patches that causally drive that activation. The central evidence is an insertion test: when patches are inserted into a blank image in ERF order, the feature activation recovers faster and more completely than when patches are inserted in activation-magnitude order. If correct, CaFE would make vision-model feature interpretation more faithful and would expose hidden context dependencies, such as a feature that fires only when a face and spilled pills co-occur, even though its maximal activation patch sits far from the pills.

What carries the argument

The central object is the Effective Receptive Field (ERF), defined as the score map ERF_k(I) = {(p, A(p | z_k, I)) : p in I}, where A is the attribution of input patch p to the scalar SAE feature activation z_k(I). The ERF is computed by backpropagating relevance from the target SAE neuron through the SAE encoder and the vision transformer layers, with AttnLRP used inside transformer layers to distribute relevance along attention edges. This mechanism converts the question 'where does the feature fire?' into 'what image evidence causes it to fire?', and the insertion test then measures whether those causal patches actually recover the activation.

What would settle it

A concrete test: run the same insertion protocol with ERF maps computed by AttnLRP versus ERF maps whose patch order is randomly permuted (or replaced by a constant map). If the permuted maps match the AttnLRP insertion AUC, the attribution carries no causal information beyond what activation ranking already provides. A second check: ablate the top-ERF patches and compare the drop in target feature activation against ablating top-activation patches; if ablating ERF patches does not yield a larger drop, the ERF does not contain the causal drivers.

Watch

Extended reading notes

Core claim

The paper's central claim is that causal, ERF-based explanations are more faithful and semantically precise than activation-based explanations for vision SAE features. Concretely, for each scalar SAE feature activation, CaFE treats that activation as the target of an input-attribution computation—backpropagating relevance through the SAE encoder and the vision transformer—and defines the effective receptive field as the attribution map over input patches. Across CLIP-ViT-L/14 layers, these ERF maps frequently diverge from naive activation maps, particularly for non-local features that emerge in later layers (peaking around layer 22, where roughly 14% of the first 100 features are non-local).

Load-bearing premise

The method assumes that the attribution method, AttnLRP, faithfully traces which input patches causally drive a single SAE feature's scalar activation; if AttnLRP's relevance distribution does not reflect the transformer's true information flow, then the ERF is not truly causal and the insertion test only shows a better proxy.

Editorial extensions

If this is right

Non-local SAE features, especially in higher transformer layers, should not be interpreted by their activation peaks alone; ERF maps are needed to avoid mislabeling.
Existing activation-based feature-labeling pipelines may systematically mislabel context-dependent features, and CaFE provides a way to detect and correct that.
The plug-and-play attribution interface means the same ERF framework can accommodate different attribution methods, with AttnLRP currently giving the best results on transformers.
ERF-guided selection also improves insertion recovery for localized features, so attribution can sharpen interpretations even where activation maps seem adequate.
Computing ERFs requires extra forward and backward passes, and manual labeling of non-local features remains subjective; scaling automated criteria is open future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same ERF-based causal-reading approach could be applied to language-model SAE features, where context mixing across tokens creates an analogous gap between high-activation tokens and causally important text spans.
A fully automated non-local-feature detector could be built by measuring the statistical divergence between activation maps and ERF maps, replacing the paper's manual inspection of the first 100 features per layer.
The insertion-test validation could be supplemented with deletion tests—removing ERF-high patches and checking whether the target activation drops more than when removing activation-high patches—to strengthen the causal claim.
If ERF is truly causal, then the fraction of non-local features per layer (near zero in early layers, rising to ~14% at layer 22) offers a quantitative signature of how self-attention builds compositional, context-dependent representations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This paper has a genuinely useful observation and a plausible method, but the causal claim is a step ahead of the evidence. The observation is that some SAE features in CLIP-ViT, mainly in later layers, show high activation patches that are not the patches actually driving the feature. The proposed fix, CaFE, is to backprop an attribution map from the SAE feature activation to the input and call that the effective receptive field. That is a natural extension of existing tools, not a new type of explanation, but applying it to SAE features fills a real gap, and the qualitative examples (the 'Despair' feature that fires for a frowning face plus spilled pills; the 'roaring face' that needs eyes and nose) are convincingly non-trivial. The layer-wise trend, non-local features are rare before layer 9 and rise to roughly 14% at layer 22, is a useful empirical anchor, even if it comes from a manual review of the first 100 features per layer.

The problems are mostly in the quantitative support. The abstract says CaFE 'recovers or suppresses' feature activations, but only the insertion (recovery) test is shown. No suppression or removal results appear anywhere. Fig. 3 has no error bars or sample sizes, so I cannot tell whether AttnLRP's margin over the activation baseline is stable. The insertion protocol itself starts from a blank image; for a ViT trained on full natural images, that is a distribution shift, and the same patch may drive a different feature when planted out of context. So the insertion test is evidence that AttnLRP gives a better ordering for that specific procedure, not full proof of causality in the intact image. A removal test on real images, or a causal intervention that activates and deactivates the feature by manipulating the ERF region, would be the right next step. The manual annotation is subjective, but the paper owns that in the limitations section. One small thing: the ERF concept is attributed to the authors' own prior work [3] rather than the original receptive-field literature, which is worth correcting.

That said, the paper is not fatally flawed. The direction is sound, the method is simple and reproducible, and the qualitative findings are likely to be useful to anyone auditing vision models. The soft spots are addressable with more experiments and stricter reporting. I think this deserves a serious referee, with the expectation of a major revision that adds a suppression/removal test, error bars, and a less overreaching statement of the causal claim.

Referee Report

4 major / 4 minor

Summary. The paper proposes Causal Feature Explanation (CaFE), a method for interpreting sparse autoencoder (SAE) features in vision transformers. Instead of relying on the patches where a feature activation is highest, CaFE computes an input-attribution map (e.g., AttnLRP) for each target activation z_k, calls this map the Effective Receptive Field (ERF), and uses it to identify the image patches that causally drive the activation. The authors train Matryoshka SAEs on CLIP/ViT-L/14 patch embeddings, present qualitative examples of non-local features whose ERFs differ from their activation maps, and report an insertion test comparing ERF-guided patch selection against activation-ranked patch selection. They conclude that ERF-based explanations are more faithful and semantically precise.

Significance. If the central claim holds, CaFE is a useful contribution to vision-SAEs interpretability: it highlights that top-activation patches can be misleading for non-local features and offers a concrete attribution-based alternative. The qualitative examples are compelling and the method is simple to reproduce. However, the quantitative evidence is currently too thin to support the causal and 'recovers or suppresses' wording. The paper's value will depend on tightening the experimental validation, particularly by adding removal/suppression tests and proper statistical reporting.

major comments (4)

[§4.1 / Abstract / Conclusion] The abstract and conclusion claim CaFE 'recovers or suppresses' feature activations and that ERF-based explanations are 'causal'. The only quantitative validation is an insertion test starting from a blank image; no removal/suppression experiment (e.g., occluding the ERF patches in the intact image and measuring the drop in z_k) is presented. This gap is load-bearing for the causal claim and must be addressed."
[Fig. 3 / §4.1] The claim that CaFE 'highly outperformed' the activation baseline is not verifiable as reported: the figure shows curves without error bars, confidence intervals, or sample sizes (number of features/images). The area under the insertion curve metric is not defined. Please report mean ± std across features/images, the number of trials, and a paired significance test.
[§3.2 / §4.1] The method assumes that AttnLRP (or other attribution) faithfully reflects the causal flow from input patches to z_k. The insertion test uses a blank canvas, which is far out-of-distribution for CLIP/ViT-L/14—self-attention and positional encodings mix global context. Demonstrating that ERF patches better recover z_k in this artificial setting does not establish that those same patches caused the activation in the original image. A complementary removal test on intact images, or a control with random patches, is necessary to distinguish true causal drivers from a better proxy for the insertion procedure.
[§4.2 / Fig. 5] The quantitative claim that non-local features peak at layer 22 (≈14% of features) is based on manual review of the first 100 features per layer, an approach later acknowledged as subjective in §4.3. Yet no inter-annotator agreement or objective criterion for 'spatially inconsistent' is given. Please provide a reproducible scoring rule or report agreement statistics.

minor comments (4)

[Abstract / §5] Typos: 'levarages' → 'leverages'; 'mounth' → 'mouth'; 'empircally' → 'empirically'.
[§3.1] The experimental setup says '5 × 108 image patches'; this should be formatted as 5 × 10^8.
[References] The paper cites only its own prior work [3] for the Effective Receptive Field concept, which is a well-established notion with earlier origins (e.g., Luo et al., 2016). Please cite the original ERF literature and clarify the relation to [3].
[Fig. 3] The figure would benefit from axis labels and a clear explanation of what is plotted (insertion ratio vs. normalized activation, or AUC). Also specify how many features and images were used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the central claim is evaluated by an independent insertion protocol, not defined into the method.

full rationale

The paper defines the Effective Receptive Field as an attribution map A(p|z_k,I) in Eq. (1) and then tests whether ERF-ranked patches recover the SAE feature activation in an insertion test. This is a genuine empirical prediction: the attribution scores are computed on the original image, while the insertion test measures z_k on partially reconstructed inputs. Nothing in the construction forces AttnLRP or any attribution method to outperform activation-ranked patches; the comparison is falsifiable and external to the method's definition. No parameter is fitted to the insertion outcome, and no equation equates the target claim to an input of the pipeline. The only self-citation is reference [3] for the ERF label and for insertion-test precedent, but the protocol is also anchored to the external Samek et al. [8] and the attribution method AttnLRP is from the external reference [1]. The definition in Eq. (1) is self-contained. Concerns that the blank-image insertion test is out-of-distribution or that suppression results are not reported are validity/robustness issues, not circularity. Therefore the paper's derivation chain does not reduce to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim has no invented entities; it relies on existing attribution methods and a redefinition of ERF as an attribution map. The main unstated load-bearing premise is that attribution equals causality, which is not proven in the paper.

free parameters (3)

SAE sparsity coefficient lambda
Reconstruction-sparsity tradeoff held fixed across layers following prior work; exact value not reported, but it controls which features the SAE learns.
SAE feature dimension m
Number of Matryoshka SAE features per layer; not reported in the text, affects feature granularity.
Insertion test patch protocol parameters
Number of patches inserted, step size, and number of images/features used for AUC are not specified; the choice shapes the reported improvement.

assumptions (4)

domain assumption Self-attention mixes information across the image, so top-activation patches can be co-occurring rather than causal.
Stated in the Introduction as motivation for CaFE (Fig 1 and Section 1).
domain assumption Input attribution methods, particularly AttnLRP, reflect the true causal contribution of input patches to the target feature activation.
Section 3.2 defines ERF as the attribution map, implicitly assuming attribution equals causation.
domain assumption Insertion test AUC is a valid measure of causal fidelity of an explanation map.
Section 4.1 uses insertion tests as the evaluation protocol for causality.
domain assumption SAE features trained on patch embeddings are meaningful units for semantic interpretation.
Imported from prior SAE works [4,5] without re-derivation; the paper's analysis operates on these features.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Causal Interpretation of Sparse Autoencoder Features in Vision." pith.science (2026). https://pith.science/paper/DEORSDNW

@misc{pith2026250900749,
  author       = {Pith},
  title        = {Pith review of: Causal Interpretation of Sparse Autoencoder Features in Vision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DEORSDNW}},
  note         = {Machine review of arXiv:2509.00749}
}

read the original abstract

Understanding what sparse auto-encoder (SAE) features in vision transformers truly represent is usually done by inspecting the patches where a feature's activation is highest. However, self-attention mixes information across the entire image, so an activated patch often co-occurs with-but does not cause-the feature's firing. We propose Causal Feature Explanation (CaFE), which leverages Effective Receptive Field (ERF). We consider each activation of an SAE feature to be a target and apply input-attribution methods to identify the image patches that causally drive that activation. Across CLIP-ViT features, ERF maps frequently diverge from naive activation maps, revealing hidden context dependencies (e.g., a "roaring face" feature that requires the co-occurrence of eyes and nose, rather than merely an open mouth). Patch insertion tests confirm that CaFE more effectively recovers or suppresses feature activations than activation-ranked patches. Our results show that CaFE yields more faithful and semantically precise explanations of vision-SAE features, highlighting the risk of misinterpretation when relying solely on activation location.

Figures

Figures reproduced from arXiv: 2509.00749 by the authors.

**Figure 2.** Method overview. The feature ’Despair’ highly activated at the patch at background. Using attribution method, we can find which [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Causality validation. We compare causality of our CaFE method with baseline (naive activation-based patch ranking). We also [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Qualitative examples of non-local SAE features and their ERFs at the points of highest activation across different layers. Even [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: The number of non-local SAE features across layers. The non-local SAE features become rarer as layer number decreases. We manually inspected the number of non-local features out of the first 100 features. vations increasingly difficult to interpret without ERF. 4.3. Di…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

[3]

Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818--2829, 2023

2023
[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Attnlrp: attention-aware layer-wise relevance propagation for transformers

Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnlrp: attention-aware layer-wise relevance propagation for transformers. arXiv preprint arXiv:2402.05602, 2024

arXiv 2024
[4]

Respect the model: Fine-grained and Robust Explanation with Sharing Ratio Decomposition

Sangyu Han, Yearim Kim, and Nojun Kwak. Respect the model: Fine-grained and robust explanation with sharing ratio decomposition. arXiv preprint arXiv:2402.03348, 2024

work page Pith review arXiv 2024
[5]

Sparse autoencoders find highly interpretable features in language models

Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024

2024
[6]

Sparse autoencoders reveal selective remapping of visual concepts during adaptation

Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In The Thirteenth International Conference on Learning Representations, 2025

work page 2025
[7]

A unified approach to interpreting model predictions

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017

2017
[8]

Sparse autoencoders learn monosemantic features in vision-language models, 2025

Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models, 2025

work page 2025

Show all 12 references

[9]

Evaluating the visualization of what a deep neural network has learned

Wojciech Samek, Alexander Binder, Gr \'e goire Montavon, Sebastian Lapuschkin, and Klaus-Robert M \"u ller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28 0 (11): 0 2660--2673, 2016

2016
[10]

Sparse autoencoders for scientifically rigorous interpretation of vision models, 2025

Samuel Stevens, Wei-Lun Chao, Tanya Berger-Wolf, and Yu Su. Sparse autoencoders for scientifically rigorous interpretation of vision models, 2025

2025
[11]

Axiomatic attribution for deep networks

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319--3328. PMLR, 2017

2017
[12]

Interpreting CLIP with hierarchical sparse autoencoders

Vladimir Zaigrajew, Hubert Baniecki, and Przemyslaw Biecek. Interpreting CLIP with hierarchical sparse autoencoders. In Forty-second International Conference on Machine Learning, 2025

2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [3]

Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818--2829, 2023

2023

[2] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[3] [2]

Attnlrp: attention-aware layer-wise relevance propagation for transformers

Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnlrp: attention-aware layer-wise relevance propagation for transformers. arXiv preprint arXiv:2402.05602, 2024

arXiv 2024

[4] [4]

Respect the model: Fine-grained and Robust Explanation with Sharing Ratio Decomposition

Sangyu Han, Yearim Kim, and Nojun Kwak. Respect the model: Fine-grained and robust explanation with sharing ratio decomposition. arXiv preprint arXiv:2402.03348, 2024

work page Pith review arXiv 2024

[5] [5]

Sparse autoencoders find highly interpretable features in language models

Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024

2024

[6] [6]

Sparse autoencoders reveal selective remapping of visual concepts during adaptation

Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation. In The Thirteenth International Conference on Learning Representations, 2025

work page 2025

[7] [7]

A unified approach to interpreting model predictions

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017

2017

[8] [8]

Sparse autoencoders learn monosemantic features in vision-language models, 2025

Mateusz Pach, Shyamgopal Karthik, Quentin Bouniot, Serge Belongie, and Zeynep Akata. Sparse autoencoders learn monosemantic features in vision-language models, 2025

work page 2025