Pith. sign in

REVIEW 4 major objections 5 minor 100 references

The SuperActivator Mechanism: Transformers Concentrate Reliable Concept Signals in the Tail

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Concept signals in transformers live in a sparse high-activation tail — the rest is noise.

desk verdict Strong empirical case that the tail of in-concept activations is a reliable detection signal for visual concepts; text modality rests on synthetic labels and the abstract overclaims a proof that never appears. read the letter →

arxiv 2512.05038 v2 pith:WQEWQ7PH submitted 2025-12-04 cs.LG

Cassandra Goldberg , Chaehyeon Kim , Adam Stein , Eric Wong This is my paper
classification cs.LG
keywords conceptvectorsdetectionSuperActivatormechanismtokenactivationdistributionstransformerinterpretabilitysparsesignalsmaxpoolinglocalization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that although concept vectors produce noisy, overlapping activations for in-concept and out-of-concept tokens, transformers systematically push the most reliable evidence for a concept into a tiny set of extreme token activations, which it calls SuperActivators. It proves that concept-aligned attention heads amplify activation gaps multiplicatively, so the strongest activations grow fastest, and shows empirically across vision and language transformers that thresholding the global in-concept tail — typically the top 5–10% of in-concept activations — detects concept presence more accurately than standard pooling, [CLS]-score, or prompting baselines, by up to 0.14 F1. The same tail signal improves concept localization: attributions computed against the average embedding of local SuperActivators align better with ground-truth masks and are more faithful to model behavior. A reader should care because it turns a known nuisance, noisy concept activations, into a simple detection rule that needs only sample-level labels and a per-concept quantile threshold.

What carries the argument

SuperActivator threshold — τ^{super}_{c,δ}=Q_{1−δ}(S^+_{val,c}), the (1−δ) quantile of in-concept validation activations; tokens scoring above it are SuperActivators. The detection rule is max-pooling over token activations with this threshold, which is what turns the tail into a concept detector. The theoretical engine is a proof that concept-aligned attention heads amplify pairwise activation gaps multiplicatively, making extreme activations grow faster than typical ones; this is offered as the reason the tail is sparse, consistent across depth, and present in both modalities.

What would settle it

Run the identical SuperActivator detector on a human-annotated, token-level text corpus for a concept like sarcasm or emotion, with no LLM-generated context. If the top 5–10% in-concept activation tail does not separate from out-of-concept activations, or F1 falls to baseline levels, the mechanism's text-domain claim collapses. A simpler check: train on LLM-labeled positive tokens and test on concept-positive sentences with LLM-stylistic cues such as hashtags, emoji, or tag-wrapped spans removed; a sharp performance drop would indicate the tail encodes label-generation style rather than the co

Watch

Extended reading notes

Core claim

The central claim is the SuperActivator Mechanism: for a concept vector v_c, the token activation scores s_c(z)=<z, v_c> mostly overlap between in-concept and out-of-concept tokens, but the in-concept distribution D^in_c develops a positive tail that separates from D^out_c with depth; the tokens in that tail, called SuperActivators, appear in most concept-positive samples and are therefore reliable indicators of presence. Detection reduces to max-pooling token activations and thresholding at the global in-concept quantile τ = Q_{1−δ}(S^+_{val,c}) with small δ. The paper further proves that concept-aligned attention heads multiplicatively amplify pairwise activation gaps, so already-extreme a

Load-bearing premise

The token-level in/out concept partition is trusted as ground truth — and for the text datasets that partition comes from an LLM's generated or wrapped spans, so if embeddings encode the generator's style rather than the concept, the tail signal is partly a dataset artifact.

Editorial extensions

If this is right

  • Concept detection can be done with almost no tuning: fixing the tail at 10% of in-concept activations and learning a single threshold from sample-level labels nearly matches the fully tuned SuperActivator detector and beats all baselines.
  • Concept localization improves by using the mean embedding of local SuperActivators as the attribution objective, yielding higher alignment with human masks and better insertion/deletion faithfulness than the global concept vector.
  • The tail signal persists across supervised and unsupervised concept extraction, including mean prototypes, linear separators, k-means, and sparse autoencoders, so the mechanism is tied to how transformers encode semantics rather than to one concept-vector construction.
  • SuperActivators are not position-dependent, so the signal is semantic rather than an artifact of where tokens appear in the input.
  • The amplification theorem gives a testable reason concept separability grows with depth and plateaus in middle layers before final-layer compression.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the three text datasets define token-level concepts with LLM-generated labels — one fully synthetic, two augmented with machine-written neutral context — the tail signal for sarcasm and emotion may be partly a stylistic signature of the label generator rather than the concept itself; human-labeled span data would settle this.
  • If the mechanism is as general as claimed, the tail threshold should also work for detecting concepts discovered entirely without labels on new domains and non-transformer architectures; the paper tests transformers only.
  • Extreme-value reasoning suggests the SuperActivator threshold is an estimate of a high quantile of a heavy-tailed distribution; this connects to outlier and novelty detection and could be formalized to set sparsity δ without validation.
  • The coverage result implies the detector inherits label noise: positive spans that lack any extreme token will be systematically missed, which may explain the weaker tail coverage the paper admits for text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the 'SuperActivator Mechanism,' the observation that for transformer concept vectors, the extreme high-activation tail of the in-concept token distribution provides a sparse but reliable signal for concept presence. The authors define a SuperActivator threshold as the (1−δ) quantile of validation-set in-concept activations, aggregate token activations by max-pooling, and threshold at this quantile for concept detection. They report consistent F1 improvements over standard aggregators and prompting baselines across four image datasets and three text datasets, several models (CLIP, LLaMA, Gemma, Qwen), multiple concept-vector types (mean prototypes, linear separators, k-means, k-means separators, SAEs), and multiple layers. They also propose using the average embedding of local SuperActivators as an attribution objective, reporting improved alignment with ground-truth masks and better insertion/deletion scores. The Appendix contains extensive ablations on layer depth, sparsity, token position, and a fixed-sparsity variant (N@Tail) that requires only sample-level labels and nearly matches the tuned method.

Significance. If the empirical claims hold, the paper identifies a practically useful and broadly applicable property of transformer concept representations: instead of pooling all token activations, focusing on the extreme tail yields better concept detection and attribution. The strongest strengths are the breadth of the evaluation (7 datasets, 5 concept-vector types, 4-5 model families, layer and sparsity ablations), the inclusion of error bars, the released code, and especially the N@Tail variant in Appendix L, which shows that the main detection benefit survives without per-concept tuning of the sparsity parameter. These are substantial. However, the cross-modality generality claim is weakened by the reliance on GPT-4o-generated token-level labels for the text datasets, and the abstract asserts a proof that does not appear in the manuscript. The attribution results are also partly circular because the SuperActivator objective is derived from the same ground-truth labels used for evaluation. With appropriate qualification and additional controls, the image-domain results and the detection methodology would be a solid contribution.

major comments (4)
  1. [Abstract and §1] The abstract claims: 'we prove that concept-aligned attention heads multiplicatively amplify pairwise activation gaps, with already-extreme activations growing fastest.' No theorem, proof, or proof sketch appears anywhere in the main text or appendices. This is an unsupported central claim as stated. Either provide the proof (or a precise theorem with assumptions) or remove/soften the claim to an empirical observation.
  2. [§4.1, Appendix C.4] The text-domain evaluation is confounded by the use of GPT-4o for token-level supervision. In the fully synthetic Sarcasm dataset, both sarcastic and neutral sentences are GPT-generated, so the SuperActivator threshold τ = Q_{1−δ}(S+_val) may be isolating GPT-4o's stylistic markers rather than a general concept signal. For iSarcasm and GoEmotions, the out-of-concept tokens come from GPT-written neutral context, so the separation between the tail of Din_c and Dout_c could reflect human-vs-GPT stylometric differences. This directly undermines the headline claim of a modality-independent transformer property. Appendix B itself concedes that for text not all in-concept samples contain a tail token. Please add a control using human-written neutral context or a human-annotated token-level text dataset, or explicitly qualify the text results as potentially dataset-specific.
  3. [§4.3, Appendix N] The attribution evaluation is partly circular. SuperActivators are defined as the top δ of validation in-concept activations, where δ is calibrated per concept to maximize detection F1, and at test time they are tokens exceeding that label-derived threshold. The attribution objective is then the mean embedding of these SuperActivators, and the resulting maps are evaluated against the same ground-truth masks used to define in-concept tokens. It is not surprising that they align better than a global concept vector. The paper needs a control that uses an equally label-informed objective without the tail selection — e.g., the mean embedding of all positive tokens, or a random subset of positive tokens — to show that the improvement is due to the tail property rather than to label leakage into the attribution objective.
  4. [§3.2 and Appendix H/I] The claim that detection 'peaks when using only 5–10% of in-concept token activations' is in part definitional, because δ directly controls the quantile of the in-concept distribution used as the threshold. Figure 5 and the Appendix H/I curves are calibration curves for a family of quantile thresholds, not independent evidence of a mechanism. The N@Tail variant in Appendix L mitigates the tuning concern, and I would encourage the authors to present that as the primary practical evidence for the tail being sufficient. The text should be carefully worded to avoid presenting the optimal-δ analysis as an unanticipated discovery.
minor comments (5)
  1. [Appendix B, last paragraph] The sentence 'there appears to be activations in the tail of Din_c that are well-separated from Din_c' should read 'well-separated from Dout_c'; as written it is self-referential and likely a typo.
  2. [Table 2 and Appendix N.1] The citation for MFABA is inconsistent: the main Table 2 cites [67] for MFABA, but [67] is FullGrad, while Appendix N.1 correctly cites [89] for MFABA. Please correct the reference.
  3. [Appendix O, Table 12] The COCO SAE SuperAct row reports F1 = 0.582 ± 0.000, which is suspiciously precise; if this is due to a degenerate single concept or rounding, please clarify.
  4. [Appendix C.4] The augmentation prompts include some duplicated phrases (e.g., 'remaining 100%') and inconsistent punctuation. A careful copyedit would improve readability.
  5. [Figure 13 caption] The caption says 'SuperActivators' after all three concept activation rows, but only one concept's tail is marked. Clarify which concept the green boxes refer to.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical SuperActivator claims are evaluated on held-out test data and against an independent out-of-concept distribution; flagged limitations are data-validity and proof-completeness issues, not circular reductions.

full rationale

Walking the derivation chain, the central definition τ^super_{c,δ}=Q_{1−δ}(S+_val,c) (Sec. 3.2) does make the SuperActivator set the top δ of in-concept validation activations, and δ is later calibrated on validation F1 (Sec. 4.2, App. D). But the paper's claims are not forced by this definition: detection F1 is computed on held-out test samples; tail separation is additionally quantified against the independent q0.98(D_out_c) (Figs. 3–4, App. B); and the finding that optimal δ is small is an empirical property of the location of the argmax, not an identity. Baselines are calibrated with the same validation procedure, so the 'up to 0.14 F1' comparison is fair. Two non-circular concerns are worth flagging: (i) Appendix C.4 states text token-level labels are generated 'via controlled prompting of GPT-4o,' so the three text results may partly reflect GPT-4o's stylistic markers rather than the concepts themselves; this is a validity/generality limitation, not a reduction of the equations to their inputs. (ii) The Abstract claims 'we prove that concept-aligned attention heads multiplicatively amplify pairwise activation gaps,' but no such proof appears in the body — an omitted proof/overclaim, again not circular. The only author self-citation ([68]) appears in a Related Work survey and is not load-bearing. Hence no circularity; score 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central claim rests on nested modeling choices: concept vectors as directions (linear separability), token-level labels defining the in/out distributions (for text, GPT-4o-generated), and per-concept validation calibration of sparsity, layer, and threshold. The mechanism's core ingredient — thresholding at the global in-concept quantile — is transparent; the main external assumptions are label quality and transfer of calibrated thresholds. The abstract's promised amplification theorem, which would reduce the reliance on these assumptions, is absent from the body.

free parameters (5)
  • δ (sparsity level) = per-concept 2–10% typical; iSarcasm ~40%; fixed variant 10%
    Defines the SuperActivator threshold τ = Q_{1−δ}(S+_val); selected per concept on the validation set to maximize detection F1 (Section 3.2, Appendices H/I).
  • Model layer per concept = chosen from a fixed depth grid per concept and per method
    Layer calibration on validation (Appendix C.1); each detection method gets its own best layer, so comparisons are internally fair but the search surface is large.
  • Unsupervised concept matching oracle = best validation F1 unit per ground-truth concept
    Each ground-truth concept is paired with the unsupervised unit achieving the highest validation F1 (Appendix C.2); optimistic selection that can inflate absolute reported performance.
  • k-means k = 1000 token-level, 50 CLS-level
    Chosen by an elbow curve (Appendix C.2); affects the quality of unsupervised concept vectors for both the method and baselines.
  • Text dataset construction probabilities = 20% / 15% / random structural choices
    GPT-4o augmentation pipelines (Appendix C.4); style choices that could imprint artifacts into the token-level labels defining the 'in-concept' distribution.
assumptions (5)
  • domain assumption Concepts are representable as directions in embedding space; dot-product activation measures concept presence (linearity hypothesis)
    Section 2 Preliminaries; standard TCAV-style assumption inherited rather than tested in this paper.
  • domain assumption Token-level ground-truth labels correctly partition in-concept vs out-of-concept tokens
    Section 2, Appendices C.3–C.4; for the text datasets these labels are GPT-4o-generated, so correctness is not externally verified.
  • domain assumption Out-of-concept distribution Dout_c built only from samples without c avoids self-attention leakage
    Section 2; a reasonable design choice, but unvalidated — leakage within positive samples is excluded by construction, shaping both Dout_c and detection behavior.
  • domain assumption Validation-calibrated (δ, layer, threshold) transfer to the test distribution
    Standard calibration assumption; no distribution-shift or OOD analysis is provided.
  • domain assumption Embedding normalization/mean-centering using training statistics preserves relevant concept structure
    Appendix C.1; affects comparability of activation scores across layers and models.
invented entities (1)
  • SuperActivator tokens
    purpose: The top δ% of in-concept token activations, claimed to carry the reliable concept signal; used as the threshold basis for detection and as the attribution objective.
    A definitional construct (T_super = {z : s_c(z) ≥ Q_{1−δ}(S+_val)}) rather than a discovered entity with external predictions. Operational evidence for its usefulness comes only from this paper's datasets; the 'Mechanism' label implies a causal dynamic that is not demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The SuperActivator Mechanism: Transformers Concentrate Reliable Concept Signals in the Tail." pith.science (2026). https://pith.science/paper/WQEWQ7PH

@misc{pith2026251205038,
  author       = {Pith},
  title        = {Pith review of: The SuperActivator Mechanism: Transformers Concentrate Reliable Concept Signals in the Tail},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQEWQ7PH}},
  note         = {Machine review of arXiv:2512.05038}
}
read the original abstract

Concept vectors aim to enhance model interpretability by linking internal representations with human-understandable semantics, but their practical utility is often limited by noisy and inconsistent activations. In this work, we uncover the SuperActivator Mechanism: a transformer dynamic that amplifies concept activation gaps, concentrating the most reliable concept evidence into a small set of high-activation tokens. To develop a theoretical understanding of this mechanism, we prove that concept-aligned attention heads multiplicatively amplify pairwise activation gaps, with already-extreme activations growing fastest. We find that this amplification is not just theoretical, but also occurs empirically on large-scale models: while in- and out-of-concept activation distributions overlap considerably, the in-concept distribution develops a positive tail clearly separated from the noise. These high-tail tokens, which we call SuperActivators, appear consistently across concept-positive samples, making them reliable indicators of concept presence. Accordingly, SuperActivator-based detection improves F1 by up to 0.14 over standard concept activation aggregators and prompting baselines across image and text modalities, models, layers, and concept extraction techniques, demonstrating the generality and practicality of our insights. Further empirical analysis demonstrates that the most reliable SuperActivators are sparse, with detection typically peaking when using only 5-10% of in-concept token activations, and capture more faithful localized semantics than global concept vectors.

Figures

Figures reproduced from arXiv: 2512.05038 by the authors.

Figure 1
Figure 1. The SuperActivator Mechanism concentrates the most informative concept signals into a sparse set of in-concept activations. These signals reliably distinguish true concept occurrences even when concept activation heatmaps spuriously highlight absent concepts or fail to fully capture present ones. This example shows LLaMA-3.2-11B-Vision-Instruct linear separator concept activations on a COCO image; examples for all i… view at source ↗
Figure 2
Figure 2. Transformers express concept activations inconsistently, making it difficult to distinguish in-concept [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Din c and Dout c become more distinct with depth, though the separation is concentrated in a small subset of tokens in the tail of Din c . Shown here are activation distributions for three linear separator concepts from LLaMA-3.2-11B-Vision-Instruct on the OpenSurfaces dataset (left), as well as the proportion of Din c activations exceeding q0.98(Dout c ) across layers (right). 3 The SuperActivator Mechanism Yields … view at source ↗
Figures from the paper (36 more)
Figure 4
Figure 4. Figure 4: Most true-concept images in the OpenSurfaces dataset have at least one Llama￾3.2-11b-Vision-Instruct linear separator activa￾tion in the high-activation tail of Din c , well separated from q0.98(Dout c ). To study these limitations empirically, we focus our anal￾ysis o…
Figure 5
Figure 5. Figure 5: SuperActivator-based concept detection is most effective when using only a small fraction of the most highly activated tokens (5–10%). This figure presents the number of LLaMA-3.2-11B-Vision-Instruct linear separator concept vectors that achieve their strongest F1 scor…
Figure 6
Figure 6. Figure 6: SuperActivators yield attribution masks that better align with the ground-truth concept regions. Shown are attribution maps for the concept Person in a COCO image using LLaMA-based linear separator concepts. Maps are computed with LIME attribution relative to the Perso…
Figure 7
Figure 7. Figure 7: CLEVR – Visualization of Concept Activations and SuperActivators Original Animal Car Bench Person Motorcycle Chair Activation Score SuperActivators [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: COCO – Visualization of Concept Activations and SuperActivators 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: OpenSurfaces – Visualization of Concept Activations and SuperActivators Original Horse Muzzle Wood Dog Beak Sky Activation Score SuperActivators [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Pascal – Visualization of Concept Activations and SuperActivators 19 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Sarcasm – Visualization of Concept Activations and SuperActivators (sarcastic and non-sarcastic version of same sentiment) 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Sarcasm – Visualization of Concept Activations and SuperActivators (non-sarcastic and sarcastic text samples) Original Text (Anger highlighted): WHAT THE HELL! I opened up the new software update, and it seems like they've moved all the settings around again. Anger Ac…
Figure 13
Figure 13. Figure 13: Augmented GoEmotions SuperActivator Example 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Across all image and text datasets, models, and concept types, there appears to be high magnitude [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]
Figure 15
Figure 15. Figure 15: SuperActivator detection across image datasets. [PITH_FULL_IMAGE:figures/full_fig_p037_15.png]
Figure 16
Figure 16. Figure 16: SuperActivator detection across text datasets. [PITH_FULL_IMAGE:figures/full_fig_p038_16.png]
Figure 17
Figure 17. Figure 17: CLEVR 2% 4% 15% 25% 28% 40% 46% 52% 65% 67% 78% 88% 90% 100% % Through Model 0 15 30 45 60 75 90 105 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p039_17.png]
Figure 18
Figure 18. Figure 18: Coco 2% 4% 15% 25% 28% 40% 46% 52% 65% 67% 78% 88% 90% 100% % Through Model 0 10 20 30 40 50 60 70 80 90 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p039_18.png]
Figure 19
Figure 19. Figure 19: OpenSurfaces 39 [PITH_FULL_IMAGE:figures/full_fig_p039_19.png]
Figure 20
Figure 20. Figure 20: Pascal 3% 4% 19% 21% 34% 39% 50% 57% 66% 75% 81% 93% 97% 100% % Through Model 0 1 2 3 4 5 6 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p040_20.png]
Figure 21
Figure 21. Figure 21: Sarcasm 3% 4% 19% 21% 34% 39% 50% 57% 66% 75% 81% 93% 97% 100% % Through Model 0 1 2 3 4 5 6 7 8 9 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p040_21.png]
Figure 22
Figure 22. Figure 22: iSarcasm 3% 4% 19% 21% 34% 39% 50% 57% 66% 75% 81% 93% 97% 100% % Through Model 0 8 16 24 32 40 48 56 64 # of Concepts SuperTok CLS MeanTok LastTok [PITH_FULL_IMAGE:figures/full_fig_p040_22.png]
Figure 23
Figure 23. Figure 23: GoEmotions 40 [PITH_FULL_IMAGE:figures/full_fig_p040_23.png]
Figure 24
Figure 24. Figure 24: Image Domain – Optimal Sparsity over Layers [PITH_FULL_IMAGE:figures/full_fig_p041_24.png]
Figure 25
Figure 25. Figure 25: Text Domain – Optimal Sparsity over Layers [PITH_FULL_IMAGE:figures/full_fig_p042_25.png]
Figure 26
Figure 26. Figure 26: Image Domain – Detection F1 over Sparsity Level δ 43 [PITH_FULL_IMAGE:figures/full_fig_p043_26.png]
Figure 27
Figure 27. Figure 27: Text Domain – Detection F1 over Sparsity Level δ 44 [PITH_FULL_IMAGE:figures/full_fig_p044_27.png]
Figure 28
Figure 28. Figure 28: Cumulative distribution functions showing, for each concept and on average across a dataset, the [PITH_FULL_IMAGE:figures/full_fig_p045_28.png]
Figure 29
Figure 29. Figure 29: Image Domain – SuperActivator Position Distribution [PITH_FULL_IMAGE:figures/full_fig_p046_29.png]
Figure 30
Figure 30. Figure 30: Text Domain – SuperActivator Position Distribution [PITH_FULL_IMAGE:figures/full_fig_p047_30.png]
Figure 31
Figure 31. Figure 31: CLEVR – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 49 [PITH_FULL_IMAGE:figures/full_fig_p049_31.png]
Figure 32
Figure 32. Figure 32: COCO – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 50 [PITH_FULL_IMAGE:figures/full_fig_p050_32.png]
Figure 33
Figure 33. Figure 33: Broden-OpenSurfaces – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 51 [PITH_FULL_IMAGE:figures/full_fig_p051_33.png]
Figure 34
Figure 34. Figure 34: Broden-Pascal – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 52 [PITH_FULL_IMAGE:figures/full_fig_p052_34.png]
Figure 35
Figure 35. Figure 35: Sarcasm – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers Concept: Sarcastic Original Text: there's no better way to wake up than having one dog jump directly on your stomach and knock the wind out of you while the other drop a dead rodent on the end of th…
Figure 36
Figure 36. Figure 36: iSarcasm – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 53 [PITH_FULL_IMAGE:figures/full_fig_p053_36.png]
Figure 37
Figure 37. Figure 37: GoEmotions – SuperActivators Across LLaMA-3.2-11B-Vision-Instruct Layers 54 [PITH_FULL_IMAGE:figures/full_fig_p054_37.png]
Figure 38
Figure 38. Figure 38: SuperActivators produce attribution maps that better match the true sarcastic cues. Shown are token-level attributions for the concept Sarcasm on a sample from the Sarcasm dataset, using LLaMA token-level linear-separator concepts with LIME-based attribution. Red indi…
Figure 39
Figure 39. Figure 39: For SAEs The strongest globally applicable concept signals are not concentrated in a very sparse [PITH_FULL_IMAGE:figures/full_fig_p066_39.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

100 extracted references · 4 canonical work pages

  1. [1]

    Meaningfully debugging model mistakes using conceptual counterfactual explanations, 2022

    Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations, 2022. URLhttps://arxiv.org/abs/2106.12723

    arXiv 2022

  2. [2]

    Understanding intermediate layers using linear classifier probes,

    Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes,

  3. [3]

    Probing for constituency structure in neural language models

    David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. Probing for constituency structure in neural language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6738–6757, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis...

    doi:10.18653/v1/2022 2022

  4. [4]

    Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 2020

    David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network.Proceedings of the National Academy of Sciences, 2020. ISSN 0027-8424. doi: 10.1073/pnas.1907375117. URL https://www.pnas. org/content/early/2020/08/31/1907375117

    doi:10.1073/pnas.1907375117 2020

  5. [5]

    Maxpoolbert: Enhancing bert classification via layer- and token-wise aggregation.ArXiv, abs/2505.15696, 2025

    Maike Behrendt, Stefan Sylvius Wagner, and Stefan Harmeling. Maxpoolbert: Enhancing bert classification via layer- and token-wise aggregation.ArXiv, abs/2505.15696, 2025. URL https: //api.semanticscholar.org/CorpusID:278782887

    arXiv 2025

  6. [6]

    Opensurfaces: A richly annotated catalog of surface appearance.ACM Transactions on Graphics (SIGGRAPH), 32(4), 2013

    Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Opensurfaces: A richly annotated catalog of surface appearance.ACM Transactions on Graphics (SIGGRAPH), 32(4), 2013

    2013

  7. [7]

    Show and tell: Visually explainable deep neural nets via spatially- aware concept bottleneck models, 2025

    Itay Benou and Tammy Riklin-Raviv. Show and tell: Visually explainable deep neural nets via spatially- aware concept bottleneck models, 2025. URLhttps://arxiv.org/abs/2502.20134

    arXiv 2025

  8. [8]

    a is b" fail to learn

    Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Llms trained on "a is b" fail to learn "b is a", 2024. URL https://arxiv.org/abs/2309.12288

    arXiv 2024

Show all 100 references
  1. [9]

    Anthropic Research Preprint, 2023

    Trenton Bricken, Adly Templeton, Jonathan Batson, Brian Chen, Adam Jermyn, Tom Conerly, andet al.Towards monosemanticity: Decomposing language models with dictionary learning. Anthropic Research Preprint, 2023. Available at Anthropic’s website

    2023

  2. [10]

    Persona vectors: Monitoring and controlling character traits in language models.ArXiv, abs/2507.21509, 2025

    Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Monitoring and controlling character traits in language models.ArXiv, abs/2507.21509, 2025. URL https://api. semanticscholar.org/CorpusID:280337840

    2025 arXiv

  3. [11]

    Concept whitening for interpretable image recognition.Nature Machine Intelligence, 2:772 – 782, 2020

    Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition.Nature Machine Intelligence, 2:772 – 782, 2020. URLhttps://api.semanticscholar.org/CorpusID:211031886

    2020

  4. [12]

    Evaluation of bert and albert sentence embedding performance on downstream nlp tasks, 2021

    Hyunjin Choi, Judong Kim, Seongho Joe, and Youngjune Gwon. Evaluation of bert and albert sentence embedding performance on downstream nlp tasks, 2021. URLhttps://arxiv.org/abs/2101.10642

    2021 arXiv

  5. [13]

    Sparse autoencoders find highly interpretable features in language models.ArXiv, abs/2309.08600, 2023

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models.ArXiv, abs/2309.08600, 2023. URL https: //api.semanticscholar.org/CorpusID:261934663

    2023 arXiv

  6. [14]

    Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders, 2025

    Bartosz Cywi ´ nski and Kamil Deja. Saeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders, 2025. URLhttps://arxiv.org/abs/2501.18052

    2025 arXiv

  7. [15]

    Discovering latent concepts learned in bert, 2022

    Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. Discovering latent concepts learned in bert, 2022. URLhttps://arxiv.org/abs/2205.07237

    2022 arXiv

  8. [16]

    Goemotions: A dataset of fine-grained emotions

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemotions: A dataset of fine-grained emotions. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 4040–4054, 2020. 11

    2020

  9. [17]

    From colors to classes: Emergence of concepts in vision transformers.ArXiv, abs/2503.24071, 2025

    Teresa Dorszewski, Lenka Tvetkov’a, Robert Jenssen, Lars Kai Hansen, and Kristoffer Wickstrøm. From colors to classes: Emergence of concepts in vision transformers.ArXiv, abs/2503.24071, 2025. URL https://api.semanticscholar.org/CorpusID:277467666

    2025 arXiv

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale.ArXiv, abs/2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2010 arXiv

  11. [19]

    Toy models of superposition, 2022

    Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposi...

    2022 arXiv

  12. [20]

    Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge, 2010

    2010

  13. [21]

    Craft: Concept recursive activation factorization for explainability.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2711–2721, 2022

    Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2711...

    2023

  14. [22]

    Explaining groups of instances with shap-iq

    Thomas Fel, Alexandre Jullien, David Vigouroux, Remi Cadene, Thomas Nicodeme, Matthieu Laly, Asma Fermanian, Benjamin Audit, and Thomas Scantamburlo. Explaining groups of instances with shap-iq. InInternational Conference on Artificial Intelligence and Statistics, pages 6467–6...

    2023

  15. [23]

    Efros, and Jacob Steinhardt

    Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting clip’s image representation via text-based decomposition.ArXiv, abs/2310.05916, 2023. URL https://api.semanticscholar.org/ CorpusID:263829688

    2023 arXiv

  16. [25]

    Scaling and evaluating sparse autoencoders, 2024

    Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders, 2024. URL https://arxiv.org/ abs/2406.04093

    2024 arXiv

  17. [26]

    Automating interpretability: Discovering and testing visual concepts learned by neural networks.ArXiv, abs/1902.03129, 2019

    Amirata Ghorbani, James Wexler, and Been Kim. Automating interpretability: Discovering and testing visual concepts learned by neural networks.ArXiv, abs/1902.03129, 2019. URL https://api. semanticscholar.org/CorpusID:59842921

    1902 arXiv

  18. [27]

    Multimodal neurons in artificial neural networks.Distill, 2021

    Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks.Distill, 2021. doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons

    2021 doi

  19. [28]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  20. [29]

    Bowman, and Evan Hubinger

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Sam...

    2024 arXiv

  21. [30]

    An evaluation of gpt models for phe- notype concept recognition.BMC Medical Informatics and Decision Making, 24(30), 2024

    Tudor Groza, Harrison Caufield, Daniel Gration, et al. An evaluation of gpt models for phe- notype concept recognition.BMC Medical Informatics and Decision Making, 24(30), 2024. doi: 10.1186/s12911-024-02439-w. URLhttps://doi.org/10.1186/s12911-024-02439-w. 12

    2024 doi

  22. [31]

    Openclip, July 2021

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/zenodo.5143773. I...

    2021 doi

  23. [32]

    Billion-scale similarity search with GPUs.IEEE Transactions on Big Data, 7(3):535–547, 2019

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data, 7(3):535–547, 2019

    2019

  24. [33]

    Lawrence Zitnick, and Ross Girshick

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogniti...

    2017

  25. [34]

    Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668–2677. PMLR, 2018

    2018

  26. [35]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Hal Daumé III and Aarti Singh, editors,Proceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine ...

    2020

  27. [36]

    Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URLhttps://arxiv.org/abs/2408.05147

    2024 arXiv

  28. [37]

    Sparse autoencoders reveal selective remapping of visual concepts during adaptation, 2025

    Hyesu Lim, Jinho Choi, Jaegul Choo, and Steffen Schneider. Sparse autoencoders reveal selective remapping of visual concepts during adaptation, 2025. URLhttps://arxiv.org/abs/2412.05276

    2025 arXiv

  29. [38]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean Conference on Computer Vision (ECCV), 2014

    2014

  30. [39]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

  31. [40]

    Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering.ArXiv, abs/2311.06668, 2023. URL https://api.semanticscholar.org/CorpusID:265149781

    2023 arXiv

  32. [41]

    A unified approach to interpreting model predictions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. InAdvances in neural information processing systems 30, 2017

    2017

  33. [42]

    Calm: A causality-guided framework for generating local and global model explanations

    Divyanshu Mahajan, Chenhao Tan, and Matthew Turek. Calm: A causality-guided framework for generating local and global model explanations. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1215–1224, 2021

    2021

  34. [43]

    Promises and pitfalls of black-box concept learning models.ArXiv, abs/2106.13314, 2021

    Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models.ArXiv, abs/2106.13314, 2021. URL https://api.semanticscholar. org/CorpusID:235652059

    2021 arXiv

  35. [44]

    Detecting high-stakes interactions with activation probes.ArXiv, abs/2506.10805, 2025

    Alex McKenzie, Urja Pawar, Phil Blandfort, William Bankes, David Krueger, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov. Detecting high-stakes interactions with activation probes.ArXiv, abs/2506.10805, 2025. URLhttps://api.semanticscholar.org/CorpusID:279318482. 13

    2025

  36. [45]

    Linguistic regularities in continuous space word representations

    Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. InNorth American Chapter of the Association for Computational Linguistics, 2013. URL https://api.semanticscholar.org/CorpusID:7478738

    2013

  37. [46]

    Springer Nature Switzerland, 2023

    Georgii Mikriukov, Gesina Schwalbe, Christian Hellert, and Korinna Bade.Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability, page 499–524. Springer Nature Switzerland, 2023. ISBN 9783031440670. doi: 10.1007/978-3-031-44067-0_26. URL h...

    2023 doi

  38. [47]

    Large language models: A survey, 2025

    Shervin Minaee, Tomas Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, Xavier Amatriain, and Jianfeng Gao. Large language models: A survey, 2025. URL https://arxiv.org/abs/2402.06196

    2025 arXiv

  39. [48]

    Fraser, and Svetlana Kiritchenko

    Isar Nejadgholi, Esma Balkır, Kathleen C. Fraser, and Svetlana Kiritchenko. Towards procedural fairness: Uncovering biases in how a toxic language classifier uses sentiment information, 2022. URL https://arxiv.org/abs/2210.10689

    2022 arXiv

  40. [49]

    Alison Noble, and Yarin Gal

    Angus Nicolson, Lisa Schut, J. Alison Noble, and Yarin Gal. Explaining explainability: Recommenda- tions for effective use of concept activation vectors, 2025. URLhttps://arxiv.org/abs/2404.03713

    2025 arXiv

  41. [50]

    Zoom in: An introduction to circuits.Distill, 2020

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits.Distill, 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in

    2020 doi

  42. [51]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://openai.com/index/gpt-4o-system-card/ . Model documentation and safety evaluation

    2024

  43. [52]

    isarcasm: A dataset of intended sarcasm

    Silviu Oprea and Walid Magdy. isarcasm: A dataset of intended sarcasm. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020

    2020

  44. [53]

    Disentangling neuron representations with concept vectors

    Laura O’Mahony, Vincent Andrearczyk, Henning Müller, and Mara Graziani. Disentangling neuron representations with concept vectors. In2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 3770–3775, 2023. doi: 10.1109/CVPRW59228.2023.00390

    2023

  45. [54]

    MacArthur, and Christopher R

    Enrico Parisini, Tapabrata Chakraborti, Chris Harbron, Ben D. MacArthur, and Christopher R. S. Banerji. Leakage and interpretability in concept-based models, 2025. URL https://arxiv.org/abs/2504. 14094

    2025

  46. [55]

    Rise: Randomized input sampling for explanation of black-box models

    Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. InProceedings of the British Machine Vision Conference (BMVC), 2018

    2018

  47. [56]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://ar...

    2021 arXiv

  48. [57]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016

    2016

  49. [58]

    Steering llama 2 via contrastive activation addition.ArXiv, abs/2312.06681, 2023

    Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition.ArXiv, abs/2312.06681, 2023. URL https://api. semanticscholar.org/CorpusID:266174252

    2023 arXiv

  50. [59]

    Roboflow100-vl: A multi-domain object detection benchmark for vision-language models

    Peter Robicheaux, Matvei Popov, Anish Madan, Isaac Robinson, Joseph Nelson, Deva Ramanan, and Neehar Peri. Roboflow100-vl: A multi-domain object detection benchmark for vision-language models. ArXiv, abs/2505.20612, 2025. URLhttps://api.semanticscholar.org/CorpusID:278910603

    2025

  51. [60]

    A conversation with bing’s chatbot left me deeply unsettled.The New York Times

    Kevin Roose. A conversation with bing’s chatbot left me deeply unsettled.The New York Times. URL https://www.nytimes.com/2023/02/16/technology/bing-sydney-microsoft-ai-chatbot.html. 14

    2023

  52. [61]

    Friedrich

    Johannes Rückert, Asma Ben Abacha, Alba Garcia Seco de Herrera, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Henning Müller, and Christoph M. Friedrich. Overview of imageclefmedical 2023 – caption prediction and concept detection. InCLEF 2023: Conferen...

    2023

  53. [62]

    Large language models encode semantics in low-dimensional linear subspaces, 2025

    Baturay Saglam, Paul Kassianik, Blaine Nelson, Sajana Weerawardhena, Yaron Singer, and Amin Karbasi. Large language models encode semantics in low-dimensional linear subspaces, 2025. URL https://arxiv.org/abs/2507.09709

    2025

  54. [63]

    Visual-tcav: Concept-based attribution and saliency maps for post-hoc explainability in image classification.ArXiv, abs/2411.05698,

    Antonio De Santis, Riccardo Campi, Matteo Bianchi, and Marco Brambilla. Visual-tcav: Concept-based attribution and saliency maps for post-hoc explainability in image classification.ArXiv, abs/2411.05698,

    arXiv

  55. [64]

    LAION-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. L...

    2022

  56. [65]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017

    2017

  57. [66]

    Turner, and Luis Espinosa-Anke

    Zara Siddique, Liam D. Turner, and Luis Espinosa-Anke. Dialz: A python toolkit for steering vectors,

  58. [67]

    Full-gradient representation for neural network visualization

    Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. In Advances in Neural Information Processing Systems 32, 2019

    2019

  59. [68]

    Towards compositionality in concept learning.ArXiv, abs/2406.18534, 2024

    Adam Stein, Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong. Towards compositionality in concept learning.ArXiv, abs/2406.18534, 2024

    2024 arXiv

  60. [69]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017

    2017

  61. [70]

    URLhttps://arxiv.org/abs/2505.06262

    arXiv

  62. [71]

    Pooling and attention: What are effective designs for llm-based embedding models?, 2024

    Yixuan Tang and Yi Yang. Pooling and attention: What are effective designs for llm-based embedding models?, 2024. URLhttps://arxiv.org/abs/2409.02727

    2024 arXiv

  63. [72]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le L...

    2024 arXiv

  64. [73]

    Investigating task-specific prompts and sparse autoencoders for activation monitoring, 2025

    Henk Tillman and Dan Mossing. Investigating task-specific prompts and sparse autoencoders for activation monitoring, 2025. URLhttps://arxiv.org/abs/2504.20271

    2025 arXiv

  65. [74]

    From noise to narrative: Tracing the origins of hallucinations in transformers, 2025

    Praneet Suresh, Jack Stanley, Sonia Joseph, Luca Scimeca, and Danilo Bzdok. From noise to narrative: Tracing the origins of hallucinations in transformers, 2025. URL https://arxiv.org/abs/2509.06938

    2025

  66. [75]

    Disentangled representation learning,

    Xin Wang, Hong Chen, Si’ao Tang, Zihao Wu, and Wenwu Zhu. Disentangled representation learning,

  67. [76]

    Manning, and Christopher Potts

    Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outperform sparse autoencoders, 2025. URLhttps://arxiv.org/abs/2501.17148

    2025 arXiv

  68. [77]

    Discovering fine-grained visual-concept relations by disentangled optimal transport concept bottleneck models, 2025

    Yan Xie, Zequn Zeng, Hao Zhang, Yucheng Ding, Yi Wang, Zhengjue Wang, Bo Chen, and Hongwei Liu. Discovering fine-grained visual-concept relations by disentangled optimal transport concept bottleneck models, 2025. URLhttps://arxiv.org/abs/2505.07209

    2025 arXiv

  69. [78]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL https://arxiv. org/abs/2308.10248

    2024 arXiv

  70. [79]

    Unmasking and quantifying racial bias of large language models in medical report generation.Communications Medicine, 4(1), September 2024

    Yifan Yang, Xiaoyu Liu, Qiao Jin, Furong Huang, and Zhiyong Lu. Unmasking and quantifying racial bias of large language models in medical report generation.Communications Medicine, 4(1), September 2024. ISSN 2730-664X. doi: 10.1038/s43856-024-00601-z. URL http://dx.doi.org/10....

    2024 doi

  71. [80]

    URLhttps://arxiv.org/abs/2211.11695

    arXiv

  72. [81]

    Latent concept-based explanation of nlp models.ArXiv, abs/2404.12545, 2024

    Xuemin Yu, Fahim Dalvi, Nadir Durrani, and Hassan Sajjad. Latent concept-based explanation of nlp models.ArXiv, abs/2404.12545, 2024. URL https://api.semanticscholar.org/CorpusID: 269282778

    2024 arXiv

  73. [82]

    A unified joint matrix factorization framework for data integration

    Lihua Zhang and Shihua Zhang. A unified joint matrix factorization framework for data integration. ArXiv, abs/1707.08183, 2017. URLhttps://api.semanticscholar.org/CorpusID:21228616

    2017 arXiv

  74. [83]

    Uncovering safety risks of large language models through concept activation vector, 2024

    Zhihao Xu, Ruixuan Huang, Changyu Chen, and Xiting Wang. Uncovering safety risks of large language models through concept activation vector, 2024. URLhttps://arxiv.org/abs/2404.12038

    2024 arXiv

  75. [84]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models, 2025. URL https://arxiv.org/abs/2506. 05176

    2025

  76. [85]

    On completeness-aware concept-based explanations in deep neural networks.Advances in neural information processing systems, 33:20554–20565, 2020

    Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks.Advances in neural information processing systems, 33:20554–20565, 2020

    2020

  77. [86]

    El-vit: Probing vision transformer with interactive visualization, 2024

    Hong Zhou, Rui Zhang, Peifeng Lai, Chaoran Guo, Yong Wang, Zhida Sun, and Junjie Li. El-vit: Probing vision transformer with interactive visualization, 2024. URLhttps://arxiv.org/abs/2401.12666

    2024 arXiv

  78. [87]

    Frequency-based distortions in contextualized word embeddings, 2021

    Kaitlyn Zhou, Kawin Ethayarajh, and Dan Jurafsky. Frequency-based distortions in contextualized word embeddings, 2021. URLhttps://arxiv.org/abs/2104.08465

    2021 arXiv

  79. [88]

    Attributionlab: Faithfulness of feature attribution under controllable environments

    Yang Zhang, Yawei Li, Hannah Brown, Mina Rezaei, Bernd Bischl, Philip Torr, Ashkan Khakzar, and Kenji Kawaguchi. Attributionlab: Faithfulness of feature attribution under controllable environments. arXiv preprint arXiv:2310.06514, 2023. URLhttps://arxiv.org/abs/2310.06514. 16

    2023 arXiv

  80. [89]

    Zhiyu Zhu, Huaming Chen, Jiayu Zhang, Xinyi Wang, Zhibo Jin, Minhui Xue, Dongxiao Zhu, and Kim-Kwang Raymond Choo. Mfaba: A more faithful and accelerated boundary-based attribution method for deep neural networks.Proceedings of the AAAI Conference on Artificial Intelligence, 3...

    2024 doi

  81. [90]

    Interpretable basis decomposition for visual explanation

    Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Interpretable basis decomposition for visual explanation. InProceedings of the European Conference on Computer Vision (ECCV), pages 119–134, 2018

    2018

  82. [93]

    Explore spurious correlations at the concept level in language models for text classification, 2024

    Yuhang Zhou, Paiheng Xu, Xiaoyu Liu, Bang An, Wei Ai, and Furong Huang. Explore spurious correlations at the concept level in language models for text classification, 2024. URL https://arxiv. org/abs/2311.08648

    2024 arXiv

  83. [95]

    Byun, Zifan Wang, Alex Troy Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zico Kolter, and Dan Hendrycks

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Troy Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, Zi...

    2023 arXiv

  84. [96]

    Mean Prototypes[ 90]: Each concept vector is defined as the average embedding of all positive examples, vc = 1 |X +c | ∑ x∈X +c z(x)

  85. [97]

    For training, we balance positive and negative samples and use BCEWithLogitsLoss with the Adam optimizer (learning rate 0.01)

    Linear Separators (LinSep)[ 34]: For each concept c, we train a linear model (without bias) to dis- tinguish positives from negatives. For training, we balance positive and negative samples and use BCEWithLogitsLoss with the Adam optimizer (learning rate 0.01). We train for up...

  86. [98]

    The choice of k was determined experimentally using an elbow curve

    K-Means Prototypes[ 26, 15]: We cluster embeddings using FAISS GPU [32] with Euclidean distance, a maximum of 300 iterations, and k=1000 for token-level embeddings and k=50 for [CLS] embeddings. The choice of k was determined experimentally using an elbow curve. Token-level em...

  87. [99]

    Is the concept ofcpresent in the following?x

    Cluster-Based Separators (K-LinSep): We first assign soft labels to embeddings based on their K- means cluster membership, then train linear separators with the same procedure described above to predict whether an embedding belongs to a given cluster. The normal vectors of the...

    2017

  88. [100]

    Neutral Filler Machine

    Step 1: Generation.A “Neutral Filler Machine” prompt is used to generate five diverse neutral-context options embedding the original emotional comment

  89. [101]

    {emotional_comment}

    Step 2: Selection.A “Grader” prompt evaluates the five drafts and selects the best single option according to neutrality and naturalness. To increase variation, a random structure is sampled per comment: • 50% chance:[Emotion][Context] • 25% chance:[Context][Emotion] • 25% cha...

  90. [2018]

    URLhttps://arxiv.org/abs/1610.01644

    arXiv

  91. [2024]

    URLhttps://api.semanticscholar.org/CorpusID:273950563

  92. [2025]

    URLhttps://transformer-circuits.pub/2025/attribution-graphs/biology.html

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.