Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Debiasing CLIP: Interpreting and Correcting Bias in Attention Heads

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that CLIP's spurious correlations can be located in a handful of attention heads and corrected at inference time by mean-ablating spurious heads and injecting class-discriminative directions into class heads, yielding…

desk verdict The empirical recipe is promising but the central derivation in Eq. 10 is sign-inconsistent; the paper needs a major correction to its Locate step before the mechanistic story can be trusted. read the letter →

arxiv 2505.17425 v1 pith:7RRDY7EW submitted 2025-05-23 cs.CV cs.CL

Wei Jie Yeo , Rui Mao , Moloud Abdar , Erik Cambria , Ranjan Satapathy This is my paper · ORCID
classification cs.CVcs.CL
keywords CLIPdebiasingattentionheadinterpretabilityspuriouscorrelationlogitlensmeanablationknowledgeinjectionworst-groupaccuracyvisiontransformers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that CLIP's biases toward spurious attributes such as image background or gender are not diffuse across the model but are concentrated in specific attention heads, and that those heads can be found and corrected without retraining. It introduces Locate-Then-Correct (LTC), which uses per-head logit-lens scores on wrongly versus correctly classified samples from the negatively associated subgroup to isolate heads encoding the spurious attribute, and by symmetry heads encoding the target class. Mean-ablating the spurious heads and projecting class-relevant heads onto LLM-generated discriminative text directions improves worst-group accuracy by more than 50% relative to zero-shot CLIP on Waterbirds, and substantially reduces gender bias on GenderBias-VL and FairFace. The corrections are interpretable: the selected heads show high attribution to the expected attribute in both text and image visualizations.

What carries the argument

The central object is the contrastive logit-lens score per attention head. For each head, the logit lens projects the head's residual-stream contribution onto CLIP's text-embedding space to score how much that head supports a target class; the score is the difference between the logit for the target class and the logit for the contrasting class. LTC then splits the negative subgroup into correctly and wrongly classified samples, computes these per-head scores on each subset, and subtracts them so that class-relevant contributions cancel and the positive remainder isolates spurious heads (Eq. 10). A one-hot sparsification step (Eq. 7) keeps only the top contributing heads, and the framework corrects them by mean ablation and by projecting class heads onto LLM-generated discriminative text directions.

What would settle it

Run LTC's head selection on Waterbirds but ablate the same number of heads chosen by the top marginal scores $V_{NW}$ alone, without the contrastive subtraction in Eq. 10; if that control reproduces LTC's worst-group improvement, then the contrastive mechanism is not the operative ingredient. A complementary check is to inject a known spurious attribute into a synthetic subset of attention heads and verify that Eq. 10 recovers exactly those heads.

Watch

Extended reading notes

Core claim

The paper's central claim is that attention heads carrying spurious attributes can be identified by a contrastive logit-lens score, $V_S = \sigma(V_{NW} - V_{NC})$ (Eq. 10), applied to samples from the negative subgroup where the spurious attribute is mismatched with the class. Here $V_{NW}$ measures per-head class-contribution scores on wrongly classified negative samples and $V_{NC}$ on correctly classified negative samples; the subtraction cancels class-relevant contributions and leaves the spurious contribution, while swapping the terms recovers class-relevant heads. The paper then shows that mean-ablating the spurious heads and orthogonally projecting the class heads onto discriminative directions derived from LLM-generated text differences improves worst-group accuracy by over 50% relative to zero-shot CLIP, narrows the performance gap between positive and negative subgroups, and transfers across datasets and model sizes.

Load-bearing premise

The argument stands on the premise that subtracting per-head class-contribution scores of correctly classified mismatched samples from those of wrongly classified mismatched samples cancels the class component and leaves only the spurious component; if the two kinds of heads are not cleanly separable in these scores, the selected heads need not be specifically spurious.

Editorial extensions

If this is right

  • Debiasing can be performed entirely at inference time, with no gradient updates or retraining of the CLIP encoder.
  • The located heads transfer across benchmarks: spurious heads found on Waterbirds are reused on CounterAnimal, and gender heads found on GenderBias-VL transfer to FairFace.
  • LTC composes with parameter-tuning methods: applying it on top of JTT further improves worst-group accuracy and reduces the subgroup gap beyond JTT alone.
  • Targeted head-level corrections outperform whole-representation projection baselines such as Roboshot and Ortho-Cali on the tested benchmarks, and are more robust to variation in LLM-generated discriminative features.
  • The identified heads are interpretable: text and image attribution show that class heads encode class features, spurious heads encode the spurious attribute, and a third set of heads encodes the association between the two.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same contrastive logit-lens score could serve as a bias audit for other decomposition-friendly transformers, including language models, since it only requires per-component logit-lens readings and a split of correctly and wrongly classified samples.
  • Editorial inference: the paper's finding that background-bias heads encode an association between spurious and target attributes rather than the spurious attribute alone suggests that ablating those heads may erase useful context knowledge; a testable alternative is to steer or edit the association heads instead of deleting them.
  • Editorial caution: because head selection uses the negative-group split, the reported gains could in principle come from removing any high-impact head; a control experiment ablating heads ranked by marginal logit-lens score without the contrastive subtraction would tell whether the contrast is the operative ingredient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Locate-Then-Correct (LTC), a training-free method to debias CLIP at the level of individual attention heads. The method decomposes the image representation into attention-head contributions (Sec. 3.2), then uses a contrastive logit-lens score (Eq. 10) to locate 'spurious' heads that activate strongly on wrongly classified negative-group samples relative to correctly classified ones, and 'target' heads that encode class information. Debiasing is done by mean-ablating spurious heads and projecting target heads onto LLM-generated class-discriminative directions (knowledge injection). The authors report improvements in worst-group accuracy on Waterbirds, CounterAnimal, GenderBias-VL, and FairFace over non-training baselines, plus interpretability analyses (SHAP, TextSpan, heatmaps) to support the mechanistic claim.

Significance. The idea of performing debiasing via localized attention-head intervention is timely and potentially practical: it requires no fine-tuning, is transparent, and the code is released. The empirical gains on Waterbirds and GenderBias (e.g., ViT-B/16 WG 49.7→73.3, BT 74.0→10.0) are substantial if they hold under a corrected derivation. However, the central Locate step is presented with a sign-inconsistent formula that, taken literally, cannot select any heads; the reported results therefore currently rest on an unspecified implementation. The paper also does not fully address the circularity of selecting heads from the same error split used for evaluation. Because the mechanistic explanation is the paper's main novelty, these issues are load-bearing.

major comments (4)
  1. [Sec. 4.2, Eq. (10)] The selection formula in Eq. (10) is inconsistent with the paper's own definitions. Under Def. 4.1 and the text in Sec. 4.2, for GN we have asy = -1, so V = VY - VS. The paper states that for correctly classified GN samples (NC), VY_NC > 0 and VS_NC < 0, and for wrongly classified GN samples (NW), VY_NW < 0 and VS_NW > 0; hence V_NC = VY_NC - VS_NC > 0 and V_NW = VY_NW - VS_NW < 0. Then V_NW - V_NC is strictly negative for every head, so the mask σ(V) = 1(V > γ) with γ = 1/|P*_GNW ∪ P*_GNC| > 0 returns the empty set. The heads listed later (e.g., L10H10 for ZSY in Table 13) cannot be produced by Eq. (10) as written. Replacing it by V_NC - V_NW makes the expression positive but then it equals (VY_NC - VY_NW) + (VS_NW - VS_NC), a sum of two positive terms that does not isolate VS. The derivation must be corrected and the actual implemented formula stated; as it stands, the mechanistic justification for the selected heads does not follow from the math.
  2. [Sec. 4.2, Def. 4.1 and Sec. 5.1] The contrastive selection in Eq. (10) is built from the same wrong/correct split on GN that defines the success metric (worst-group accuracy). A head whose logit-lens score is high when the model errs on GN will be selected regardless of whether it encodes the spurious attribute (e.g., background) rather than any other error-inducing feature. The interpretation evidence in Sec. 6 (SHAP, TextSpan) is qualitative and shown for only a handful of heads; it does not quantify how often the located heads coincide with attribute-encoding heads across the three backbones and two bias types. Moreover, the CounterAnimal evaluation in Sec. 5.1 reuses ZS from Waterbirds rather than locating CounterAnimal-specific spurious heads, so it does not test the locator's generalization. Please provide a validation that breaks this circularity, for example by showing that the same heads are selected when the wrong/correct split is computed on a different spurious attribute, or by comparing the located heads against a null distribution of equally important random heads.
  3. [Sec. 4.2, paragraph after Eq. (10)] The paragraph after Eq. (10) states that because V_NW and V_NC are 'normalized to 1', one obtains VS|NW > VS|NC and VY|NW < VY|NC, which is used to claim that the difference isolates VS. The normalization is not defined (per-head, per-layer, or across the head matrix?), and no derivation is given for why normalization cancels the VY terms. If the normalization is across heads (summing to 1), then the per-head values are scaled by dataset-specific constants, and the subtraction V_NW - V_NC still mixes VY and VS differences under the sign conventions above. Please provide the explicit algebra or an empirical check (e.g., ground-truth spurious labels) that the selected heads' scores track the attribute rather than the error status.
  4. [Sec. 4, first paragraph; Eq. (8)] The decomposition in Eq. (8) assumes that contributions from MLP layers and heads outside P* are 'minimal' (epsilon), citing Gandelsman et al. However, the paper does not quantify the residual epsilon for the OpenCLIP ViT-B/L/H models used, even though the corrective edits are applied only to attention heads and then summed to form EI(I). If MLP contributions are non-negligible, the headline worst-group improvements cannot be attributed to the head-level intervention as cleanly as claimed. Please report the fraction of the final representation norm contributed by all attention heads versus MLPs, and the fraction covered by the selected heads.
minor comments (6)
  1. [Abstract] The phrase 'achieving over a >50% gain' is redundant and ambiguous; specify whether the gain is relative or absolute and clarify the comparison baseline.
  2. [Sec. 5.1, CounterAnimal] For CounterAnimal, the 'gap' is defined between easy and hard subsets rather than between positive and negative groups; state this explicitly in the main text and clarify why Worst-Group accuracy is not reported for this dataset.
  3. [Fig. 10 caption] The caption states 'x refers to ZS', but the text describes dashed lines for top-state selection; please describe what the x-markers and dashed lines represent in both Fig. 10 and Fig. 11.
  4. [Sec. 6, Spurious Association] The instruction to 'change Eq. 10 to σ(VNC)' is not a well-formed replacement; specify the exact formula used to locate ZS when predicting the spurious attribute S.
  5. [Eq. (6)] The notation (y) for the alternate class is used without definition; introduce a symbol such as y' and define it before Eq. (6).
  6. [Table 13] The TextSpan statements for individual heads are informative but the method for obtaining them for specific heads is not described; add a pointer to the TextSpan procedure or an appendix for reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. 10's 'isolation' of spurious heads is self-definitional and, under the paper's own sign conventions, selects no heads.

  1. self definitional [Section 4.2, Eq. 10 and Def. 4.1]
    "We formulate the contrastive solution to isolate VS as: VS = σ(V_NW− V_NC), ... In Eq. 6, V_NW < 0 since (y)=y∗ ≠ ˆy, and vice versa for V_NC > 0. Referencing Eq. 8, this leaves V_NW= VY − VS + ϵ <0, and V_NC= VY − VS + ϵ >0 as asy = −1. This leads to VS > VY for V_NW and opposite for V_NC. Additionally, both V_NW and V_NC are normalized to 1, leading to VS|N W> VS|N Cand VY |N W< VY |N C. Thus, Eq. 10 isolates the positive VS terms while filtering out the negative VY terms."

    Def. 4.1 already defines VS as the contribution that is negative in correctly classified GN samples and positive in wrongly classified GN samples. Eq. 10 then 'locates' spurious heads by exactly this wrong-vs-correct GN contrast, so the identification is a restatement of the definition. Worse, the paper's own inequalities imply V_NW − V_NC = (VY_NW−VY_NC) + (VS_NC−VS_NW) < 0 for every head, because VY_NW<0<VY_NC and VS_NC<0<VS_NW; hence σ(V>γ) with γ≥0 selects no heads, and the published heads (L10H10 for ZSY) cannot arise from Eq. 10.

full rationale

The experimental core of LTC (mean-ablating selected heads and injecting class directions) is evaluated against external baselines on held-out test sets; those WG gains are not logically forced by the head-selection rule, because test accuracy is computed on data not used to select PS/PY. The linear-decomposition backbone is cited from external work (Elhage et al. 2021; Gandelsman et al. 2023), and the self-citation Yeo et al. 2024 is not load-bearing. The circularity lies upstream in the localization derivation. Def 4.1 defines VS by its sign pattern on correctly/wrongly classified GN samples, and Eq.10 'locates' spurious heads by the same wrong-vs-correct GN contrast, so the mechanistic 'discovery' that the selected heads encode spurious attributes is a restatement of the definition rather than an independent derivation. Moreover, the paper's own inequalities V_NW<0<V_NC imply V_NW−V_NC<0 for every head, making σ(V>0) empty as written; the published heads (e.g., L10H10) therefore cannot be produced by Eq.10. Any sign-corrected version (V_NC−V_NW) still mixes VY and VS differences unless one assumes the VY cancellation that the derivation is supposed to establish. The SHAP and TextSpan checks do provide some independent evidence about the content of the selected heads, so the identifiability claim is not wholly tautological; but the load-bearing algebraic step that purports to isolate VS from VY is, on the paper's own equations, either empty or an assumption in disguise. Score 6 reflects this partial by-construction reduction of the central mechanism.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The method's central claims rest mostly on known linear-decomposition mathematics plus a paper-specific mixture model and threshold heuristics. The head-level corrections are not derived from the data alone; they assume a clean Y/S separation that the paper posits rather than proves.

free parameters (4)
  • mask threshold gamma = 1 / |P*_GNW union P*_GNC| (formula, not tuned)
    Threshold in Sec.4.2 decides which heads count as spurious or class-relevant; the authors state no cross-validation was used, but the threshold is data-dependent and arbitrary.
  • number of LLM text features per class (Nu) = not reported
    Knowledge injection in Sec.4.3 adds Nu text-difference directions per class; Nu and the GPT4-o prompt affect KI strength and are not specified.
  • CounterAnimal counterfactual class mapping = top-1 misclassified class per CA class, from the easy subset
    Appendix A.1 optimizes the y:y pairing on DE and uses pseudo-labels; this mapping is needed for KI and is not part of the core contrastive selection.
  • worst-group size for GenderBias = top 10 occupations
    Section 5.1 defines GN as the 10 most biased occupations in zero-shot inference; the magnitude of reported bias reduction depends on this choice.
assumptions (5)
  • domain assumption Attention head outputs add linearly to the final image representation (Eq.3-4).
    Taken from Elhage et al. and Gandelsman et al.; exact for the residual stream but only approximate for CLIP after projection and skip connections; the paper relies on it for head-level edits.
  • domain assumption MLP contributions and non-selected heads are small enough to ignore (epsilon in Eq.8).
    Section 4 states MLP layers have limited direct impact; the sparsity of V via one-hot argmax assumes most heads contribute noise.
  • ad hoc to paper V(y) decomposes as VY + asy VS + epsilon with separate target and spurious state sets.
    Eq.8 and Def.4.1 postulate the mixture and sign behavior; this is the core modeling assumption and is not derived.
  • ad hoc to paper Normalized VNW and VNC cancel VY such that VNW - VNC isolates VS.
    Section 4.2 asserts 'both VNW and VNC are normalized to 1' and concludes the difference isolates spurious terms; no proof is given.
  • domain assumption LLM-generated visual difference directions are valid and unbiased class-discriminative features.
    Section 4.3 uses GPT4-o to produce text features for projection; the paper does not measure the fidelity of these directions.
invented entities (1)
  • ZSY (spurious-association state)
    purpose: Represents attention states that encode the association between spurious attribute S and class Y, rather than S alone; used to explain why ablating ZS but not ZSY removes background-class correlation while preserving background classification.
    Introduced in Sec.6 based on SHAP and ablation curves within the paper; no external or falsifiable handle is provided, so it remains a paper-internal construct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Debiasing CLIP: Interpreting and Correcting Bias in Attention Heads." pith.science (2026). https://pith.science/paper/7RRDY7EW

@misc{pith2026250517425,
  author       = {Pith},
  title        = {Pith review of: Debiasing CLIP: Interpreting and Correcting Bias in Attention Heads},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RRDY7EW}},
  note         = {Machine review of arXiv:2505.17425}
}
abstract

Multimodal models like CLIP have gained significant attention due to their remarkable zero-shot performance across various tasks. However, studies have revealed that CLIP can inadvertently learn spurious associations between target variables and confounding factors. To address this, we introduce \textsc{Locate-Then-Correct} (LTC), a contrastive framework that identifies spurious attention heads in Vision Transformers via mechanistic insights and mitigates them through targeted ablation. Furthermore, LTC identifies salient, task-relevant attention heads, enabling the integration of discriminative features through orthogonal projection to improve classification performance. We evaluate LTC on benchmarks with inherent background and gender biases, achieving over a $>50\%$ gain in worst-group accuracy compared to non-training post-hoc baselines. Additionally, we visualize the representation of selected heads and find that the presented interpretation corroborates our contrastive mechanism for identifying both spurious and salient attention heads. Code available at https://github.com/wj210/CLIP_LTC.

Figures

Figures reproduced from arXiv: 2505.17425 by the authors.

Figure 1
Figure 1. Left: Linear decomposition of image representations into individual attention head activations [Elhage et al., 2021]. Spurious states (background: land) activate more strongly on images with opposing attributes, whereas target states (class: waterbird) activate on images with matching attributes. Right: LTC identifies and corrects these states: mean-ablation mitigates spurious states, while knowledge injection enhan… view at source ↗
Figure 2
Figure 2. Image visualization: Localized representations of [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Normalized SHAP values for Y and S. Text Interpretation. TextSpan grounds visual repre￾sentations onto generic text statements, which are not task-specific. Instead, we use GPT4-o to generate cap￾tions c for each image and apply SHAP [Lundberg, 2017] to the identified states ZS, ZSY , ZY . SHAP assigns an importance score ϕ to each text token in the caption. The prediction logit is the similarity score between the s… view at source ↗
Figures from the paper (25 more)
Figure 4
Figure 4. Figure 4: Z_S: Ablating states encoding S, Z_SY: Association between S and Y . TS: TextSpan [Left]: Predicting S as the target class. [Right]: Predicting both S and Y . Dataset: Waterbirds [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Prediction margins in Waterbirds. Model: ViT-B/16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Prediction margins in Waterbirds. Model: ViT-L/14 16 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Prediction margins in Waterbirds. Model: ViT-H/14 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Qualitative analysis on gender bias of top 10 occupations vs proportion of male workforce. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Accuracy of the occupation-dominated gender of each occupation. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Analysis on % sub-group samples for locating spurious and classification states against performance. Left: ViT-B/16, Middle: ViT-L/14 Right: ViT-H/14. Dash lines refer to selecting the top contributing state in ZY and ZSY . x refers to ZS. Metric: Accuracy (↑). Datase…
Figure 11
Figure 11. Figure 11: Analysis on % sub-group samples for locating spurious and classification states against performance. Dash lines refer to selecting the top contributing state in ZY and ZSY . x refers to ZS. Metric: Bias (↓). Dataset: Genderbias [PITH_FULL_IMAGE:figures/full_fig_p020_…
Figure 12
Figure 12. Figure 12: , 15 and 18 refers to the target class Y ’s contribution distribution in Waterbirds. The top and bottom heatmaps correspond to samples in GNC and GNW [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: V scores across the last 4 layers and all heads for ViT-B/16. Layer-wise in the y-axis and head-wise in x-axis. Top: VNC , Bottom: VNW . Dataset: Genderbias 21 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: VC scores across the last 4 layers and all head for predicting the spurious attribute: background directly for ViT-B/16. Layer-wise in the y-axis and head-wise in x-axis. Dataset: Waterbirds [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: V scores across the last 4 layers and all heads for ViT-L/14. Layer-wise in the y-axis and head-wise in x-axis. Top: VNC , Bottom: VNW . Dataset: Waterbirds 22 [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: V scores across the last 4 layers and all heads for ViT-L/14. Layer-wise in the y-axis and head-wise in x-axis. Top: VNC , Bottom: VNW . Dataset: Genderbias [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: VC scores across the last 4 layers and all head for predicting the spurious attribute: background directly for ViT-L/14. Layer-wise in the y-axis and head-wise in x-axis. Dataset: Waterbirds 23 [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: V scores across the last 8 layers and all heads for ViT-H/14. Layer-wise in the y-axis and head-wise in x-axis. Top: VNC , Bottom: VNW . Dataset: Waterbirds [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: V scores across the last 8 layers and all heads for ViT-H/14. Layer-wise in the y-axis and head-wise in x-axis. Top: VNC , Bottom: VNW . Dataset: Genderbias 24 [PITH_FULL_IMAGE:figures/full_fig_p024_19.png]
Figure 20
Figure 20. Figure 20: VC scores across the last 4 layers and all head for predicting the spurious attribute: background directly for ViT-H/14. Layer-wise in the y-axis and head-wise in x-axis. Dataset: Waterbirds 25 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Normalized SHAP values towards text features belonging to [PITH_FULL_IMAGE:figures/full_fig_p027_21.png]
Figure 22
Figure 22. Figure 22: Image visualization: Localized representations of [PITH_FULL_IMAGE:figures/full_fig_p030_22.png]
Figure 23
Figure 23. Figure 23: Image visualization: Localized representations of [PITH_FULL_IMAGE:figures/full_fig_p031_23.png]
Figure 24
Figure 24. Figure 24: Image visualization: Localized representations of [PITH_FULL_IMAGE:figures/full_fig_p032_24.png]
Figure 25
Figure 25. Figure 25: Normalized SHAP values towards text features belonging to [PITH_FULL_IMAGE:figures/full_fig_p033_25.png]
Figure 26
Figure 26. Figure 26: Image visualization: Localized representations of [PITH_FULL_IMAGE:figures/full_fig_p035_26.png]
Figure 27
Figure 27. Figure 27: Image visualization: Localized representations of [PITH_FULL_IMAGE:figures/full_fig_p036_27.png]
Figure 28
Figure 28. Figure 28: Image visualization: Localized representations of [PITH_FULL_IMAGE:figures/full_fig_p037_28.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Multifaceted Analysis of Social Biases in Large Language Models

    cs.CY 2025-12 conditional novelty 4.0 of 10

    Four widely used LLMs exhibit distinct, measurable political, ideological, alliance, language, and gender biases across five probing tasks.

Reference graph

Works this paper leans on

27 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    Zero-shot robustification of zero-shot models with foundation models

    Dyah Adila, Changho Shin, Linrong Cai, and Frederic Sala. Zero-shot robustification of zero-shot models with foundation models. arXiv preprint arXiv:2309.04344,

    arXiv

  2. [4]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee,

    2009

  3. [9]

    https://distill.pub/2021/multimodal-neurons

    doi: 10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons. Siobhan Mackenzie Hall, Fernanda Gonçalves Abrantes, Hanwen Zhu, Grace Sodunke, Aleksandar Shtedritski, and Hannah Rose Kirk. Visogender: A dataset for benchmarking gender bias in image-text pronoun resolution. Advances in Neural Information Processing Systems, 36,

    doi:10.23915/distill.00030 2021

  4. [10]

    MiniCPM: Unveiling the potential of small language models with scalable training strategies

    10 Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395,

    arXiv

  5. [11]

    URL https://doi.org/10.5281/zenodo. 5143773. Nick Jiang, Anish Kachinthaya, Suzie Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. arXiv preprint arXiv:2410.02762,

    arXiv

  6. [12]

    Fairface: Face attribute dataset for balanced race, gender, and age

    Kimmo Kärkkäinen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908.04913,

    arXiv 1908

  7. [14]

    Visual classification via description from large language models

    Sachit Menon and Carl V ondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183,

    arXiv

  8. [15]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217,

    arXiv

Show all 27 references
  1. [16]

    Accessed: 2024-11-27

    URL https://www.lesswrong.com/ posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens . Accessed: 2024-11-27. Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill,

    2024

  2. [18]

    Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization

    Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731,

    1911 arXiv

  3. [19]

    Causal mediation analysis for interpreting neural NLP: The case of gender bias

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural NLP: The case of gender bias. arXiv preprint arXiv:2004.12265,

    2004 arXiv

  4. [20]

    GenderBias-VL: Benchmarking gender bias in vision language models via counterfactual probing

    Yisong Xiao, Aishan Liu, QianJia Cheng, Zhenfei Yin, Siyuan Liang, Jiapeng Li, Jing Shao, Xi- anglong Liu, and Dacheng Tao. GenderBias-VL: Benchmarking gender bias in vision language models via counterfactual probing. arXiv preprint arXiv:2407.00600,

    arXiv

  5. [21]

    Mitigating spurious correla- tions in multi-modal models during fine-tuning

    Yu Yang, Besmira Nushi, Hamid Palangi, and Baharan Mirzasoleiman. Mitigating spurious correla- tions in multi-modal models during fine-tuning. In International Conference on Machine Learning, pages 39365–39379. PMLR, 2023a. Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel...

    arXiv

  6. [23]

    visual" with

    We adopt a 2-layer non-linear probe as the classifier for ERM, JTT and JTT-LTC, with the hidden layer dimension as128 for ViT-B/16 and256 for ViT-L/14 and ViT-H/14. Cont Adapter uses a 2-layer adapter instead. We ran the Cont Adapter with the original hyperparameters and only ...

    2023

  7. [24]

    B.4 Spurious effects in Negative sub-group The correct discovery and distinction between ZY and ZS/ZSY are hinged on the belief that the two can be separated more accurately in GN

    Overall, we find that using the mask effectively finds states most probable for encoding the respective representations. B.4 Spurious effects in Negative sub-group The correct discovery and distinction between ZY and ZS/ZSY are hinged on the belief that the two can be separate...

  8. [25]

    features

    Fig. 21 displays the normalized SHAP values for the set of aggregated states representing {SY, S, Y} for Waterbirds or {S, Y} for Genderbias. Referencing the prompt, we form Y from “features” and the annotated “species” of the bird for Waterbirds and occupation class for Gende...

    2023

  9. [26]

    Left: ViT-B/16, Middle: ViT-L/14, Right: ViT-H/14Dataset: Waterbirds Table 11: Top text features for ZSY , ZY , ZS

    Figure 21: Normalized SHAP values towards text features belonging to Y and S. Left: ViT-B/16, Middle: ViT-L/14, Right: ViT-H/14Dataset: Waterbirds Table 11: Top text features for ZSY , ZY , ZS. Dataset: Waterbirds. Model: ViT-B/16 and ViT-L/14 ViT-B/16 ViT-L/14 ZSY ZY ZS ZSY Z...

    2023

  10. [27]

    bioreactor

    Dataset: Waterbirds Model Top Localized States ViT-B/16 ZSY : L10H10 ZY : L11H5 ZS: L11H6 Tranquil boating on a lake Photo of a reptile Photo taken in Namib Desert Peaceful rural farmland Image with a seagull Photo taken in the Alaskan mountains Serene garden pond An image wit...

    2023

  11. [1000]

    polar bear

    The multi-class nature of this task introduces a unique challenge: determining the appropriate counterfactual label y for a given class label y. Unlike binary classification, where a single y, y pairing suffices for KI, multi-class tasks involve multiple pairings for each clas...

    2023

  12. [2009]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    2010 arXiv

  13. [2015]

    A unified approach to interpreting model predictions

    Scott Lundberg. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874,

    arXiv

  14. [2017]

    https://distill.pub/2017/feature-visualization

    doi: 10.23915/distill.00007. https://distill.pub/2017/feature-visualization. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natu...

    2017 doi

  15. [2020]

    Debiasing vision-language models via biased prompts

    Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka. Debiasing vision-language models via biased prompts. arXiv preprint arXiv:2302.00070,

    arXiv

  16. [2021]

    https://transformer-circuits.pub/2021/framework/index.html. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Mart...

    2021

  17. [2022]

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian

    https://transformer-circuits.pub/2022/toy_ model/index.html. Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. Advances in Neural Information Processing Systems, 36,

    2022

  18. [2023]

    CLIP the bias: How useful is balancing data in multimodal learning? arXiv preprint arXiv:2403.04547,

    Ibrahim Alabdulmohsin, Xiao Wang, Andreas Steiner, Priya Goyal, Alexander D’Amour, and Xiaohua Zhai. CLIP the bias: How useful is balancing data in multimodal learning? arXiv preprint arXiv:2403.04547,

    arXiv

  19. [2024]

    Interpreting CLIP’s image representation via text-based decomposition

    Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting CLIP’s image representation via text-based decomposition. arXiv preprint arXiv:2310.05916,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.