Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Rethinking Explainability in the Era of Multimodal AI

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that per-modality heatmaps, attention views, and circuit traces systematically misrepresent how multimodal models decide, and that the field should stop relying on them.

desk verdict A timely position paper with a real point about unimodal explanations, but the formal desiderata are vacuous as stated and cannot do the evaluative work the paper claims. read the letter →

arxiv 2506.13060 v1 pith:O2FT256R submitted 2025-06-16 cs.AI cs.LG

Chirag Agarwal This is my paper · ORCID
classification cs.AIcs.LG
keywords multimodalexplainabilityunimodalexplanationscross-modalinfluenceGrangermodalitysynergisticfaithfulnessunifiedstabilityattentionnotexplanationexplainableAI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that existing explainability tools answer the wrong question when applied to multimodal models. Heatmaps, token highlights, attention views, and circuit traces are all built for a single input stream, so they cannot represent the interaction where one modality changes the role of another. The paper's position is that such unimodal explanations can be plausible yet systematically wrong, and that the field should stop treating them as evidence about multimodal decisions. To give the field something to build toward, it proposes three properties a multimodal explanation should satisfy: modality influence, synergistic faithfulness, and unified stability. If the paper is right, the safety case for high-stakes multimodal systems depends on developing and checking explanations against these cross-modal properties.

What carries the argument

The carrying object is a formal definition of a multimodal explanation as a collection of per-modality explanation objects $E = \{E_1, \dots, E_M\}$, paired with three quantitative tests. The first test compares the change $\Delta E_m$ in the explanation for modality $m$ with the change $\Delta f_m$ in the model's output when that modality is replaced by a neutral reference, requiring $\Delta E_m \geq \alpha_m \Delta f_m - \beta_m$ for all inputs. The second test builds kept and removed versions of each modality from the explainer's own output and requires the model's prediction to stay close under kept inputs and degrade under removed inputs. The third test imposes Lipschitz-style bounds: explanations for one modality must vary slowly under semantic-preserving changes to the same modality and under small perturbations to any other modality. These definitions carry the argument because they turn the complaint that unimodal explainers miss cross-modal influence into concrete criteria that a future multimodal explainer could be checked against.

What would settle it

Take any multimodal model and a deliberately uninformative explainer that always returns constant attribution, so $\Delta E_m = 0$. Because the proposed criterion allows choosing $\beta_m$ large enough to satisfy $\Delta E_m \geq \alpha_m \Delta f_m - \beta_m$, the explainer passes the Granger test while contributing nothing; observing this on a real model would falsify the claim that the desiderata separate faithful from misleading explanations.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multimodal model's output is a joint function of its inputs, so an explanation that treats each input type separately cannot represent the cross-modal influence behind a decision. The paper states this as a categorical position: the community should stop relying on unimodal explainability techniques for multimodal models. As support, it points to cases where unimodal attributions look plausible while the model actually relies on a different modality, such as a clinical system that predicts pneumonia from the word 'cough' while ignoring a chest X-ray, and to experiments showing that gradient-based and attention-based explanations do not track what vision-language models use. The paper then formalizes what a proper multimodal explanation must do through three desiderata, each stated as a quantitative condition on a model and its per-modality explanation objects.

Load-bearing premise

The proposal assumes that inequalities with adjustable constants can actually separate good from bad explainers; as written, a bounded explainer can satisfy them by choosing a tiny multiplier and a large offset, so the tests bind only if the constants are fixed or constrained.

Editorial extensions

If this is right

  • Per-modality heatmaps, attention maps, and token highlights should no longer be accepted as explanations of a multimodal model's decision.
  • Evaluation of multimodal explainers should include ablation-based modality-influence tests, sufficiency and necessity perturbation tests, and cross-modal stability bounds.
  • Shortcut detection becomes possible in principle: a criterion that tracks ablation changes can reveal when a model ignores one modality entirely, even if each unimodal map looks plausible.
  • Future multimodal model design should incorporate explainability early, for example through explicit alignment modules or disentangled representations that expose modality contributions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: The three desiderata are declared in a modality-agnostic form, so benchmarking them on synthetic text-image, audio-text, or graph-language tasks with known modality contributions is a direct next step the paper calls for but does not execute.
  • Editorial extension: The Granger-style influence condition could be used as a screening test on existing deployed models: any explainer showing near-zero importance for a modality whose ablation changes the output would signal a hidden dependence or a broken explainer, without retraining.
  • Editorial extension: The stability bounds suggest a practical stress test for any candidate multimodal explainer using synonyms, brightness shifts, or other imperceptible perturbations; unimodal explainers would be expected to fail these bounds already at small magnitudes, which would make the paper's position empirically checkable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This position paper argues that unimodal explainability methods (saliency maps, attention visualizations, circuit traces, and similar per-modality tools) systematically misrepresent the decisions of multimodal models because they fail to capture cross-modal interactions. The authors propose three desiderata for multimodal explanations—Granger-style modality influence, synergistic faithfulness, and unified stability—formalized in Eqs. (2)-(7) of Section 3. The paper supports its position with illustrative examples (e.g., a clinical chest X-ray plus text report), a discussion of attention's non-causal nature, a critique of mechanistic interpretability tools, and a small original experiment applying logit lens to a graph-language model. It concludes with a call to stop relying on unimodal explanations in multimodal settings.

Significance. If the central claim were rigorously established, the paper would have practical importance for explainable AI, especially in high-stakes domains. The paper usefully compiles many relevant references and highlights real limitations of per-modality attributions. It also ships a small qualitative experiment (logit lens on a graph-language model), which is a concrete attempt to demonstrate the failure of a unimodal tool. However, the paper's constructive contribution depends on the formal desiderata being meaningful constraints, and as stated they are vacuous. The empirical claim of 'systematic' misrepresentation is not supported by a systematic study. Thus, the significance is contingent on substantial revision of the formalism and evidence.

major comments (4)
  1. [Section 3, Eq. (2)] The Granger Modality Influence condition ΔEm(x) ≥ αm Δfm - βm with existential αm > 0 and βm ≥ 0 is vacuous: any explainer with bounded output (including the constant zero explainer) satisfies it by setting βm = αm max_x Δfm, which makes the right-hand side non-positive. The text claims this inequality 'prevents an explainer from overstating or understating a modality's role,' but that is not true. To make the criterion meaningful, the constants must be fixed a priori (e.g., universal across explainers, or determined by the evaluation protocol) or the inequality must be replaced by a scale-invariant condition.
  2. [Section 3, Eqs. (6)-(7)] The Unified Stability conditions are similarly vacuous. In Eq. (6), the Lipschitz constant L'_m is existential; on a compact input space every Lipschitz continuous explainer satisfies the bound with some L'_m, so the condition rules out no method. In Eq. (7), the tolerance ϵ_{m→n} and Lipschitz constant L_{n,m} are also existential; one can set ϵ_{m→n}=0, making the condition trivially true, or choose L_{n,m} large enough to absorb any bounded change. As a result, the desiderata cannot serve as 'evaluation criteria for comparing explanation methods' as claimed in Section 3.
  3. [Abstract and Section 2] The central claim that unimodal explanations 'systematically misrepresent' multimodal decisions is supported only by illustrative examples (Figures 1-2), anecdotal citations (e.g., Neo et al., Casey et al., Wu et al.), and one qualitative logit-lens experiment. No quantitative evidence is provided about how frequently or severely unimodal explanations misrepresent decisions across different models, tasks, or methods. To justify the strong 'stop relying' conclusion, the paper should either report systematic experiments or explicitly frame the claim as a testable hypothesis rather than an established fact.
  4. [Section 3, Eqs. (4)-(5)] The synergistic faithfulness conditions depend on thresholds ϵ_keep_m and ϵ_remove_m, but the paper gives no procedure for setting these thresholds. Without a principled choice, the sufficiency and necessity tests are not operational: any explainer can pass or fail by adjusting the thresholds. The paper should specify how the thresholds are determined (e.g., fixed a priori, or relative to the model's output distribution) and how the resulting criteria distinguish between explainers.
minor comments (6)
  1. [Section 3, Notations] There is a typo in 'modelf is trained on'—it should be 'model f is trained on.'
  2. [Figure 4 caption] The caption refers to 'the perturbed x2 image' when the perturbation is applied to x1; this should read 'perturbed x1 image.'
  3. [Section 3, III) Unified Stability] The phrase 'that satisfies the following bounds' is ungrammatical; it should be 'that satisfy the following bounds.'
  4. [Section 3, II) Faithfulness] The opening definition of faithfulness uses the phrase 'where the irrelevant input features ... are masked or zeroed out,' which is circular unless 'irrelevant' is defined independently of the explainer; the later Keep/Remove construction resolves this, but the initial sentence should be clarified.
  5. [Figure 2 and Appendix A] The logit-lens experiment is described only qualitatively; it would be clearer if the text explained why tokens like 'bomb' and 'neighbor' are considered evidence of failure, and how the 'static across layers' observation is quantified.
  6. [References] The in-text citation 'Till, O'Neill and Bui, 2024' merges two separate references (Demian Till and Charles O'Neill with Thang Bui); this should be corrected to two distinct citations.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: a position paper whose conclusions are argued from case studies and external evidence; self-citations are non-load-bearing and the formal desiderata are proposed definitions rather than derived predictions.

full rationale

This is a position paper, not a derivation, and the circularity analysis must be calibrated accordingly. The central claim — that unimodal explanations systematically misrepresent multimodal decisions and the community should stop relying on them — is argued, not derived. It rests on worked examples (multimodal sentiment analysis, the clinical X-ray/text case, the logit-lens study on TEA-GLM trained on Cora, and the concept-score shift in Fig. 4) and on external literature (Neo et al. 2024; Kang et al. 2025; Jain and Wallace 2019; Wiegreffe and Pinter 2019; Casey et al. 2021; Alvarez-Melis and Jaakkola 2018; Wu et al. 2023). No parameter is fitted to a subset of data and then renamed as a prediction, and no result is proven from inputs that contain the conclusion. The formal desiderata in Section 3 (Eqs. 2–7) are stated definitions of desired properties — Granger-style modality influence, synergistic faithfulness, unified stability — and the paper explicitly frames them as “evaluation criteria” and “design guidelines,” with the conclusion calling the whole discussion “a conceptual foundation,” not an empirical derivation. The several self-citations (Agarwal and Nguyen 2020; Agarwal et al. 2022a,b,c, 2023, 2024; Seth et al. 2025) support background assertions about attribution methods, stability, and plausibility-versus-faithfulness; none is the unique or load-bearing support for the paper’s call to abandon unimodal explainability, which is independently motivated. The skeptic’s objection has force but is not a circularity objection: Eq. (2) (dEm >= am dfm - bm with existential am>0, bm>=0) and Eqs. (6)–(7) (existential Lipschitz and tolerance constants) can indeed be satisfied vacuously by any bounded explainer, so the proposed framework may fail to constrain anything; that is a rigor and evaluation weakness, not a case of an input definition that produces the conclusion. Per the rules that vacuity and “not standard consensus” belong to correctness rather than circularity, the circularity burden here is low. The score of 2 sits at the bottom of the 0–2 band and reflects the presence of incidental, non-load-bearing self-citations, while the steps list is empty because no circular step reduces by construction to the paper’s own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or computational entities; it introduces three normative properties. The main hidden degrees of freedom are the existential constants in the desiderata, listed as free parameters because they let any bounded explainer satisfy the formal conditions.

free parameters (3)
  • Granger constants αm, βm = unspecified, existentially quantified
    In Eq. (2), αm>0 and βm≥0 can be made arbitrarily small/large respectively, which makes the inequality satisfiable by any bounded explainer; this is a free degree of freedom that weakens the desideratum.
  • Faithfulness thresholds ϵ_keep_m, ϵ_remove_m = unspecified
    Eqs. (4)-(5) require existence of thresholds only; loose thresholds allow almost any explainer to pass sufficiency and necessity tests.
  • Stability tolerances {ϵ_{m→n}, η_m} and Lipschitz constants {L_{n,m}, L'_m} = unspecified
    Eqs. (6)-(7) quantify over tolerances and Lipschitz constants; for any bounded explanation, sufficiently large constants satisfy the bounds, so the stability requirement is trivially satisfiable.
assumptions (4)
  • domain assumption A multimodal model's behavior is not the sum of its parts and cannot be understood from per-modality explanations alone.
    This is the paper's central premise, asserted in Section 2.1 and used to justify the need for new desiderata; it is argued by example rather than proven.
  • domain assumption Replacing one modality with a neutral reference xref_m is a valid causal intervention that measures how much the model relies on that modality.
    Section 3, Eq. (2). The paper assumes ablations to a neutral reference isolate modality influence without confounding by distribution shift.
  • domain assumption Explanation objects from different modalities can be combined by set union into one composite explanation (Eq. 1).
    Section 3. This assumes compatibility and comparability of heterogeneous explanation spaces, which the paper itself notes as an open challenge.
  • standard math Lipschitz continuity of explanation maps is an appropriate formalization of explanation stability.
    Section 3, Eq. (6)-(7). This imports a standard notion, and the paper gives no justification that Lipschitz bounds capture the intended semantic stability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Explainability in the Era of Multimodal AI." pith.science (2026). https://pith.science/paper/O2FT256R

@misc{pith2026250613060,
  author       = {Pith},
  title        = {Pith review of: Rethinking Explainability in the Era of Multimodal AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O2FT256R}},
  note         = {Machine review of arXiv:2506.13060}
}
read the original abstract

While multimodal AI systems (models jointly trained on heterogeneous data types such as text, time series, graphs, and images) have become ubiquitous and achieved remarkable performance across high-stakes applications, transparent and accurate explanation algorithms are crucial for their safe deployment and ensure user trust. However, most existing explainability techniques remain unimodal, generating modality-specific feature attributions, concepts, or circuit traces in isolation and thus failing to capture cross-modal interactions. This paper argues that such unimodal explanations systematically misrepresent and fail to capture the cross-modal influence that drives multimodal model decisions, and the community should stop relying on them for interpreting multimodal models. To support our position, we outline key principles for multimodal explanations grounded in modality: Granger-style modality influence (controlled ablations to quantify how removing one modality changes the explanation for another), Synergistic faithfulness (explanations capture the model's predictive power when modalities are combined), and Unified stability (explanations remain consistent under small, cross-modal perturbations). This targeted shift to multimodal explanations will help the community uncover hidden shortcuts, mitigate modality bias, improve model reliability, and enhance safety in high-stakes settings where incomplete explanations can have serious consequences.

Figures

Figures reproduced from arXiv: 2506.13060 by the authors.

Figure 1
Figure 1. Unimodal XAI (U-XAI) vs. Multimodal XAI (M-XAI) Attributions in Clinical Decision Support. (A) A multimodal diagnostic model that processes both a patient’s clinical text report (symp￾toms, physical exam) and their chest X-ray to conclude, “The patient has Pneumonia,” but a clinician, examining the case, notes no radiographic evidence of pneumonia. (B) Applying unimodal explainabil￾ity method separately to each moda… view at source ↗
Figure 2
Figure 2. Interpreting Graph Language Models using Logit Lens [nostalgebraist, 2020]. The “logit lens” provides a simple interpretability lens, designed to analyze token predictions across layers. Here, we use it to analyze a graph language multimodal model (GLM), which projects graph embeddings into the LLM’s embedding space as specialized tokens (gi) and is typically trained for node classification and link prediction (foll… view at source ↗
Figure 3
Figure 3. Tracing and Quantifying Node Importance via Circuit Interventions in LLMs [Hanna et al., 2025]. The figure illustrates the method of circuit tracing to identify and quantify the importance of intermediate nodes in a transformer-based LLM’s reasoning process. The task involves answering the prompt: “Fact: the capital of the state containing Dallas is”, with the model initially outputting "Austin" (42%) as the top pre… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Evaluating Modal-based Stability with Concept-Based Explanations. Given a mul￾timodal captioning model and an image x1 paired with text x2, we test cross-modal stability by applying a small, output-invariant perturbation δ1 to the image. While both the original x1 and …
Figure 5
Figure 5. Figure 5: Interpreting Graph Language Models using Logit Lens. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CHARM: Charge Calibration and Acoustic Rescue for LLM-based Multimodal Sarcasm Detection

    cs.SD 2026-07 conditional novelty 6.0 of 10

    Symmetric charged prompts cancel zero-shot LLM sarcasm bias; acoustic late fusion with openSMILE and Omni probes lifts weak backbones up to +0.382 Macro-F1 across English and Chinese.

  2. Decoding the Multimodal Maze: A Systematic Review on the Adoption of Explainability in Multimodal Attention-based Models

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    A systematic review of 55 papers finds explainability for multimodal attention-based models is dominated by attention-weight visualizations, while evaluation remains mostly qualitative and non-standardized.

Reference graph

Works this paper leans on

69 extracted references · 62 canonical work pages · cited by 2 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv, 2024

    work page 2024

  2. [2]

    Explaining image classifiers by removing input features using generative models

    Chirag Agarwal and Anh Nguyen. Explaining image classifiers by removing input features using generative models. In ACCV, 2020

    work page 2020

  3. [3]

    Rethinking stability for attribution-based explanations

    Chirag Agarwal, Nari Johnson, Martin Pawelczyk, Satyapriya Krishna, Eshika Saxena, Marinka Zitnik, and Himabindu Lakkaraju. Rethinking stability for attribution-based explanations. In ICLR Workshop on PAIR 2Struct , 2022 a

    work page 2022

  4. [4]

    Openxai: Towards a transparent evaluation of model explanations

    Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. NeurIPS, 2022 b

    work page 2022

  5. [5]

    Probing gnn explainers: A rigorous theoretical and empirical analysis of gnn explanation methods

    Chirag Agarwal, Marinka Zitnik, and Himabindu Lakkaraju. Probing gnn explainers: A rigorous theoretical and empirical analysis of gnn explanation methods. In AISTATS, 2022 c

    work page 2022

  6. [6]

    Evaluating explainability for graph neural networks

    Chirag Agarwal, Owen Queen, Himabindu Lakkaraju, and Marinka Zitnik. Evaluating explainability for graph neural networks. Scientific Data, 2023

    work page 2023

  7. [7]

    Faithfulness vs

    Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. Faithfulness vs. plausibility: On the (un) reliability of explanations from large language models. arXiv, 2024

    work page 2024

  8. [8]

    On the robustness of interpretability methods

    David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv, 2018

    work page 2018

Show all 69 references
  1. [9]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  2. [10]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018

    2018

  3. [11]

    Mechanistic interpretability for ai safety--a review

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety--a review. arXiv, 2024

    2024

  4. [12]

    Interpreting clip with sparse linear concept embeddings (splice)

    Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio Calmon, and Himabindu Lakkaraju. Interpreting clip with sparse linear concept embeddings (splice). NeurIPS, 2024

    2024

  5. [13]

    A systematic review of natural language processing applied to radiology reports

    Arlene Casey, Emma Davidson, Michael Poon, Hang Dong, Daniel Duma, Andreas Grivas, Claire Grover, V \' ctor Su \'a rez-Paniagua, Richard Tobin, William Whiteley, et al. A systematic review of natural language processing applied to radiology reports. BMC Medical Informatics and...

    2021

  6. [14]

    Fleurs: Few-shot learning evaluation of universal representations of speech

    Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. Fleurs: Few-shot learning evaluation of universal representations of speech. In IEEE Spoken Language Technology Workshop, 2022

    2022

  7. [15]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv, 2023

    2023

  8. [16]

    A survey of the state of explainable AI for natural language processing

    Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. A survey of the state of explainable AI for natural language processing. In AACL, 2020

    2020

  9. [17]

    Interpretable explanations of black boxes by meaningful perturbation

    Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In ICCV, 2017

    2017

  10. [18]

    Attention in natural language processing

    Andrea Galassi, Marco Lippi, and Paolo Torroni. Attention in natural language processing. IEEE Transactions on Neural Networks and learning systems, 2020

    2020

  11. [19]

    Scaling and evaluating sparse autoencoders

    Leo Gao, Tom Dupr \'e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv, 2024

    2024

  12. [20]

    Multimodal neurons in artificial neural networks

    Gabriel Goh, Nick Cammarata †, Chelsea Voss †, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi:10.23915/distill.00030. https://distill.pub/2021/multimodal-neurons

    2021 doi

  13. [21]

    Localizing model behavior with path patching

    Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv, 2023

    2023

  14. [22]

    circuit-tracer

    Michael Hanna, Mateusz Piotrowski, Jack Lindsey, and Emmanuel Ameisen. circuit-tracer. https://github.com/safety-research/circuit-tracer, 2025. The first two authors contributed equally and are listed alphabetically

    2025

  15. [23]

    Sparse autoencoders can interpret randomly initialized transformers

    Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison. Sparse autoencoders can interpret randomly initialized transformers. arXiv, 2025

    2025

  16. [24]

    How to use and interpret activation patching

    Stefan Heimersheim and Neel Nanda. How to use and interpret activation patching. arXiv, 2024

    2024

  17. [25]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv, 2024

    2024

  18. [26]

    Attention is not explanation

    Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv, 2019

    2019

  19. [27]

    Towards better explanations of class activation mapping

    Hyungsik Jung and Youngrock Oh. Towards better explanations of class activation mapping. In ICCV, 2021

    2021

  20. [28]

    See what you are told: Visual attention sink in large multimodal models

    Seil Kang, Jinyeong Kim, Junhyeok Kim, and Seong Jae Hwang. See what you are told: Visual attention sink in large multimodal models. arXiv, 2025

    2025

  21. [29]

    Are sparse autoencoders useful? a case study in sparse probing

    Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv, 2025

    2025

  22. [30]

    Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

    Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In ICML, 2018

    2018

  23. [31]

    Visual explanations from hadamard product in multimodal deep networks

    Jin-Hwa Kim and Byoung-Tak Zhang. Visual explanations from hadamard product in multimodal deep networks. arXiv, 2017

    2017

  24. [32]

    Sparse autoencoders work on attention layer outputs

    Connor Kissane, Robert Krzyzanowski, Arthur Conmy, and Neel Nanda. Sparse autoencoders work on attention layer outputs. In Alignment Forum, 2024

    2024

  25. [33]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, 2020

    2020

  26. [34]

    Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, A...

    2025

  27. [35]

    The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025

    Llama. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, 2025. URL https://ai.meta.com/blog/llama-4-multimodal-intelligence/

    2025

  28. [36]

    Dime: Fine-grained interpretations of multimodal models via disentangled local explanations

    Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, and Louis-Philippe Morency. Dime: Fine-grained interpretations of multimodal models via disentangled local explanations. In AIES, 2022

    2022

  29. [37]

    Towards principled evaluations of sparse autoencoders for interpretability and control

    Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv, 2024

    2024

  30. [38]

    K-sparse autoencoders

    Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv, 2013

    2013

  31. [39]

    Cora Dataset , 2017

    Andrew McCallum. Cora Dataset , 2017. URL https://doi.org/10.18738/T8/HUIG48

    2017 doi

  32. [40]

    Dual attention networks for multimodal reasoning and matching

    Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In CVPR, 2017

    2017

  33. [41]

    Attribution patching: Activation patching at industrial scale

    Neel Nanda. Attribution patching: Activation patching at industrial scale. URL: https://www. neelnanda. io/mechanistic-interpretability/attribution-patching, 2023

    2023

  34. [42]

    Towards interpreting visual information processing in vision-language models

    Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. arXiv, 2024

    2024

  35. [43]

    Interpreting gpt: the logit lens — lesswrong, 2020

    nostalgebraist. Interpreting gpt: the logit lens — lesswrong, 2020. URL https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens

    2020

  36. [44]

    Sparse autoencoders enable scalable and reliable circuit identification in language models

    Charles O'Neill and Thang Bui. Sparse autoencoders enable scalable and reliable circuit identification in language models. arXiv, 2024

    2024

  37. [45]

    Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging

    Aur \'e lie Pahud de Mortanges, Haozhe Luo, Shelley Zixin Shu, Amith Kamath, Yannick Suter, Mohamed Shelan, Alexander P \"o llinger, and Mauricio Reyes. Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging. NPJ digital medic...

    2024

  38. [46]

    Multimodal explanations: Justifying decisions and pointing to the evidence

    Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In CVPR, 2018

    2018

  39. [47]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. NeurIPS, 2023

    2023

  40. [48]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In KDD, 2016

    2016

  41. [49]

    Multimodal explainable artificial intelligence: A comprehensive review of methodological advances and future research directions

    Nikolaos Rodis, Christos Sardianos, Panagiotis Radoglou-Grammatikis, Panagiotis Sarigiannidis, Iraklis Varlamis, and Georgios Th Papadopoulos. Multimodal explainable artificial intelligence: A comprehensive review of methodological advances and future research directions. IEEE...

    2024

  42. [50]

    Grad-cam: visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localization. IJCV, 2020

    2020

  43. [51]

    Towards a systematic evaluation of hallucinations in large-vision language models, 2025

    Ashish Seth, Dinesh Manocha, and Chirag Agarwal. Towards a systematic evaluation of hallucinations in large-vision language models, 2025

    2025

  44. [52]

    A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models

    Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms of large language models. arXiv, 2025

    2025

  45. [53]

    Explain and improve: Lrp-inference fine-tuning for image captioning models

    Jiamei Sun, Sebastian Lapuschkin, Wojciech Samek, and Alexander Binder. Explain and improve: Lrp-inference fine-tuning for image captioning models. Information Fusion, 2022

    2022

  46. [54]

    A review of multimodal explainable artificial intelligence: Past, present and future

    Shilin Sun, Wenbin An, Feng Tian, Fang Nan, Qidong Liu, Jun Liu, Nazaraf Shah, and Ping Chen. A review of multimodal explainable artificial intelligence: Past, present and future. arXiv, 2024

    2024

  47. [55]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017

    2017

  48. [56]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv, 2023

    2023

  49. [57]

    true features

    Demian Till. Do sparse autoencoders find" true features"? 2024. URL https://www. lesswrong. com/posts/QoR8noAB3Mp2KBA4B/do-sparse-autoencoders-find-true-features

    2024

  50. [58]

    Interpretable multi-modal hate speech detection

    Prashanth Vijayaraghavan, Hugo Larochelle, and Deb Roy. Interpretable multi-modal hate speech detection. arXiv, 2021

    2021

  51. [59]

    Llms as zero-shot graph learners: Alignment of gnn representations with llm token embeddings

    Duo Wang, Yuan Zuo, Fengzhi Li, and Junjie Wu. Llms as zero-shot graph learners: Alignment of gnn representations with llm token embeddings. NeurIPS, 2024

    2024

  52. [60]

    Interpretability-based multimodal convolutional neural networks for skin lesion diagnosis

    Sutong Wang, Yunqiang Yin, Dujuan Wang, Yanzhang Wang, and Yaochu Jin. Interpretability-based multimodal convolutional neural networks for skin lesion diagnosis. IEEE Transactions on Cybernetics, 2021 a

    2021

  53. [61]

    M2lens: Visualizing and explaining multimodal models for sentiment analysis

    Xingbo Wang, Jianben He, Zhihua Jin, Muqiao Yang, Yong Wang, and Huamin Qu. M2lens: Visualizing and explaining multimodal models for sentiment analysis. IEEE TVCG, 2021 b

    2021

  54. [62]

    Logitlens4llms: Extending logit lens analysis to modern large language models

    Zhenyu Wang. Logitlens4llms: Extending logit lens analysis to modern large language models. arXiv, 2025

    2025

  55. [63]

    Measuring cross-modal interactions in multimodal models

    Laura Wenderoth, Konstantin Hemker, Nikola Simidjievski, and Mateja Jamnik. Measuring cross-modal interactions in multimodal models. In AAAI, 2025

    2025

  56. [64]

    Attention is not not explanation

    Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In EMNLP-IJCNLP, 2019

    2019

  57. [65]

    Audio-text models do not yet leverage natural language

    Ho-Hsiang Wu, Oriol Nieto, Juan Pablo Bello, and Justin Salamon. Audio-text models do not yet leverage natural language. In ICASSP. IEEE, 2023

    2023

  58. [66]

    Visual entailment: A novel task for fine-grained image understanding

    Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv, 2019

    2019

  59. [67]

    Gnnexplainer: Generating explanations for graph neural networks

    Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. NeurIPS, 2019

    2019

  60. [68]

    Towards best practices of activation patching in language models: Metrics and methods

    Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In ICLR, 2024

    2024

  61. [69]

    Vlm ^ 2 -bench: A closer look at how well vlms implicitly link explicit matching visual cues

    Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, and Yi R Fung. Vlm ^ 2 -bench: A closer look at how well vlms implicitly link explicit matching visual cues. arXiv, 2025

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.