Pith. sign in

REVIEW 3 major objections 6 minor 104 references

Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that feature, data, and component attribution are the same three techniques aimed at different aspects, and that a unified view would let the three interpretability communities trade methods and results.

desk verdict Useful cross-attribution taxonomy and a plausible unification argument that overstates its coverage; deserves a serious referee with revisions. read the letter →

arxiv 2501.18887 v3 pith:SA2NGJYG submitted 2025-01-31 cs.LG cs.AI

Shichang Zhang , Tessa Han , Usha Bhalla , Himabindu Lakkaraju This is my paper · ORCID
classification cs.LGcs.AI
keywords attributionmethodsfeaturedatacomponentmechanisticinterpretabilityexplainableAIdata-centricunification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that three apparently separate families of AI interpretability—feature attribution, which explains predictions via input features; data attribution, which traces predictions back to training points; and component attribution, which localizes behavior to internal neurons or layers—are fundamentally the same enterprise. Its central claim is that these methods differ primarily in their perspectives rather than techniques: all three assign scores to aspects of the model through perturbations, gradients, or linear approximations, with the only real difference being which aspect is scored. The paper backs this with a taxonomy mapping representative methods from each community onto the same three technique columns, and it points out that the empty cells in this table are research opportunities: techniques proven in one attribution type can be transplanted into another. It also argues that the three communities share concepts such as noise smoothing and path tracing, plus common evaluation criteria, and that a unified view advances model editing, steering, and regulation. If the position holds, attribution research stops being three vocabularies and becomes one field with exchangeable results.

What carries the argument

The load-bearing machinery is a three-column classification of attribution methods, summarized in Table 1, in which every method is placed into one of three technique categories—perturbation, gradient, or linear approximation—across three attribution targets (features, data, components). The formal anchor is the shared attribution function $g$: each problem asks for a score $\phi_i(x)$, $\psi_j(x)$, or $\gamma_k(x)$ measuring how an aspect contributes to $f(x)$. A secondary mechanism is the local function approximation (LFA) framework, which the paper cites to show that at least eight feature attribution methods are instances of fitting an interpretable model $g$ over a local neighborhood with a specific loss $\ell$; the paper then hypothesizes that the same function-approximation lens may unify data and component attribution. The taxonomy does the work: it converts the unification claim into a checkable table of methods, and the empty cells in the table become the paper's proposed research agenda.

What would settle it

Compile a comprehensive list of attribution methods from all three communities and check whether each falls cleanly into perturbation, gradient, or linear-approximation categories; if a substantial fraction (sparse autoencoder features, simulation-based data attribution, reinforcement-learning data valuation) resist this classification and have no counterpart in the other communities, the claim that the three types differ only in perspective fails.

Watch

Extended reading notes

Core claim

The central discovery claimed by the paper is that feature, data, and component attribution are unified: for a model $f$ with input $x$, training data $D_{\mathrm{train}}$, and components $c$, all three problems seek an attribution function $g$ that assigns scores to a chosen set of aspects—features $x_i$, training points $x^{(j)}$, or components $c_k$—for the same test output $f(x)$. The paper demonstrates that popular methods in each community fall into the same three technique categories: perturbation-based methods that modify the aspect and measure output change; gradient-based methods that use first- or second-order derivatives of the loss or output; and linear approximation methods that fit a linear surrogate model. It further asserts that these categories capture the majority of representative methods, that the communities already share high-level concepts such as smoothing and path tracing, and that they evaluate attributions with the same counterfactual, task-specific, and human criteria. On this basis the paper claims the fragmentation of the field is a matter of perspective, not substance.

Load-bearing premise

The position rests on the premise that the three technique categories—perturbation, gradient, and linear approximation—cover the majority of representative methods in each attribution community, and the paper does not define that universe or quantify coverage; if the methods it excludes (sparse autoencoders, simulation-based and reinforcement-learning data attribution) turn out to be central rather than marginal, the unification would cover only a curated subset.

Editorial extensions

If this is right

  • Methods developed for one attribution type can be transplanted to the other two; for example, second-order Hessian techniques used in feature and data attribution could be applied to component attribution.
  • A common evaluation framework—counterfactual, task-specific, and human—can be used to compare attributions across the three types, reducing the fragmentation of metrics.
  • Model editing and steering can combine the localization power of component attribution with data attribution's ability to select corrective training points and feature attribution's identification of spurious patterns.
  • Shared challenges such as computational cost, inconsistency, and evaluation fairness can be attacked once, with solutions transferring across attribution types.
  • Empty cells in the unified table become concrete research directions, such as applying advanced game-theoretic notions to data and component attribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable consequence the paper leaves implicit: if the unification holds, attribution benchmarks could be merged across communities, letting a data attribution method be scored with the same counterfactual fidelity metrics used for feature attribution.
  • The paper's hypothesis that data and component attribution are also local function approximation suggests a derivation: work out the equivalent of its Table 3 (neighborhood–loss pairs) for representative DA and CA methods, which would confirm or bound the unification.
  • The empty cells in the unified table double as predictions: a technique that works in one attribution type should have a working analogue in the other two; where the analogue fails, the difference may be more than perspective.
  • A regulatory reading follows naturally: if the three attribution types are one technique family, transparency obligations such as the right to explanation could be met by a single unified attribution standard rather than separate ones per community, though the paper does not develop this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This position paper argues that feature attribution (FA), data attribution (DA), and component attribution (CA) are unified: they all seek an attribution function over different aspects, and their dominant methods share three techniques—perturbations, gradients, and linear approximations. The paper formalizes the three attribution problems in §2, classifies representative methods in Table 1, discusses shared concepts and evaluation criteria in §3.4, proposes research directions for interpretability and broader AI in §4, and addresses alternative views in §5. The main evidence is a curated taxonomy rather than a formal equivalence, and the paper explicitly acknowledges several method families that fall outside the taxonomy.

Significance. The paper's organizational contribution is real: Table 1 offers a compact map of a fragmented literature, and the explicit discussion of alternative views and exceptions is a strength. The local-function-approximation framework for FA is already established (Han et al., 2022) and is accurately summarized. If the coverage claim could be quantified, the paper would provide a useful bridge for cross-community transfer, with concrete payoffs in model editing, steering, and regulation. However, as submitted, the central claim rests on an unmeasured representativeness assertion and on category assignments that are not fully operationalized.

major comments (3)
  1. [§5 and Appendix E.3] The sentence in §5 that the three-way categorization 'captures the majority of representative methods' is not operationalized, and the manuscript's own appendices list substantial method families outside the taxonomy: tree-based surrogates for FA (Appendix C.5), simulation-based GPTfluence and RL-based DVRL for DA (Appendix D.6), and SAE-based dictionary learning for CA (Appendix E.3). SAEs are a dominant approach in current mechanistic interpretability, and the paper's exclusion is not clean: Appendix E.3 says SAEs are not strictly component attribution because they learn new components, yet the included method Sparse Feature Circuits (Marks et al., 2024) builds directly on SAE-discovered features. The paper should define the universe of attribution methods, report coverage quantitatively, and justify why the excluded families are peripheral; otherwise the abstract's claim that FA, DA, and CA 'differ primarily in perspectives rather than techniques' describes a curated subset rather than attribution methods as a whole.
  2. [Table 1 and §4.1.2] The assignment of each method to a single 'core technique' needs a stated criterion. Several methods are multi-technique by the paper's own account: LIME is presented as both perturbation-based and linear approximation (§3.1), KernelSHAP is game-theoretic perturbation but also connected to linear approximation (Appendix C.2), and Datamodel/TRAK are described as linear approximations that can be viewed as perturbation-based (§3.2 and Appendix D.5). Without an operational rule, such as 'the technique whose output directly determines the attribution score' or 'the formal class in which the estimator is derived', the cells in Table 1 are not robust, and the empty-cell transfer agenda in §4.1.2 inherits this ambiguity.
  3. [§2] The formal commonality that FA, DA, and CA 'all seek an attribution function g that assigns scores to specific aspects' is too weak to support the unification claim, since any scoring function over features, data points, or components satisfies it. The paper should either state a stronger formal condition, such as all three being instances of counterfactual or local approximation over a common mathematical structure, or explicitly label the formalization as a perspective rather than a theorem. This matters because the later claims of shared evaluation criteria and theoretical transfer in §3.4 and §4.1.2 lean on this formal framing.
minor comments (6)
  1. [Notation (§2, §3.1.1)] The symbol g denotes the attribution function in §2 and also the linear model in the LFA framework; this overload is confusing and should be clarified, for example by writing g_attrib and g_lin.
  2. [Appendix D] The first paragraph of Appendix D contains the typo 'suprisingly' and should read 'surprisingly'.
  3. [Appendix E.1] The sentence 'ACDC is tries to find a subnetwork' is ungrammatical and should read 'ACDC tries to find a subnetwork'.
  4. [Table 1] The Perturbation/Direct row lists many methods without clear substructure; consider grouping entries by citation or by subfamily to improve readability.
  5. [§3.2 and Table 1] The text attributes GradDot and GradCos to Charpiat et al. [2019], while Table 1 lists them under Pruthi et al. [2020]; this citation mismatch should be reconciled.
  6. [References] The reference list relies heavily on arXiv preprints; this is acceptable for a position paper, but flagging peer-reviewed versions where available would improve usability.

Circularity Check

1 steps flagged · score 3.0 of 10

Table 1 is populated with methods that “align with the unified view,” making the taxonomy’s support partly self-definitional; no fitted predictions or load-bearing self-citation are present.

  1. self definitional [Appendix B, paragraph introducing Table 1; see also §5 Alternative views]
    "In Table 1, we summarize the attribution methods discussed in this paper, which we believe are the representative ones and align with the unified view we presented."

    The paper's demonstration that FA, DA, and CA share core techniques is carried by Table 1 and the three-category taxonomy. The table's population, however, was selected to align with the conclusion: the paper says these are the methods that “align with the unified view we presented.” The later claim that the taxonomy “captures the majority of representative methods” (§5) is therefore about a sample defined by the conclusion rather than an independently enumerated universe. Non-fitting families are acknowledged in the appendices (tree surrogates for FA, simulation/RL methods for DA, sparse autoencoders for CA) and set aside by component definition or by calling them “other techniques,” so the generalization to “majority” is not derived from an external standard.

full rationale

Mild circularity arises from evidence curation, not from a formal derivation. The paper is a position paper, and its central claim—that feature, data, and component attribution share perturbation/gradient/linear-approximation techniques—is supported by classifying concrete methods; those classifications are checkable and are not derived from the conclusion. The LFA unification in §3.1.1 cites the authors' own prior work (Han et al. 2022), but it is not load-bearing for the three-way technique claim and is used mainly to suggest an analogy for DA and CA. The main circular component is the selection of Table 1's methods “to align with the unified view,” with the §5 “majority of representative methods” assertion unquantified and the appendix exceptions acknowledged. This reduces the force of the taxonomy as evidence but does not make the paper's substantive observations equivalent to their inputs. No fitted parameters are renamed as predictions, and no uniqueness theorem is imported from the authors, so the score stays below 6.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no numerical free parameters and no physical or ontological entities. Its central claim rests on four interpretive assumptions: the aspect-based formalization of attribution (Section 2), the three-technique taxonomy (Section 3), the conjectural extension of LFA to DA and CA (Section 4.1.2), and the representativeness of Table 1 (Section 5). The representativeness assumption is the most fragile because excluded methods are acknowledged but not systematically handled.

assumptions (4)
  • domain assumption The attribution problem can be uniformly formalized as a function g assigning scores to arbitrary aspects (features, data points, or components).
    Section 2 defines FA, DA, and CA through one attribution function g and states they 'differ only in the choice of aspects.' This re-framing is assumed before any evidence is presented.
  • ad hoc to paper Perturbation, gradient, and linear approximation form the right level of abstraction for comparing all three attribution types.
    The central thesis depends on this classification. Section 3 and Table 1 choose these three families, and Section 5 concedes other techniques exist but calls them exceptions without a formal selection criterion.
  • ad hoc to paper The local function approximation framework, previously established for FA, extends to DA and CA.
    Section 4.1.2 says 'One may hypothesize that DAs perform function approximation... and that CAs perform function approximation...' This is a conjecture, not a proven theorem.
  • domain assumption The methods listed in Table 1 are representative enough to support a claim about attribution methods as a whole.
    Section 5 states 'our categorization captures the majority of representative methods' without defining the selection universe or quantifying coverage. Appendices D.6 and E.3 list excluded methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability." pith.science (2026). https://pith.science/paper/SA2NGJYG

@misc{pith2026250118887,
  author       = {Pith},
  title        = {Pith review of: Towards Unified Attribution in Explainable AI, Data-Centric AI, and Mechanistic Interpretability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SA2NGJYG}},
  note         = {Machine review of arXiv:2501.18887}
}
read the original abstract

The increasing complexity of AI systems has made understanding their behavior critical. Numerous interpretability methods have been developed to attribute model behavior to three key aspects: input features, training data, and internal model components, which emerged from explainable AI, data-centric AI, and mechanistic interpretability, respectively. However, these attribution methods are studied and applied rather independently, resulting in a fragmented landscape of methods and terminology. This position paper argues that feature, data, and component attribution methods share fundamental similarities, and a unified view of them benefits both interpretability and broader AI research. To this end, we first analyze popular methods for these three types of attributions and present a unified view demonstrating that these seemingly distinct methods employ similar techniques (such as perturbations, gradients, and linear approximations) over different aspects and thus differ primarily in their perspectives rather than techniques. Then, we demonstrate how this unified view enhances understanding of existing attribution methods, highlights shared concepts and evaluation criteria among these methods, and leads to new research directions both in interpretability research, by addressing common challenges and facilitating cross-attribution innovation, and in AI more broadly, with applications in model editing, steering, and regulation.

Figures

Figures reproduced from arXiv: 2501.18887 by the authors.

Figure 1
Figure 1. The three types of attribution: FA, DA, and CA. While each type seeks to attribute a model’s output to a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

104 extracted references · 30 canonical work pages

  1. [1]

    Sanity checks for saliency maps

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018

    2018

  2. [2]

    The computational complexity of circuit discovery for inner interpretability

    Federico Adolfi, Martina G Vilas, and Todd Wareham. The computational complexity of circuit discovery for inner interpretability. arXiv preprint arXiv:2410.08025, 2024

    arXiv 2024

  3. [3]

    Openxai: Towards a transparent evaluation of model explanations

    Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. Advances in neural information processing systems, 35: 0 15784--15799, 2022

    2022

  4. [4]

    Towards the unification and robustness of perturbation and gradient based explanations

    Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Steven Wu, and Himabindu Lakkaraju. Towards the unification and robustness of perturbation and gradient based explanations. In International Conference on Machine Learning, pages 110--119. PMLR, 2021

    2021

  5. [5]

    A unified view of gradient-based attribution methods for deep neural networks

    Marco Ancona, Enea Ceolini, Cengiz \"O ztireli, and Markus Gross. A unified view of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104, 2017

    arXiv 2017

  6. [6]

    Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai

    Alejandro Barredo Arrieta, Natalia D \' az-Rodr \' guez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garc \' a, Sergio Gil-L \'o pez, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58: 0 82--115, 2020

    2020

  7. [7]

    On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

    Sebastian Bach, Alexander Binder, Gr \'e goire Montavon, Frederick Klauschen, Klaus-Robert M \"u ller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10 0 (7): 0 e0130140, 2015

    2015

  8. [8]

    Training data attribution via approximate unrolling

    Juhan Bae, Wu Lin, Jonathan Lorraine, and Roger Baker Grosse. Training data attribution via approximate unrolling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3NaqGg92KZ

    2024

Show all 104 references
  1. [9]

    How to explain individual classification decisions

    David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert M \ A z ller. How to explain individual classification decisions. Journal of Machine Learning Research, 11 0 (Jun): 0 1803--1831, 2010

    2010

  2. [10]

    Model interpretability through the lens of computational complexity

    Pablo Barcel \'o , Mika \"e l Monet, Jorge P \'e rez, and Bernardo Subercaseaux. Model interpretability through the lens of computational complexity. Advances in neural information processing systems, 33: 0 15487--15498, 2020

    2020

  3. [11]

    Relatif: Identifying explanatory training samples via relative influence

    Elnaz Barshan, Marc-Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pages 1899--1909. PMLR, 2020

    1909

  4. [12]

    Local vs

    Shahaf Bassan, Guy Amir, and Guy Katz. Local vs. global interpretability: A computational complexity perspective. arXiv preprint arXiv:2406.02981, 2024

    2024 arXiv

  5. [13]

    Understanding the role of individual units in a deep neural network

    David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 117 0 (48): 0 30071--30078, 2020

    2020

  6. [14]

    Mechanistic interpretability for ai safety--a review

    Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety--a review. arXiv preprint arXiv:2404.14082, 2024

    2024 arXiv

  7. [15]

    Impossibility theorems for feature attribution

    Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. Proceedings of the National Academy of Sciences, 121 0 (2): 0 e2304406120, 2024

    2024

  8. [16]

    From shapley values to generalized additive models and back

    Sebastian Bordt and Ulrike von Luxburg. From shapley values to generalized additive models and back. In International Conference on Artificial Intelligence and Statistics, pages 709--745. PMLR, 2023

    2023

  9. [17]

    Towards monosemanticity: Decomposing language models with dictionary learning

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023

    2023

  10. [18]

    Truth is universal: Robust detection of lies in llms

    Lennart B \"u rger, Fred A Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in llms. Advances in Neural Information Processing Systems, 37: 0 138393--138431, 2024

    2024

  11. [19]

    Low-complexity probing via finding subnetworks

    Steven Cao, Victor Sanh, and Alexander M Rush. Low-complexity probing via finding subnetworks. arXiv preprint arXiv:2104.03514, 2021

    2021 arXiv

  12. [20]

    On training data influence of gpt models

    Yekun Chai, Qingyi Liu, Shuohuan Wang, Yu Sun, Qiwei Peng, and Hua Wu. On training data influence of gpt models. arXiv preprint arXiv:2404.07840, 2024

    2024 arXiv

  13. [21]

    Input similarity from the neural network perspective

    Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 32, 2019

    2019

  14. [22]

    Learning to explain: An information-theoretic perspective on model interpretation

    Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-theoretic perspective on model interpretation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Procee...

    2018

  15. [23]

    L-shapley and c-shapley: Efficient model interpretation for structured data

    Jianbo Chen, Le Song, Martin J Wainwright, and Michael I Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. arXiv preprint arXiv:1808.02610, 2018 b

    2018 arXiv

  16. [24]

    Towards automated circuit discovery for mechanistic interpretability

    Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adri \`a Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36: 0 16318--16352, 2023

    2023

  17. [25]

    Detection of influential observation in linear regression

    R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19 0 (1): 0 15--18, 1977

    1977

  18. [26]

    Characterizations of an empirical influence function for detecting influential cases in regression

    R Dennis Cook and Sanford Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22 0 (4): 0 495--508, 1980

    1980

  19. [27]

    Residuals and influence in regression

    R Dennis Cook and Sanford Weisberg. Residuals and influence in regression. NY: Chapman and Hall, 1982

    1982

  20. [28]

    Explaining by removing: A unified framework for model explanation

    Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22 0 (209): 0 1--90, 2021

    2021

  21. [29]

    Are neural nets modular? inspecting functional modularity through differentiable weight masks

    R \'o bert Csord \'a s, Sjoerd van Steenkiste, and J \"u rgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. arXiv preprint arXiv:2010.02066, 2020

    2010 arXiv

  22. [30]

    Sparse autoencoders find highly interpretable features in language models

    Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023

    2023 arXiv

  23. [31]

    Real time image saliency for black box classifiers

    Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pages 6970--6979, 2017

    2017

  24. [32]

    Editing factual knowledge in language models

    Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint arXiv:2104.08164, 2021

    2021 arXiv

  25. [33]

    The shapley taylor interaction index

    Kedar Dhamdhere, Ashish Agarwal, and Mukund Sundararajan. The shapley taylor interaction index. arXiv preprint arXiv:1902.05622, 2019

    1902 arXiv

  26. [34]

    Visualizing higher-layer features of a deep network

    Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341 0 (3): 0 1, 2009

    2009

  27. [35]

    Interpretable explanations of black boxes by meaningful perturbation

    Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429--3437, 2017

    2017

  28. [36]

    Causal abstractions of neural networks

    Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34: 0 9574--9586, 2021

    2021

  29. [37]

    Dissecting recall of factual associations in auto-regressive language models

    Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767, 2023

    2023 arXiv

  30. [38]

    Patchscope: A unifying framework for inspecting hidden representations of language models

    Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscope: A unifying framework for inspecting hidden representations of language models. arXiv preprint arXiv:2401.06102, 2024

    2024 arXiv

  31. [39]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning, pages 2242--2251. PMLR, 2019

    2019

  32. [40]

    Neuron shapley: Discovering the responsible neurons

    Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Advances in neural information processing systems, 33: 0 5922--5932, 2020

    2020

  33. [41]

    Studying large language model generalization with influence functions

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296, 2023

    2023 arXiv

  34. [42]

    A survey of methods for explaining black box models

    Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca Giannotti. A survey of methods for explaining black box models. arXiv preprint arXiv:1802.01933, 2018

    2018 arXiv

  35. [43]

    F ast IF : Scalable influence functions for efficient model interpretation and debugging

    Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. F ast IF : Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computatio...

    2021 doi

  36. [44]

    Training data influence analysis and estimation: A survey

    Zayd Hammoudeh and Daniel Lowd. Training data influence analysis and estimation: A survey. Machine Learning, 113 0 (5): 0 2351--2403, 2024

    2024

  37. [45]

    Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations

    Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  38. [46]

    Data cleansing for models trained with sgd

    Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data cleansing for models trained with sgd. Advances in Neural Information Processing Systems, 32, 2019

    2019

  39. [47]

    Does localization inform editing? surprising differences in causality-based localization vs

    Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Significance tests for neural networks

    Enguerrand Horel and Kay Giesecke. Significance tests for neural networks. Journal of Machine Learning Research, 21 0 (227): 0 1--29, 2020

    2020

  41. [49]

    Computationally efficient feature significance and importance for predictive models

    Enguerrand Horel and Kay Giesecke. Computationally efficient feature significance and importance for predictive models. In Proceedings of the Third ACM International Conference on AI in Finance, pages 300--307, 2022

    2022

  42. [50]

    Datamodels: Predicting predictions from training data

    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622, 2022

    2022 arXiv

  43. [51]

    Explaining explanations: Axiomatic feature interactions for deep networks

    Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22 0 (104): 0 1--54, 2021

    2021

  44. [52]

    How can i explain this to you? an empirical study of deep neural network explanation methods

    Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava. How can i explain this to you? an empirical study of deep neural network explanation methods. Advances in neural information processing systems, 33: 0 4211--4222, 2020

    2020

  45. [53]

    Efficient task-specific data valuation for nearest neighbor algorithms

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. arXiv preprint arXiv:1908.08619, 2019

    1908 arXiv

  46. [54]

    Scalability vs

    Ruoxi Jia, Fan Wu, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2021

  47. [55]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885--1894. PMLR, 2017

    2017

  48. [56]

    Concept bottleneck models

    Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338--5348. PMLR, 2020

    2020

  49. [57]

    The disagreement problem in explainable machine learning: A practitioner's perspective

    Satyapriya Krishna*, Tessa Han*, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspective. Transactions on Machine Learning Research (TMLR), 2024

    2024

  50. [58]

    Beta shapley: a unified and noise-reduced data valuation framework for machine learning

    Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 8780--8802. PMLR, 2022

    2022

  51. [59]

    Optimal ablation for interpretability

    Maximilian Li and Lucas Janson. Optimal ablation for interpretability. arXiv preprint arXiv:2409.09951, 2024

    2024 arXiv

  52. [60]

    Measuring the effect of training data on deep learning predictions via randomized experiments

    Jinkun Lin, Anqi Zhang, Mathias L \'e cuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Measuring the effect of training data on deep learning predictions via randomized experiments. In International Conference on Machine Learning, pages 13468--13504. PMLR, 2022

    2022

  53. [61]

    A unified approach to interpreting model predictions

    Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4768--4777, 2017

    2017

  54. [62]

    The geometry of truth: Emergent linear structure in large language model representations of true/false datasets

    Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023

    2023 arXiv

  55. [63]

    Sparse feature circuits: Discovering and editing interpretable causal graphs in language models

    Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024

    2024 arXiv

  56. [64]

    Locating and editing factual associations in gpt

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35: 0 17359--17372, 2022

    2022

  57. [65]

    Coresets for data-efficient training of machine learning models

    Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950--6960. PMLR, 2020

    2020

  58. [66]

    Fast model editing at scale

    Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. arXiv preprint arXiv:2110.11309, 2021

    2021 arXiv

  59. [67]

    The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability

    Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. arXiv pr...

    2024

  60. [68]

    Attribution patching: Activation patching at industrial scale

    Neel Nanda. Attribution patching: Activation patching at industrial scale. URL: https://www. neelnanda. io/mechanistic-interpretability/attribution-patching, 2023

    2023

  61. [69]

    Operationalizing the blueprint for an ai bill of rights: Recommendations for practitioners, researchers, and policy makers

    Alex Oesterling, Usha Bhalla, Suresh Venkatasubramanian, and Himabindu Lakkaraju. Operationalizing the blueprint for an ai bill of rights: Recommendations for practitioners, researchers, and policy makers. arXiv preprint arXiv:2407.08689, 2024

    2024 arXiv

  62. [70]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022

    2022 arXiv

  63. [71]

    Trak: Attributing model behavior at scale

    Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186, 2023

    2023 arXiv

  64. [72]

    High dimensional model explanations: An axiomatic approach

    Neel Patel, Martin Strobel, and Yair Zick. High dimensional model explanations: An axiomatic approach. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 401--411, 2021

    2021

  65. [73]

    Direct and indirect effects

    Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373--392. Association for Computing Machinery and Morgan & Claypool Publishers, 2022

    2022

  66. [74]

    Rise: Randomized input sampling for explanation of black-box models

    V Petsiuk. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421, 2018

    2018 arXiv

  67. [75]

    Estimating training data influence by tracing gradient descent

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 0 19920--19930, 2020

    2020

  68. [76]

    A practical review of mechanistic interpretability for transformer-based language models

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646, 2024

    2024

  69. [77]

    Toward transparent ai: A survey on interpreting the inner structures of deep neural networks

    Tilman R \"a uker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. In 2023 ieee conference on secure and trustworthy machine learning (satml), pages 464--483. IEEE, 2023

    2023

  70. [78]

    Why should i trust you?: Explaining the predictions of any classifier

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135--1144. ACM, 2016

    2016

  71. [79]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part...

    2015

  72. [80]

    Mechanistic? arXiv preprint arXiv:2410.09087, 2024

    Naomi Saphra and Sarah Wiegreffe. Mechanistic? arXiv preprint arXiv:2410.09087, 2024

    2024 arXiv

  73. [81]

    Scaling up influence functions

    Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8179--8186, 2022

    2022

  74. [82]

    Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450, 2016

    Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450, 2016

    2016 arXiv

  75. [83]

    Decomposing and editing predictions by modeling model computation

    Harshay Shah, Andrew Ilyas, and Aleksander Madry. Decomposing and editing predictions by modeling model computation. arXiv preprint arXiv:2404.11534, 2024

    2024 arXiv

  76. [84]

    A value for n-person games

    Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2, 1953

    1953

  77. [85]

    Learning important features through propagating activation differences

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pages 3145--3153. PMlR, 2017

    2017

  78. [86]

    Deep inside convolutional networks: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013

    2013 arXiv

  79. [87]

    Smoothgrad: removing noise by adding noise

    Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vi \'e gas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825, 2017

    2017 arXiv

  80. [88]

    Limetree: Interactively customisable explanations based on local surrogate multi-output regression trees

    Kacper Sokol and Peter Flach. Limetree: Interactively customisable explanations based on local surrogate multi-output regression trees. arXiv, 2020

    2020

  81. [89]

    Striving for simplicity: The all convolutional net

    Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014

    2014 arXiv

  82. [90]

    A simple and effective pruning approach for large language models

    Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023

    2023 arXiv

  83. [91]

    The many shapley values for model explanation

    Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International conference on machine learning, pages 9269--9278. PMLR, 2020

    2020

  84. [92]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arXiv preprint arXiv:1703.01365, 2017

    2017 arXiv

  85. [93]

    Attribution patching outperforms automated circuit discovery

    Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348, 2023

    2023 arXiv

  86. [94]

    Bias and confidence in not quite large samples

    John Tukey. Bias and confidence in not quite large samples. Ann. Math. Statist., 29: 0 614, 1958

    1958

  87. [95]

    Investigating gender bias in language models using causal mediation analysis

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33: 0 12388--12401, 2020

    2020

  88. [96]

    Position: An inner interpretability framework for ai inspired by lessons from cognitive neuroscience

    Martina G Vilas, Federico Adolfi, David Poeppel, and Gemma Roig. Position: An inner interpretability framework for ai inspired by lessons from cognitive neuroscience. arXiv preprint arXiv:2406.01352, 2024

    2024 arXiv

  89. [97]

    Data banzhaf: A robust data valuation framework for machine learning

    Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pages 6388--6421. PMLR, 2023

    2023

  90. [98]

    Interpretability in the wild: a circuit for indirect object identification in gpt-2 small

    Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022

    2022 arXiv

  91. [99]

    Gradient based feature attribution in explainable ai: A technical review

    Yongjie Wang, Tong Zhang, Xu Guo, and Zhiqi Shen. Gradient based feature attribution in explainable ai: A technical review. arXiv preprint arXiv:2403.10415, 2024

    2024 arXiv

  92. [100]

    If you like shapley then you’ll love the core

    Tom Yan and Ariel D Procaccia. If you like shapley then you’ll love the core. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 5751--5759, 2021

    2021

  93. [101]

    Data valuation using reinforcement learning

    Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference on Machine Learning, pages 10842--10851. PMLR, 2020

    2020

  94. [102]

    Visualizing and understanding convolutional networks

    Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818--833. Springer, 2014

    2014

  95. [103]

    Gstarx: Explaining graph neural networks with structure-aware cooperative games

    Shichang Zhang, Yozen Liu, Neil Shah, and Yizhou Sun. Gstarx: Explaining graph neural networks with structure-aware cooperative games. Advances in Neural Information Processing Systems, 35: 0 19810--19823, 2022

    2022

  96. [104]

    Representation engineering: A top-down approach to ai transparency

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.