Pith. sign in

REVIEW 3 major objections 5 minor 61 references

Evaluating Model Explanations without Ground Truth

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AXE scores explanations by whether their top features recover the model's prediction, with no ground truth needed.

desk verdict AXE is a genuinely new and promising ground-truth-free evaluation idea, but Algorithm 1 leaks the explained point into the k-NN training set; the empirical claims hold only after a leave-one-out fix is stated and implemented. read the letter →

arxiv 2505.10399 v1 pith:LHNDIQ2L submitted 2025-05-15 cs.AI cs.LG

Kaivalya Rawal , Zihao Fu , Eoin Delaney , Chris Russell This is my paper · ORCID
classification cs.AIcs.LG
keywords explainableAIfeatureimportanceexplanationsexplanationevaluationground-truthagnostick-nearestneighborsfairwashingdetectionlocalmodelfaithfulness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Competing local explanations of the same prediction often contradict one another, and existing evaluation metrics either need a "ground-truth" explanation nobody can supply or reward sensitivity to off-manifold perturbations. This paper argues that neither approach is adequate and proposes three principles an evaluation should satisfy: explanations should depend on the datapoint and the model, and should not depend on off-manifold model behavior. To meet them, it introduces AXE, a score that measures whether the top-n features named by an explanation can be used by a k-nearest-neighbor model to recover the model's prediction. The paper reports that AXE ranks a set of real explanations coherently, agrees across its own hyperparameters, and detects adversarially "fairwashed" explanations—explanations that hide a protected attribute's role—in every one of ten tested settings, a result the authors attribute to AXE's avoidance of sensitivity-based assumptions.

What carries the argument

The carrying object is a per-datapoint k-nearest-neighbor accuracy check on the explanation's top-n features. For each explained point, AXE extracts the n features with largest absolute importance, fits a separate k-NN model to the model's predictions over the dataset using only those features, and records whether that model reproduces the explained point's prediction; the dataset-level quality score is the average accuracy, and an area-under-curve over n turns the top-n scores into a single scalar. The deliberate separation of k-NN models per datapoint is what makes the score local, and training on the dataset's own points is what keeps it on-manifold.

What would settle it

A direct test is a user study: give humans the same top-n features and ask them to predict the model's output, then compare their success rate with AXE scores; if the two rankings diverge on real datasets, the predictiveness proxy is not the useful notion. A cheaper computational check is to rerun Algorithm 1 exactly as written, without excluding the explained point from the k-NN training set, since with that point present distance-based recovery becomes trivially easy and would inflate every explanation's score.

Watch

Extended reading notes

Core claim

The paper's central claim is that explanation quality for local feature-importance methods can be measured without any ground-truth explanation and without sensitivity analysis. The proposed measure, AXE, takes a datapoint, its model prediction, and a candidate explanation, keeps only the features the explanation marks most important, and asks whether a k-NN model trained on those features over the dataset recovers the model's prediction. A good explanation is one whose top features separate model predictions in feature space. The paper shows that this operationalizes the user-study finding that useful explanations help users predict model behavior, and it argues AXE satisfies local contextualization, model relativism, and on-manifold evaluation. In fairwashing experiments built on an adversarial attack, AXE scored truthful explanations above manipulated ones in all ten settings, which the paper states is the first evaluation metric able to do so.

Load-bearing premise

AXE's entire validity rests on the premise that a local explanation is good exactly when the model's prediction at that point can be recovered by a k-NN model built on the dataset using the explanation's top-n features, a proxy for human usefulness that the paper does not test against human behavior.

Editorial extensions

If this is right

  • Practitioners can rank competing local explanations for any tabular model using only the dataset and model outputs, with no oracle explanation.
  • AXE can serve as an audit signal: explanations that hide a protected attribute's contribution receive lower scores than explanations that surface it, so a regulator could flag fairwashed explanations.
  • For neural networks, where coefficient-based ground truths do not exist, AXE provides a comparison metric that the paper shows is stable across its own hyperparameters.
  • The three principles give later evaluation methods a checklist: local contextualization, model relativism, and on-manifold evaluation.
  • Because AXE measures on-manifold predictiveness rather than sensitivity, it avoids rewarding explainers that merely match a particular perturbation scheme.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the proxy holds up in human tests, AXE-style scores could become a cheap selection criterion in model documentation and explanation dashboards, something the paper motivates but does not itself demonstrate.
  • The paper's pseudocode omits a leave-one-out step; unless the released implementation excludes the explained point when training the k-NN model, the reported near-perfect scores could be an artifact of self-matching, so the code should be checked before relying on the numbers.
  • The same k-NN recovery idea extends naturally to regression targets by replacing classification accuracy with a continuous agreement measure, and to explanation types beyond top-n feature sets, such as rules or counterfactuals.
  • A testable consequence of the paper's account is that high-AXE explanations should be the ones human users find most useful in prediction tasks; if user studies show no such link, the metric would need to be recalibrated rather than abandoned.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that existing evaluation frameworks for local feature-importance explanations are either ground-truth-dependent or sensitivity-based, and proposes three principles---local contextualization, model relativism, and on-manifold evaluation---for a desirable evaluation framework. It then introduces AXE, a metric that scores an explanation by training a k-NN model on the top-n features identified by that explanation and measuring how accurately the model prediction can be recovered. The paper motivates AXE with a synthetic example, applies it to detect explanation fairwashing, and compares it with prior metrics on OpenXAI datasets. It claims that AXE is the first evaluation metric to detect fairwashing perfectly on the tested suite.

Significance. If the central claim holds, AXE is a genuinely useful contribution: it provides a ground-truth-agnostic, on-manifold way to compare local explanations, with a clear operationalization of the idea that good explanations help users predict model behavior. The paper is clearly written, ships code, and includes both synthetic examples and real-data comparisons. The main caveats are the unstated leave-one-out protocol in Algorithm 1, the lack of statistical reporting in the fairwashing experiment, and the unvalidated k-NN proxy for human simulatability. These issues are fixable, and the core idea is worth publishing after the experimental protocol is corrected.

major comments (3)
  1. [Section 4.1, Table 2] As written, Algorithm 1 trains the k-NN model M_k^i on X_f constructed from the full dataset X, which includes the explained point x_i, and then queries M_k^i at x_i. For k=1 the nearest neighbor is x_i itself, so every explanation receives an AXE score of exactly 1; for small k, the self-neighbor systematically inflates all scores and compresses differences between explanations. The illustrative example in Section 3.2 (q(e_b) ~ 0.5 for a wide range of k) and the fairwashing results in Table 2 cannot be reproduced by the literal algorithm: with self-inclusion, the E_phi and E_psi explanations would also receive near-perfect scores from their own points. The paper never states a leave-one-out step. Because the central claim is that AXE measures how well the top-n features generalize in recovering model behavior, the algorithm must exclude x_i from the training set for its own prediction (or equivalently exclude the zero-distance self-match at query time), and the reported experiments must be rerun under that protocol.
  2. [Section 4.1, Table 2] The claim that AXE is 'the first evaluation metric to be able to do this perfectly' is not supported by the experimental reporting. Table 2 contains a single run per dataset/model/foil configuration, with no error bars, no number of repetitions, and no statement of the AXE hyperparameter k used (only n=1 is fixed). The PGI/PGU baseline numbers are also reported without the perturbation-width hyperparameter, which Section 3.2 itself shows to be decisive for these metrics. I therefore cannot assess whether the 100% success rate is robust or an artifact of a favorable hyperparameter choice or of the train/query overlap in Algorithm 1. The authors should report standard errors or confidence intervals, the k value(s) used, and the PGI/PGU width(s), and should recompute the table after fixing Algorithm 1.
  3. [Section 3.1] The load-bearing premise of AXE is that a good local explanation is one whose top-n features allow a k-NN model to recover the model's prediction. The cited user studies support the broader idea that useful explanations help people predict model behavior, but they do not establish that k-NN accuracy is a faithful proxy for human simulatability; in particular, k-NN ignores feature magnitude, sign, and human cognitive factors. If AXE is proposed as a formal operationalization, the paper should state this more carefully and provide at least a small human-subject validation or a clear argument for why the proxy is sufficient. Without this, the empirical comparisons only show that AXE agrees with the authors' chosen synthetic and fairwashing constructions, not that it tracks explanation quality in the human-centered sense invoked in the introduction.
minor comments (5)
  1. [Table 2, German Credit m_L (2 foils) PGU row] The entry '0-0.524' appears to be a typo for '-0.524'; please fix.
  2. [Section 3.1, cache-size discussion] The sentence 'For a dataset with ν features' should read 'For a dataset with N features and ν datapoints'; the notation is inconsistent and confusing.
  3. [Section 4.2, Figures 6 and 7] The notation AXE1, AXE3, AXE5, AXE9 is ambiguous: the paper uses AXE_k^n elsewhere, and the reader cannot tell whether the subscript refers to k or n. Please make the notation consistent.
  4. [Section 3.2, Figure 5] The caption misspells 'reliably' as 'relibably'.
  5. [Section 3.2, k-NN decision rule] The illustrative example treats 'average near 0.5' as poor accuracy, but Algorithm 1 uses hard accuracy on predicted labels; please clarify the k-NN decision rule (majority vote versus averaged score) and the tie-breaking procedure.

Circularity Check

1 steps flagged · score 6.0 of 10

AXE's Algorithm 1 trains the k-NN on the query point itself, making its 'predictions' equal to the target by construction.

  1. fitted input called prediction [Section 3.1, Algorithm 1 (lines 4-6); reported results in Table 2]
    "for each datapoint x_i and explanation e_i in (X,E) do ... Create X_f with subset of features f_imp from X ... Train K-NN model M_k^i with inputs X_f and target Y_preds ... Obtain prediction \hat{y}_i from M_k^i for datapoint x_i"

    X_f is built from the full dataset X (line 4), so it contains the row x_i for the point being explained. M_k^i is trained on X_f (line 5) then queried at x_i (line 6). For k=1, the nearest neighbour of x_i is itself (distance 0), so \hat{y}_i = y_i for every explanation; for small k the self-neighbour dominates. Accuracy is therefore 1.0 (or near 1.0) regardless of e_i: AXE is constant by construction, not a predictiveness measure. Table 2 values (1.000 vs 0.680) cannot be reproduced from Algorithm 1 as written; they silently require leave-one-out exclusion of x_i. The central claim that AXE recovers m(x) from top-n features thus reduces to memorizing the target.

full rationale

Aside from this construction issue, I find no other circular step. The proposed principles are normative rather than derived from AXE; the fairwashing benchmark's correct answers are fixed by the adversarially modified model's true decision rule, not by AXE; and AXE is compared against external OpenXAI baselines without fitting parameters to a target. However, the central algorithmic step in Algorithm 1 is circular in the literal sense: the k-NN is trained on the same point it is asked to predict, forcing accuracy near 1.0 for every explanation and making the reported discriminative scores unattainable as written. This reduces the claimed prediction to memorization by construction, so a score of 6 is warranted; if leave-one-out were explicitly added, the remaining framework would be self-contained.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

AXE rests on a normative assumption about explanation usefulness, on k-NN accuracy as the operationalization, and on scope restrictions. The hyperparameters k and n are user-selected. No new particles, forces, or entities are introduced.

free parameters (2)
  • AXE top-n
    Number of top features retained for the k-NN model. User-selected; paper uses n=1 for fairwashing and an AUC over all n in baselines.
  • AXE k (neighbors)
    Number of neighbors for the k-NN model. Paper reports k=1,3,5,9 and claims robustness, but gives no selection criterion.
assumptions (4)
  • domain assumption A good local explanation is one whose top features predict the model output well.
    This is the core operationalization of AXE, motivated by cited user studies but not tested against human subjects here.
  • ad hoc to paper The explained point is not included in the k-NN training set.
    Required for the reported non-trivial scores, but never stated in Algorithm 1 or Section 3.1.
  • domain assumption The three principles (local contextualization, model relativism, on-manifold evaluation) are the right normative standards.
    The paper argues for these principles but they are not derived from first principles.
  • domain assumption Binary classification and tabular data are the relevant scope.
    AXE is defined for binary models and the experiments cover tabular datasets only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Model Explanations without Ground Truth." pith.science (2026). https://pith.science/paper/LHNDIQ2L

@misc{pith2026250510399,
  author       = {Pith},
  title        = {Pith review of: Evaluating Model Explanations without Ground Truth},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LHNDIQ2L}},
  note         = {Machine review of arXiv:2505.10399}
}
read the original abstract

There can be many competing and contradictory explanations for a single model prediction, making it difficult to select which one to use. Current explanation evaluation frameworks measure quality by comparing against ideal "ground-truth" explanations, or by verifying model sensitivity to important inputs. We outline the limitations of these approaches, and propose three desirable principles to ground the future development of explanation evaluation strategies for local feature importance explanations. We propose a ground-truth Agnostic eXplanation Evaluation framework (AXE) for evaluating and comparing model explanations that satisfies these principles. Unlike prior approaches, AXE does not require access to ideal ground-truth explanations for comparison, or rely on model sensitivity - providing an independent measure of explanation quality. We verify AXE by comparing with baselines, and show how it can be used to detect explanation fairwashing. Our code is available at https://github.com/KaiRawal/Evaluating-Model-Explanations-without-Ground-Truth.

Figures

Figures reproduced from arXiv: 2505.10399 by the authors.

Figure 1
Figure 1. Different Explainers Yield Different Explanations: A neural network predicts diabetes on the “Pima Indians” dataset [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Explanation Generation (a) and Evaluation (b): AXE [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Violations of local contextualization and model relativism: Plots showing explanation quality [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Synthetic Data and Model for AXE and PGI evalu [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Comparing explanations using AXE and PGI: By definition [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Evaluating the Quality of Explanations for Neural Networks: For a fair comparison across evaluation metrics, we [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Evaluating the Quality of Explanations for Logistic Regression: For a fair comparison across evaluation metrics, we [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 25 canonical work pages

  1. [1]

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity checks for saliency maps. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (Montréal, Canada) (NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 9525–9536

    2018

  2. [2]

    Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. 2024. OpenXAI: towards a transparent evaluation of post hoc model explanations. In Proceedings of the 36th International Conference on Neural Information Processing Systems (New Orleans, LA, USA) (NIPS ’22). Curran Associa...

    work page 2024

  3. [3]

    Ulrich Aïvodji, Hiromi Arai, Sébastien Gambs, and Satoshi Hara. 2024. Character- izing the risk of fairwashing. In Proceedings of the 35th International Conference on Neural Information Processing Systems (NIPS ’21) . Curran Associates Inc., Red Hook, NY, USA, Article 1136, 13 pages

    work page 2024

  4. [4]

    Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. 2020. Evaluating saliency map explanations for convolutional neural networks: a user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 275–285. ...

    arXiv 2020

  5. [5]

    Brian Barr, Noah Fatsi, Leif Hancox-Li, Peter Richter, Daniel Proano, and Caleb Mok. 2023. The Disagreement Problem in Faithfulness Metrics. arXiv:2311.07763 [cs.LG] https://arxiv.org/abs/2311.07763

    arXiv 2023

  6. [6]

    Barry Becker and Ronny Kohavi. 1996. Adult. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5XW20

    doi:10.24432/c5xw20 1996

  7. [7]

    Gajos, and Elena L

    Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. 2020. Proxy tasks and subjective measures can be misleading in evaluating explainable AI systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces (Cagliari, Italy) (IUI ’20). Association for Computing Machinery, New York, NY, USA, 454–464. doi:10.1145/3...

    arXiv 2020

  8. [8]

    Carvalho, Eduardo M

    Diogo V. Carvalho, Eduardo M. Pereira, and Jaime S. Cardoso. 2019. Machine Learning Interpretability: A Survey on Methods and Metrics. Electronics 8, 8 (2019). doi:10.3390/electronics8080832

    doi:10.3390/electronics8080832 2019

Show all 61 references
  1. [9]

    Arjun Chandrasekaran, Viraj Prabhu, Deshraj Yadav, Prithvijit Chattopadhyay, and Devi Parikh. 2018. Do explanations make VQA models more predictable to a human?. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Ellen Riloff, David Chia...

    2018

  2. [10]

    Victor Chang, Jozeene Bailey, Qianwen Ariel Xu, and Zhili Sun. 2022. Pima Indi- ans diabetes mellitus classification based on machine learning (ML) algorithms. Neural Comput Appl , (March 2022), 1–17

    2022

  3. [11]

    Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. 2023. Machine Expla- nations and Human Understanding. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT ’23). As- sociation for Computing Machinery, New York, N...

    2023 doi

  4. [12]

    Julien Colin, Thomas FEL, Remi Cadene, and Thomas Serre. 2022. What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods. In Advances in Neural Information Processing Systems , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K...

    2022

  5. [13]

    Cover and P

    T. Cover and P. Hart. 1967. Nearest neighbor pattern classification. IEEE Trans- actions on Information Theory 13, 1 (1967), 21–27. doi:10.1109/TIT.1967.1053964

    1967

  6. [14]

    Bach, and Himabindu Lakkaraju

    Jessica Dai, Sohini Upadhyay, Ulrich Aivodji, Stephen H. Bach, and Himabindu Lakkaraju. 2022. Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations. In Proceedings of the 2022 AAAI/ACM Confer- ence on AI, Ethics, and Society (Oxford, ...

    2022 doi

  7. [15]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Ima- geNet: A large-scale hierarchical image database. In2009 IEEE Conference on Com- puter Vision and Pattern Recognition . , , 248–255. doi:10.1109/CVPR.2009.5206848

    2009

  8. [16]

    Finale Doshi-Velez and Been Kim. 2017. Towards A Rigorous Science of Inter- pretable Machine Learning. arXiv:1702.08608 [stat.ML] https://arxiv.org/abs/ 1702.08608

    2017 arXiv

  9. [17]

    Thomas Fel, Rémi Cadène, Mathieu Chalvidal, Matthieu Cord, David Vigouroux, and Thomas Serre. 2024. Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. In Proceedings of the 35th International Conference on Neural Information Processin...

    2024

  10. [18]

    FICO. 2022. Explainable Machine Learning Challenge. https://community.fico. com/s/explainable-machine-learning-challenge?tabset-158d9=3

    2022

  11. [19]

    Evelyn Fix and J. L. Hodges. 1989. Discriminatory Analysis. Nonparametric Discrimination: Consistency Properties. International Statistical Review / Revue Internationale de Statistique 57, 3 (1989), 238–247. http://www.jstor.org/stable/ 1403797

    1989

  12. [20]

    Ruth Fong, Mandela Patrick, and Andrea Vedaldi. 2019. Understanding Deep Networks via Extremal Perturbations and Smooth Masks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . , , 2950–2958. doi:10.1109/ ICCV.2019.00304

    2019

  13. [21]

    Fong and Andrea Vedaldi

    Ruth C. Fong and Andrea Vedaldi. 2017. Interpretable Explanations of Black Boxes by Meaningful Perturbation. In 2017 IEEE International Conference on Computer Vision (ICCV). , , 3449–3457. doi:10.1109/ICCV.2017.371

    2017 doi

  14. [22]

    Benjamin Fresz, Lena Lörcher, and Marco Huber. 2024. Classification Metrics for Image Explanations: Towards Building Reliable XAI-Evaluations. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (Rio de Janeiro, Brazil) (FAccT ’24). Associat...

    2024

  15. [23]

    Gilpin, David Bau, Ben Z

    Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining Explanations: An Overview of Interpretability of Machine Learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA). , , 80–89. d...

    2018

  16. [24]

    Thomas Hartley, Kirill Sidorov, Christopher Willis, and David Marshall. 2020. Explaining Failure: Investigation of Surprise and Expectation in CNNs. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). , , 56–65. doi:10.1109/CVPRW50498.2020.00014

    2020

  17. [25]

    Peter Hase and Mohit Bansal. 2020. Evaluating Explainable AI: Which Algorith- mic Explanations Help Users Predict Model Behavior?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel ...

    2020 doi

  18. [26]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Under- standing. Proceedings of the International Conference on Learning Representations (ICLR) , (2021),

    2021

  19. [27]

    Hans Hofmann. 1994. Statlog (German Credit Data). UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5NC77

    1994 doi

  20. [28]

    Sara Hooker, Dumitru Erhan, Pieter jan Kindermans, and Been Kim. 2018. Evalu- ating Feature Importance Estimates. arXiv , (2018), . https://arxiv.org/pdf/1806. 10758.pdf

    2018

  21. [29]

    Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. 2021. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recogn. Lett. 150, C (Oct. 2021), 228–234. doi:10.1016/j.patrec.2021.06.030

    2021 doi

  22. [30]

    Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viegas, and Michael Terry. 2019. XRAI: Better Attributions Through Regions. In 2019 IEEE/CVF International Con- ference on Computer Vision (ICCV) . , , 4947–4956. doi:10.1109/ICCV.2019.00505

    2019

  23. [31]

    Satyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, and Himabindu Lakkaraju. 2024. The Disagreement Problem in Explainable Ma- chine Learning: A Practitioner’s Perspective. Transactions on Machine Learning Research , (2024), . https://openreview.net/forum?id=jESY2WTZCe

    2024

  24. [32]

    Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. https://www.propublica.org/ article/how-we-analyzed-the-compas-recidivism-algorithm

    2016

  25. [33]

    Vera Liao and Kush R

    Q. Vera Liao and Kush R. Varshney. 2021. Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. CoRR abs/2110.10790 (2021). arXiv:2110.10790 https://arxiv.org/abs/2110.10790

    2021 arXiv

  26. [34]

    Zachary C. Lipton. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue 16, 3 (June 2018), 31–57. doi:10.1145/3236386.3241340

    2018

  27. [35]

    Yang Liu, Sujay Khandagale, Sujay Khandagale, Colin White, and Willie Neiswanger. 2021. Synthetic Benchmarks for Scientific Research in Explain- able Machine Learning. In Proceedings of the Neural Information Processing Sys- tems Track on Datasets and Benchmarks , J. Vanschore...

    2021

  28. [36]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural FAccT ’25, June 23–26, 2025, Athens, Greece Kaivalya Rawal, Zihao Fu, Eoin Delaney, and Chris Russell Information Processi...

    2017

  29. [37]

    Charles Marx, Flavio Calmon, and Berk Ustun. 2020. Predictive Multiplicity in Classification. In Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 119) , Hal Daumé III and Aarti Singh (Eds.). PMLR, , 6765–6774....

    2020

  30. [38]

    Chuizheng Meng, Loc Trinh, Nan Xu, James Enouen, and Yan Liu. 2022. Inter- pretability and fairness evaluation of deep learning models on MIMIC-IV dataset. Scientific Reports 12, 1 (03 May 2022), 7166. doi:10.1038/s41598-022-11012-2

    2022 doi

  31. [39]

    Vishwali Mhasawade, Salman Rahman, Zoé Haskell-Craig, and Rumi Chunara

  32. [40]

    Christoph Molnar. 2022. Interpretable Machine Learning (2 ed.). leanpub, . https: //christophm.github.io/interpretable-ml-book

    2022

  33. [41]

    Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. 2020. Interpretable Machine Learning – A Brief History, State-of-the-Art and Challenges. In ECML PKDD 2020 Workshops, Irena Koprinska, Michael Kamp, Annalisa Appice, Cor- rado Loglisci, Luiza Antonie, Albrecht Zimmerman...

    2020

  34. [42]

    Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. 2023. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. ACM Comput....

    2023 doi

  35. [43]

    Martin Pawelczyk, Klaus Broelemann, and Gjergji Kasneci. 2020. Learning Model- Agnostic Counterfactual Explanations for Tabular Data. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 3126–3132. doi:10...

    2020

  36. [44]

    Vitali Petsiuk, Abir Das, and Kate Saenko. 2018. RISE: Randomized Input Sampling for Explanation of Black-box Models. CoRR abs/1806.07421 (2018), . arXiv:1806.07421 http://arxiv.org/abs/1806.07421

    2018 arXiv

  37. [45]

    Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wort- man Wortman Vaughan, and Hanna Wallach. 2021. Manipulating and Mea- suring Model Interpretability. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CH...

    2021

  38. [46]

    Luyu Qiu, Yi Yang, Caleb Chen Cao, Yueyuan Zheng, Hilary Ngai, Janet Hsiao, and Lei Chen. 2022. Generating Perturbation-based Explanations with Robustness to Out-of-Distribution Data. In Proceedings of the ACM Web Conference 2022 (Virtual Event, Lyon, France) (WWW ’22). Associ...

    2022

  39. [47]

    Michael Redmond. 2002. Communities and Crime. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C53W3X

    2002 doi

  40. [48]

    Why Should I Trust You?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16...

    2016

  41. [49]

    Richard D Riley and Gary S Collins. 2023. Stability of clinical prediction models developed using statistical or machine learning methods. Biom J 65, 8 (July 2023), e2200302

    2023

  42. [50]

    Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2017. Evaluating the Visualization of What a Deep Neural Network Has Learned. IEEE Transactions on Neural Networks and Learning Systems 28, 11 (2017), 2660–2673. doi:10.1109/TNN...

    2017

  43. [51]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In 2017 IEEE International Con- ference on Computer Vision (ICCV) . , , 618–6...

    2017 doi

  44. [52]

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning im- portant features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML’17). JMLR.org, , 3145–3153

    2017

  45. [53]

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Tra...

    2014 arXiv

  46. [54]

    Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju

  47. [55]

    Viégas, and Martin Watten- berg

    Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Watten- berg. 2017. SmoothGrad: removing noise by adding noise. CoRR abs/1706.03825 (2017), . arXiv:1706.03825 http://arxiv.org/abs/1706.03825

    2017 arXiv

  48. [56]

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 70) , Doina Precup and Yee Whye Teh (Eds.). PMLR, , 3319–3328. ...

    2017

  49. [57]

    Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. 2018. Top-Down Neural Attention by Excitation Backprop. Int. J. Comput. Vision 126, 10 (oct 2018), 1084–1102. doi:10.1007/s11263-017-1059-x

    2018 doi

  50. [58]

    Gandomi, Fang Chen, and Andreas Holzinger

    Jianlong Zhou, Amir H. Gandomi, Fang Chen, and Andreas Holzinger. 2021. Evaluating the Quality of Machine Learning Explanations: A Survey on Methods and Metrics. Electronics 10, 5 (2021), . doi:10.3390/electronics10050593

    2021 doi

  51. [1042]

    doi:10.18653/v1/D18-1128

    doi

  52. [2019]

    CoRR abs/1911.02508 (2019),

    How can we fool LIME and SHAP? Adversarial Attacks on Post hoc Explanation Methods. CoRR abs/1911.02508 (2019), . arXiv:1911.02508 http: //arxiv.org/abs/1911.02508

    2019 arXiv

  53. [2024]

    In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Trans- parency (Rio de Janeiro, Brazil)(FAccT ’24)

    Understanding Disparities in Post Hoc Machine Learning Explanation. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Trans- parency (Rio de Janeiro, Brazil)(FAccT ’24). Association for Computing Machinery, New York, NY, USA, 2374–2388. doi:10.1145/363...

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.