Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods

T0 review · 4 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper presents xai_evals, a Python package aimed at unifying the generation, benchmarking, and evaluation of post-hoc local explanations for machine learning models, across tabular and image data.

desk verdict Software-announcement paper whose promised package and experiments are absent; the wrapper idea is sensible, but this draft doesn't support its central claim. read the letter →

arxiv 2502.03014 v1 pith:YPIVCVJS submitted 2025-02-05 cs.LG cs.AIcs.ET

Pratinav Seth , Yashwardhan Rathore , Neeraj Kumar Singh , Chintan Chitroda , Vinay Kumar Sankarapu This is my paper · ORCID
classification cs.LGcs.AIcs.ET
keywords explainableAIpost-hocexplanationsfeatureattributionexplanationevaluationbenchmarkingSHAPLIMEGrad-CAM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that xai_evals, a new Python package, fills a gap in explainable AI by bringing explanation generation and explanation evaluation into one tool. It integrates widely used attribution methods--SHAP, LIME, Grad-CAM, Integrated Gradients, and DlBacktrace--and measures their output with metrics such as faithfulness, sensitivity, comprehensiveness, sufficiency, and robustness. The intended payoff is a standardized way to compare explanation methods across tabular and image models, so that researchers and practitioners can judge not only whether a model is accurate but whether its reasons are trustworthy. If the package works as described, it would give the field a common benchmarking surface for post-hoc local explanations.

What carries the argument

The central machinery is the package's paired class structure: illustration classes such as SHAPExplainer, LIMEExplainer, TorchImageExplainer, and DlBacktrace explainers produce attribution maps, while ExplanationMetricsTabular and ExplanationMetricsImage quantify them using perturbation-based formulas--faithfulness compares output change under feature perturbation to attribution magnitude, sensitivity measures attribution change under noise, comprehensiveness and sufficiency mask the top-k features, and image metrics such as MPRT average output changes over perturbed pixels.

What would settle it

Install the package from the PyPI URL given in the abstract, run the Section 3.6 IRIS example exactly as written, and check that the computed metrics reproduce Table 3 (for SHAP on the IRIS Random Forest: faithfulness -0.568218, infidelity 0.000048, sensitivity 0.00328, and so on). If installation fails, the package is absent from PyPI, or the values do not match, the paper's central claim is not supported.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a single Python package, xai_evals, can combine explanation generation and evaluation for both tabular and image data, wrapping established attribution methods and perturbation-based metrics, and that this combination addresses the fragmentation of existing explainability tooling. The package is described as supporting classical machine learning models and deep learning models through a unified API, with separate paths for tabular and image modalities.

Load-bearing premise

The load-bearing premise is that the xai_evals package actually exists in installable form and reliably computes the listed metrics; the paper gives no working installation command, repository identifier, or reported experiments to confirm this.

Editorial extensions

If this is right

  • A practitioner could run SHAP, LIME, and Integrated Gradients on the same model and dataset through one API and read their faithfulness, sensitivity, and complexity scores from a single table.
  • Tabular and image models could be evaluated under the same metric conventions, making cross-modal comparisons of explanation quality possible.
  • Teams deploying models in regulated settings could use the generated metric values as documentation for why a prediction was made.
  • The package would lower the barrier to running explanation evaluations, since it removes the need to assemble separate libraries for generation and evaluation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an editorial extension: the paper states that benchmarking experiments showed the package is effective, but the text reports no such experiments; verifying effectiveness on a fixed dataset is a natural and necessary next step.
  • As an editorial extension: because the package builds on existing open-source explanation and metric libraries, its numerical outputs inherit those libraries' version-specific behaviors, so comparing releases may require care.
  • As an editorial extension: a testable extension of the package's own logic would be to run all supported explainers on one tabular and one image benchmark and publish the metric table, which the present report does not provide.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript presents xai_evals, a Python package claimed to generate, benchmark, and evaluate post-hoc local explanations for tabular and image data. It describes supported explainers (SHAP, LIME, Grad-CAM, Integrated Gradients, and the authors' DlBacktrace), a set of tabular and image metrics, and two usage examples with small output tables. The paper does not contain the empirical evaluation promised in the Introduction; Section 4 is a regulatory discussion, and the Conclusion refers to benchmarking experiments that never appear. Section 3.3 states the package can be installed 'once the package is published' and prints a corrupted pip command, so the package itself cannot be obtained or tested from the manuscript.

Significance. If the framework existed and worked as described, it would address a real need in XAI evaluation, and the related-work survey in Section 2.2 is a useful synthesis. The authors are transparent about depending on Quantus, Captum, and tf-explain for large parts of the implementation. The central claim, however, is unsubstantiated: no installation artifact, no benchmark results, and no correct or reproducible metric definitions are provided. Because the paper's contribution is the package and its evaluation, the current submission does not establish that the package exists or that any of its promised functionality is effective. It may be a basis for a future system paper, but as written it is a proposal, not a validated contribution.

major comments (4)
  1. [Section 4 and Section 5] Section 1 promises that Section 4 'presents experimental results demonstrating the effectiveness of the package,' but Section 4 is a discussion of GDPR and the EU AI Act with no data, tables, or protocols. Section 5 then states that 'the results of our benchmarking experiments showed that xai_evals is effective,' yet no such experiments appear anywhere in the manuscript. The central claim of demonstrated effectiveness is therefore unsupported.
  2. [Section 3.3.1] The only pip command shown is 'pip install xa i_ ev al s', which is not executable, and the surrounding text says installation can be done 'once the package is published.' No repository URL, commit hash, or version is provided. Since the paper's object of study is the package, the absence of any verifiable artifact means none of the integration or metric claims can be checked. The abstract's PyPI link does not remedy this, as no installation or import is demonstrated from it.
  3. [Section 3.5.1 and Table 3] The Monotonicity formula is 1/(n-1) times the sum of indicators of sign(a_i) = sign(a_{i+1}), which tests sign agreement between adjacent feature attributions, not whether attributions move with model output; it does not implement the stated definition. The tabular Faithfulness formula is an unnormalized sum of |f(x)-f(x'_i)| |a_i|, whereas the image Faithfulness Correlation in Section 3.5.2 is normalized by the sum of |a_i|, so equivalent-sounding metrics are not comparable across modalities. Table 3 reports Faithfulness = -0.568218 even though the formula is a nonnegative sum, and reports an 'infidelity' column with no definition in Section 3.5. These issues call into question whether the metric computations are correct.
  4. [Section 3.6.3] The evaluation example sets task='binary-classification' while using the three-class IRIS dataset and the same RandomForestClassifier used for multiclass classification in Section 3.6.1. This example, together with the corrupted whitespace in all code listings, suggests the usage snippets have not been run; combined with the missing package, the illustrative Tables 2 and 3 cannot be reproduced.
minor comments (4)
  1. [Title and Section 2.1.1] There are numerous typographical and formatting artifacts, including 'F RAMEWORK' and 'POST-H OC' in the title, 'Explainbility' in Section 2.1.1, and code listings with inserted spaces such as 'lo ad _i ri s ()'.
  2. [Table 3] The 'infidelity' column is reported in Table 3 but no formula or reference for it is given in Section 3.5 or anywhere else.
  3. [Figures 1 and 2] Figure 1 is referenced as the library overview but has no caption or descriptive text in the body, and Figure 2 is described only by a short caption; neither figure is discussed in enough detail to understand the design or the overlay method.
  4. [Section 3.5.1 and Section 3.5.2] The metrics would benefit from explicit perturbation and masking protocols; formulas such as Comprehensiveness and Sufficiency are underspecified with respect to the choice of S_k, the masking rule, and the construction of x_focused.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular structure: the paper is a software-system description whose claims are existence and capability claims, not derived quantities.

full rationale

xai_evals is a package report, not a derivation chain. Its central claims are that the package exists, integrates certain explanation methods, and computes certain metrics. None of these claims is shown to be equivalent to its own inputs: there are no fitted parameters, no predicted values derived from fitted data, and no equation that reduces to itself by construction. The only self-citation is to the authors' own DlBacktrace method (Sankarapu et al. 2024), cited as one of several integrated explanation techniques; it is not used as evidence that the package is correct or effective. The conclusion that "benchmarking experiments showed that xai_evals is effective" is unsupported because Section 4 contains no benchmark data, but an unsupported empirical claim is a reproducibility/evidence gap, not circularity. Similarly, the missing installable package and corrupted pip command undermine the existence claim, but they do not make any argument circular. The metric definitions are stated directly with formulas; even if some formulas are unusual or inconsistent with standard definitions, they are not derived from the package's own outputs. Overall, the manuscript's weaknesses are evidential and practical, not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The report introduces no fitted parameters. It relies on the validity of external software libraries and metrics, on the functional existence of the package, and on the authors' own DlBacktrace method, with no independent verification.

assumptions (4)
  • domain assumption The xai_evals package is installable and functional as claimed.
    Section 3.3 gives pip install instructions, but the command text is corrupted and package availability is not verified.
  • domain assumption Attribution methods from SHAP, LIME, Captum, tf-explain, and DlBacktrace produce correct attributions.
    The whole framework builds on these; no verification is provided within this paper.
  • domain assumption Evaluation metrics from Quantus and from cited papers (Yeh et al., Adebayo et al.) are valid and implemented correctly.
    Section 3.5 lists metrics; image metrics are explicitly credited to Quantus in the acknowledgments, and tabular metric formulas contain potential errors.
  • domain assumption DlBacktrace (Sankarapu et al., 2024, same research group) is a valid explanation method.
    The paper integrates this method but provides no independent assessment; it is a self-citation.
invented entities (1)
  • DlBacktrace explainer integration
    purpose: Provides backpropagation-based layer-wise attributions in the package.
    Presented as a feature of xai_evals; its only supporting reference is the authors' own prior arXiv preprint.

how reviews work

0 comments
Cite this review

Pith. "Pith review of xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods." pith.science (2026). https://pith.science/paper/YPIVCVJS

@misc{pith2026250203014,
  author       = {Pith},
  title        = {Pith review of: xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPIVCVJS}},
  note         = {Machine review of arXiv:2502.03014}
}
read the original abstract

The growing complexity of machine learning and deep learning models has led to an increased reliance on opaque "black box" systems, making it difficult to understand the rationale behind predictions. This lack of transparency is particularly challenging in high-stakes applications where interpretability is as important as accuracy. Post-hoc explanation methods are commonly used to interpret these models, but they are seldom rigorously evaluated, raising concerns about their reliability. The Python package xai_evals addresses this by providing a comprehensive framework for generating, benchmarking, and evaluating explanation methods across both tabular and image data modalities. It integrates popular techniques like SHAP, LIME, Grad-CAM, Integrated Gradients (IG), and Backtrace, while supporting evaluation metrics such as faithfulness, sensitivity, and robustness. xai_evals enhances the interpretability of machine learning models, fostering transparency and trust in AI systems. The library is open-sourced at https://pypi.org/project/xai-evals/ .

Figures

Figures reproduced from arXiv: 2502.03014 by the authors.

Figure 1
Figure 1. xai_evals Library Overview. 3 Package Overview The xai_evals package provides a comprehensive suite of functionalities to facilitate model interpretability and explainability. The main features of the package include: • Compatibility with Various Models: The package supports both classical machine learning models, such as those in scikit-learn (e.g., RandomForest, LogisticRegression, etc.), as well as deep learning … view at source ↗
Figure 2
Figure 2. Illustration of Overlay GradCAM attribution map over Image Sample. The attribution map highlights the [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training Large Language Models for Self-Explanation Faithfulness

    cs.LG 2026-07 conditional novelty 6.0 of 10

    RL fine-tuning with a counterfactual mention/influence reward raises LLM self-explanation faithfulness (Phi-CCT) from near zero to ~0.66 in-distribution for two 8B models, with partial transfer to held-out tasks.

  2. Faithfulness to Refusal: A Causal Audit of Neuron Selectors

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A causal audit via neuron-row zeroing shows attribution methods (LRP, IG) faithfully identify dispensable neurons and can install refusal behavior, while rank-stability proxies systematically miss selector failures.

  3. Position: Intelligent Coding Systems Should Write Programs with Justifications

    cs.SE 2025-08 conditional novelty 4.0 of 10

    A position paper advocating that intelligent coding systems should accompany code with justified explanations that are cognitively aligned and semantically faithful.

Reference graph

Works this paper leans on

19 extracted references · 12 canonical work pages · cited by 3 Pith papers

  1. [1]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Neural Information Processing Systems, 2017. URL https://api.semanticscholar.org/CorpusID:21889700

    work page 2017

  2. [2]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should i trust you?”: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016. URL https://api.semanticscholar.org/CorpusID:13029170

    work page 2016

  3. [3]

    Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128: 0 336 -- 359, 2016. URL https://api.semanticscholar.org/CorpusID:15019293

    work page 2016

  4. [4]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017. URL https://api.semanticscholar.org/CorpusID:16747630

    work page 2017

  5. [5]

    Dlbacktrace: A model agnostic explainability for any deep learning models, 2024

    Vinay Kumar Sankarapu, Chintan Chitroda, Yashwardhan Rathore, Neeraj Kumar Singh, and Pratinav Seth. Dlbacktrace: A model agnostic explainability for any deep learning models, 2024. URL https://arxiv.org/abs/2411.12643

    arXiv 2024

  6. [6]

    Interpretability needs a new paradigm, 2024

    Andreas Madsen, Himabindu Lakkaraju, Siva Reddy, and Sarath Chandar. Interpretability needs a new paradigm, 2024. URL https://arxiv.org/abs/2405.05386

    arXiv 2024

  7. [7]

    From flexibility to manipulation: The slippery slope of xai evaluation, 2024

    Kristoffer Wickstrøm, Marina Marie-Claire Höhne, and Anna Hedström. From flexibility to manipulation: The slippery slope of xai evaluation, 2024. URL https://arxiv.org/abs/2412.05592

    arXiv 2024

  8. [8]

    M4: a unified xai benchmark for faithfulness evaluation of feature attribution methods across metrics, modalities and models

    Xuhong Li, Mengnan Du, Jiamin Chen, Yekun Chai, Himabindu Lakkaraju, and Haoyi Xiong. M4: a unified xai benchmark for faithfulness evaluation of feature attribution methods across metrics, modalities and models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    work page 2023

Show all 19 references
  1. [9]

    Openxai: Towards a transparent evaluation of model explanations, 2024

    Chirag Agarwal, Dan Ley, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations, 2024. URL https://arxiv.org/abs/2206.11104

    2024 arXiv

  2. [10]

    Anna Hedström, Leander Weber, Dilyara Bareeva, Daniel Krakowczyk, Franz Motzkus, Wojciech Samek, Sebastian Lapuschkin, and Marina M. C. Höhne. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations and beyond, 2023. URL https://arxiv.org/a...

    2023 arXiv

  3. [11]

    Fairx: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability, 2024

    Md Fahim Sikder, Resmi Ramachandranpillai, Daniel de Leng, and Fredrik Heintz. Fairx: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability, 2024. URL https://arxiv.org/abs/2406.14281

    2024 arXiv

  4. [12]

    Captum: A unified and generic model interpretability library for pytorch, 2020

    Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. Captum: A unified and generic model interpretability library for pytorch, 2020. URL h...

    2020 arXiv

  5. [13]

    tf-explain, 2021

    Raphael Meudec. tf-explain, 2021

    2021

  6. [14]

    Inseq: An interpretability toolkit for sequence generation models

    Gabriele Sarti, Nils Feldhus, Ludwig Sickert, and Oskar van der Wal. Inseq: An interpretability toolkit for sequence generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Association for...

    2023 doi

  7. [15]

    Umang Bhatt, Adrian Weller, and Jos \'e M. F. Moura. Evaluating and aggregating feature-based model explanations. In International Joint Conference on Artificial Intelligence, 2020. URL https://api.semanticscholar.org/CorpusID:218486810

    2020

  8. [16]

    Inouye, and Pradeep Ravikumar

    Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar. On the (in)fidelity and sensitivity for explanations. arXiv: Learning, 2019. URL https://api.semanticscholar.org/CorpusID:166227838

    2019

  9. [17]

    Goodfellow, Moritz Hardt, and Been Kim

    Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Neural Information Processing Systems, 2018. URL https://api.semanticscholar.org/CorpusID:52938797

    2018

  10. [18]

    o m, Leander Weber, Sebastian Lapuschkin, and Marina M.-C. H \

    Anna Hedstr \"o m, Leander Weber, Sebastian Lapuschkin, and Marina M.-C. H \"o hne. Sanity checks revisited: An exploration to repair the model parameter randomisation test. ArXiv, abs/2401.06465, 2024. URL https://api.semanticscholar.org/CorpusID:266977162

    2024 arXiv

  11. [19]

    Jaakkola

    David Alvarez-Melis and T. Jaakkola. Towards robust interpretability with self-explaining neural networks. ArXiv, abs/1806.07538, 2018. URL https://api.semanticscholar.org/CorpusID:49324194

    2018 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.