Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Challenges in Evaluating Explanation Methods for Static and Evolving Data

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Explainable AI is producing many new explanation methods while evaluating them too narrowly, and this paper argues the field should shift toward systematic human-grounded studies and evaluation that tracks how explanations behave as data…

desk verdict A coherent workshop position that reinforces existing critiques of XAI evaluation; its own survey is too thinly reported to carry the preference claim. read the letter →

arxiv 2608.06351 v1 pith:PBFRQ6A2 submitted 2026-08-06 cs.AI

Jerzy Stefanowski This is my paper · ORCID
classification cs.AI
keywords ExplainableAIXAIevaluationHuman-groundedSaliencymapsCounterfactualexplanationsConceptdriftEvolvingdatastreamsBiasdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that explainable AI (XAI) is bottlenecked by how it is evaluated, not by how many methods it produces: new explanation methods keep appearing while their usefulness is checked in narrow, mostly automatic ways. It makes the case through two static-image case studies and two evolving-data studies. In a 148-respondent human survey, three saliency-map methods received clearly different preference scores, and preferences shifted with the animal and the respondent's XAI experience, showing that "which explanation is best" depends on context. For dynamic settings, the paper claims that static explanations go stale and that concept drift should be explained by watching prototypes and group counterfactuals move over time. If this is right, trustworthy AI deployment in high-stakes fields needs a much broader evaluation practice than current benchmarks provide.

What carries the argument

The argument runs on a three-level evaluation taxonomy distinguishing application-grounded, human-grounded, and functionally-grounded evaluation, plus a set of case studies that put the taxonomy to work. The load-bearing mechanisms are: saliency maps and Concept-wise Relevance Propagation for locating bias in image models; a uniform-visualization online ranking survey for comparing explanation methods; an ensemble-of-explainers approach with a Pareto front and Ideal Point Method for choosing counterfactuals; and prototype-displacement measures, feature parallel plots, and cosine similarity between evolving group-counterfactual vectors for detecting and localizing concept drift. Together these mechanisms show how evaluation changes when the audience, the visualization format, and the time dimension are taken seriously.

What would settle it

Rerun the ranking task with the same three methods but change only the rendering style, such as outline-plus-highlight versus heatmap, while keeping the underlying saliency maps identical; if the ProtoPNet preference of 1045 versus 763 versus 643 points shrinks, reverses, or tracks the style rather than the method, the survey's conclusion that ProtoPNet is the most useful explanation would be an artifact of interface design.

Watch

Extended reading notes

Core claim

The central claim is that XAI's bottleneck is evaluation, not invention. The paper asserts that in XAI there is a trend toward constantly introducing more and more new methods while assessment of their usefulness is too limited, that human-grounded evaluation is neglected, and that existing measures focus on static settings while explanations for evolving data require taking into account longer temporal dynamics and the co-evolution of data, models, and explanations. It supports this with case studies: a bias-detection pipeline using Concept-wise Relevance Propagation saliency maps and concept unlearning, a human survey in which ProtoPNet outscored ACE and RISE overall but not for every animal or every user group, and counterfactual and prototype-based analyses showing that drift can be tracked through moving explanation vectors. The paper also concludes that there is no single definitive best counterfactual method and that evaluation should embrace multiple criteria, including Pareto-style selection.

Load-bearing premise

The load-bearing premise is that the 148-person online survey, with its fixed set of 10 animal images and a custom uniform visualization, actually measures which explanation method is more useful rather than which visualization style looks more intuitive.

Editorial extensions

If this is right

  • Deployment of XAI in high-stakes settings should treat human-grounded evaluation as a required component, not an optional supplement, because method rankings change with the audience and the task.
  • No explanation method should be declared best on the basis of automatic proxy metrics alone; the same method can win for one animal or task and lose for another.
  • Counterfactual evaluation should stop looking for a single winner and instead use ensembles plus Pareto-style multi-criteria selection, which also matches how humans weigh trade-offs.
  • For evolving data streams, explanations must be evaluated over time: prototype movement and group-counterfactual vector changes can reveal drift location and feature-level causes before model accuracy degrades.
  • Research should invest in benchmark datasets with annotated ground-truth explanations, because proxy measures can pass while missing the features a ground-truth rule actually uses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If visualization format drives ranking as much as method identity, then many published saliency-map comparisons may be measuring interface design; a controlled rendering swap would settle this. The paper itself flags format standardization as a challenge but does not test format as an independent variable.
  • The same temporal machinery of prototype displacement and group-counterfactual cosine similarity could be transferred from stream classifiers to continual learning and online reinforcement learning, where explanations also need to track non-stationarity.
  • A natural extension is to define 'temporal fidelity' and 'temporal stability' metrics for explanations, analogous to the static proxy measures the paper criticizes, so that drift in explanation quality can be monitored automatically alongside drift in model accuracy.
  • The multi-criteria/Pareto view of counterfactual quality suggests a general evaluation stance: replace 'best method' competitions with trade-off surfaces, and let the user's task pick the point on the surface.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper is a position/talk paper arguing that the evaluation of explainable AI (XAI) methods is insufficiently rigorous: too many new methods are introduced, human-grounded evaluation is neglected, and most existing measures are designed for static settings, whereas real-world systems face evolving data and concept drift. The author supports this position with two case studies: (1) the DetoxAI system, which uses saliency maps and concept unlearning for debiasing image classifiers under quantitative fairness metrics, and (2) a human-grounded survey comparing ACE, ProtoPNet, and RISE on animal image classification, reporting ProtoPNet as the overall preference. The paper then discusses multi-criteria counterfactual selection, prototype-based drift measures, and group counterfactual drift analysis, and closes with open issues for XAI evaluation, including the need for temporal dynamics and more systematic human studies.

Significance. If the central position is accepted, it would push the XAI community toward broader evaluation practices: systematic human studies, multiple complementary metrics, and explicit treatment of temporal dynamics in evolving data. The paper is valuable as a concise synthesis of evaluation taxonomies and as a concrete illustration of why static evaluation is limited. It also gives credit to the author's own prior work on prototype drift measures and group counterfactuals, which are timely contributions. However, the paper's positive empirical evidence is thin and self-referential; the only direct empirical demonstration, the human survey, lacks inferential statistics and controls, so the significance is conditional on the reader accepting the position on general grounds rather than on the presented evidence.

major comments (3)
  1. [Section 3.2, paragraphs beginning 'We decided to develop an online survey' and 'A total of 148 completed surveys were…] The human-grounded study is the paper's main empirical illustration, but it does not report any inferential statistics: there are no confidence intervals, significance tests, effect sizes, or correlation coefficients for the claimed 'strong correlations' (e.g., ACE with elephant trunk, ProtoPNet with hippo mouth or tiger stripes). The headline ranking (ProtoPNet 1,045 vs ACE 763 vs RISE 643) could plausibly lie within sampling variability, so the claim that ProtoPNet is preferred overall is not supported by the reported evidence. Additionally, because all methods were converted to a custom 'uniform graphical interface' (colored line and highlighting), the comparison evaluates the author's rendering of each method rather than the methods as commonly used by the community; this confound should be discussed as a limitation or the study should be re-analyzed and re-reported with appropriate statistical controls.
  2. [Section 3.1, DetoxAI case study] The automated evaluation case study claims that 'most of them led to the elimination of the network's undesirable focus on biases' based on qualitative visual inspection of saliency maps and 'improvement of selected group-level fairness metrics,' but the actual metrics, the number of methods tested, the magnitude of improvements, and the statistical uncertainty are not reported; all details are deferred to reference [28]. Because the paper uses DetoxAI as an example of more rigorous automated evaluation, the absence of these quantitative results makes the case study unverifiable as presented. At minimum, the paper should report the actual fairness metric values and describe how the qualitative claim about saliency maps was assessed.
  3. [Sections 4 and 5, paragraphs on [25], [13], and [26]] The positive proposals for multi-criteria counterfactual selection, prototype drift measures, and group counterfactual drift analysis are presented only by reference to the author's own prior works; no experimental protocol, parameter settings, datasets, or quantitative results are summarized in this manuscript. Given that the central argument is that XAI evaluation must be broadened and made more rigorous, the reader cannot verify whether these methods meet the standards the paper itself calls for. These claims should be framed as preliminary illustrations, and the key evaluation evidence from [13], [25], and [26] should be summarized in enough detail to allow the reader to judge the strength of the support.
minor comments (6)
  1. [Abstract and footnote 1] The footnote in the abstract states 'accepted for a publication ... 2016' while the text refers to IJCAI-ECAI 2026 and the arXiv date is 2026; the year appears to be a typo and should be corrected.
  2. [Section 3.1, sentence beginning 'In in the CelebA dataset'] There is a duplicated word 'in' in 'In in the CelebA dataset'; this should be corrected.
  3. [Section 5, sentence beginning 'However, most these methods'] The phrase 'most these methods' is missing the preposition 'of' and should read 'most of these methods'.
  4. [Section 6, final paragraph] The phrase 'air space exploration' is likely intended to be 'aerospace exploration' or 'space exploration'.
  5. [Reference 31] Reference [31] contains the typo 'Proceedins' in the proceedings title; it should read 'Proceedings'.
  6. [General] Several measurements are referenced only by external papers (e.g., the fairness metrics in [28] and the prototype measures in [13]); adding brief definitions or a short table of the main measures would make the paper more self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper is an experience/position piece whose self-citations refer to independently published, externally falsifiable experiments.

full rationale

This is not a derivation paper: it is an experience/position talk discussing challenges in XAI evaluation. Its central claim (that XAI evaluation is too limited and human-grounded evaluation is neglected) is supported by external reviews and position papers ([16], [20]), not by the author's own prior work. Where the paper does rely on the author's previous works ([13], [25], [26], [27], [28]), those works are presented as published, peer-reviewed empirical studies with experimental results on external datasets and standard fairness metrics; they are independent evidence rather than a self-referential premise. The human-grounded survey in Section 3.2 is an empirical case study, not a quantity derived from its own inputs; its reported "strong correlations" lack reported coefficients and significance tests, and the uniform visualization may threaten construct validity, but these are methodological weaknesses, not circular reductions. No equation is fitted and then renamed a prediction, no uniqueness theorem is imported from the authors' own prior work, and no ansatz is smuggled in via self-citation. The paper also explicitly concedes limitations (e.g., trust was not analyzed), and its positive recommendations are framed as open challenges rather than forced conclusions. Accordingly, no load-bearing step reduces to its own input by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new fitted parameters or invented entities. Its load-bearing assumptions are domain assumptions from the XAI and concept-drift literature, plus the author's own prior definitions of drift-explanation measures. The evidence for the success of those measures is self-cited.

assumptions (3)
  • domain assumption The Doshi-Velez and Kim taxonomy (application-, human-, function-based evaluation) is a valid basis for classifying XAI evaluation.
    Invoked in Section 2 as the organizing framework for evaluating explanations; the paper does not question this taxonomy.
  • domain assumption Validity, proximity, sparsity, actionability, and plausibility are the appropriate desiderata for counterfactual explanations.
    Listed in Section 4 as standard properties of counterfactuals, following the cited literature [9,30]; these are assumed without derivation.
  • domain assumption Concept drift is adequately captured by changes in data distributions and model performance, and prototype/group-counterfactual measures track these changes.
    Section 5 assumes that the proposed prototype distances and counterfactual vector cosine similarities are meaningful proxies for drift causes; this assumption is from the author's prior works [13,26] and not independently validated in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Challenges in Evaluating Explanation Methods for Static and Evolving Data." pith.science (2026). https://pith.science/paper/PBFRQ6A2

@misc{pith2026260806351,
  author       = {Pith},
  title        = {Pith review of: Challenges in Evaluating Explanation Methods for Static and Evolving Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PBFRQ6A2}},
  note         = {Machine review of arXiv:2608.06351}
}
read the original abstract

This paper addresses the limitations of Explainable Artificial Intelligence (XAI) with respect to insufficient evaluation. They are illustrated through the DetoxAI image recognition system for bias detection and concept unlearning. Then, an example of a human-grounded evaluation of methods for explaining image classification is presented. The paper further explores methods for adapting explanations to evolving data streams with concept drift. Experiences with adapting counterfactuals for this problem are discussed. Finally it is related to the challenges of tracking the co-evolution of data, models, and explanations.\footnote{This paper has been accepted for a publication in J.Nalepa (ed) Explainable AI in Space. Proceedings of EASi 2026 Workshop at IJCAI-ECAI 2026 Bremen, Springer CCIS vol 3107 (2016).}

Figures

Figures reproduced from arXiv: 2608.06351 by the authors.

Figure 1
Figure 1. The left panel shows raw face images, and the right panel, corresponding CRP saliency maps. In the saliency maps, red hues indicate positive relevance the true class, while blue hues indicate negative contributions - more details in [28]. In particular image neural networks may develop internal representations that encode not only useful high-level features but also harmful biases [3]. In [28] we have considered sev… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 24 canonical work pages

  1. [28]

    Investigating the Relationship Between Debiasing and Artifact Removal using Saliency Maps

    Sztukiewicz, L., Stępka, I., Wiliński, M., Stefanowski, J.: Investigating the relation- ship between debiasing and artifact removal using saliency maps. arXiv preprint arXiv:2503.00234 (2025)

    work page Pith review arXiv 2025

  2. [13]

    Information Fusion p

    Karolczak, J., Stefanowski, J.: Explaining data changes with prototypes: A measure-driven approach. Information Fusion p. 103602 (2025)

    work page 2025

  3. [25]

    International Journal of Applied Mathematics and Computer Science34(1), 119– 133 (2024)

    Stepka, I., Lango, M., Stefanowski, J.: A multi-criteria approach for selecting an explanation from the set of counterfactuals produced by an ensemble of explainers. International Journal of Applied Mathematics and Computer Science34(1), 119– 133 (2024)

    work page 2024

  4. [26]

    Stępka, I., Stefanowski, J.: Explaining concept drift through the evolution of group counterfactuals. In: Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2025, Porto, Portugal, September 15–19, 2025, Revised Selected Papers, Part IV. pp. 413 – 427. Springer Nature (2026)

    work page 2026

  5. [1]

    Nature Machine Intelligence5, 1006–1019 (09 2023)

    Achtibat, R., Dreyer, M., Eisenbraun, I., Bosse, S., Wiegand, T., Samek, W., Lapuschkin, S.: From attribution maps to human-understandable explanations through concept relevance propagation. Nature Machine Intelligence5, 1006–1019 (09 2023). https://doi.org/10.1038/s42256-023-00711-8

    doi:10.1038/s42256-023-00711-8 2023

  6. [2]

    Information fusion58, 82–115 (2020)

    Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al.: Explainable artifi- cial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion58, 82–115 (2020)

    2020

  7. [3]

    Advances in Neural Information Processing Systems29(2016) 12 Jerzy Stefanowski

    Bolukbasi, T., Chang, K.W., Zou, J.Y., Saligrama, V., Kalai, A.T.: Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in Neural Information Processing Systems29(2016) 12 Jerzy Stefanowski

    work page 2016

  8. [4]

    In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence

    Byrne, R.M.: Counterfactuals in explainable artificial intelligence (xai): Evidence from human reasoning. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. pp. 6276–6282 (2019)

    work page 2019

Show all 31 references
  1. [5]

    Doshi-Velez,F.,Kim,B.:Towardsarigorousscienceofinterpretablemachinelearn- ing (2017), https://arxiv.org/abs/1702.08608

    2017 arXiv

  2. [6]

    Master’s thesis, Poznan University of Technology, Faculty of Computer Science and Telecommunication (2023)

    Frackowiak, Z.: Comparing methods explaining predictions of image neural net- works. Master’s thesis, Poznan University of Technology, Faculty of Computer Science and Telecommunication (2023)

    2023

  3. [7]

    Accepted for IJCAI-2026 Conference Main Track, also arXiv preprint arXiv:2405.17642 (2024)

    Furman, O., Wielopolski, P., Lenkiewicz, Ł., Stefanowski, J., Zięba, M.: Unifying perspectives: Plausible counterfactual explanations on global, group-wise, and lo- cal levels. Accepted for IJCAI-2026 Conference Main Track, also arXiv preprint arXiv:2405.17642 (2024)

    2024 arXiv

  4. [8]

    Artificial Intelligence291, 103428 (02 2021)

    Guidotti, R.: Evaluating local explanation methods on ground truth. Artificial Intelligence291, 103428 (02 2021). https://doi.org/10.1016/j.artint.2020.103428

    2021

  5. [9]

    Data Mining and Knowledge Discovery pp

    Guidotti, R.: Counterfactual explanations and how to find them: literature review and benchmarking. Data Mining and Knowledge Discovery pp. 1–55 (04 2022)

    2022

  6. [10]

    ACM computing surveys (CSUR)51(5), 1–42 (2018)

    Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM computing surveys (CSUR)51(5), 1–42 (2018)

    2018

  7. [11]

    Statistical Science 21(1), 1–14 (2006)

    Hand, D.J.: Classifier technology and the illusion of progress. Statistical Science 21(1), 1–14 (2006). https://doi.org/10.1214/088342306000000060

    2006 doi

  8. [12]

    Neurocomputing555, 126640 (2023)

    Hinder, F., Vaquet, V., Brinkrolf, J., Hammer, B.: Model-based explanations of concept drift. Neurocomputing555, 126640 (2023)

    2023

  9. [14]

    ACM Computing Surveys55(2), 1–38 (2022)

    Kaur, D., Uslu, S., Rittichier, K.J., Durresi, A.: Trustworthy artifi- cial intelligence: A review. ACM Computing Surveys55(2), 1–38 (2022). https://doi.org/10.1145/3491209

    2022 doi

  10. [15]

    In: Pro- ceedings of the Fortieth AAAI Conference on Artificial Intelligence

    Kavouras, L., Psaroudaki, E., Tsopelas, K., Rontogiannis, D., Theologitis, N., Sacharidis, D., Giannopoulos, G., Tomaras, D., Markou, K., Gunopulos, D.: GLANCE: Global actions in a nutshell for counterfactual explainability. In: Pro- ceedings of the Fortieth AAAI Conference on...

    2026

  11. [16]

    Applied Sciences12(19) (2022)

    Lopes, P., Silva, E., Braga, C., Oliveira, T., Rosado, L.: Xai systems evaluation: A review of human and computer-centred methods. Applied Sciences12(19) (2022)

    2022

  12. [17]

    IEEE Transactions on Knowledge and Data EngineeringPP, 1–1 (10 2018)

    Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., Zhang, G.: Learning under concept drift: A review. IEEE Transactions on Knowledge and Data EngineeringPP, 1–1 (10 2018). https://doi.org/10.1109/TKDE.2018.2876857

    2018

  13. [18]

    ACM Computing Surveys54(6) (Jul 2021)

    Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys54(6) (Jul 2021)

    2021

  14. [19]

    Frontiers in Artificial Intelligence (2022)

    Mertes, S., Huber, T., Weitz, K., Heimerl, A., André, E.: Ganterfac- tual—counterfactual explanations for medical non-experts using generative adver- sarial learning. Frontiers in Artificial Intelligence (2022)

    2022

  15. [20]

    In: Proceedings of the 43rd International Conference on Machine Learning (ICML) (2026)

    Moshkovitz, M., Srinivas, S., Semenova, L., Frost, N., Rashtchian, C., Boreiko, V., Zhang, S., Lakkaraju, H., Rudin, C., Vaughan, J.W.: Position: Explainability research must prioritize foundations over ad-hoc methods. In: Proceedings of the 43rd International Conference on Ma...

    2026

  16. [21]

    Algorithms18(7) (2025) Challenges in evaluating explanation methods

    Pelosi, D., Cacciagrano, D., Piangerelli, M.: Explainability and interpretability in concept and data drift: A systematic literature review. Algorithms18(7) (2025) Challenges in evaluating explanation methods ... 13

    2025

  17. [22]

    ACM Comput

    Read, J., Zliobaite, I.: Supervised learning from data streams: An overview and update. ACM Comput. Surv.57(12) (Jul 2025). https://doi.org/10.1145/3737279, https://doi.org/10.1145/3737279

    2025 doi

  18. [23]

    Computers and Electrical Engineering118, 109370 (2024)

    Sadeghi, Z., Alizadehsani, R., CIFCI, M.A., Kausar, S., Rehman, R., Mahanta, P., Bora, P.K., Almasri, A., Alkhawaldeh, R.S., Hussain, S., Alatas, B., Shoeibi, A., Moosaei, H., Hladík, M., Nahavandi, S., Pardalos, P.M.: A review of explainable artificial intelligence in healthc...

    2024

  19. [24]

    In: Asian Conference on Intelligent Information and Database Systems ACIIDS 2023

    Stefanowski, J.: Multi-criteria approaches to explaining black box machine learning models. In: Asian Conference on Intelligent Information and Database Systems ACIIDS 2023. pp. 195–208. Springer (2023)

    2023

  20. [27]

    In: Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases

    Stępka, I., Sztukiewicz, L., Wiliński, M., Stefanowski, J.: Detoxai: A python toolkit for debiasing deep learning models in computer vision. In: Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases. pp. 502–505. Springer (2025)

    2025

  21. [29]

    IEEE AccessPP, 1–1 (01 2025)

    Tahir, H., Jabbarpour, M., Vo, B., Kowalczyk, R., Barr, J., Bessell, T.: Toward explainable ai in spacecraft health monitoring: Comparative benchmarking of anomaly detection models and open-source datasets. IEEE AccessPP, 1–1 (01 2025). https://doi.org/10.1109/ACCESS.2025.3639551

    2025

  22. [30]

    ACM Computing Surveys (2020)

    Verma, S., Boonsanong, V., Hoang, M., Hines, K., Dickerson, J., Shah, C.: Coun- terfactual explanations and algorithmic recourses for machine learning: A review. ACM Computing Surveys (2020)

    2020

  23. [31]

    In: Proceedins of ECAI 2024 Conference, pp

    Wielopolski, P., Furman, O., Stefanowski, J., Zięba, M.: Probabilistically plausible counterfactual explanations with normalizing flows. In: Proceedins of ECAI 2024 Conference, pp. 954 – 961. IOS Press (2024)

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.