{"id":"777fc72e-34ad-40bf-ab17-8112e6073274","arxiv_id":"2505.10515","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PnPXAI automatically selects and tunes post hoc explanation methods for neural networks across vision, language, time series, and tabular data.","lead":"PnPXAI is a software framework that automatically detects a neural network's architecture, recommends which explainability (XAI) methods to use, and tunes their hyperparameters. The authors test it on medical and financial tasks and report that users find it convenient, though the validation is limited.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Default correctness metric ABPC is imported from a demographic-parity paper and never defined or validated as explanation faithfulness, so the optimizer's 'optimal explanations' claim is unsupported.","rationale":"The paper is a systems contribution with a public codebase, plausible modular design, and qualitative demonstrations across modalities; I read the central claim charitably as 'a non-expert can obtain trustworthy, optimized explanations in a plug-and-play manner.' The most load-bearing link is not the detector or even the hand-authored Table 2 mapping, because those errors would be localized and easy to smoke-test. It is the evaluator/optimizer loop: the only quantitative source of 'optimality' is the default ABPC objective, which is borrowed from demographic-parity fairness work, never defined, and never validated as a faithfulness metric. If ABPC does not track true explanation quality, every downstream 'optimal explanation' claim collapses, and user satisfaction ratings cannot compensate. The reader's weakest_assumption bundles this concern with the Table 2 mapping; I partially agree because both are validation gaps, but ABPC is the load-bearing one since it undermines the quantitative meaning of the entire optimization phase. The issue is empirically checkable and addressable, so the engineering contribution need not be rejected; however, the central claim as stated is not currently established. CONDITIONAL remains the appropriate verdict, and my read does not change it.","tokens_in":9364,"tokens_out":5847,"duration_ms":59464,"concrete_test":"Re-run the Figure 2 grid search on the LiTS liver-tumor task, recording for each explainer and hyperparameter trial the exact ABPC score used by the optimizer plus the paper's two ground-truth fidelity metrics: relevance mass accuracy and relevance rank accuracy against the tumor/healthy-liver masks. Compute Spearman rank correlations between ABPC and each fidelity metric per explainer. If ABPC is not consistently and positively correlated with ground-truth fidelity, the optimizer is selecting hyperparameters by an unvalidated proxy. Independently, re-run the same optimization with relevance mass accuracy as the objective and compare the selected hyperparameters; material divergence would show the reported 'optimized' sets depend on an unvalidated default metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that PnPXAI produces 'optimal explanations' rests on the evaluator/optimizer loop, whose default objective is ABPC. In the Liver Tumor Detection use case, the paper states: 'ABPC (Han et al. 2023) is chosen as the default objective with the evaluation property of correctness as it measures whether the explanation is faithful to the model's decision without requiring the ground truth attribution mask.' This is a missing-support passage: ABPC is never defined or adapted in the paper. The cited reference is a demographic-parity fairness paper, where ABPC is a distribution-level disparity metric between protected groups, not an explanation-faithfulness metric. Without a formal definition or an empirical validation that ABPC tracks attribution quality, the hyperparameter optimizer is maximizing an unmotivated scalar. Figure 2 reports that optimized hyperparameters improve ground-truth relevance mass/rank accuracy, but on one dataset and without reporting any correlation between ABPC and those fidelity metrics. Figure 3's claim that explainer rankings 'align' with ABPC is asserted without displaying ABPC values. If the optimizer's objective is not a valid fidelity measure, then 'optimal' explanations are not established in any quantitative sense, and the paper's central promise to non-expert users is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PnPXAI, an open-source explainable AI (XAI) framework that claims to be universal across data modalities and neural network architectures. The framework comprises five modules: a detector that builds a symbolic graph of a user-provided model, a recommender that filters applicable explanation methods using a hand-authored mapping table, an explainer pool, an evaluator that scores explanations with quantitative metrics, and a hyperparameter optimizer that tunes explanation methods via grid search. The central claim is that a few lines of code produce 'optimal explanations' without user expertise. Validation consists of a 31-participant user survey, a liver tumor detection use case with CT ground-truth masks, an acute kidney injury (AKI) detection use case with known biomarkers on MIMIC-III, and a bank account fraud detection interface.","tokens_in":9588,"tokens_out":4389,"duration_ms":43528,"significance":"If the central claims were fully supported, PnPXAI would be a useful contribution: it lowers the barrier for non-expert practitioners to apply and tune explanation methods, and the modular design with public code, API documentation, and a demo is a genuine practical asset. The liver tumor and AKI use cases provide positive evidence that hyperparameter optimization improves ground-truth relevance accuracy and that recommended explainers identify expected biomarkers. However, the strongest claim—that PnPXAI produces 'optimal explanations'—currently rests on an undefined default objective (ABPC) and on a hand-authored recommender mapping that is not validated. The absence of comparisons against existing frameworks such as Captum, OmniXAI, AutoXAI, and OpenXAI also makes the 'universal' claim unsubstantiated. These issues are fixable in a revision, but they are load-bearing for the paper's main contribution.","major_comments":[{"comment":"The paper states in the Liver Tumor Detection use case that 'ABPC (Han et al. 2023) is chosen as the default objective with the evaluation property of correctness as it measures whether the explanation is faithful to the model's decision without requiring the ground truth attribution mask,' but ABPC is never defined anywhere in the manuscript. The cited reference is a demographic parity paper in which ABPC is a distribution-level disparity metric, not an explanation faithfulness metric. Because the optimizer maximizes this undefined objective for all recommended explainers, the claim that PnPXAI produces 'optimal explanations' is unsupported. The authors must define ABPC for attribution maps, state precisely how it is computed from the model and explanation, and validate it as a faithfulness measure (for example, by reporting its correlation with the ground-truth relevance mass and rank accuracy used in Figure 2).","section":"Evaluator / Liver Tumor Detection"},{"comment":"The quantitative validation of hyperparameter optimization reports relevance mass accuracy and rank accuracy improvements for most explainers, but it is based on a single model (ResNet50) on a single sliced-CT dataset, with no error bars, no repeated runs, and no statistical significance tests. Furthermore, the figure does not show the ABPC values for default versus optimized settings, so the reader cannot verify that the ABPC objective being optimized is actually aligned with the ground-truth fidelity metrics. Please provide per-seed or per-slice statistics, significance tests, and an explicit ABPC-versus-ground-truth correlation, or otherwise restrict the conclusion to the reported cases.","section":"Hyperparameter Optimizer / Figure 2"},{"comment":"Table 2 is the sole basis for the recommender module, yet there is no empirical or theoretical justification that the mapping of explanation methods to modalities and architectures is correct or complete. For example, the table lists AttentionRollout and TransformerAttribution only for transformers, but it does not explain why these methods are not applicable to recurrent or convolutional architectures. Since the recommender's output is a core claimed advantage over other frameworks, a wrong or incomplete mapping would mislead non-expert users. Please provide a validation protocol (e.g., sanity checks on representative models of each architecture type, or references establishing applicability) and ideally a comparison of recommended versus non-recommended explainers on a common benchmark.","section":"Recommender / Table 2"},{"comment":"The text claims that 'the line graph in Figure 3 depicts better identification of the most important features by LIME, Integrated Gradients, which aligns with the highest scores in ABPC metric,' but no ABPC scores or ABPC-based rankings are displayed in Figure 3, and no quantitative measure of alignment (correlation, rank overlap, or similar) is provided. Without the ABPC values and an alignment statistic, the claimed coherence between the evaluator and known biomarkers is an assertion rather than a demonstrated result. Please report the actual ABPC values for the explainers and the alignment measure used.","section":"Acute Kidney Injury Detection / Figure 3"},{"comment":"The paper's 'universal' claim is supported only by the feature-checklist comparison in Table 1; there is no experimental comparison against Captum, OmniXAI, AutoXAI, Xaitk-Saliency, or OpenXAI on a common task. Since the novelty is framed as overcoming limitations of these frameworks, a direct comparison on at least one task (e.g., the liver tumor or AKI use case) is needed to substantiate the claimed advantages in flexibility, number of supported methods, and recommendation quality. Without such a comparison, the contribution is demonstrated only in absolute terms, not relative to existing tools.","section":"Introduction / Table 1"}],"minor_comments":[{"comment":"The table header contains a typo: 'A verage' should be 'Average'.","section":"Table 3"},{"comment":"The caption contains a typo: 'Comparision' should be 'Comparison'.","section":"Table 1 caption"},{"comment":"The phrase 'As shown in the top center of Figure 2' is ambiguous; the intended panel (hyperparameter optimization trials, relevance mass accuracy, or relevance rank accuracy) should be named explicitly.","section":"Figure 2 text"},{"comment":"The code snippet would benefit from a brief explanation of the expected data batch format, since the input_extractor, label_extractor, and target_extractor lambdas are the only documentation of the plug-and-play interface.","section":"Code 1"},{"comment":"The paper refers to the Co-12 categories but does not specify which concrete metrics instantiate correctness, continuity, and compactness, nor which of the three properties ABPC is supposed to belong to; a short table of the implemented metrics and their definitions would improve reproducibility.","section":"Evaluator section"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful and reproducible open-source artifact, and the ground-truth-oriented use cases are a positive step. However, the undefined ABPC objective is a load-bearing issue for the 'optimal explanations' claim, and the lack of comparisons with existing frameworks weakens the universality claim. I would be willing to review a revised version that defines and validates ABPC, adds statistical rigor to the hyperparameter optimization experiments, and provides at least one direct comparison against existing XAI frameworks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"PnPXAI is a competent integration of existing XAI tooling, with a public codebase and one genuinely informative use case. The biggest weakness is also the easiest to fix: the default correctness metric ABPC is imported from a demographic-parity paper, never defined or validated as a faithfulness metric, and the paper's 'optimal explanations' claim rests on it. Read it for the detector and the liver-tumor ground-truth evaluation; treat the optimizer's objective as unsupported.\n\nWhat is new is modest but real. The detector traverses the symbolic graph and locates layers for layer-wise explainers (LRP, RAP) on custom architectures—the piece I haven't seen in Captum, OmniXAI, or AutoXAI. The recommender mapping table and the AutoExplanation wrapper make the framework easy to use, and the code, docs, and demo are public. The liver-tumor use case is the strongest evidence: hyperparameter optimization improves relevance mass and rank accuracy against LiTS masks, and the heatmaps look sensible. That is real, non-circular validation for the tuning component.\n\nThe soft spots are proportionate. ABPC is the main one. The paper says it measures faithfulness without a ground-truth mask, but gives no definition or adaptation, and the cited paper is about group fairness. The authors report no correlation between ABPC and their own ground-truth metrics, so there is no evidence the optimizer maximizes something that tracks explanation quality. That is load-bearing. Fixing it means defining ABPC in XAI terms, validating it against ground-truth metrics, or replacing it.\n\nOther issues are smaller. There are no comparisons against AutoXAI or OmniXAI. The mapping table is hand-authored with no correctness or completeness check, and the 'universal' framing overclaims relative to that table. The survey is 31 participants, mostly XAI-experienced, no significance tests. None are fatal for an integration paper, but they cap the claims. The reader's circularity concern is wrong on the core: the use cases use external ground truth; the circularity is in using the framework's own evaluator without validating that evaluator.\n\nWho benefits? Practitioners in medicine, finance, or applied fields who want a low-effort XAI pipeline, and XAI tool builders. It is not the place for new explanation methods or metrics. I would send it to peer review with a requirement to define and validate ABPC (or replace it) and add at least one baseline comparison. I would not cite it until the metric issue is resolved.","headline":"PnPXAI is a useful integration with real engineering and a good liver-tumor ground-truth example, but relying on an undefined fairness-paper metric (ABPC) for 'optimal' explanations is a load-bearing gap.","tokens_in":10112,"tokens_out":5636,"would_cite":false,"duration_ms":48022,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PnPXAI claims to auto-detect a model's architecture and return tuned explanations with no user configuration.","keywords":["explainable AI","XAI framework","post hoc explanation","attribution methods","hyperparameter optimization","model-agnostic","automatic explanation","plug-and-play"],"falsifier":"Take a model with known ground-truth attributions, run every explanation method the framework does and does not recommend, and check whether the recommended set contains the best-scoring method under the framework's own metric; a case where an excluded method beats all recommended ones would falsify the recommender. A second check: if tuning hyperparameters to maximize the internal score decreases agreement with ground-truth masks, the optimizer's objective is not tracking faithfulness.","tokens_in":9188,"feed_emoji":"🧩","tokens_out":6417,"duration_ms":58837,"temperature":0.7,"pith_summary":"The paper introduces PnPXAI, a framework intended to make post hoc explanation of neural networks a plug-and-play operation: a user supplies a trained model and dataset, and the framework returns attribution maps without manual configuration. It argues that existing XAI libraries force users to know which explainers fit their architecture and how to set hyperparameters, blocking adoption in real-world settings. PnPXAI automates this by detecting the model architecture from a symbolic graph, filtering a pool of explanation methods against a mapping table of modalities and architecture families, and tuning each method's hyperparameters against quantitative scores. If the central claim holds, practitioners in medicine, finance, and other fields can obtain and trust explanations without specialist XAI expertise.","feed_headline":"One framework auto-selects and tunes XAI methods for any model","feed_subtitle":"It detects the architecture, recommends explainers, and optimizes them so practitioners need no XAI expertise.","key_machinery":"The central mechanism is the recommender's mapping table, which declares which explanation methods are applicable to each combination of data modality (vision, language, structured data, time series) and architecture family (linear, convolutional, recurrent, transformer, decision tree). The detector produces the architecture side of the key by tracing a symbolic graph of the user's model, including residual connections. The evaluator scores candidate explanations with three quantified properties (correctness, continuity, compactness), and the optimizer grid-searches hyperparameters to maximize the default correctness score. The claim of universality rests on this table: any model that can be classified into a row yields a recommended method list automatically.","core_discovery":"The paper's central claim is that a single framework can automatically do what currently requires an XAI specialist: identify the structure of an arbitrary neural network, decide which explanation methods can run on it, and optimize each method's hyperparameters on the user's own data so the resulting attribution maps are as faithful as possible. The claim is realized as AutoExplanation, which takes a model and dataloader, runs a detector that builds a symbolic graph of layer types and residual connections, consults a recommender that intersects the detected data modality with the detected architecture family, and hands the shortlisted explainers to an evaluator and grid-search optimizer. The default optimization objective is a correctness score called ABPC, chosen because it measures faithfulness without needing ground-truth masks. Validity is argued through use cases where optimized heatmaps align with liver-tumor segmentation masks and known clinical biomarkers, and through a survey in which participants rated the automatic recommendation as the most important feature.","pith_inferences":["The mapping-table design makes the framework a natural benchmark harness: if the table were extended with new explainers, the same pipeline could compare any candidate method against the recommended set on the same models and data.","The authors' stated future direction, applying the pipeline to generative language models, is a direct test of the universality claim because attention-only and decoder architectures would need new rows in the table.","A community-maintained mapping table could drift from the actual applicability of explainers as new architectures appear, so the framework's usefulness will depend on regular empirical revalidation of each row.","The user survey measures satisfaction, not explanation quality, so the strongest evidence for the framework's core claim comes from the medical and financial ground-truth case studies; extending those to more domains would strengthen the universal claim."],"forward_implications":["A user with no XAI background can obtain attribution maps for vision, language, tabular, and time-series models with a few lines of code.","Model-specific explainers (gradient-based, CAM-based, relevance propagation, attention-based) become usable alongside model-agnostic ones, widening the pool of candidate explanations.","Because applicability is defined by the mapping table, adding a new explanation method only requires registering its supported modalities and architectures in that table.","Hyperparameter choices that visibly alter attribution quality, such as the superpixel mask for perturbation-based methods, are tuned automatically rather than left to user intuition.","Quantitative evaluation is integrated into the pipeline, so each recommended explanation comes with scores for correctness, continuity, and compactness rather than an unjudged heatmap."],"supporting_citations":[{"why":"Supplies ABPC, the default correctness objective used by the optimizer to judge explanations.","marker":"(Han et al. 2023)"},{"why":"Prior automatic XAI framework limited to model-agnostic methods; PnPXAI's recommender is positioned as extending this scope.","marker":"(Cugny et al. 2022)"},{"why":"Non-automatic XAI library whose method pool and sub-optimal explanation problem motivate the recommender and optimizer.","marker":"(Yang et al. 2022)"},{"why":"Baseline framework with hard-coded implementations and layer-specific restrictions, used as a comparison point.","marker":"(Agarwal et al. 2022)"},{"why":"Justifies using quantitative evaluation instead of human-grounded judgments in the evaluator module.","marker":"(Adebayo et al. 2018)"},{"why":"Provides ground-truth relevance metrics and evidence that hyperparameters affect explanation quality.","marker":"(Arras, Osman, and Samek 2022)"},{"why":"Liver tumor segmentation dataset used for ground-truth validation of optimized attribution heatmaps in the medical use case.","marker":"(Bilic et al. 2023)"},{"why":"Co-12 evaluation taxonomy from which the evaluator selects correctness, continuity, and compactness properties.","marker":"(Nauta et al. 2023)"}],"fun_headline_variants":["XAI for all models: auto-detect, recommend, and tune explainers","Plug-and-play XAI framework auto-selects and optimizes explainers","Universal XAI: detects architecture, picks methods, tunes them","One framework automates XAI method selection and tuning","AutoXAI: recommends and optimizes explanations for any neural net"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's promise of 'optimal' explanations rests on an unvalidated assumption: that its hand-made table of which explainers work on which models is correct, and that its default score really measures explanation faithfulness.","fun_headline_variants_meta":{"raw":{"variants":["XAI for all models: auto-detect, recommend, and tune explainers","Plug-and-play XAI framework auto-selects and optimizes explainers","Universal XAI: detects architecture, picks methods, tunes them","One framework automates XAI method selection and tuning","AutoXAI: recommends and optimizes explanations for any neural net"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1432,"prompt_tokens":943,"completion_tokens":489,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":398}},"tokens_in":559,"tokens_out":489,"duration_ms":4368,"temperature":1.0,"reasoning_tokens":398,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:07:17.835837+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model with known ground-truth attributions, run every explanation method the framework does and does not recommend, and check whether the recommended set contains the best-scoring method under the framework's own metric; a case where an excluded method beats all recommended ones would falsify the recommender. A second check: if tuning hyperparameters to maximize the internal score decreases agreement with ground-truth masks, the optimizer's objective is not tracking faithfulness.","supporting_citations":[{"cited_title":"Retiring $\\Delta$DP: New Distribution-Level Metrics for Demographic Parity","cited_arxiv_id":"2301.13443","evidence_quote":"Supplies ABPC, the default correctness objective used by the optimizer to judge explanations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior automatic XAI framework limited to model-agnostic methods; PnPXAI's recommender is positioned as extending this scope."},{"cited_title":"OmniXAI: A Library for Explainable AI","cited_arxiv_id":"2206.01612","evidence_quote":"Non-automatic XAI library whose method pool and sub-optimal explanation problem motivate the recommender and optimizer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline framework with hard-coded implementations and layer-specific restrictions, used as a comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ground-truth relevance metrics and evidence that hyperparameters affect explanation quality."},{"cited_title":"B.; Vorontsov, E.; Ben-Cohen, A.; Kaissis, G.; Szeskin, A.; Jacobs, C.; Mamani, G","cited_arxiv_id":null,"evidence_quote":"Liver tumor segmentation dataset used for ground-truth validation of optimized attribution heatmaps in the medical use case."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Co-12 evaluation taxonomy from which the evaluator selects correctness, continuity, and compactness properties."}],"review_version":1}