Pith. sign in

REVIEW 5 major objections 5 minor 4 references

EvoXplain: When Machine Learning Models Agree on Predictions but Disagree on Why -- Measuring Mechanistic Multiplicity Across Training Runs

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EvoXplain claims that a training pipeline can yield several distinct explanation basins even when predictions are identical, so a single attribution profile is one draw from a distribution.

desk verdict The framework is plausible, but the paper's key empirical claim is undercut by its own Table 1. read the letter →

arxiv 2512.22240 v5 pith:CQP5RSML submitted 2025-12-23 cs.LG cs.AI

Chama Bensmail This is my paper · ORCID
classification cs.LGcs.AI
keywords mechanisticmultiplicityexplanatorybasinsSHAPattributionstabilitytrainingpipelineinterpretabilityRashomoneffectmodelretrainingexplanationdistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EvoXplain asks whether a training pipeline produces one stable explanation or several distinct ones when the same model class is retrained many times. Its central claim is that predictive agreement does not imply mechanistic agreement: on the Adult Income benchmark, a fixed deep-network pipeline with identical hyperparameters and data splits repeatedly converged to two separate clusters of SHAP attribution vectors, with nearly identical accuracy but different feature-level reasoning. On the Breast Cancer benchmark, the same architecture collapsed to one basin, while logistic regression moved between two basins as regularization strength varied. The paper concludes that explanatory stability is an emergent property of the admissible training pipeline, not an intrinsic property of model class, so a single attribution profile should be read as one draw from a distribution of explanations. The paper is careful to state that basins are diagnostics, not claims about causal mechanisms.

What carries the argument

The load-bearing object is the explanatory basin: a cluster of unit-normalized attribution vectors produced by repeated instantiations of the same admissible training pipeline. EvoXplain's pipeline is: train R models from the same model class under admissible variation; compute a global SHAP attribution vector for each model over a boundary set near the decision threshold; L2-normalize the vectors; measure pairwise differences with cosine distance; run k-means for k=2..K_max and select k* by a silhouette threshold τ; if k*=1, explanations are judged to converge to one basin; otherwise each basin gets an empirical support p_j and the normalized Shannon entropy H quantifies how spread the runs

What would settle it

Compute pairwise cosine similarities within and between the two Adult-DNN basins and compare against a null distribution from random label permutation in the same unit-normalized space; if between-basin similarity is not statistically separated from within-basin similarity, the two-basin result fails. A second check is to repeat the protocol with a different attribution method, such as gradient-based saliency; if the two-basin structure disappears, the reported multiplicity is an artifact of SHAP rather than a property of the pipeline.

Watch

Extended reading notes

Core claim

On the Adult Income benchmark, a fixed deep-network pipeline — same architecture, data split, and hyperparameters, with only random seeds varied — repeatedly produced two well-separated clusters of SHAP attribution vectors across five independent splits. Mean accuracy was stable (≈0.853, std ≤0.002) in all runs, and normalized mechanistic entropy was high (0.925–0.977), so both clusters were substantially occupied. For individual test instances, models from different clusters produced nearly identical probabilities while assigning different attributions to features such as capital-gain. On the Breast Cancer Wisconsin dataset the same DNN architecture collapsed to one basin (k*=1, entropy 0),

Load-bearing premise

The load-bearing empirical premise is that two clusters of attribution vectors with near-perfect cosine similarity are genuinely different mechanisms; if those near-identical directions are just high-dimensional noise, the k-means separation is spurious and the basin claim collapses.

Editorial extensions

If this is right

  • A single attribution profile from one trained model cannot be treated as the model's reasoning; it is one draw from a distribution of admissible explanations.
  • Averaged or consensus explanations can describe no actual model when basins are separable — the paper calls this a ghost explanation.
  • Audits and governance reviews should report basin structure alongside accuracy, because accuracy alone cannot certify explanation stability.
  • Hyperparameter selection can silently switch a linear model between distinct explanation regimes without changing predictive performance.
  • Predictive agreement at the level of individual decisions does not guarantee that two models reasoned the same way.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step the paper only sketches would be to run EvoXplain on counterfactual or rule-based explanations; the basin machinery does not depend on SHAP specifically.
  • The near-unit cosine similarities reported between runs (CosMean 0.962–1.000) mean the two-basin claim should be tested against a null distribution of clusterings in high-dimensional unit-normalized space before it is used in high-stakes settings; the paper does not supply that null model.
  • If the basin structure survives that stress test, a testable extension is to seed training near each basin and observe whether trajectories stay confined, turning descriptive basins into a predictive account of mechanism accessibility.
  • For foundation-model fine-tuning, the framing suggests that functionally similar models should be checked for explanatory basins before releasing a single checkpoint; the paper points to this as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes EvoXplain, a diagnostic framework that assesses explanation stability by repeatedly training a fixed model family under an admissible pipeline, computing SHAP attribution vectors, L2-normalizing them, and clustering them with k-means for k=2..Kmax. The number of basins k* is selected by a silhouette threshold tau, and mechanistic entropy plus basin centroids are used to characterize whether explanations converge to one basin or separate into multiple. Experiments on Adult Income and Breast Cancer Wisconsin with DNNs and Logistic Regression claim three regimes: stable convergence, explanatory multiplicity, and conditional multiplicity. The central assertion is that predictive agreement does not imply mechanistic agreement and that interpretability should be measured at the population level of trained models rather than on a single instance.

Significance. The question addressed is timely and important: whether equally accurate models can systematically differ in their attribution profiles is directly relevant to applied ML, model auditing, and the Rashomon/underspecification literature. The proposed framework is simple, model-agnostic, and the authors provide a public repository, which are strengths. If the empirical results were valid, EvoXplain would be a practical tool for deciding when a single explanation is misleading. The paper is also appropriately cautious about not treating basins as causal mechanisms. However, the headline Adult-DNN result rests on clustering evidence that is internally inconsistent and lacks any null-model comparison, and the robustness analysis is deferred to a missing appendix. As written, the central empirical claim is not yet supported.

major comments (5)
  1. [Table 1, Section 7.2.1] CosMin and CosMean are never defined. If they are pairwise similarities among the 200 attribution vectors, Split 101's values of 1.000/1.000 imply all vectors have identical direction, making a two-cluster solution with silhouette 0.407 impossible. If they are similarities between basin centroids, CosMin=1.000 means the two centroids are identical, contradicting the claim of 'distinct predictive mechanisms.' Either interpretation undercuts the Adult-DNN two-basin result. These metrics must be defined and reconciled with the clustering output.
  2. [Section 4.4, Appendix A] The threshold tau is never reported, and Kmax and the boundary-set bandwidth are also unspecified. Section 4.4 states that conclusions are 'qualitatively stable for reasonable variations of tau (see Appendix A),' but Appendix A only points to a repository and does not contain the promised robustness analysis. Since k* is defined directly by the condition max S(k) >= tau, the central claims are not reproducible or falsifiable without reporting tau and a sensitivity analysis over tau, Kmax, and the boundary-set construction.
  3. [Sections 4.3, 6.5] No null model is provided for the k-means/silhouette procedure. In a 100-dimensional unit-normalized space, k-means will split any point cloud, and silhouette values in the 0.4-0.6 range can arise from a single unimodal distribution on the sphere. The claim that the two Adult-DNN basins are 'distinct predictive mechanisms' requires comparing the observed separation with a null distribution (e.g., permuted or bootstrap-resampled attribution vectors). Without such a baseline, the clustering evidence is ambiguous.
  4. [Section 7.2.2, Figure 4] The semantic interpretation of basins is not quantified. Table 1 reports CosMean values of 0.962-1.000, so the centroid profiles of the two basins are extremely similar in direction, yet the text claims 'clear and systematic differences' across features. No effect sizes, confidence intervals, or permutation tests support this interpretation. Visual inspection of a PCA projection cannot establish two distinct predictive strategies when the reported geometric similarities are near one.
  5. [Abstract vs. Sections 6.1, 7] The abstract describes evaluation on a TCGA pan-cancer cohort and states that all models reach about 98% accuracy, while the full text evaluates Adult Income and Breast Cancer Wisconsin and reports Adult accuracy around 0.853. This is a substantial internal inconsistency that prevents the reader from knowing which experiments are actually being reported. It must be resolved before the paper can be evaluated.
minor comments (5)
  1. [Section 4.1] The statement that Gini-based feature importance was initially tried but is not reported could be removed or accompanied by a citation/repo entry; as written, it is a dangling remark.
  2. [Section 6.4] The boundary set construction is underspecified: the reference classifier, the probability band width around 0.5, and the resulting number of boundary samples are not given. This affects the comparability of attribution vectors across runs.
  3. [Section 7.3] For the logistic regression regime, only a qualitative description is given; no silhouette score, entropy value, or C values are reported in the text or table. Table 2 merely says 'varies,' which is too vague.
  4. [Figure 6] The color scale and exact numerical values of the cosine similarity matrix are not provided in the text; the caption says 'typically exceeding 0.90' but the figure is not reproducible from the caption.
  5. [Notation] The same symbol k* is used for the detected number of basins and as the index of the argmax in Section 4.4; consider distinguishing the selected k from the candidate k values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EvoXplain is an operational diagnostic, not a derivation; its basin definitions are explicit, and no fitted parameter is renamed as a prediction.

full rationale

EvoXplain's pipeline is transparently operational rather than derivational. Section 4.4 defines the basin count by a stated silhouette-threshold rule ('select k⋆ = ... if max_k S(k) ≥ τ; 1 otherwise') and explicitly says that 'This clustering procedure operationally defines explanatory basins as regions of explanation space repeatedly reached under admissible retraining.' The reported basin counts, entropy values, and centroids are outputs of that published decision rule, not predictions of an externally defined quantity. No fitted parameter is later relabeled as a prediction: τ is a user-set detection threshold, and the paper does not claim to predict an independently defined basin count from fitted values. There are no load-bearing self-citations; the cited Rashomon, underspecification, and SHAP-instability literature is external background, and no uniqueness theorem or rescaling ansatz is imported from the authors' prior work. The manuscript's own limitations and weak spots—Section 5's caveat that basins are not claimed to be true causal mechanisms, Section 4.4's deferred τ-robustness check that Appendix A does not actually present (it points to the repository), and Table 1's undefined CosMin/CosMean values that appear internally inconsistent with a two-basin reading for split 101—are validity, reproducibility, and evidence-assessment concerns, not cases where the paper's conclusions reduce to their own inputs by construction. The boundary-set construction in Section 6.4 also ensures illustrative instances sit near p = 0.5, but that is a sampling choice for an existence illustration rather than a circular derivation. Accordingly, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The framework rests on a small number of hand-chosen preprocessing and clustering decisions (τ, Kmax, boundary set), and its central object—the 'explanatory basin'—is an operational construct defined by those choices. There is no null-model calibration to show that the detected clusters are more structured than random noise.

free parameters (5)
  • Silhouette threshold τ = not specified
    Determines whether a multi-basin solution is accepted (k*>1 requires max silhouette ≥ τ). The paper does not report τ in the main text and defers to missing Appendix A.
  • Maximum cluster count Kmax = not specified
    Upper bound of candidate k in k-means; affects which basin counts are considered. No value given.
  • Boundary set bandwidth = not specified
    Test instances are selected if predicted probability falls within an unspecified band around 0.5; this selection determines the explanation vectors.
  • Number of runs R per split = 100–several hundred; 200 for Adult-DNN
    Sample size of the explanation distribution; no power analysis for clustering stability.
  • DNN hyperparameters (hidden sizes, dropout, learning rate) = hidden sizes 100 and 50, Adam, ReLU
    Architecture choices are fixed but their impact on basin structure is not studied.
assumptions (5)
  • domain assumption Cosine distance between normalized attribution vectors reflects mechanistic difference.
    Section 4.3 assumes that the direction of SHAP values captures explanatory logic; no validation of this geometric premise.
  • ad hoc to paper k-means clusters with silhouette above threshold correspond to distinct predictive mechanisms.
    The operational definition of 'explanatory basin' is the output of k-means with silhouette threshold; this is a methodological stipulation, not an externally validated taxonomy.
  • domain assumption SHAP is a valid probe of explanation structure.
    SHAP is used as the explanation method; the paper acknowledges it may not be correct, but the basin structure is claimed to reflect model differences.
  • domain assumption Repeated training runs with different seeds are independent samples of the admissible pipeline.
    Section 6.2 varies seeds, initialization, minibatch order, dropout; the paper assumes these are sufficient to represent the pipeline's explanation distribution.
  • domain assumption The boundary set (test points near 0.5 predicted probability) is representative of explanation structure.
    Section 6.4 restricts explanation computation to a boundary set without justifying that full-test-set explanations would give the same basin structure.
invented entities (2)
  • Explanatory basin
    purpose: A cluster of attribution vectors from repeated training runs that is claimed to correspond to a distinct predictive mechanism.
    Defined entirely by k-means + silhouette within the paper; no external handle (e.g., causal validation) is provided.
  • Mechanistic entropy
    purpose: Normalized Shannon entropy of cluster supports, used to quantify how evenly runs spread across basins.
    Simply normalized entropy of cluster frequencies; a derived summary, not an independent entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvoXplain: When Machine Learning Models Agree on Predictions but Disagree on Why -- Measuring Mechanistic Multiplicity Across Training Runs." pith.science (2026). https://pith.science/paper/CQP5RSML

@misc{pith2026251222240,
  author       = {Pith},
  title        = {Pith review of: EvoXplain: When Machine Learning Models Agree on Predictions but Disagree on Why -- Measuring Mechanistic Multiplicity Across Training Runs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CQP5RSML}},
  note         = {Machine review of arXiv:2512.22240}
}
read the original abstract

Machine learning models are primarily judged by predictive performance, especially in applied genomics, where explanations are read as biological findings. In practice, reported gene panels are stabilised by averaging, ranking, or taking consensus over the many models a pipeline produces across cross-validation folds, tuning grids, and repeated runs. This raises an overlooked question: when two models achieve high accuracy, do they rely on the same internal logic, or reach the same outcome via different mechanisms? We introduce EvoXplain, a diagnostic framework that measures whether a pipeline's explanation is uniquely determined across repeated training and model selection. Rather than analysing a single trained model, EvoXplain treats explanations as samples drawn from the training and model selection pipeline itself, without aggregating predictions or constructing ensembles, and examines whether they form a single coherent explanatory basin or separate into multiple structured basins. We evaluate EvoXplain on a TCGA pan-cancer cohort and a within-cancer breast-cancer subtype task, using elastic-net Logistic Regression and gradient-boosted trees. Although all models reach about 98% accuracy, explanation structure differs across pipelines. Holding the data split fixed and varying only the regularisation strength, equally accurate Logistic Regression models separate into a few discrete, reproducible basins that recur across 100 data splits and carry distinct biological content, while the gradient-boosted pipeline converges to one basin. The same multiplicity appears within a single cancer subtype, from the ordinary tuning step alone. EvoXplain makes explanatory structure visible, revealing when an averaged consensus corresponds to no single trained model, and reframes interpretability as a property of the training pipeline rather than of any single model.

Figures

Figures reproduced from arXiv: 2512.22240 by the authors.

Figure 1
Figure 1. Conceptual illustration of explanatory regimes detected by EvoXplain. Each point represents a normalized attribution vector obtained from an independently trained model instantiation under the same admissible training pipeline. Attribution vectors are projected into a low-dimensional explanation space using principal component analysis (PCA) for visualization. Clusters correspond to explanatory basins—regions of exp… view at source ↗
Figure 2
Figure 2. Stable explanatory convergence in the BC-DNN pipeline. Each point represents the [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Explanatory multiplicity in the Adult-DNN pipeline. Each point represents the normal [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Centroid attribution comparison between explanatory basins in the Adult-DNN [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Instance-level explanatory disagreement between explanatory basins in the Adult-DNN [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Cross-split stability of explanatory basins in the Adult-DNN pipeline. Each cell shows [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]
Figure 7
Figure 7. Figure 7: Conditional basin accessibility in logistic regression on the Breast Cancer Wisconsin [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 1 linked inside Pith

  1. [3]

    Cynthia Rudin

    doi: 10.23915/distill.00024.001. Cynthia Rudin. Stop explaining black box models for high stakes decisions and use interpretable models instead.Nature Machine Intelligence, 1:206–215,

    doi:10.23915/distill.00024.001

  2. [13]

    Predictive multiplicity in algorithmic fairness: Wrestling with a wicked problem

    Daniel Umbach, Amanda Coston, and Kenneth Holstein. Predictive multiplicity in algorithmic fairness: Wrestling with a wicked problem. InProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (F AccT),

    2023

  3. [2020]

    The BIG argument for AI safety cases.arXiv preprint arXiv:2503.11705,

    Ibrahim Habli, Richard Hawkins, Colin Paterson, Philippa Ryan, Yan Jia, Mark Sujan, and John McDermid. The BIG argument for AI safety cases.arXiv preprint arXiv:2503.11705,

    arXiv

  4. [2025]

    doi: 10.1007/978-3-032-05461-6

    doi:10.1007/978-3-032-05461-6

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.