REVIEW 3 cited by
Unrestricted Permutation forces Extrapolation: Variable Importance Requires at least One More Model, or There Is No Free Variable Importance
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper reviews and advocates against the use of permute-and-predict (PaP) methods for interpreting black box functions. Methods such as the variable importance measures proposed for random forests, partial dependence plots, and individual conditional expectation plots remain popular because they are both model-agnostic and depend only on the pre-trained model output, making them computationally efficient and widely available in software. However, numerous studies have found that these tools can produce diagnostics that are highly misleading, particularly when there is strong dependence among features. The purpose of our work here is to (i) review this growing body of literature, (ii) provide further demonstrations of these drawbacks along with a detailed explanation as to why they occur, and (iii) advocate for alternative measures that involve additional modeling. In particular, we describe how breaking dependencies between features in hold-out data places undue emphasis on sparse regions of the feature space by forcing the original model to extrapolate to regions where there is little to no data. We explore these effects across various model setups and find support for previous claims in the literature that PaP metrics can vastly over-emphasize correlated features in both variable importance measures and partial dependence plots. As an alternative, we discuss and recommend more direct approaches that involve measuring the change in model performance after muting the effects of the features under investigation.
Forward citations
Cited by 3 Pith papers
-
Walkable to Whom? Capturing Subjective Variability in Walkability Perception Using Multimodal Deep Learning
A user-conditioned deep learning model that combines sidewalk images with rater attributes predicts individual walkability ratings better than image-only models, and sidewalk imagery yields higher walkability scores t...
-
Causal SHAP: Feature Attribution with Dependency Awareness through Causal Discovery
Causal SHAP replaces SHAP's independence assumption with a PC-discovered causal graph and IDA-derived causal strengths, zeroing out features that are correlated but not causal.
-
IMEX Interaction-Based Model Explanation
IMEX's PCS metric — normalized leave-one-feature-out prediction variation — recovers known feature structure on three synthetic datasets, while its interaction metric PCI remains unvalidated.
Discussion (0). Continue with ORCID to comment.