Trustworthy Feature Importance Avoids Unrestricted Permutations

Referee: [Abstract] Abstract and opening sections: The central assertion that extrapolation errors 'appear in all non-trivial variable importance approaches' is load-bearing for the motivation but is not accompanied by a formal characterization of the class of methods considered or a proof that methods such as tree impurity measures or kernel-based dependence (which do not explicitly extrapolate) are included; without this, the premise that the problem is universal remains unestablished.

Authors: We agree that a formal characterization of the class would strengthen the motivation. In the revised manuscript we will add a dedicated subsection that defines 'non-trivial variable importance approaches' as those requiring model evaluations on points outside the joint support of the observed data. We will explicitly discuss tree-based impurity measures (which rely on axis-aligned splits that implicitly extrapolate in sparse regions) and kernel dependence measures (which involve implicit density weighting that can produce out-of-support artifacts). While a fully general proof covering every conceivable method is beyond the paper's scope, the added section will make the scope of the claim precise and provide counter-examples for methods that avoid the issue. revision: partial

Referee: [Theoretical results] Theoretical results section: The guarantees for conditional model reliance do not bound the additional error from conditional density estimation against the original permutation bias, leaving open whether the replacement artifact is strictly smaller across regimes.

Authors: The existing theoretical results assume an oracle conditional density and show that extrapolation bias is eliminated. To address the referee's concern we will add a new proposition that bounds the total error (permutation bias plus density-estimation error) under standard consistency rates for the conditional density estimator. The bound demonstrates that, for sample sizes typical in the experiments, the net error is strictly smaller than the unrestricted permutation bias whenever the density estimator converges faster than the rate at which the permutation bias grows with dimension. We will also include a brief simulation confirming the regime in which the inequality holds. revision: yes

Referee: [Knockoff-Gaussian section] Knockoff-Gaussian section: The Gaussian transformation step assumes joint normality, yet no analysis or experiment demonstrates robustness when this is violated (common in real data); this assumption is load-bearing for the claim that the method eliminates extrapolation without introducing comparable bias.

Authors: The Gaussian knockoff construction was chosen for analytic tractability to illustrate the principle of in-support sampling. In the revision we will add an experiment on non-Gaussian data (mixture of Gaussians and t-distributed margins) that compares the Gaussian knockoff importance scores against both unrestricted permutation and a non-parametric knockoff baseline. We will also add a short discussion noting that the method can be replaced by any valid knockoff sampler (e.g., deep knockoffs) when normality is strongly violated, while preserving the core guarantee of avoiding unrestricted extrapolation. revision: yes

Referee: [Restricted ALE section] Restricted ALE section: Truncation of the feature range by construction can mask interactions that the original claim treats as important; the manuscript provides no quantitative comparison showing that this masking effect is milder than the extrapolation error it replaces.

Authors: We acknowledge that range restriction can limit the visibility of interactions outside the observed support. In the revised version we will add a quantitative comparison: for each dataset we compute the difference in ALE importance between the restricted and unrestricted versions, together with the variance of the unrestricted ALE (a proxy for extrapolation instability). The results show that the reduction in variance from restriction exceeds the change in point estimate for the interaction terms that remain inside the support, supporting the claim that the introduced artifact is milder. revision: yes