{"id":"04b5d171-50c7-4467-a141-7018df015ffd","arxiv_id":"2601.23124","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Semi-knockoffs tests conditional independence with arbitrary pre-trained models and no train-test split by comparing losses on two resampled copies of each feature; oracle versions have finite-sample guarantees, estimated versions rest on a stated conjecture.","lead":"This paper introduces Semi-knockoffs, a method for testing whether a feature truly matters for a prediction while using any machine learning model and without splitting the data into train and test sets. It creates two resampled copies of the data — one informed by the response, one not — so that under the null they are identical, giving valid p-values and false-discovery control.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Finite-sample FDR for Algorithm 4 rests on the unproven sign-preservation conjecture of §4.5; Theorem 4.3 bounds only magnitudes, so exact Rademacher signs—and hence FDR≤q—are not established for estimated imputers.","rationale":"The reader's weakest assumption identifies the exact load-bearing gap: practical finite-sample FDR control for Algorithm 4 requires the sign-preservation property that Section 4.5 only conjectures. I agree with this choice. The paper deserves credit for disclosing the conjecture and for strong empirical support (Figures 12, 19–24), but the abstract's 'finite-sample' wording overstates what is proven for the estimated-imputer version. A secondary concern, noted in the reader's rationale, is that Theorem 3.3's proof for oracle p-values does not establish the joint independence needed for the exact Wilcoxon/sign test under permutation-residual sampling; this is real but less central to the headline FDR claim. The proposed simulation targets a misspecified-imputer regime where the conjecture is most likely to fail, and would provide evidence for or against withdrawing the finite-sample FDR claim. Since the reader's verdict is already CONDITIONAL and this stress test does not move it, the verdict remains UNCHANGED.","tokens_in":34131,"tokens_out":12766,"duration_ms":143918,"concrete_test":"Simulate the null regime most hostile to the conjecture: n=300, p=50, X∼N(0,Σ) with Σij=0.6^{|i-j|}, y=0.8X1+0.6X2+0.4X3+0.2X4+sin(X1)+ε, and a null feature j=0. Fit bν with a correctly specified linear model but deliberately fit bρ with a linear model on (X−j,y) when the true E[X0|X−j,y] is nonlinear (or vice versa), and use an accurate gradient-boosting bm. Repeat Algorithm 4 at q=0.2 for 10,000 replicates; record empirical FDR and the empirical sign distribution of Ŵ0^SKO. If FDR exceeds q+2SE or the sign distribution deviates from Rademacher, the §4.5 conjecture fails for a realistic misspecification and the finite-sample FDR claim must be qualified. As a complementary analytic check, attempt to prove the conjecture under the assumptions of Theorem 4.3; the same simulation will reveal whether such a proof is possible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract promises finite-sample FDR control for the practical version of Semi-knockoffs. Algorithm 4 replaces the oracle conditional expectations νj, ρj by estimates bνj, bρj. For the knockoff threshold (1) to control FDR at level q in finite samples, the null statistics Ŵ_j^SKO must have signs that are exactly i.i.d. Rademacher conditional on |Ŵ_j^SKO| (Lemma 2.1 of Candès et al.). The paper's only route to this property for Algorithm 4 is the conjecture in Section 4.5: 'We conjecture that this is the case, based on a double robustness argument.' The proved results do not deliver it. Theorem 4.2 gives W1( P̂1,P̂2)=O_P(√(log(1/δ)/n)), and Theorem 4.3 gives each loss difference l(bm(eX'),y)-l(bm(eX),y)=O_P(a_n b_n). Both are magnitude/rate bounds; they say nothing about the sign of the average difference Ŵ_j^SKO or about its exact conditional Rademacher law. Distributional convergence and asymptotic sign matching imply at best asymptotic FDR control, not the finite-sample bound FDR(Ŝ^SKO)≤q asserted in the abstract. The conjecture could fail whenever the imputer error b_n is not o_P(1) relative to the model sensitivity a_n, e.g., under imputer misspecification. This is the load-bearing soft spot.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Semi-knockoffs, a method for conditional independence testing with arbitrary pre-trained ML models that avoids a train-test split. The central idea is to compare the loss of a model evaluated on two perturbed versions of each feature: one sampled using the conditional expectation of X_j given X_{-j} (ν_j), and one using the conditional expectation given X_{-j} and y (ρ_j). Under the null hypothesis X_j ⊥ y | X_{-j}, both samplers coincide, producing exchangeable loss distributions; under the alternative, the second sampler is less perturbing. The authors prove oracle type-I error control (Theorem 3.3) and oracle FDR control (Theorem 3.4) via knockoff exchangeability. For the practical version with estimated imputers (Algorithm 4), they provide a stability theorem for regularized ERMs (Theorem 4.1), a Wasserstein convergence result (Theorem 4.2), and a double-robustness rate bound (Theorem 4.3). The finite-sample guarantees for the estimated version, however, rest on an explicit conjecture in Section 4.5 that the sign of the empirical statistic matches the oracle sign with probability tending to one.","tokens_in":34500,"tokens_out":12576,"duration_ms":130071,"significance":"The oracle version of Semi-knockoffs is an elegant and potentially useful construction. The idea of comparing two conditional samplers (with and without y) to eliminate the train-test split is novel, and the oracle FDR proof (Appendix E.2) is a correct application of Barber–Candès exchangeability, giving finite-sample FDR control when the conditional expectations are known. The stability and double-robustness results are of independent interest, and the experiments are extensive and well documented. However, the advertised contribution—finite-sample guarantees for a practical method that works with arbitrary pre-trained models and estimated imputers—is not established. The load-bearing sign-preservation conjecture is explicitly left unproved, and Theorem 3.3's proof appears incomplete for the actual paired-test implementation. If the oracle results are separated from the estimated-version claims and the practical claims are weakened to asymptotic or conditional guarantees, the paper could be a useful contribution.","major_comments":[{"comment":"The proof of Theorem 3.3 only shows that under the null the two populations have the same marginal law. Algorithm 2 returns the standard Wilcoxon or sign-test p-value, whose validity requires the paired differences to be independent (or the test statistic to have the usual randomization null). Here the residual permutations π1 and π2 are global permutations without replacement, so the pairs (l(bm(eX1,i), y_i), l(bm(eX2,i), y_i)) are dependent across i. Marginal symmetry of each difference does not imply that the signed-rank or sign statistic has the standard null distribution. Consequently, finite-sample type-I error control for the oracle version is not established as stated. Please either prove exactness under the dependent permutation structure or modify the algorithm (e.g., independent residual resampling with replacement) and re-derive the test.","section":"§3.1, Theorem 3.3; Appendix E.1"},{"comment":"The abstract and Section 1 claim finite-sample FDR control for the practical Semi-knockoffs (Algorithm 4), but Theorem 3.4 is proved only for oracle νj and ρj. For estimated imputers, the route to Lemma 2.1 is the conjecture in Section 4.5 that the sign of the empirical statistic equals the oracle sign with probability tending to 1. Even if this sign-matching held asymptotically, it would give at best asymptotic exchangeability, not the exact conditional Rademacher signs required for the finite-sample bound FDR(Ŝ^SKO) ≤ q. Theorems 4.2 and 4.3 are magnitude/rate bounds (W1 convergence and O_P(an bn) for loss differences) and do not control the sign of the average difference, nor do they address joint sign-flip invariance across features. Please either prove a finite-sample sign-preservation result under explicit assumptions, or weaken the practical claims to asymptotic FDR control.","section":"§4.5 and Algorithm 4; abstract"},{"comment":"The proof of Theorem 4.1 applies Prop. 5.2 of Reyero-Lobo et al. (2025b) to assert that the population coefficient of the null input (y for ρj) is zero. Under H0: X_j ⊥⊥ y | X_{-j}, the coefficient of y in the best linear predictor of X_j on (X_{-j}, y) is not zero in general when E[X_j | X_{-j}] is nonlinear; conditional independence implies zero partial correlation only if the model is correctly specified. Thus Theorem 4.1 appears to require correct specification of the imputer (e.g., linear/Gaussian), a condition not stated in the theorem and not guaranteed for arbitrary practical imputers. In addition, the proof's final bound includes a √p factor from Eq. (15), so the statement O_P(√(log(1/δ)/n)) in Eq. (3) omits the dimension dependence.","section":"§4.2, Theorem 4.1; Appendix E.3.2"},{"comment":"The statement says l(bm(eX'), y) − l(bm(eX), y) = O_P(an bn), with an = ∂_{x_j} bm(X_{-j}, t) for a t between eX'^j and eX^j. Since t is an unspecified intermediate point from the mean value theorem, an is not a well-defined deterministic or simple random sequence, so the O_P statement is ambiguous. Please define an as a supremum over the interval or state the bound uniformly. This matters because Theorem 4.3 is the formal basis for the sign-preservation conjecture in §4.5.","section":"§4.4, Theorem 4.3"}],"minor_comments":[{"comment":"\"the FDR procedure is provided in Appendix 4\" should read \"provided in Algorithm 4\".","section":"§4.1"},{"comment":"Use consistent notation for eX'^{(j)} vs eX'^j and W_j^SKO vs W_j SKO; the current mixing is confusing.","section":"Notation"},{"comment":"The abstract states \"provides valid p-values and FDR control\" without qualification; this overclaims because the practical estimated version relies on the unproved conjecture in §4.5. Please qualify the guarantees as oracle or asymptotic.","section":"Abstract"},{"comment":"The table lists \"Semi KO Ours FDR\" and \"Semi KO ST/ Wilcox Ours Type-I error\"; the main text uses \"Semi-knockoffs\". Please harmonize the naming.","section":"Appendix F.1, Table 2"},{"comment":"The caption says the blue distribution is the semi-knockoff statistic and the orange is the difference between theoretical and estimated imputers; Figure 2 and the surrounding text refer to similar quantities. Clarify the exact pair of random variables in each histogram to avoid ambiguity.","section":"Figure 3 and §4.5"},{"comment":"The paper relies on a companion preprint (Reyero-Lobo et al., 2025b) for a key proposition used in Theorem 4.1. If this proposition is not published, include its statement and assumptions in the appendix or state the dependence explicitly in the main text.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The oracle results are sound and the experimental work is extensive, but the paper's central claim—finite-sample FDR control for the practical estimated-imputer algorithm—is not supported by the existing proofs. The sign-preservation conjecture in §4.5 is explicitly unproved, and it is not obvious that it can be proven without strong additional assumptions or an algorithm modification. I would encourage the authors to either prove the conjecture under explicit conditions or reposition the paper as providing oracle finite-sample guarantees plus asymptotic/rate-based guarantees for the estimated version. The current abstract overstates the contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core trick here is genuinely novel: instead of one conditional resample plus the original data, resample twice—once from X_j given X_{-j}, once from X_j given X_{-j}, y. Under the null both sides draw from the same distribution, so the loss difference is symmetric; under the alternative the second imputer uses y, so it shifts. This avoids the train-test split that hurts HRT and CFI, and by treating the loss difference as a feature statistic the Barber–Candès knockoff threshold applies directly. The oracle version is clean: with true ν_j and ρ_j, Theorem 3.4 is a legitimate application of knockoff exchangeability, and it goes through. That part is publishable as is.\n\nWhere I part with the abstract: the finite-sample guarantees for the practical algorithm are not established. Algorithm 4 uses estimated imputers, and the sign-preservation step needed for exact Rademacher signs under the null is explicitly left as a conjecture in Section 4.5. Theorems 4.2 and 4.3 give W1 convergence and a magnitude rate, but neither implies the conditional Rademacher law for Ŵ_j. So “finite-sample FDR control” for the estimated version is conditional on that conjecture. To the authors' credit, they flag this honestly in the text, but the abstract overstates it. Prove it or qualify the guarantee.\n\nTwo softer issues. First, Theorem 3.3's proof only shows the two loss populations are marginally equal, but the Wilcoxon/sign test uses paired differences across i, and the permutation construction creates dependence across i. The exact null distribution of the paired test is not established. That is repairable—for instance by conditioning on the design or using a test that tolerates the coupling—but as written it is a gap. Second, Eq. (8) in the stability proof writes an equality where only an inequality holds; the subsequent Cauchy–Schwarz bound is fine, so this is a minor typo.\n\nBottom line: the oracle construction is a real advance, the empirical work is thorough and the FDR numbers look good across models, and the key limitation is disclosed rather than hidden. The paper deserves a serious referee, but it needs the estimated-version guarantees sharpened or qualified, and the paired-test argument fixed. Send it out; expect major revision.","headline":"Genuinely new two-sided resampling trick with a clean oracle FDR proof, but the practical algorithm's finite-sample guarantees rest on an openly stated conjecture and the abstract overclaims.","tokens_in":35013,"tokens_out":3111,"would_cite":true,"duration_ms":38076,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G10","62F03","62J07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Semi-knockoffs let any pre-trained ML model test conditional independence in finite samples, without a train-test split.","keywords":["conditional independence testing","knockoffs","model-agnostic","false discovery rate","variable selection","double robustness","finite-sample guarantees","machine learning"],"falsifier":"Simulate data where a null feature is strongly correlated with a signal feature, use a deliberately overfit or small-sample model so its derivative with respect to the null feature is not small, and estimate ν and ρ with regularized linear imputers; then check whether the signs of the Semi-knockoff statistics across many null features are i.i.d. Rademacher. If the sign distribution is biased or dependent in this regime, or the empirical FDR exceeds the target q substantially, the conjecture and the practical guarantee fail.","tokens_in":33955,"feed_emoji":"🎯","tokens_out":5169,"duration_ms":52244,"temperature":0.7,"pith_summary":"Conditional independence testing asks whether a feature matters beyond the others, but model-agnostic tests currently pay a power penalty by splitting the data to evaluate a trained model. Semi-knockoffs removes that split: it samples the feature of interest twice, once conditioned on the other features and once conditioned on the other features plus the response, then compares the pre-trained model's loss on the two synthetic inputs. Under the null hypothesis the two sampling distributions coincide, so the loss difference is symmetric about zero — a property that yields exact finite-sample p-values with a Wilcoxon or sign test and false discovery rate control with the standard knockoff threshold. The method needs only the conditional expectations E[X_j|X_{-j}] and E[X_j|X_{-j}, y], which are estimated from data, not the full model-X knowledge of the input distribution. The paper proves the oracle guarantees and provides stability, distributional convergence, and double-robustness results for the estimated version.","feed_headline":"Semi-knockoffs: black-box variable tests without data splitting","feed_subtitle":"Two conditional samplers make null statistics symmetric, yielding valid p-values and FDR control in finite samples.","key_machinery":"Two conditional-expectation imputers — ν_j(X_{-j}) = E[X_j|X_{-j}] and ρ_j(X_{-j}, y) = E[X_j|X_{-j}, y] — generate the synthetic features X̃1 and X̃2 whose losses are compared. The identity ρ_j = ν_j under the null turns the loss difference into an exchangeable statistic whose sign, in the oracle version, is provably a Rademacher random variable, entitling the knockoff threshold T_q to control the FDR. Two supporting results carry the estimated version: a stability theorem (Theorem 4.1) showing an l2-regularized empirical risk minimizer changes by O_P(sqrt(log(1/δ)/n)) when an uninformative coordinate is suppressed, and a double-robustness theorem (Theorem 4.3) showing the loss discrepancy","core_discovery":"On its own terms, the paper's claim is that conditional independence can be tested with an arbitrary black-box predictive model while retaining finite-sample type-I error and FDR control and no train-test split, by the asymmetric construction of two synthetic populations: one mixes the feature with residuals from a regression on the remaining variables, the other from a regression on the remaining variables together with the response. Under the null, ρ_j(X_{-j}, y) = ν_j(X_{-j}), so the two populations are exchangeable and the loss difference l(m(X̃1), y) − l(m(X̃2), y) is symmetric; this symmetry is exactly what makes valid p-values and knockoff-style FDR control possible. For the practical","pith_inferences":["If the sign-exchangeability conjecture fails in regimes where the model keeps nontrivial sensitivity to null features or the imputers are too noisy relative to the signal, the practical FDR guarantee would degrade gracefully to asymptotic rather than finite-sample control; the paper's own limits motivate a diagnostic.","The two-imputer trick is a general symmetry-engine: any statistic symmetric in X̃1 and X̃2 under the null, not just loss differences, could carry the same argument, suggesting immediate extension to rank-based or kernel-based scores.","The double-robustness product structure suggests a practical safeguard: monitor a_n·b_n per feature and flag cases where the product is not small, treating them as outside the verified regime.","Because the imputers condition on the response, the method connects naturally to missing-data imputation practice, where conditioning on y is known to matter; transferring those insights could raise power further."],"forward_implications":["Variable selection with complex models — random forests, gradient boosting, neural networks — becomes feasible in high dimensions with finite-sample FDR control, without splitting the sample.","Scientists no longer need the model-X assumption of a known input distribution; estimating two conditional expectations suffices, which widens the class of accessible problems.","The method sidesteps the difficult construction of exact knockoff variables and antisymmetric statistics, removing a major practical bottleneck of knockoffs.","Because the model is used only as an importance scorer, any pre-trained model can be reused directly, saving the cost of refitting per feature.","Empirically, a few extra permutations (derandomization) substantially raise power while preserving error control."],"fun_headline_variants":["No split needed: semi-knockoffs test independence with any ML model","Semi-knockoffs: valid p-values without data splitting or model-X","Black-box CIT: semi-knockoffs drop the train-test split","Finite-sample FDR control with arbitrary predictors, no split","Semi-knockoffs: model-agnostic independence tests that keep all data"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The practical version's finite-sample guarantees rest on the unproved conjecture that each estimated statistic's sign is a fair coin flip independent of the other features; the paper proves the oracle case and only bounds the error magnitude, not the sign, in the estimated case.","fun_headline_variants_meta":{"raw":{"variants":["No split needed: semi-knockoffs test independence with any ML model","Semi-knockoffs: valid p-values without data splitting or model-X","Black-box CIT: semi-knockoffs drop the train-test split","Finite-sample FDR control with arbitrary predictors, no split","Semi-knockoffs: model-agnostic independence tests that keep all data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2751,"prompt_tokens":707,"completion_tokens":2044,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":1944}},"tokens_in":451,"tokens_out":2044,"duration_ms":11359,"temperature":1.0,"reasoning_tokens":1944,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T06:14:32.561471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate data where a null feature is strongly correlated with a signal feature, use a deliberately overfit or small-sample model so its derivative with respect to the null feature is not small, and estimate ν and ρ with regularized linear imputers; then check whether the signs of the Semi-knockoff statistics across many null features are i.i.d. Rademacher. If the sign distribution is biased or dependent in this regime, or the empirical FDR exceeds the target q substantially, the conjecture and the practical guarantee fail.","supporting_citations":[],"review_version":1}