{"id":"75ef5f74-ca5a-4564-ab41-6959010ef907","arxiv_id":"2412.00613","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RL-TST improves two-sample testing by autoencoding all unlabeled data first, then training a discriminative model on the labeled split, which boosts test power on MNIST, ImageNet, and synthetic benchmarks.","lead":"This paper introduces a two-step recipe for deciding whether two data samples come from different distributions: first learn a compact representation of all the data without using labels, then train a classifier or kernel on a labeled portion, and finally run a permutation test. This is meant to raise statistical power by using hidden structure in the test data without breaking the test's error guarantees.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Type-I control on MNIST/ImageNet is asserted but never reported; if RL-TST's empirical level exceeds alpha on those benchmarks, the Table 3 power gains are not meaningful.","rationale":"The reader's weakest_assumption focuses on whether the permutation test remains valid after two-phase training. I checked this directly: under H0, the label vector is independent of the data values; Phase 1 is a permutation-symmetric, label-free function of the pooled sample; Phase 2 uses only Str labels; and Algorithm 1 computes permutations with the model M fixed. Hence, conditional on the data and on M, the original Ste labeling is uniform over the remaining label counts, so the permutation p-value is exact. Biggs et al.'s principle extends to this pipeline, and the paper's reliance on it is not a hidden flaw, though a short proof would remove all doubt. The real soft spot is empirical. The paper advertises 'keeping Type-I error near nominal' as part of the central claim, but the only reported level curve is HDGM-S at d=2. For the image benchmarks where the claimed improvements are largest, Appendix C.1 substitutes an unquantified sentence. If those H0 experiments in fact show inflated rejection rates for RL-TST, then Table 3's power numbers are not evidence of higher sensitivity; they are evidence of an oversized test. This is exactly the kind of missing support that a conditional verdict should demand. My recommendation stays CONDITIONAL, matching the reader's verdict, but for a more concrete reason: require the H0 level table for MNIST/ImageNet before treating the headline claim as established.","tokens_in":22321,"tokens_out":18424,"duration_ms":197464,"concrete_test":"Run Algorithm 1 under H0 on MNIST (real vs real and fake vs fake) and ImageNet (fake vs fake) for RL-C2ST, RL-C2ST-L, and RL-MMD-D at alpha=0.05 for M=200,400,600,800,1000, using at least 100 independent seeds and the same hyperparameters as Table 3. Report empirical rejection rates; for 100 trials the 95% binomial tolerance around 0.05 is roughly [0.007, 0.093], or use 1000 trials for tighter bounds. If any RL-TST variant exceeds the upper tolerance, or if its rate is significantly higher than the baselines' rates, Type-I control fails and the power comparisons in Table 3 cannot be interpreted as sensitivity gains.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: higher test power and Type-I error near nominal. The power part is supported by Table 3 and Figure 4, but the second part is supported only by Figure 4(b), which reports level for HDGM-S at d=2. For MNIST and ImageNet, Appendix C.1 states that 'All methods are tested with a reasonable type-I error rate' without giving a single number. This matters because the strongest RL-TST gains appear at small sample sizes on these image benchmarks, e.g., RL-C2ST-L 0.491 vs MMD-D 0.290 at M=200 on MNIST, and 0.400 vs 0.230 on ImageNet. If the RL variants reject H0 on real-vs-real or fake-vs-fake data at a rate well above 0.05, those power advantages are an artifact of an oversized test rather than a real sensitivity gain. The reader's permutation-validity worry does not by itself land: under H0 the labels are independent of the data, Phase 1 is a label-free symmetric function of the pooled sample, and Algorithm 1 conditions on the fixed model before permuting Ste labels, so the permutation null is exact. The missing empirical Type-I evidence is therefore the load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RL-TST, a two-phase representation-learning framework for non-parametric two-sample testing. In Phase 1, an autoencoder is trained on the pooled unlabeled data (SP ∪ SQ) to produce 'inherent representations'; in Phase 2, either a classifier or a deep kernel is trained on a labeled training split applied to these representations to produce 'discriminative representations'; in Phase 3, a permutation test is run on the held-out test split using the resulting statistic (C2ST accuracy, C2ST-L logit distance, or MMD with a deep kernel). The authors argue that using the full unlabeled data is legitimate under the no-sample-index principle and that the manifold assumption justifies the two-phase design. They report experiments on HDGM, MNIST, and ImageNet comparing RL-C2ST, RL-C2ST-L, and RL-MMD-D against C2ST, C2ST-L, MMD-D, and MMD-FUSE, and find that the RL variants often have higher average test power, with Type-I error around 0.05 reported for HDGM-S at d=2.","tokens_in":22550,"tokens_out":10058,"duration_ms":95953,"significance":"If the reported power gains are accompanied by correct Type-I control, RL-TST is practically valuable: it is a simple, model-agnostic pretraining step that can be added to existing classifier- or kernel-based tests, and the gains at small sample sizes on image benchmarks are substantial. The paper's strengths are its broad empirical comparison against current state-of-the-art methods, its inclusion of two autoencoder variants, its check of Type-I error on one HDGM setting, and its explicit comparison with semi-supervised learning alternatives. However, the image-benchmark Type-I evidence is absent, the exact permutation-test validity of the full pipeline is not formally established, and the code link is missing, so the central claim 'higher test power while keeping Type-I near nominal' is not yet fully supported for all benchmarks. I do not see circularity or an internal inconsistency in the empirical comparisons.","major_comments":[{"comment":"The empirical Type-I error of RL-TST is never reported for the MNIST and ImageNet benchmarks, and for HDGM it is reported only for d=2 in Figure 4(b). Appendix C.1 states only that 'All methods are tested with a reasonable type-I error rate' for both image benchmarks, without a single number. This is load-bearing because the largest reported power gains occur at small sample sizes on these benchmarks (e.g., RL-C2ST-L 0.491 vs. MMD-D 0.290 at M=200 on MNIST, and RL-C2ST-L 0.400 vs. MMD-FUSE 0.230 at M=200 on ImageNet). If the RL variants reject H0 on real-vs-real or fake-vs-fake data at a rate well above 0.05, those gains would be an artifact of an oversized test. Please report empirical level under H0, with standard errors, for MNIST, ImageNet, and HDGM d=10, or provide a theorem that, for the exact Algorithm 1 pipeline, the permutation test has exact level alpha regardless of the unsupervised Phase 1.","section":"Section 4 / Table 3 / Appendix C.1"},{"comment":"The Type-I validity of the exact two-phase procedure is asserted rather than proven. The paper invokes the no-sample-index principle of Biggs et al. (2023), but that result does not directly cover RL-TST because Phase 2 trains a supervised classifier or kernel on the labeled split Str before the permutation test on Ste. A short conditioning argument would fix this: under H0, conditional on the full pooled feature values and on the Str labels, the Ste labels are exchangeable, so with the model M fixed the permutation test is exact; and Phase 1 is a label-free function of the pooled features, so it does not break exchangeability. Please state this lemma or an equivalent theorem in the main text, since the central claim includes control of Type-I errors.","section":"Algorithm 1 / Section 3.1"}],"minor_comments":[{"comment":"The reproducibility statement says 'All the reproducible code can be found in the anonymous link' but no link or repository identifier appears in the text; please provide a working URL.","section":"Appendix C.8"},{"comment":"Table 2 and Figure 3 report test power without error bars, and the footnote in Table 2 rejects standard deviations because each trial outcome is 0 or 1; that reason is invalid, since a proportion over 100 trials has standard error sqrt(p(1-p)/100).","section":"Table 2 / Figure 3"},{"comment":"In Algorithm 1, line 3 updates only phi ('phi* <- arg min_phi R(fa, X)'), whereas Eq. (6) defines a joint minimization over phi and psi; please align the pseudocode with the objective and state how the decoder is used or discarded.","section":"Algorithm 1 / Eq. (6)"},{"comment":"The sentence 'only holistic approaches, such as BYOL, can effectively capture the IRs of whole data' is an unsupported design claim and should be either cited or qualified.","section":"Section 3.1"},{"comment":"Theorem D.4 is not self-contained: it quantifies over 'any arbitrarily small Delta_mu,ml' but then states bounds involving sqrt(ln(4/delta)/(8 mu)) without defining the relationship, and the proof introduces an undefined 's' in 'ln(4s/delta)'; please rewrite the statement and proof or label the section as heuristic.","section":"Appendix D.1 / Theorem D.4"},{"comment":"The permutation-testing paragraph says the labels are permuted 'for n times', but n already denotes the sample size; use a separate symbol such as nperm for the number of permutations.","section":"Section 2 / Permutation Testing"}],"recommendation":"major_revision","confidential_remarks":"The reader's concern about the permutation null appears answerable: under H0, conditional on all feature values and on the Str labels, the Ste labels are exchangeable, and Phase 1 does not use labels, so a short conditioning lemma would prove exact level for Algorithm 1. The missing empirical Type-I results for MNIST and ImageNet are the main obstacle; I would not reject the paper on circularity or on the reader's weakest assumption, but I would require the level evidence and a proof sketch before acceptance. The missing code link should also be fixed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the exact pipeline: pretrain an autoencoder or WAE on the full unlabeled pooled sample, then fine-tune a classifier or deep kernel on a labeled split, then run a permutation test on the held-out features. That exact two-phase combination is not in the cited prior work, and the paper deserves credit for the consistency of the empirical gains. On HDGM, MNIST-vs-DCGAN, and ImageNet-vs-StyleGAN-XL, the RL variants beat C2ST, C2ST-L, MMD-D, and MMD-FUSE at most sample sizes, with the strongest gains at small M. The HDGM level plot shows Type-I error near 0.05, which is real evidence. The discussion of why mainstream SSL methods fail for two-sample testing is also sensible and useful.\n\nThe main soft spot is exactly where the stress-test note lands. Appendix C.1 says all methods on MNIST and ImageNet maintain a “reasonable type-I error rate” but gives no numbers. Since the headline gains appear at M=200–400 on those benchmarks, an oversized test under H0 would make those power numbers meaningless. This is fixable and should be the first thing demanded in revision. The reader’s permutation-validity worry, by contrast, does not really land: under H0 the labels are independent of the data, Phase 1 is a label-free symmetric function of the pooled sample, and Algorithm 1 conditions on the fitted model before permuting the held-out labels, so the permutation null is exact in principle. The gap is empirical, not logical.\n\nSmaller issues: the promised code is only described as an anonymous link, not accessible in the text; Figure 3 and Table 2 lack error bars; and the theory in Appendix D is borrowed from Balcan-Blum and does not add a level-control proof for the exact two-phase procedure. None of these are fatal, but together they make the paper more of a strong empirical contribution than a complete methodological one.\n\nWho gets value: anyone working on representation learning for nonparametric two-sample tests, and practitioners who want a drop-in pretraining step before C2ST or MMD-D. It deserves a serious referee, and my own verdict would be conditional: report Type-I error on MNIST and ImageNet, make the code available, and add error bars to the missing figures. Then the empirical claim is solid enough to stand.","headline":"Solid empirical paper: the two-phase pretraining trick plausibly buys real power, but the missing Type-I numbers on the image benchmarks are the one gap standing between this and a clean accept.","tokens_in":23123,"tokens_out":1781,"would_cite":true,"duration_ms":20661,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"RL-TST, a two-phase framework that first learns inherent representations from the entire unlabeled dataset and then learns discriminative representations on a labeled training split, achieves higher test power than C2ST, C2ST-L, MMD-D…","keywords":["two-sample testing","representation learning","maximum mean discrepancy","classifier two-sample test","permutation test","test power","autoencoder","deep kernel"],"falsifier":"Run RL-TST under the null hypothesis $P=Q$ for many trials (say 500) on a dataset where the autoencoder is trained on the pooled data including test inputs, and measure the empirical rejection rate at $\\alpha=0.05$. If the empirical Type-I error is substantially above the nominal level (e.g., $>0.10$) for a large sample size where the test is well powered, the central claim of controlled level fails. As a sharper check, compare the permutation null distribution of the statistic under the full two-phase procedure to the null obtained when the encoder is trained only on the training split; any systematic inflation would indicate the supervised phase distorts the permutation null.","tokens_in":22109,"feed_emoji":"📊","tokens_out":10021,"duration_ms":146583,"temperature":0.7,"pith_summary":"Two-sample tests ask whether two datasets come from the same distribution, and modern versions learn representations of the data before testing. This paper argues that the common practice of splitting data and learning representations only on the training half throws away useful structure, and proposes RL-TST, which first learns inherent representations from the entire unlabeled pool of both samples using an autoencoder, then learns discriminative representations by training a classifier or deep kernel on a labeled training split. The central claim is that this two-phase use of the test set (without sample labels) raises test power over strong baselines while keeping the false-rejection rate near the nominal level, and the paper backs this with experiments on synthetic Gaussian mixtures, MNIST, and ImageNet. A sympathetic reader should care because, if the claim holds, practitioners can get more sensitive two-sample tests at a fixed sample size by adding a label-free pretraining step before their usual testing procedure.","feed_headline":"Unlabeled test data can boost two-sample test power","feed_subtitle":"A two-phase method uses unlabeled test inputs to learn better representations, beating four baselines on key benchmarks.","key_machinery":"The central mechanism is the two-phase RL-TST pipeline, defined in Algorithm 1. Phase 1 learns a featurizer $\\phi^*$ by minimizing a reconstruction objective (e.g., autoencoder MSE) on the unlabeled pool $S_{unl}$; these are the inherent representations (IRs). Phase 2 freezes $\\phi^*$ and trains either an MLP $g$ on $S_{tr}$ with binary cross-entropy or a deep kernel $k_{\\phi^*}$ by maximizing the estimated test-power ratio $\\hat{J}_{DR}$ from Eq. (10), producing discriminative representations (DRs). Phase 3 applies a permutation test to any of the three statistics (1), (2), or (3) on the test split. The framework leans on the no-sample-index principle, established in prior work on kernel fusion, which says representation learning on pooled data without labels does not inflate the Type-I error of permutation tests, and on a dimension-versus-sample-size result for kernel two-sample tests to explain why lower-dimensional inherent representations let MMD-based tests detect higher-order distributional differences.","core_discovery":"On its own terms, the paper's discovery is that in non-parametric two-sample testing, the unlabeled test split can be safely and profitably used for representation learning: RL-TST first trains an autoencoder on the whole pooled dataset $S_{unl} = S_P \\cup S_Q$ to obtain inherent representations (IRs) that reflect the underlying data manifold, then fine-tunes either a classifier $g$ or a deep kernel $k$ on the labeled training split $S_{tr}$ over these IRs to obtain discriminative representations (DRs), and finally runs a permutation test on the held-out test split using the DRs. The paper claims this simultaneously uses the manifold structure of the test data and the discriminative signal of the training data, and reports consistent power gains over C2ST, C2ST-L, MMD-D, and MMD-FUSE on HDGM, MNIST, and ImageNet benchmarks, with Type-I error near $\\alpha=0.05$.","pith_inferences":["The paper's level-control story depends on the unsupervised Phase 1 using the test inputs without labels; an untested extension is whether the subsequent supervised Phase 2 on the training split preserves the permutation null, so the Type-I error should be re-checked at larger $N$ and with more permutation trials than the reported 100-trial averages.","The dimension-versus-sample-size argument suggests that any dimensionality-reducing transformation of the pooled data should improve MMD-based tests; a cheap test of this would replace the autoencoder with PCA or a random projection and compare power, which would separate the benefit of manifold learning from mere dimension reduction.","The framework's reliance on the manifold assumption implies it could underperform on intrinsically high-dimensional or discrete data; a natural extension is to make the first phase adaptive to the estimated intrinsic dimension of the input space."],"forward_implications":["At a fixed sample size, adding an autoencoder pretraining step on the pooled unlabeled sample before supervised training increases test power for C2ST, C2ST-L, and MMD-D implementations.","Because the first phase uses the whole dataset without labels, the method avoids the training/test trade-off that comes from splitting scarce data, so less data is needed to reach a given power.","The framework is modular: any autoencoder-based representation learner can be inserted in Phase 1, and any classifier- or kernel-based test can be wrapped, so future representation learning advances can transfer directly to two-sample testing.","The empirical results indicate that on real-versus-generated image benchmarks (MNIST, ImageNet), RL-C2ST-L and RL-MMD-D reach full power at smaller sample counts than the fused-kernel baseline."],"supporting_citations":[{"why":"Supplies the no-sample-index principle that justifies training representation learners on the unlabeled pooled test data without inflating Type-I error; also provides the MMD-FUSE baseline.","marker":"Biggs et al. [2023]"},{"why":"Provides the MMD-D deep-kernel method that RL-TST wraps, its training objective $\\hat{J}_{DR}$, and the HDGM/MNIST/ImageNet benchmarks the paper compares on.","marker":"Liu et al. [2020]"},{"why":"Defines C2ST and its accuracy statistic (1); its theorem on C2ST test power underlies the paper's power analysis of RL-C2ST.","marker":"Lopez-Paz and Oquab [2017]"},{"why":"Introduces C2ST-L, the logit-based statistic (2) that RL-C2ST-L improves.","marker":"[Cheng and Cloninger, 2020]"},{"why":"Defines MMD and its unbiased U-statistic (3), the base statistic for all MMD-based variants in the paper.","marker":"Gretton et al. [2012a]"},{"why":"Provides the dimension-and-sample-order result the paper cites to explain why lower-dimensional representations let MMD tests detect higher-order moment discrepancies.","marker":"[Yan and Zhang, 2023]"}],"fun_headline_variants":["Unlabeled test data sharpens two-sample tests","Two-phase learning boosts test power with test set","Manifold-aware representations improve two-sample testing","Whole-data representation learning lifts two-sample power","Test set aids representation for stronger two-sample tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The permutation test in Algorithm 1 stays valid when the encoder has been trained on the full unlabeled pool $S_{unl} = S_P \\cup S_Q$ that includes the test inputs and then the classifier or kernel is trained on the labeled training split; the paper relies on the no-sample-index principle for the unsupervised phase but does not prove level control for this exact two-phase combination.","fun_headline_variants_meta":{"raw":{"variants":["Unlabeled test data sharpens two-sample tests","Two-phase learning boosts test power with test set","Manifold-aware representations improve two-sample testing","Whole-data representation learning lifts two-sample power","Test set aids representation for stronger two-sample tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2920,"prompt_tokens":944,"completion_tokens":1976,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1906}},"tokens_in":560,"tokens_out":1976,"duration_ms":14812,"temperature":1.0,"reasoning_tokens":1906,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:10:42.068601+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RL-TST under the null hypothesis $P=Q$ for many trials (say 500) on a dataset where the autoencoder is trained on the pooled data including test inputs, and measure the empirical rejection rate at $\\alpha=0.05$. If the empirical Type-I error is substantially above the nominal level (e.g., $>0.10$) for a large sample size where the test is well powered, the central claim of controlled level fails. As a sharper check, compare the permutation null distribution of the statistic under the full two-phase procedure to the null obtained when the encoder is trained only on the training split; any systematic inflation would indicate the supervised phase distorts the permutation null.","supporting_citations":[{"cited_title":"MMD - FUSE : Learning and Combining Kernels for Two - Sample Testing Without Data Splitting","cited_arxiv_id":null,"evidence_quote":"Supplies the no-sample-index principle that justifies training representation learners on the unlabeled pooled test data without inflating Type-I error; also provides the MMD-FUSE baseline."},{"cited_title":"Sutherland","cited_arxiv_id":null,"evidence_quote":"Provides the MMD-D deep-kernel method that RL-TST wraps, its training objective $\\hat{J}_{DR}$, and the HDGM/MNIST/ImageNet benchmarks the paper compares on."},{"cited_title":"A survey on deep semi-supervised learning","cited_arxiv_id":null,"evidence_quote":"Provides the dimension-and-sample-order result the paper cites to explain why lower-dimensional representations let MMD tests detect higher-order moment discrepancies."}],"review_version":1}