{"id":"91bd451e-5b6a-49a6-a56f-2cb448480762","arxiv_id":"2507.12022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DOV4MM detects whether a masked pre-trained model was trained on a given dataset via relative embedding reconstruction difficulty, reporting p<0.05 in tests on ImageNet-1K and WikiText-103.","lead":"This paper introduces DOV4MM, a method that checks whether a black-box AI model was pre-trained on a specific dataset by measuring how hard it is to reconstruct masked information in the model's embeddings. It could help dataset owners detect unauthorized use of their data in popular masked models like MAE and BERT.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DOV4MM's test statistic collapses to Rp−Rv, so it only detects memorization of the specific validation split Dv; partial use of Dpub excluding Dv would return 'legal', and the claimed verification scope is broader than what is tested.","rationale":"The most load-bearing point is not the broad empirical statement of Observation 1, which the paper supports across ten image and four language models, but the precise quantity the test actually measures. Because the relative baseline cancels, the test is a membership test on the defender's validation split. The paper's own ablation in Table 3h shows that raw R fails while ΔR works, which is consistent with the cancellation, yet the experiments never vary the overlap between D_pub and the pre-training set. The reader's weakest assumption (Observation 1) is related but less sharp: the specific failure mode is not strong regularization but the fact that D_v itself must be memorized. The early-stopping, fine-tuning, and adaptive-attack experiments in §4.6 and §C all keep full overlap, so they do not probe this structural dependence. I would keep the verdict conditional: the method is useful for the exact all-or-nothing scenario it tests, the code is released, and the baseline comparisons are informative. However, the claims should be qualified and the partial-overlap experiment added. The statistical-test inconsistency (Algorithm 1 uses ttest_ind despite the paired-test formulas in §A.4) is a secondary correctness issue that should be fixed, but it is unlikely to reverse the extreme p-values, so it is not the load-bearing concern.","tokens_in":24905,"tokens_out":12135,"duration_ms":138282,"concrete_test":"Repeat the ImageNet-50 protocol (MAE ViT-B/16, D_t = 20k, D_v = remaining, K = 30, N = 1024, mask ratio 75%) but pre-train M_s only on D_t, holding D_v and D_pvt out of pre-training. Run DOV4MM exactly as specified. If the one-tailed paired p-value exceeds 0.05, the method fails to detect partial use of D_pub, and the central claim must be restricted to models trained on all of D_pub.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (4) of the supplement defines the paired difference as d_k = ΔRk_pt − ΔRk_vt, and since ΔR_xt = R_x − R_t by Definition 2, this equals Rk_p − Rk_v. The decoder-training baseline R_t cancels exactly. Consequently, the one-sided paired t-test is a direct comparison of reconstruction difficulty between the private set D_pvt and the validation split D_v. A rejection of H0 means only that the specific samples in D_v are easier to reconstruct than the private samples. In every \"illegal\" experiment, M_s was pre-trained on the whole of D_pub, so D_v was always part of the pre-training data by construction. If a suspect instead pre-trains on a subset of D_pub that does not include D_v — for example, only on D_t, or on an arbitrary 80% that misses part of D_v — then D_v is unseen, R_v is not lowered, and the test should return p > 0.05 even though data from D_pub was used. No partial-overlap experiment is reported in §4 or the supplement. The abstract's claim that DOV4MM ascertains whether M_s was \"pre-trained on a particular unlabeled dataset\" is therefore only supported for the special case of full overlap with the defender's random holdout; the claimed scope is not matched by the test.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DOV4MM, a black-box dataset ownership verification method for pre-trained masked models. The defender splits a public dataset into a training subset Dt and a validation subset Dv, trains a light decoder on Dt to reconstruct masked embeddings of the suspicious encoder, and computes per-iteration reconstruction difficulties for Dt, Dv, and a private set Dpvt. A one-sided paired t-test is then applied to the relative difficulties ΔRvt and ΔRpt to decide whether the suspicious model was pre-trained on the public dataset. The method is evaluated on ImageNet subsets and ImageNet-1K with ten masked image modeling methods, and on WikiText-103 subsets with four masked language models, claiming significant rejection for illegal cases and non-rejection for legal cases, with ablations and interference-resistance experiments.","tokens_in":25222,"tokens_out":4464,"duration_ms":54278,"significance":"If the central claim holds, DOV4MM is a useful addition to dataset protection: it is the first verification method tailored to masked models, works in a black-box setting, does not rely on backdoor watermarks, and the experimental scope is broad (ten MIM methods, four MLM methods, multiple architectures, fine-tuning, early stopping, and an adaptive attack). The authors also provide code and report detailed p-values in the supplementary material, which strengthens reproducibility. The significance is tempered by the fact that, as shown below, the tested statistic reduces to a comparison of the defender's holdout split against the private split, so the scope of the verified claim is narrower than the abstract states.","major_comments":[{"comment":"The test statistic collapses to a direct comparison between the private set and the validation split, so the decoder baseline Dt cancels out. In the supplement, Eq. (4) defines d_k = ΔR^k_pt − ΔR^k_vt; substituting Definition 2 gives d_k = (R^k_p − R^k_t) − (R^k_v − R^k_t) = R^k_p − R^k_v. Thus the test rejects H0 when the private samples are harder to reconstruct than the specific validation samples Dv, not when the model has used Dpub as a whole. In every illegal experiment, Ms was pre-trained on the whole public dataset, so Dv was always a subset of the pre-training data. If a suspect pre-trained on a subset of Dpub that does not include Dv (for example, only on Dt or on an arbitrary 80% that misses Dv), the test would return p > 0.05 despite the use of data from Dpub. No partial-overlap experiment is reported in §4 or the supplement. The abstract's claim that DOV4MM ascertains whether Ms was 'pre-trained on a particular unlabeled dataset' is therefore supported only for the special case of full overlap with the defender's random holdout. The authors should either narrow the claim to verification of the entire dataset (or of the holdout split) or provide evidence, theoretical or experimental, that partial overlap with Dpub is detected.","section":"§3.3, Definition 2, and Supplement Eq. (4)"},{"comment":"Algorithm 1 calls scipy.stats.ttest_ind(ΔRpt, ΔRvt), which is an independent-samples t-test, while the surrounding text and Eqs. (4)–(6) describe a paired one-sided t-test on the K paired differences. The paired and independent tests can give materially different p-values when the paired observations are correlated, which is the very situation here because both quantities are computed on the same model and the same random masks. If the experiments used Algorithm 1, then the reported p-values are not from the stated paired test; if the experiments used the paired formulation, then Algorithm 1 and the corresponding code are misleading. This is load-bearing for the statistical claim, and the authors should report which procedure was actually run and update the code/text accordingly.","section":"Supplement §A.4, Algorithm 1"},{"comment":"Observation 1, that the reconstruction difficulty of masked information is lower for seen samples than unseen samples in the embedding space of masked models, is the load-bearing empirical premise of the entire method. It is supported by the reported experiments, but it is not characterized in scope: there is no analysis of when the gap can shrink or disappear, for example under strong regularization, heavy fine-tuning with large learning rates, or adversarial training that explicitly minimizes the seen/unseen gap. The early-stopping, fine-tuning, and adaptive-attack experiments are helpful, but they do not establish a boundary of validity. Given that the method's decision rule and all conclusions depend on this observation, the paper should state it as a testable hypothesis and report conditions under which it fails, rather than presenting it as a general invariant.","section":"§3.2.1, Observation 1"}],"minor_comments":[{"comment":"The caption of Figure 3 and the supplementary Figure 5 contain long runs of garbled glyph tokens (e.g., '/uni0000002c/uni00000031/...'); these should be cleaned to the intended text.","section":"Figure 3 caption and supplementary Figure 5"},{"comment":"In the MLM training settings, 'a learning rate of 5e-' is missing the exponent; it should read, for example, '5e-5'.","section":"Supplement §A.3"},{"comment":"Line 3 of Algorithm 1 reads 'scipy.state.ttest ind(...)', which should be 'scipy.stats.ttest_ind(...)'.","section":"Algorithm 1"},{"comment":"The fine-tuning experiment on logits replaces the embedding mask with no mask at all and uses a three-layer fully connected decoder; this variant should be described more precisely, since it is no longer the same reconstruction-difficulty definition as in Eq. (3).","section":"§4.6, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a serious empirical contribution, but its central claim as stated in the abstract is broader than what the test actually measures. The cancellation in Eq. (4) of the supplement reduces the test to a membership-inference-style comparison between the defender's holdout split and the private set. This can be fixed either by reframing the contribution as detecting full-dataset pre-training (or, more precisely, pre-training on the holdout split) or by adding experiments and analysis for partial overlap. The t-test implementation discrepancy between Algorithm 1 and the paired-test description is also concerning and should be resolved in revision. If the authors commit to the paired test, the p-values in the experiments should be recomputed with the correct procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the take: this is the first dataset ownership verification method aimed at masked pre-trained models, and the empirical story is convincing, but the headline claim is narrower than the abstract suggests. The test statistic secretly cancels the 'relative' baseline and just compares the private set against the defender's holdout, so the method verifies memorization of the specific validation split, not usage of the public dataset at large.\n\nWhat's new and good: the paper targets an important gap, since existing DOV methods are designed for supervised or contrastive models. The relative embedding reconstruction difficulty is a reasonable metric, and the evaluation is broad: ten MIM methods, four MLM methods, multiple architectures, plus early stopping, fine-tuning, and an adaptive attack. Code is available, and the efficiency is a clear win over DI4SSL. In the exact-match scenario (suspect pre-trained on all of D_pub), the method works cleanly, with p-values that are tiny for illegal cases and comfortably non-significant for legal ones.\n\nWhere the soft spots are: first, the structural one. In the supplement, d_k = ΔR_pt,k − ΔR_vt,k = (R_p,k − R_t,k) − (R_v,k − R_t,k) = R_p,k − R_v,k, so the decoder-training baseline cancels. The t-test is directly comparing reconstruction difficulty of the private set against the validation subset D_v. Every 'illegal' experiment had D_v contained in the pre-training data by construction. If a suspect trains on a subset of D_pub that misses D_v, the test would return 'legal' even though D_pub was used. No partial-overlap experiments are reported, so the paper's scope claim is not matched by the evidence. Second, Algorithm 1 uses scipy.stats.ttest_ind, an independent-samples test, rather than the paired t-test described in the text. The p-values are so extreme that the conclusions likely survive, but the implementation is inconsistent with the stated test. Third, Observation 1 is empirical with no theoretical backing; acceptable for an applied paper, but worth noting.\n\nWho this is for: dataset owners, people in data provenance, and anyone building membership-inference baselines. I'd bring it to a reading group, and I'd cite it as a method that detects holdout memorization. It deserves a serious referee, but a referee should ask for partial-overlap experiments and a corrected paired t-test. My verdict: conditional accept, not a clean one.","headline":"First DOV method for masked models, but the test statistic secretly compares private samples against the defender's holdout, so the claimed verification scope is narrower than the paper says.","tokens_in":25744,"tokens_out":5436,"would_cite":true,"duration_ms":58292,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DOV4MM verifies whether a black-box masked pre-trained model was trained on a given public dataset by comparing masked-information reconstruction difficulty for seen versus unseen samples.","keywords":["dataset ownership verification","masked modeling","pre-trained models","embedding reconstruction difficulty","membership inference","black-box verification","self-supervised learning","hypothesis testing"],"falsifier":"Pre-train a masked model on a partial mixture of the protected dataset and an unrelated one — for instance, half of $D_{pub}$ interleaved with an equal-sized foreign dataset — run DOV4MM with $D_{pub}$ as the defended dataset, and check whether the p-value still falls below 0.05; the paper reports only the fully overlapping illegal case and fully disjoint legal cases, so a partial-overlap result that exonerates a model that did train on $D_{pub}$ would falsify the central claim. A complementary probe is to train the same architecture with strong weight decay or explicit anti-memorization regularization and watch whether the $\\Delta R_{pt} - \\Delta R_{vt}$ gap collapses below the significance threshold.","tokens_in":24727,"feed_emoji":"🔍","tokens_out":13158,"duration_ms":137446,"temperature":0.7,"pith_summary":"The paper claims that a dataset owner can determine whether a suspicious black-box masked model was pre-trained on the owner's public dataset, without watermarking the data and without seeing the model's internals. The proposed DOV4MM trains a small decoder on the suspect model's own embeddings, then measures the relative embedding reconstruction difficulty — how much harder masked information is to rebuild for private unseen samples than for the public dataset's own validation samples — and feeds that measure into a one-tailed paired t-test. A p-value below 0.05 says the model was trained on the protected dataset; a p-value above 0.05 says it was not. The authors report this verdict is correct across ten masked image models on ImageNet-1K and four masked language models on WikiText-103, using only about 3% of the protected data, in settings where prior watermark-based and inference-based verification approaches fail or do not transfer.","feed_headline":"An embedding test catches masked AI models trained on your data","feed_subtitle":"Works across ten masked image models and four language models using roughly 3% of the protected data.","key_machinery":"The load-bearing object is the relative embedding reconstruction difficulty $\\Delta R$: the per-iteration gap between two embedding reconstruction difficulties measured against a common reference set. The atomic quantity is the per-sample embedding reconstruction difficulty $R(x, t, \\hat{t}, M, M_d) = \\frac{\\|(M_d(e_t) - e) \\odot (1 - \\hat{t})\\|_2^2}{\\|1 - \\hat{t}\\|_1}$, the squared reconstruction error at masked positions in the suspect model's embedding space after a decoder $M_d$ trained on that model's own outputs tries to fill them in. DOV4MM computes $\\Delta R$ for the public validation split and for the defender's private set, both relative to the training split, over $K$ sampling iterations, and submits the paired differences to a one-tailed pairwise t-test; the test's p-value is the decision statistic that converts the raw reconstruction gap into an ownership verdict.","core_discovery":"The paper's central claim is that pre-trained masked models retain a measurable memory of their training data: in the model's embedding space, masked information from samples seen during pre-training is easier to reconstruct than masked information from samples never seen (Observation 1). On that basis DOV4MM trains a decoder $M_d$ on the suspect model's own embeddings, computes the embedding reconstruction difficulty of masked positions for three sample groups — the public training subset, the public validation subset, and a private set the model never saw — and forms the relative embedding reconstruction difficulties $\\Delta R_{vt}$ and $\\Delta R_{pt}$ against the common training reference. A one-tailed paired t-test on these paired values decides ownership: rejecting the null hypothesis (p < 0.05) marks the suspect illegal; failing to reject marks it legal. The paper reports this test succeeding across ten masked image modeling methods and four masked language modeling methods, and surviving early stopping, downstream fine-tuning, masked-modeling fine-tuning, and an adaptive loss that explicitly tries to equalize seen and unseen reconstruction difficulty.","pith_inferences":["If Observation 1 is a memorization signal, its strength should track how hard the model overfit during pre-training; this predicts the test will become harder to pass on very large or highly regularized pre-training runs, a regime the paper does not examine.","The decoder is trained on the suspect's own embeddings, so the verdict fingerprints the model's representational geometry as much as the data itself; a suspect who reproduces the protected data with a different architecture or through distillation may weaken the signal, which the paper does not test.","The t-test framing invites a stronger adversary than the one studied: a suspect who knows the exact $\\Delta R$ statistic could pre-train directly against it, for example by regularizing the seen-unseen gap at precisely the masked positions the decoder probes, rather than with the generic equalization loss the paper tries.","The same private probe set and protocol could serve as a standing license-monitoring service — one decoder per suspect model, one probe set per dataset — since verifying a new suspect costs only decoder training plus feature queries, a deployment pattern the authors leave implicit."],"forward_implications":["Dataset owners can audit a suspect masked model through its output API alone, without altering or watermarking their data, which removes the main drawback of backdoor-based verification.","Verification is cheap enough for routine use: on ImageNet-1K, DOV4MM needs roughly 3% of the protected dataset (about 40,000 images) and about 353 seconds on a single GPU, whereas the dataset-inference baseline must process the entire dataset.","The test transfers across mask-based pre-training families — ten masked image models and four masked language models — indicating the seen-versus-unseen reconstruction gap is a general property of masked pre-training rather than an artifact of one architecture or modality.","Tested evasion attempts fail to break it: early stopping, downstream fine-tuning, fine-tuning with masked modeling, and a custom adaptive loss that tries to equalize reconstruction difficulty all leave the p-value below 0.05.","Because the check needs only the defender's own data and the model's feature outputs, it can be applied after the fact to models already trained or deployed, not only to models whose training a defender can influence through watermarks."],"supporting_citations":[{"why":"Supplies the masked image modeling setting (MAE) whose embedding-space reconstruction difficulty Observation 1 is about.","marker":"[22]"},{"why":"Defines the masked language modeling objective that the NLP half of the experiments extends the method to.","marker":"[12]"},{"why":"The prior contrastive-model dataset ownership verification that DOV4MM replaces; its failure on masked models motivates the new statistic.","marker":"[52]"},{"why":"DI4SSL, the dataset-inference baseline DOV4MM is compared against, which must process the entire dataset and fails on masked models.","marker":"[14]"},{"why":"PartCrop, the membership-inference baseline that operates directly on high-dimensional representations and misclassifies the tested cases.","marker":"[59]"},{"why":"CTRL, the representative backdoor-watermark baseline whose fixed trigger can fire on non-watermarked data, motivating watermark-free verification.","marker":"[28]"},{"why":"The statistical source of the one-tailed pairwise t-test that converts relative reconstruction difficulty into the p-value verdict.","marker":"[23]"},{"why":"ImageNet-1K, the main image benchmark and source of the protected subsets used in all image experiments.","marker":"[11]"},{"why":"WikiText-103, the text benchmark used to validate the method on masked language models.","marker":"[40]"}],"fun_headline_variants":["Reconstruction difficulty marks masked models that used your dataset","DOV4MM: t-test reveals if a masked model was pre-trained on your data","One statistical test catches masked models trained on your data","Masked model ownership proven by reconstruction disparity","Embedding reconstruction gap exposes masked models trained on your data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method rests on Observation 1, the empirically asserted claim that masked information is easier to reconstruct in the embedding space of samples a model saw during pre-training than of samples it never saw; the paper gives no proof of this gap, and if it shrinks or disappears under strong regularization, heavy fine-tuning, or partial overlap between the protected dataset and the model's real training data, the p-value test loses its power to separate illegal from legal models.","fun_headline_variants_meta":{"raw":{"variants":["Reconstruction difficulty marks masked models that used your dataset","DOV4MM: t-test reveals if a masked model was pre-trained on your data","One statistical test catches masked models trained on your data","Masked model ownership proven by reconstruction disparity","Embedding reconstruction gap exposes masked models trained on your data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3625,"prompt_tokens":999,"completion_tokens":2626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2543}},"tokens_in":615,"tokens_out":2626,"duration_ms":21483,"temperature":1.0,"reasoning_tokens":2543,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:55:47.641769+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pre-train a masked model on a partial mixture of the protected dataset and an unrelated one — for instance, half of $D_{pub}$ interleaved with an equal-sized foreign dataset — run DOV4MM with $D_{pub}$ as the defended dataset, and check whether the p-value still falls below 0.05; the paper reports only the fully overlapping illegal case and fully disjoint legal cases, so a partial-overlap result that exonerates a model that did train on $D_{pub}$ would falsify the central claim. A complementary probe is to train the same architecture with strong weight decay or explicit anti-memorization regularization and watch whether the $\\Delta R_{pt} - \\Delta R_{vt}$ gap collapses below the significance threshold.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Supplies the masked image modeling setting (MAE) whose embedding-space reconstruction difficulty Observation 1 is about."},{"cited_title":"Dataset ownership verification in contrastive pre-trained models,","cited_arxiv_id":null,"evidence_quote":"The prior contrastive-model dataset ownership verification that DOV4MM replaces; its failure on masked models motivates the new statistic."},{"cited_title":"Dataset inference for self- supervised models","cited_arxiv_id":null,"evidence_quote":"DI4SSL, the dataset-inference baseline DOV4MM is compared against, which must process the entire dataset and fails on masked models."},{"cited_title":"A unified membership inference method for visual self-supervised en- coder via part-aware capability","cited_arxiv_id":null,"evidence_quote":"PartCrop, the membership-inference baseline that operates directly on high-dimensional representations and misclassifies the tested cases."},{"cited_title":"An embarrassingly simple backdoor attack on self-supervised learning","cited_arxiv_id":null,"evidence_quote":"CTRL, the representative backdoor-watermark baseline whose fixed trigger can fire on non-watermarked data, motivating watermark-free verification."},{"cited_title":"Introduction to mathematical statistics","cited_arxiv_id":null,"evidence_quote":"The statistical source of the one-tailed pairwise t-test that converts relative reconstruction difficulty into the p-value verdict."},{"cited_title":"Imagenet: A large-scale hierarchical image database","cited_arxiv_id":null,"evidence_quote":"ImageNet-1K, the main image benchmark and source of the protected subsets used in all image experiments."}],"review_version":1}