REVIEW 4 major objections 6 minor 24 references
Dataset Ownership Verification in Contrastive Pre-trained Models
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A one-tailed paired T-test on embedding similarity gaps can decide, at p<0.05, whether a black-box contrastive encoder was pre-trained on a particular unlabeled dataset.
desk verdict A useful empirical signal and broad experiments, but the headline p-value claim is undermined by per-dataset tuning of the test statistic's amplification factor. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the contrastive relationship gap $d(D,\hat{D},M,T)$, defined as the collection of differences $s_i - \hat{s}_i$ between similarity sets computed on the protected dataset $D$ and a private unseen dataset $\hat{D}$ for encoder $M$ under augmentation $T$. It is built from two similarity families: the unary relationship similarity set $S_U$, which averages cosine similarities among multiple global and local augmented views of the same image, and the binary relationship similarity set $S_B$, which measures how much the all-pairs similarity distribution of a set of images changes under augmentation. Multi-scale random cropping (global $T^g$ and local $T^l$ augmentations) activates the gap from both coarse and fine spatial scales. The decision procedure is a one-tailed paired T-test comparing the suspect's gap samples $d_{sus}$ with the shadow encoder's gap samples $d_{sdw}$; rejecting the null $H_0$ at $\alpha=0.05$ declares the dataset stolen.
What would settle it
Train a set of innocent encoders that vary in architecture (for example ViT-B/16), training algorithm (for example DINO), training budget, and dataset similarity to Dpub, then run the published one-tailed paired T-test against the standard ResNet18-SimCLR-on-SVHN shadow; the central claim collapses if more than the nominal 5% of those innocent models yield p<0.05.
Extended reading notes
Core claim
The paper's central claim is that the embedding geometry of a contrastive pre-trained encoder contains a forensic signature of its unlabeled training set. The signature is formalized as the contrastive relationship gap: the difference between similarity statistics computed on the defender's public dataset and on a private dataset the suspect never saw. Two statistics define the gap: unary relationship similarity, measuring how much closer augmented views of the same image get when the image was in the training set, and binary relationship similarity, measuring how much less the pairwise similarity structure among images changes under augmentation when those images were in the training set. The paper argues that this gap is dramatically larger for an encoder trained on the protected dataset, and that comparing the suspect's gap to the gap of an innocent shadow encoder lets a one-tailed paired T-test reject the null hypothesis with p<0.05 for stolen datasets while failing to reject it for innocent ones. The authors validate this across five contrastive algorithms, CNN and ViT architectures, and datasets from CIFAR-10 to ImageNet, including robustness checks with DP-SGD training, downstream fine-tuning, and early stopping.
Load-bearing premise
The test treats one defender-selected shadow model (by default a ResNet18 trained with SimCLR on SVHN) as a faithful stand-in for what any innocent suspect's contrastive relationship gap would look like, so an innocent model whose training recipe produces systematically different gaps could be falsely accused.
Editorial extensions
If this is right
- A data owner can verify theft of an unlabeled dataset from a pre-trained encoder without modifying the dataset and with only black-box access to the model's embeddings.
- The same p-value test transfers across contrastive training algorithms (SimCLR, BYOL, SimSiam, MoCo v3, DINO) and across CNN and ViT architectures, so the defender need not know the suspect's training recipe.
- On ImageNet-scale models, verification succeeds using 0.1% of the protected training data and takes far less compute than prior dataset-inference baselines.
- The signal survives privacy-preserving training (DP-SGD), full fine-tuning for downstream tasks, and early stopping, so the verification is not limited to freshly pre-trained encoders.
- The method does not generalize to masked image modeling encoders such as MAE, which the paper identifies as the boundary of its approach.
Reading between the lines
- A natural extension the paper leaves implicit: the gap test should transfer to non-visual contrastive encoders (text, audio, graphs), because the unary and binary relationships only assume augmentation invariances, not image-specific structure.
- The single-shadow null distribution is the fragile part: a suspect trained with much more data, a much longer schedule, or a much larger architecture may show a systematically larger gap even when innocent; an ensemble of shadow models or per-architecture calibration could harden the test.
- The gap is plausibly a probe of memorization, not just membership, so varying augmentation intensity, dataset size, or regularization may make the reported p-values move in predictable ways that could be measured.
- Because each paired T-test uses K subsampled gap estimates, the effective sample size is K; a practitioner should confirm the K samples are independent draws before interpreting p<0.05 as a sharp bound.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DOV4CL, a dataset ownership verification method for self-supervised contrastive pre-trained models in a black-box setting. The method is based on two empirical observations: contrastive encoders produce more similar representations for augmentations of training samples than for unseen samples (unary relationship), and pairwise similarities among training samples are more stable under augmentation (binary relationship). The defender computes a 'contrastive relationship gap' between a suspect encoder and a shadow encoder using the defender's public and private datasets, then runs a one-tailed paired t-test with a 0.05 threshold to decide whether the suspect was pre-trained on the protected dataset. Experiments on CIFAR10, ImageNette, and ImageNet with SimCLR, BYOL, SimSiam, MoCo v3, and DINO report p-values below 0.05 for illegal suspects and above 0.05 for legal suspects, along with sensitivity, specificity, and AUROC comparisons against DI4SSL and EncoderMI. The paper also reports time cost, ablation studies, and an honest limitation section noting that the method does not work for masked image modeling. The code is publicly available.
Significance. If the statistical claims are valid, this would be a useful first step toward dataset ownership verification for contrastive pre-trained models, with the appealing properties of not modifying the dataset and requiring only a small fraction of the data. The paper is thorough in its empirical coverage across multiple architectures and contrastive algorithms, and it explicitly acknowledges limitations (e.g., MIM encoders) and provides code. However, the central evidence rests on p-values from a hypothesis test, and the test statistic depends on a hyperparameter that is tuned per dataset after seeing the data; this directly affects the validity of the headline claims. The paper also relies on a single shadow model as the null reference without establishing when that null assumption holds. The contribution is significant if these statistical issues can be resolved, but the current manuscript does not yet support the strong claim of calibrated hypothesis testing.
major comments (4)
- [Eq. (11), Appendix A.2, Appendix A.9] The amplification hyperparameter a invalidates the reported p-values as frequentist significance tests. In Eq. (11), I(·) is described as returning a when the condition holds and 1 otherwise, so a is a multiplicative weight on each gap component rather than an indicator. Appendix A.2 sets a = 10000 for CIFAR10, a = 0.1 for ImageNette, and a = 1 for ImageNet, and Appendix A.9 shows that the p-values change with a and states that the defender 'needs to set appropriate a based on his actual situation.' Since a is chosen after seeing the data and changes the test statistic, the null distribution is not fixed in advance, and the reported p-values in Tables 10-15 and Table 2 are not valid out-of-sample significance levels. The paper should either fix a before data inspection, report a sensitivity analysis across a without calling the results p-values, or use a proper calibration procedure.
- [Section 3.2.3 and Tables 12, 15] The shadow-model null assumption is not justified. The method uses a single defender-chosen shadow encoder (default ResNet18 with SimCLR trained on SVHN) as the reference in the paired t-test, and the null hypothesis is that the suspect's gap is not larger than the shadow's gap. If an innocent suspect has a different architecture, training algorithm, or training data, its gap on Dpub versus Dpvt may differ systematically from the shadow's, producing false positives. The paper tests only a limited set of combinations (mainly VGG16/ResNet18 with five contrastive methods on CIFAR10 and ImageNette), and does not provide a characterization of when the shadow is a valid null distribution. Additional experiments with more diverse innocent suspects, or a null constructed from multiple shadows, are needed to support the method's black-box claim.
- [Eqs. (1) and (11), Section 3.2.3] The definition of the contrastive relationship gap is ambiguous and does not clearly map to the t-test. Eq. (1) defines d as a set of differences, while Eq. (11) defines d as a 2-element set containing sums over unary and binary components; it is not clear whether d is a scalar, a vector, or a set, and how the paired t-test is applied to a collection of such objects after K samplings. The paper should specify the exact construction of the paired samples used in the t-test, including how the vector/set-valued d is reduced to a testable quantity.
- [Tables 10-15 and Table 2] The reported p-values are averaged over three trials, but averaging p-values by arithmetic mean is not statistically meaningful. p-values from repeated experiments should be combined using meta-analytic methods or reported as ranges; the current presentation can mislead readers about the stability of the test outcomes. The sensitivity/specificity metrics in Table 1 are computed from these p-value classifications, so the issue propagates to the claimed AUROC of 1.0.
minor comments (6)
- [Section 1] There is a typo in the first sentence: 'the the availability' should read 'the availability.'
- [Section 3.2.2] Eq. (11) uses the symbol I(·) for a function that is not an indicator; this is confusing because standard notation for an indicator is 1{·}. Renaming this function, e.g., to w(·), would improve clarity.
- [Appendix A.9] Figure 7 shows that the choice of a changes the verification outcome, and the text says this effect 'is related to the distributions of Dpub and Dsdw and is not fixed.' This is an honest statement, but it underscores the need for a principled selection rule rather than a per-dataset search.
- [Section 4.1 and Appendix A.2] The experimental setup uses a random split 'into two subsets of equal size' for Dpub and Dalt, but the paper does not describe how many random splits were used or whether the reported results are averaged across splits; this should be clarified.
- [Section 4.3] The paper says the method uses only 0.1% of ImageNet data for verification, which is a strength, but the time comparison in Table 8 reports a single run; reporting variance across multiple runs would strengthen the claim.
- [Appendix A.11] The watermark-based baseline 'DOV-CTRL' is adapted from a backdoor attack, and the comparison is useful; however, the paper does not discuss whether the watermark method modifies the dataset, which is a key difference from the proposed method and should be stated explicitly in the comparison.
Circularity Check
No significant circularity: the contrastive relationship gap is an empirical statistic defined independently of the verification outcome, and the reported p-values do not reduce by construction to the method's inputs.
full rationale
The paper's central claim is empirical, not derivational: Observations 1 and 2 are presented as observed properties of contrastive encoders, and Definition 1 together with Eq. (11) define the gap statistic without assuming the verification result. The one-tailed paired T-test against a shadow encoder is a standard statistical procedure rather than a tautology. The method does not rest on a self-citation chain; references to EncoderMI and DI4SSL are prior-art comparisons, not load-bearing derivations. The main caveats are non-circular: Appendix A.9 admits that the amplification hyperparameter a is dataset-dependent and must be set by the defender, which is a calibration and overfitting concern for the reported p-values rather than an equivalence by construction; Section 3.2.3 also relies on a single shadow model to supply the null distribution, which is a validity assumption about innocent suspects. The MIM limitation in Section 4.7 is an honest scope restriction. None of these make the core observation equivalent to its inputs, so the derivation chain is self-contained.
Assumptions & free parameters
free parameters (4)
- a (amplification factor) =
10000 (CIFAR10), 0.1 (ImageNette), 1 (ImageNet)
- kpub, kpvt (sampling sizes) =
256/128 (CIFAR10), 32/32 (ImageNette, ImageNet)
- K (number of samplings) =
30 (CIFAR10), 50 (ImageNette, ImageNet)
- M, N (global/local augmentation counts) =
M=2, N=6
assumptions (4)
- domain assumption Observation 1: contrastive pre-trained encoders produce more similar representations for augmentations of seen samples than unseen samples.
- domain assumption Observation 2: the pairwise similarity between seen samples' representations does not change significantly under augmentation, unlike unseen samples.
- domain assumption The shadow model's gap distribution is a valid null for any innocent suspect model.
- standard math The paired differences used in the T-test are approximately normal and independent across the K samplings.
Cite this review
Pith. "Pith review of Dataset Ownership Verification in Contrastive Pre-trained Models." pith.science (2026). https://pith.science/paper/7PJ5XYA3
@misc{pith2026250207276,
author = {Pith},
title = {Pith review of: Dataset Ownership Verification in Contrastive Pre-trained Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/7PJ5XYA3}},
note = {Machine review of arXiv:2502.07276}
}
abstract
High-quality open-source datasets, which necessitate substantial efforts for curation, has become the primary catalyst for the swift progress of deep learning. Concurrently, protecting these datasets is paramount for the well-being of the data owner. Dataset ownership verification emerges as a crucial method in this domain, but existing approaches are often limited to supervised models and cannot be directly extended to increasingly popular unsupervised pre-trained models. In this work, we propose the first dataset ownership verification method tailored specifically for self-supervised pre-trained models by contrastive learning. Its primary objective is to ascertain whether a suspicious black-box backbone has been pre-trained on a specific unlabeled dataset, aiding dataset owners in upholding their rights. The proposed approach is motivated by our empirical insights that when models are trained with the target dataset, the unary and binary instance relationships within the embedding space exhibit significant variations compared to models trained without the target dataset. We validate the efficacy of this approach across multiple contrastive pre-trained models including SimCLR, BYOL, SimSiam, MOCO v3, and DINO. The results demonstrate that our method rejects the null hypothesis with a $p$-value markedly below $0.05$, surpassing all previous methodologies. Our code is available at https://github.com/xieyc99/DOV4CL.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Deep learning with differential privacy
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308–318,
work page 2016
-
[9]
Proof-of-learning: Definitions and practice
Hengrui Jia, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Anvith Thudi, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning: Definitions and practice. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 1039–1056. IEEE,
work page 2021
-
[10]
Decision boundary of deep neural networks: Challenges and opportunities
12 Published as a conference paper at ICLR 2025 Hamid Karimi and Jiliang Tang. Decision boundary of deep neural networks: Challenges and opportunities. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 919–920,
work page 2025
-
[11]
Characterizing the decision boundary of deep neural networks
Hamid Karimi, Tyler Derr, and Jiliang Tang. Characterizing the decision boundary of deep neural networks. arXiv preprint arXiv:1912.11460,
arXiv 1912
-
[12]
Black-box dataset ownership verification via backdoor watermarking
Yiming Li, Mingyan Zhu, Xue Yang, Yong Jiang, Tao Wei, and Shu-Tao Xia. Black-box dataset ownership verification via backdoor watermarking. IEEE Transactions on Information Forensics and Security, 2023b. Yu Li, Lizhong Ding, and Xin Gao. On the decision boundary of deep neural networks. arXiv preprint arXiv:1808.05385,
-
[14]
Dataset inference: Ownership resolution in machine learning
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. arXiv preprint arXiv:2104.10706,
-
[15]
Reading digits in natural images with unsupervised feature learning
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp
work page 2011
-
[16]
Self-supervised method is SimCLR
Both the datasets of defender and suspect are CIFAR10, meaning p-value should be less than 0.05. Self-supervised method is SimCLR. The shadow model is a ResNet18 pre-trained on ImageWoof using SimCLR. The results demonstrate that our method remains effective even under early stopping conditions. Table 16: The results (p-values) of our method on suspicious...
work page 2023
Show all 24 references
-
[18]
Membership inference attacks against machine learning models
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pp. 3–18. IEEE,
2017
-
[20]
Proof-of-learning with incentive security
Zishuo Zhao, Zhixuan Fang, Xuechao Wang, and Yuan Zhou. Proof-of-learning with incentive security. arXiv preprint arXiv:2404.09005,
-
[21]
(2009): The CIFAR10 dataset consists of 32x32 colored images with 10 classes
14 Published as a conference paper at ICLR 2025 APPENDIX A T HE DETAILS AND ADDITIONAL SUPPLEMENTS OF EXPERIMENTS A.1 D ATASETS USED CIFAR10 Krizhevsky et al. (2009): The CIFAR10 dataset consists of 32x32 colored images with 10 classes. There are 50000 training images and 1000...
2009
-
[23]
Dpub is CIFAR10-1
A.3 D ETAILED EXPERIMENTAL RESULTS ON CIFAR10 This section presents the experimental results ( p-values) of several baselines and our method on CIFAR10. Dpub is CIFAR10-1. The results are shown in Table 10, Table 11, and Table 12, respectively. ‘Dsus’ is the dataset used to pr...
-
[512]
On ImageNette/ImageWoof, the encoder with non-ViT-S/16 architecture is pre-trained for 800 epochs, while ViT-S/16 architecture is pre-trained for 2000 epochs with a batch size of
2000
-
[2001]
Membership inference attacks from first principles
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. IEEE,
2022
-
[2002]
Imagenet: A large-scale hierarchical image database
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. Ieee,
2009
-
[2009]
An image is worth 16x16 words: Transformers for image recognition at scale
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...
2010 arXiv
-
[2011]
Scalable private learning with pate
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and ´Ulfar Erlingsson. Scalable private learning with pate. arXiv preprint arXiv:1802.08908,
-
[2017]
Very deep convolutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,
-
[2018]
Encodermi: Membership inference against pre-trained encoders in contrastive learning
Hongbin Liu, Jinyuan Jia, Wenjie Qu, and Neil Zhenqiang Gong. Encodermi: Membership inference against pre-trained encoders in contrastive learning. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 2081–2095, 2021a. Yang Liu, Zhen Zh...
2021
-
[2020]
Differential privacy
11 Published as a conference paper at ICLR 2025 Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pp. 1–12. Springer,
2025
-
[2021]
An empirical study of training self-supervised vision transformers
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9640–9649, 2021a. Xinyun Chen, Wenxiao Wang, Chris Bender, Yiming Ding, Ruoxi Jia, Bo L...
2021
-
[2022]
Proof-of-learning is currently more broken than you think
Congyu Fang, Hengrui Jia, Anvith Thudi, Mohammad Yaghini, Christopher A Choquette-Choo, Natalie Dullerud, Varun Chandrasekaran, and Nicolas Papernot. Proof-of-learning is currently more broken than you think. In 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P...
2023
-
[2023]
Model stealing attacks against inductive graph neural networks
13 Published as a conference paper at ICLR 2025 Yun Shen, Xinlei He, Yufei Han, and Yang Zhang. Model stealing attacks against inductive graph neural networks. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1175–1192. IEEE,
2025
-
[2024]
Label-only membership inference attacks
Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International conference on machine learning, pp. 1964–1974. PMLR,
1964
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.