{"id":"a614d692-b2c3-4183-98eb-d11ed0c9e40e","arxiv_id":"2505.10351","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"PartCrop uses part-level feature responses to infer membership in black-box visual self-supervised encoders, with attack accuracies of 56-79% across three datasets and three SSL paradigms.","lead":"This paper presents PartCrop, a membership inference attack that can tell, without knowing the training recipe, whether an image was part of a self-supervised encoder's training data. It works across contrastive learning and masked image modeling, and the paper also studies scaling behavior and a defense.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PartCrop's central claim assumes the black-box API returns spatial feature maps; if only pooled embeddings are exposed, the method cannot construct its response-energy feature.","rationale":"The reader's weakest_assumption identifies exactly the same premise: the attacker needs spatial feature maps rather than a pooled embedding. This is the load-bearing concern because the entire method, including the similarity distribution in Eqs. 3–5 and the KL-divergence energies, is defined over spatial locations. Without per-location features, the method cannot be applied as written. The concern does not invalidate the results within the authors' stated experimental setup, but it does limit the real-world generality of the 'black-box' claim, since many embedding APIs expose only a global vector. I agree with the reader's conditional verdict: the method is well-validated within its assumed API setting, but the threat model must be made explicit and the pooled-embedding case should be tested or acknowledged. The other issues noted by the reader, such as missing error bars and the MoCo scaling trend, are secondary and do not affect the core method's validity as strongly as the output-granularity assumption.","tokens_in":33100,"tokens_out":4629,"duration_ms":52551,"concrete_test":"Use the released code to rerun the Table 4 partial-setting attack with an API wrapper that returns only avgpooled global features for the whole image and for each crop (i.e., χ becomes a single vector, N=1). Construct membership features from the scalar similarities between the global image vector and the m part vectors, retrain the same attacker on CIFAR100 for MAE, DINO, and MoCo, and compare attack accuracy with Table 4. If any model drops to roughly 50%, PartCrop requires spatial feature-map access and Section 3 must state this restriction; if all models remain above 55%, the assumption is not load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines the adversary as having 'black-box access' to the target SSL encoder, but Section 4 Stage 1 assumes the encoder outputs a spatial feature map χ ∈ R^{H×W×D} (Eq. 1). The membership feature E is built from the similarity distribution between each part vector p_i and the N = H×W spatial locations of χ (Eqs. 3–5). This is not a pooled embedding: it requires per-location vectors. The threat model never states that the service exposes this granularity, and the paper's motivating examples (e.g., 'extract data features' through ML-as-a-service APIs) typically return a single global vector. If a real API exposes only pooled representations, PartCrop cannot compute the KL-divergence feature and the attack as described does not run. The central claim, 'without knowing how self-supervised model is trained, PartCrop generally achieves satisfying attack performance,' is therefore conditional on an unstated API assumption. This is not an internal inconsistency, but it is the least secure premise connecting the threat model to the headline result. Appendix E discusses noisy-crop limitations but does not acknowledge this output-granularity requirement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PartCrop, a membership inference attack against visual self-supervised encoders under a black-box setting in which the adversary does not know the SSL training method or its hyperparameters. The attack crops multiple image parts, feeds the whole image and the crops through the victim encoder to obtain a spatial feature map and part features, computes per-location cosine similarities, converts them into a distribution, and uses KL divergence against uniform and Gaussian benchmarks as membership features (Eqs. 1–5). A simple MLP attacker is trained on these features. Experiments across MAE, DINO, and MoCo on CIFAR10/100 and Tinyimagenet report attack accuracies of roughly 53–79%, outperforming baselines including EncoderMI, and the paper evaluates defenses (early stopping, differential privacy, shrinking crop scale range) and scaling behavior, leading to the PartCrop-v2 attacker with Tanh and RMSNorm.","tokens_in":33344,"tokens_out":4869,"duration_ms":47058,"significance":"The paper addresses a timely and practically relevant question: membership inference against SSL encoders without training-recipe knowledge. The part-aware motivation is well grounded in prior work [20], and the evaluation is broad, covering multiple SSL paradigms, architectures, datasets, and threat models. The release of code and the scaling study are additional strengths. If the threat model is refined to match real API output granularity, the method could be practically valuable. However, the current manuscript overstates the generality of some results, and the missing error bars and a contradiction in the scaling narrative weaken several quantitative claims.","major_comments":[{"comment":"The threat model in Section 3 defines only black-box access, but Stage 1 of PartCrop assumes the encoder outputs a spatial feature map χ ∈ R^{H×W×D} and the membership feature is built from per-location similarities (Eqs. 3–5). If a real black-box API exposes only a pooled or global embedding, the attack cannot be constructed. This output-granularity requirement should be stated explicitly in the threat model, justified with examples of APIs that expose feature maps, or evaluated with a pooled-embedding variant. As written, the central claim that PartCrop works 'without knowing how self-supervised model is trained' is conditional on an unstated API assumption.","section":"Sections 3–4, Eq. (1)"},{"comment":"The scaling experiments are reported without error bars even though the text states that three repeated runs were averaged, so the reader cannot assess whether the non-monotonic trends for DINO and MAE are significant. More seriously, Answer 1 states that scaling encoder training data reduces membership inference risk, with a more gradual reduction in contrastive learning, but Fig. 7(a) shows MoCo attack accuracy increasing with data scale (60.89% to 64.63% to 66.0%). This is a direct contradiction that is not resolved by the single ImageNet22K point (51.40%), which is also reported without variance. Please correct Answer 1 or provide the evidence that reconciles the stated conclusion with the plotted trend, and report error bars or confidence intervals.","section":"Section 8.2.1, Fig. 7, Answer 1"},{"comment":"Several reported cross-dataset successes are degenerate classifiers. For example, MAE CIFAR10→Tinyimagenet achieves Acc 52.13, Pre 99.51, Rec 4.14, and MoCo Tinyimagenet→CIFAR100 achieves Pre 100.00, Rec 12.96. With recall below 5%, an accuracy near 52% is essentially equivalent to an all-non-member classifier, and the F1 value is driven by the trivial precision-recall trade-off. The claim in Section 6.3 that 'PartCrop succeeds in all cross-dataset attacks' is therefore not supported by the reported metrics. Please report balanced measures such as AUC or balanced accuracy, or tune the decision threshold, and temper the generalization claim accordingly.","section":"Table 6, Section 6.3"},{"comment":"The method's performance is highly sensitive to the crop scale choice (e.g., MAE accuracy drops from 58.38% to 50.32% when the scale changes from (0.08, 0.2) to (0.01, 0.03)), and Appendix E acknowledges that the pre-defined crop scale may not suit all datasets. Since the adversary in the stated threat model does not know the training recipe, it also cannot know the optimal crop scale for a given victim model. The paper should specify how the attacker chooses the crop scale in practice (e.g., a fixed default across all datasets or a dataset-adaptive rule) and provide a sensitivity analysis across all evaluated datasets so that the 'unified' claim is not contingent on an unavailable oracle choice.","section":"Section 6.2.3, Fig. 5, Appendix E"}],"minor_comments":[{"comment":"The defense name 'PartCorp' appears in the text; it should be 'PartCrop'.","section":"Section 7"},{"comment":"The subsection heading 'Merbership Feature' contains a typo; it should be 'Membership Feature'.","section":"Section 6.2.1"},{"comment":"The caption states 'Comparisions with SupervisedMI...'; the intended word is 'Comparisons'.","section":"Table 4 caption"},{"comment":"The sentence 'To answer Question 2' should refer to Question 4, since the paragraph addresses attacker scaling rather than attacker training-data size.","section":"Section 8.3.2"},{"comment":"The KL-divergence direction and the support of the Gaussian benchmark should be defined precisely: the distributions are over the N spatial locations, and it should be clarified whether the Gaussian vector g_i is regenerated for each query or fixed across queries, since this affects the stochasticity of the membership feature.","section":"Eq. (5), Section 4 Stage 2"},{"comment":"The text says that after scaling attacker training data, DINO performance 'is still higher than that of 2000 when the size is 10000.' This is true for accuracy (58.10 vs 57.99) but false for F1 (48.86 vs 54.58), so the claim should be qualified to refer to accuracy only, or the discrepancy should be explained.","section":"Section 8.2.2, Fig. 8"},{"comment":"The variant definitions appear mislabeled: 'Narrow' replaces d=512 with d=1024 (a wider network) and 'Wide' replaces d=512 with d=256 (a narrower network). If the labels are correct, the accompanying discussion of 'too small' dimensions for the narrow variant is confusing; please clarify the naming or the definitions.","section":"Table 11"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extension of the authors' CCS 2024 paper [24]; the primary new content is the scaling study, which currently contains the most serious unresolved issues (missing error bars and the contradiction between Answer 1 and Fig. 7). The threat-model gap concerning spatial feature map exposure should be addressed before publication, as it affects the practical relevance of the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before deciding whether to spend time on it. First, the core idea is real: PartCrop infers membership in self-supervised vision encoders without knowing whether the target was trained with MAE, DINO, MoCo, or something else, and it works moderately well in the partial-knowledge setting. Second, the threat model has a load-bearing gap that the authors never state. The attack needs the encoder to return a spatial feature map χ ∈ R^{H×W×D}; the membership feature is built from similarities between each crop and every spatial location. The paper frames the adversary as having black-box API access, which in practice often means a single pooled embedding. If the API only returns a global vector, PartCrop cannot construct its response-energy feature. This is not a fatal flaw in the method, but it is a fatal flaw in how the threat model is presented, and the headline claim is unconditional.\n\nWhat is genuinely new and good: the recipe-free setting, the part-aware response feature, and the ImageNet1K scaling study with PartCrop-v2. The paper is honest that it extends a CCS 2024 paper, and the new material is disclosed cleanly. The attack beats EncoderMI on masked-image-modeling targets where EncoderMI falls to random guessing, and the defense section is a reasonable first pass. Code is available.\n\nSoft spots, in proportion: the threat model gap is the big one, and it is fixable by rewriting the assumptions. The scaling study lacks error bars even though the authors say they repeated each setting three times; some of the plotted differences are smaller than typical seed variance, so the trend claims are shakier than the prose. Answer 1 ('scaling data reduces risk') is overgeneralized: MoCo's accuracy actually goes up from 1% to 100% in Fig 7, and MAE's goes down, so the answer is at best 'it depends on the paradigm.' The ImageNet22K single point for MoCo supports the eventual-drop speculation but is thin. Finally, some shadow-setting successes are degenerate, e.g., MAE CIFAR10-to-Tinyimagenet at 52.13% accuracy with 4.14% recall; calling that a success is generous.\n\nNet: this paper deserves a serious referee. The method is novel, the experiments are broad, and the weakness is in the threat-model wording and the strength of the scaling conclusions, not in the core attack. A careful referee should push for an explicit feature-map-access assumption, error bars, and more honest language about when the attack does and does not work.\n\nRecommendation: send to peer review, with a request for major revision on the threat model and scaling presentation.","headline":"PartCrop is a genuinely novel recipe-free membership inference attack on visual SSL encoders, but its headline claim quietly assumes an API exposes spatial feature maps rather than pooled embeddings.","tokens_in":33886,"tokens_out":1899,"would_cite":true,"duration_ms":20860,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PartCrop infers training membership from self-supervised vision encoders without knowing the training recipe.","keywords":["membership inference","self-supervised learning","part-aware capability","black-box attack","visual encoder privacy","contrastive learning","masked image modeling","scaling privacy risk"],"falsifier":"Run the PartCrop pipeline unchanged against an encoder API that returns only the pooled global embedding (no spatial map) in the partial setting: if attack accuracy falls to roughly 50 percent for MAE, DINO, and MoCo, the spatial part-response mechanism is confirmed as the active signal; if accuracy stays above chance, that premise is not necessary. A complementary check is to attack a model trained on a synthetic dataset of texture-only images that contain no object parts, where the claimed part-aware capability should have nothing to attach to and accuracy should collapse toward chance.","tokens_in":32877,"feed_emoji":"🔍","tokens_out":7662,"duration_ms":67676,"temperature":0.7,"pith_summary":"The paper claims that a single black-box attack, PartCrop, can decide whether a given image was part of the training set of a visual self-supervised encoder, even when the attacker has no knowledge of the training method or its hyperparameters. It builds on the observation that self-supervised models learn a part-aware capability and that training images produce stronger part-response signals than test images. A sympathetic reader would care because real encoder APIs are black boxes, while the previous state-of-the-art attack requires knowing the training recipe and fails on masked image modeling. PartCrop reports attack accuracy well above chance across MAE, DINO, and MoCo on three datasets, and the paper further studies how data and model scaling change the attack's success.","feed_headline":"PartCrop exposes membership in self-supervised vision encoders","feed_subtitle":"Crops object parts, measures spatial response energy, and attacks MAE, DINO, and MoCo with no training-recipe knowledge.","key_machinery":"The load-bearing construction is the part-response energy. An image is passed through the encoder to obtain a spatial feature map $\\chi \\in \\mathbb{R}^{N \\times D}$; $m$ random crops at scale 0.08–0.2 are encoded into part vectors $P$. Each part vector $p_i$ is multiplied against $\\chi$ to form a similarity vector $v_i$, which is softmaxed into a probability distribution and then compared to a uniform and a gaussian reference distribution by KL divergence, giving $e_i = [e_i^u, e_i^g]$. The sorted energy vectors are concatenated into the membership feature $E \\in \\mathbb{R}^{2m}$ that trains a four-layer MLP attacker, so the method converts the part-response asymmetry between training and test data into a discriminative score without using any training recipe of the victim.","core_discovery":"The central claim is that membership leaves a measurable trace in how self-supervised encoders respond to object parts: for training images, the similarity between a cropped object part and the image's spatial feature map is more peaked than for test images, and this asymmetry persists across contrastive learning, masked image modeling, and hybrid paradigms. PartCrop harvests this trace without any training-recipe prior by randomly cropping many small patches, computing their response energy against the image feature map via KL divergence to uniform and gaussian reference distributions, and feeding the sorted energies to a small fully connected attacker. In the paper's experiments, this yields attack accuracies of 58.38/57.65/66.36 percent for MAE, 60.62/59.13/56.13 percent for DINO, and 77.20/78.84/73.77 percent for MoCo on CIFAR100/CIFAR10/Tinyimagenet, with the method also transferring across datasets in the shadow setting.","pith_inferences":["If a deployment exposes only a pooled global embedding, the spatial feature map that PartCrop needs is unavailable; a testable extension is whether attention maps or multi-crop consistency can reconstruct enough spatial response information to restore attack power.","The part-response gap between members and non-members may be a general memorization probe, useful for auditing datasets or detecting overfitting in self-supervised models beyond the binary membership question.","The scaling results imply that privacy risk tracks model capacity: publishing membership-inference accuracy alongside task accuracy in model cards could make the trade-off visible to deployers.","Because the paper shows PartCrop features are complementary to image-level EncoderMI features, a hybrid of both signals may push attack accuracy higher than either alone."],"forward_implications":["A black-box encoder service that exposes spatial feature maps can be probed for training membership even when the model, its paradigm, and its hyperparameters are hidden.","The attack transfers across datasets: an attacker trained on one public dataset can infer membership in encoders trained on other datasets, widening the threat beyond data-leakage scenarios.","Early stopping, differential privacy, and shrinking the crop scale range all reduce PartCrop's accuracy, but the paper reports only partial mitigation; MoCo remains substantially vulnerable.","Scaling behavior is non-trivial: larger encoders raise membership-inference risk, larger training corpora lower it (more slowly for contrastive learning), and attackers benefit from more training data.","PartCrop-v2, using Tanh activation and RMSNorm, stabilizes attacker scaling and avoids the random-guess collapse observed for narrow or deep attacker variants."],"supporting_citations":[{"why":"The strong baseline that assumes training-recipe knowledge; PartCrop is compared against it and outperforms it on masked image modeling.","marker":"[6]"},{"why":"Supplies the observation that self-supervised models have part-aware capability, motivating the part-crop strategy.","marker":"[20]"},{"why":"MAE, the masked image modeling victim encoder used in the attacks.","marker":"[13]"},{"why":"DINO, the contrastive vision transformer victim encoder.","marker":"[11]"},{"why":"MoCo, the contrastive CNN victim encoder.","marker":"[12]"},{"why":"Foundational membership inference threat model and the black-box setting that PartCrop adopts.","marker":"[9]"},{"why":"ImageNet1K, the large-scale dataset used for the scaling experiments.","marker":"[23]"},{"why":"RMSNorm, a component of the proposed PartCrop-v2 attacker.","marker":"[66]"}],"fun_headline_variants":["PartCrop: Membership leak in self-supervised vision encoders","Part-aware cropping reveals training data in SSL encoders","Blind membership attack on self-supervised vision models","Crop parts, detect training images in SSL encoders","PartCrop: Unified attack on self-supervised encoders"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack requires the black-box service to return a spatial feature map with per-location vectors; if only a single global embedding is exposed, the part-response energies cannot be constructed and PartCrop's core signal disappears.","fun_headline_variants_meta":{"raw":{"variants":["PartCrop: Membership leak in self-supervised vision encoders","Part-aware cropping reveals training data in SSL encoders","Blind membership attack on self-supervised vision models","Crop parts, detect training images in SSL encoders","PartCrop: Unified attack on self-supervised encoders"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00048,"raw_usage":{"total_tokens":2424,"prompt_tokens":1043,"completion_tokens":1381,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":1299}},"tokens_in":659,"tokens_out":1381,"duration_ms":10371,"temperature":1.0,"reasoning_tokens":1299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:10:45.995559+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the PartCrop pipeline unchanged against an encoder API that returns only the pooled global embedding (no spatial map) in the partial setting: if attack accuracy falls to roughly 50 percent for MAE, DINO, and MoCo, the spatial part-response mechanism is confirmed as the active signal; if accuracy stays above chance, that premise is not necessary. A complementary check is to attack a model trained on a synthetic dataset of texture-only images that contain no object parts, where the claimed part-aware capability should have nothing to attach to and accuracy should collapse toward chance.","supporting_citations":[],"review_version":1}