{"id":"7f6014b8-0362-4d25-8838-ddf051c289f9","arxiv_id":"2608.11643","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AI-art detectors trained on LDM and SD2.1 artwork miss 42% to 58% of Stable Diffusion 3.5 Medium images in a zero-shot cross-generator test.","lead":"This paper tests whether AI-art detectors trained on older Stable Diffusion models still catch images from the newer Stable Diffusion 3.5. All five detectors miss many more new images while rarely flagging human art, and the best detector is a CLIP vision model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD drop is attributed to generator shift, but prompt-pipeline shift is a confound: SD3.5m images use reverse-prompted, title-augmented prompts while ID images use a different pipeline; a control with LDM/SD2.1 under the new prompts is needed.","rationale":"The reader's weakest assumption identifies the same confound I consider most load-bearing. The paper's central contribution is demonstrating that generator shift degrades frozen-backbone linear-probe detectors. That conclusion requires the OOD AI set to differ from the ID AI set only in the generator; instead, the SD3.5m images are produced with a substantially different text-conditioning pipeline (reverse-prompted, title-augmented prompts), so the experiment conflates two variables. This is not merely a stylistic detail: prompt distribution is known to affect generated image statistics, and a linear probe trained on LDM/SD2.1 artifacts may fail on any images with different prompt-induced statistics, regardless of the underlying generator. The fact that the paper acknowledges the limitation in Section 5 strengthens rather than weakens the concern. The proposed control experiment would settle the attribution cleanly. I do not see a stronger objection: the dataset construction is otherwise careful (balanced pruning, locked thresholds, held-out human reference, deterministic seeds), and the reported drops are large and consistent across architectures. Missing error bars and unreleased code are secondary and do not change the conditional verdict. Therefore no adjustment to the reader's CONDITIONAL verdict is needed.","tokens_in":27555,"tokens_out":4650,"duration_ms":43241,"concrete_test":"Generate a matched control OOD set by running the exact Section 3.4 pipeline (CLIP Interrogator with BLIP-Large and ViT-L/14, same prompt template, same seeds) on the same 10,000 held-out human artworks, but with LDM and SD2.1 as the generators instead of SD3.5m (yielding roughly 20,000 control AI images). Evaluate all five detectors with their locked validation thresholds. If balanced-accuracy drops on the LDM/SD2.1 control are comparable to the SD3.5m drops in Table 16, the degradation is driven by prompt-pipeline shift rather than generator architecture; if the control stays near ID levels, the generator-shift interpretation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is a confound between generator identity and prompt distribution. The SD3.5m OOD images are generated from CLIP Interrogator + BLIP captions augmented with style and title (Sections 3.4.2-3.4.3), whereas the in-distribution AI-ArtBench images from LDM and SD2.1 were generated with a different, unspecified prompt pipeline. Section 5 explicitly acknowledges that 'there may still be some prompt distribution shift'. The central claim is that detectors degrade because of the new DiT-based generator architecture, but the design changes both the generator and the prompt pipeline simultaneously. If the prompt pipeline alone produces images that are more 'human-like' in the feature spaces used by frozen backbones, then even LDM/SD2.1 images generated under the new reverse-prompting pipeline might trigger similarly large false-negative rates, and the reported drops (CLIP ViT-L/14: -21.4 balanced accuracy, -0.430 recall) would not uniquely implicate the generator change. The dataset quality metrics in Table 5 (CLIPScore, FID, KID, LPIPS) do not resolve this because they measure prompt-image alignment and distributional distance within the OOD set, not the effect of prompt pipeline on detector features. Without a control OOD set from LDM/SD2.1 under the same reverse-prompted, title-augmented prompts, the headline conclusion is underdetermined.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates how AI-art detectors trained on earlier U-Net-based generators behave when evaluated on images from a newer Diffusion Transformer-based generator, Stable Diffusion 3.5 Medium. The authors construct a prompt-aligned SD3.5m dataset of 10,000 images across ten art styles using a reverse-prompting pipeline (CLIP Interrogator with BLIP captions, followed by title and style augmentation), train five frozen-backbone linear-probe detectors on a balanced subset of AI-ArtBench (LDM and SD2.1) data, and evaluate them on this OOD set with fixed validation-selected thresholds. The paper reports strong in-distribution performance, substantial OOD drops in balanced accuracy and AI recall, low OOD false-positive rates, and a per-style analysis showing that Realism is hardest and Ukiyo-e easiest. Grad-CAM visualizations on ConvNeXt-Base are used to argue that OOD false negatives arise from weak and peripheral attribution. The central claim is that current frozen-backbone detectors generalize poorly under generator shift, with the primary failure mode being missed AI images.","tokens_in":27801,"tokens_out":5373,"duration_ms":55390,"significance":"If the result holds, the paper provides a valuable empirical warning: detectors that look near-perfect on known generators can miss a large fraction of images from a newer architecture, and the false-negative asymmetry has direct operational consequences. The study has notable strengths: a balanced, source-equalized training corpus; deterministic preprocessing; frozen backbones with linear probes; fixed thresholds; per-style and source-wise analyses; and a clearly stated emerging-generator threat model. The authors also candidly acknowledge in Section 5 that prompt distribution shift may remain. However, the headline attribution of the performance drop to generator architecture is not uniquely established, because the OOD construction changes both the generator and the prompt pipeline simultaneously. The paper is therefore a useful empirical contribution, but its central claim needs either an additional control experiment or a more carefully qualified interpretation.","major_comments":[{"comment":"The central claim that generator shift causes the OOD performance drop is confounded with prompt-pipeline shift. The SD3.5m images are generated from reverse-prompted, title-augmented prompts built with CLIP Interrogator and BLIP, whereas the in-distribution AI-ArtBench images from LDM and SD2.1 were generated with a different, unspecified prompt pipeline. Because the experimental design changes both the generator and the prompt distribution at once, the large drops in recall (e.g., CLIP ViT-L/14 from about 0.997 to 0.568 in Table 16) cannot be uniquely attributed to the change in generator architecture. The acknowledgment in Section 5 that 'there may still be some prompt distribution shift' does not resolve the issue, and the dataset quality metrics in Table 5 measure prompt-image alignment and distributional distance rather than the effect of prompt pipeline on detector features. A control OOD set generated from LDM and/or SD2.1 under the same reverse-prompted, title-augmented pipeline is needed; without it, the conclusions should be reframed as applying to a combined generator-plus-prompt shift.","section":"Sections 3.4.2-3.4.4, 4.3, and 5"},{"comment":"The same CLIP ViT-L/14 model is used both as the vision encoder in the prompt-generation pipeline (CLIP Interrogator) and as the detector backbone for the best-performing model. This shared component is a potential confound for the cross-architecture comparison: the OOD images may be systematically closer to or farther from the CLIP ViT-L/14 feature distribution used by the detector, making the finding that 'CLIP ViT-L/14 performs best overall' in Section 4.3 difficult to interpret as a pure statement about detector robustness. The authors should either use a different encoder in the Interrogator (for example, a ViT-B/32 or an OpenCLIP variant) or provide a control analysis showing that the ranking of backbones is insensitive to this shared-component choice.","section":"Sections 3.4.2 and 3.6.1"}],"minor_comments":[{"comment":"Please clarify how the 10,000 held-out human samples were selected across the ten styles and whether the sampling was stratified with a fixed seed; the current text states only that the images were 'sampled across the 10 art styles.'","section":"Section 3.4.1"},{"comment":"The early-stopping behavior is described ambiguously: the text says the procedure uses patience of three epochs but that training proceeded to the full ten-epoch budget in runs that continued to improve. Please report which models, if any, were stopped early.","section":"Section 4.1"},{"comment":"The overall row is computed on pooled data rather than as an average of the per-style rows; this is stated in the text, but a table footnote would prevent reader confusion.","section":"Table 5"},{"comment":"The style name is spelled 'Ukiyoe' in Figure 15 and 'Ukiyo-e' elsewhere; please unify the spelling throughout.","section":"Section 4.4 and Figure 15"},{"comment":"For reproducibility, please provide the exact diffusers library version and the checkpoint revision or hash for stabilityai/stable-diffusion-3.5-medium, in addition to the generation parameters already listed in Table 4.","section":"Section 3.4.4"},{"comment":"The paper makes strong reproducibility claims but does not state whether the OOD dataset, extracted feature bundles, training scripts, or audit logs will be released; please add an availability statement or repository link.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The protocol is a genuine held-out evaluation and I see no circularity in the main experiment; the reason for major revision is the confound between generator identity and prompt-pipeline identity, not any internal inconsistency. The proposed control experiment, generating LDM/SD2.1 images under the same reverse-prompting pipeline, is feasible within the paper's scope and would either strengthen the headline claim or require it to be appropriately qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, it delivers a genuinely useful artifact: a 10,000-image SD3.5 Medium dataset of prompt-aligned AI artwork across ten styles, with a documented reverse-prompting pipeline. Second, the headline claim—that the performance drop is due to generator architecture shift—is real but not uniquely established, because the OOD set changes both the generator and the prompt-generation pipeline at once.\n\nThe experiment itself is done carefully. The authors prune AI-ArtBench to balance classes and styles, lock thresholds on the validation set, hold out the human reference set that supplies the reverse prompts, and evaluate five frozen-backbone linear probes under a shared protocol. The ID numbers look sane, and the OOD asymmetry (low false-positive rate on humans, recall collapsing on SD3.5m) is striking and consistent across all five detectors. ConvNeXt-Base and CLIP ViT-L/14 degrade the least, but even the best drops 21 points of balanced accuracy and misses 43% of AI images.\n\nThe main soft spot is the confound. The SD3.5m images are generated from CLIP Interrogator + BLIP captions, then augmented with style and title, whereas the AI-ArtBench training images came from a different, unspecified prompt pipeline. So the OOD test set shifts both the generator and the prompt distribution. The authors acknowledge this in Section 5, but they still attribute the drop to generator shift throughout the paper. A control set of LDM/SD2.1 images generated under the same reverse-prompted pipeline would separate the two, and without it the title claim is stronger than the evidence.\n\nSmaller issues: no error bars (with 10k images per class the main gaps are clearly significant, but per-style cells are 2k images and model rankings might wobble), no code or dataset release, and the Grad-CAM analysis is qualitative and only for one backbone. The CLIP ViT-L/14 being used both to build the prompts and to act as the best detector is worth a note, though it is not a fatal circularity—the target result is not fitted.\n\nCitation pattern is fine. This is a legitimate extension of GenImage/ArtiFact-style cross-generator work, not a new detection method. I would send it to a serious referee with a request for the control experiment (or a toned-down causal claim) and release of the benchmark and code. The core result—frozen-backbone detectors miss a large fraction of newer generator images—is solid enough to stand, even after the confound is acknowledged.","headline":"A clean, honest empirical study of cross-generator detector failure, but the generator-shift attribution is underdetermined because the OOD set also switches the prompt pipeline.","tokens_in":28368,"tokens_out":2334,"would_cite":true,"duration_ms":26550,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that frozen-backbone AI-art detectors trained on earlier U-Net diffusion generators lose 21 to 26 points of balanced accuracy on Stable Diffusion 3.5 Medium, with most errors being missed AI images rather than false…","keywords":["AI-art detection","generator shift","out-of-distribution evaluation","Stable Diffusion 3.5 Medium","Diffusion Transformer","frozen-backbone linear probe","CLIP ViT-L/14","Grad-CAM"],"falsifier":"Generate a control out-of-distribution set from Stable Diffusion 2.1 or Latent Diffusion using the same CLIP Interrogator, BLIP, and title-augmented prompt pipeline used for the SD3.5m dataset, and evaluate the five detectors at their locked thresholds. If these control images show the same large drop in balanced accuracy and recall as SD3.5m, the degradation is caused by the prompt pipeline rather than by the generator architecture; if they stay near in-distribution levels, the generator itself is the cause.","tokens_in":27315,"feed_emoji":"🎨","tokens_out":9358,"duration_ms":83734,"temperature":0.7,"pith_summary":"This paper asks whether AI-art detection remains reliable when a content producer simply switches to a newer, unseen image generator. Five frozen-backbone detectors trained on artwork from Latent Diffusion and Stable Diffusion 2.1 are evaluated, without retraining or threshold adjustment, on 10,000 prompt-aligned images from Stable Diffusion 3.5 Medium, a Diffusion Transformer model. The paper claims that every detector suffers a substantial drop in balanced accuracy, and that the dominant failure mode is missed AI images: recall on the new generator falls to between 0.419 and 0.568 while false positives on human artwork stay low. The best model, CLIP ViT-L/14, still loses 21.4 accuracy points and falls from roughly 0.997 in-distribution recall to 0.568 out-of-distribution. The consequence is that near-perfect benchmark performance on known generators can conceal weak coverage of newer generators, so detection should be treated as one layer of a broader verification system.","feed_headline":"AI-art detectors lose up to 26 points on Stable Diffusion 3.5 images","feed_subtitle":"Five detectors trained on older diffusion models miss 42–58 percent of images from the newer generator, mostly as false negatives.","key_machinery":"The central object is the prompt-aligned SD3.5m dataset, built to make generator shift measurable: 10,000 Stable Diffusion 3.5 Medium images, 1,000 per art style, generated from reverse-prompted, title-augmented captions (CLIP Interrogator with BLIP-Large) of held-out human artworks, so the OOD images match the style and approximate content of the human references while the generative architecture changes from U-Net-based latent diffusion to a Diffusion Transformer. The detectors are frozen pretrained backbones (ResNet-18/50, EfficientNet-B0, ConvNeXt-Base, CLIP ViT-L/14) with a single linear classification head, trained with weighted binary cross-entropy and evaluated at a validation-selected threshold never adjusted on OOD data. This combination is what turns a new generator into a controlled experimental variable and puts the asymmetric false-negative failure at the center of the results.","core_discovery":"On its own terms, the paper's central discovery is a generalization gap: detectors trained and tuned on one generator family do not transfer to a newer architecture. All five models are nearly flawless in-distribution, with balanced accuracy between 0.926 and 0.997; under generator shift to SD3.5m, balanced accuracy drops by 20.8 to 25.7 percentage points. The error pattern is asymmetric: false-negative rates climb steeply while false-positive rates remain low, so a deployed detector would let roughly 4,300 to 5,800 of 10,000 SD3.5m images pass as human-created. CLIP ViT-L/14 performs best overall, and Ukiyo-e remains comparatively detectable while Realism is hardest, with no model exceeding 0.694 balanced accuracy on Realism. Grad-CAM analysis shows that false negatives have weaker and more diffuse activations than successful in-distribution detections, supporting the view that the visual cues learned from LDM and SD2.1 do not cleanly transfer to SD3.5m.","pith_inferences":["Beyond the paper: because the SD3.5m images were generated with a different prompt pipeline (CLIP Interrogator, BLIP captioning, and title augmentation) than the AI-ArtBench in-distribution images, part of the measured drop may reflect prompt or content shift; a control OOD set from SD2.1 or LDM generated with the same reverse-prompting pipeline would separate the two explanations.","Beyond the paper: if generator architecture is the dominant cause, then detectors trained on low-level artifacts will keep failing as new architectures and samplers appear; a more durable detector would likely need to model the generative process itself, for instance through diffusion reconstruction error or frequency-aware features, rather than relying on frozen image features alone.","Beyond the paper: a testable extension is to train detectors on a prompt-diverse and generator-diverse mixture and measure the SD3.5m gap; if the gap shrinks, both prompt diversity and generator diversity are needed for robustness.","Beyond the paper: the asymmetric false-negative pattern suggests that in evasion-prone deployments the operating threshold should be tuned for recall, and that validation-selected thresholds may need recalibration whenever a new generator family appears."],"forward_implications":["A non-adversarial producer who merely adopts a newer public generator can evade a deployed detector; no gradient access, adversarial perturbation, or knowledge of the detector is required.","In a screening workflow, the practical failure is one-sided: synthetic images from newer generators mostly pass as human, while human artwork is rarely flagged, so high precision on known generators gives false confidence.","Strong in-distribution accuracy should not be used as evidence of deployment readiness; periodic evaluation against newly released generators is the corresponding requirement.","Style-aware evaluation is necessary, since OOD difficulty varies from Ukiyo-e, the easiest style, to Realism, the hardest, across all backbones.","Backbone choice reduces but does not close the gap: CLIP ViT-L/14 and ConvNeXt-Base degrade least, yet still lose more than 20 points of balanced accuracy."],"supporting_citations":[{"why":"Supplies the AI-ArtBench benchmark that provides the in-distribution training, validation, and ID test splits.","marker":"[49]"},{"why":"Supplies the Latent Diffusion generator family used for in-distribution training data.","marker":"[44]"},{"why":"Defines the out-of-distribution generator, Stable Diffusion 3.5 Medium, a Diffusion Transformer-based model.","marker":"[13]"},{"why":"Provides ArtBench-10, the source of the human artwork samples used for the human class.","marker":"[27]"},{"why":"Provides CLIP Interrogator, the reverse-prompting tool used to construct the prompt-aligned SD3.5m dataset.","marker":"[38]"},{"why":"Supplies the BLIP-Large captioning model used inside the reverse-prompting pipeline.","marker":"[25]"},{"why":"Supplies the CLIP ViT-L/14 backbone and the embedding space used for detection and prompt alignment.","marker":"[39]"},{"why":"Supplies the ResNet-18 and ResNet-50 backbones evaluated as CNN baselines.","marker":"[17]"},{"why":"Supplies the EfficientNet-B0 backbone evaluated as a CNN baseline.","marker":"[51]"},{"why":"Supplies the ConvNeXt-Base backbone, the best CNN detector in the study.","marker":"[29]"}],"fun_headline_variants":["AI-art detectors miss up to 58% of Stable Diffusion 3.5 art","Generator shift cuts AI-art detector accuracy by up to 26 points","New generator fools five AI-art detectors trained on older models","AI-art detectors flunk on SD3.5: 42-58% pass as human"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the out-of-distribution performance drop is caused by the change in generator architecture, not by the change in the prompt-construction pipeline used to create the SD3.5m images; the paper itself acknowledges this prompt distribution shift as a competing explanation.","fun_headline_variants_meta":{"raw":{"variants":["AI-art detectors miss up to 58% of Stable Diffusion 3.5 art","Generator shift cuts AI-art detector accuracy by up to 26 points","New generator fools five AI-art detectors trained on older models","AI-art detectors flunk on SD3.5: 42-58% pass as human"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1646,"prompt_tokens":984,"completion_tokens":662,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":579}},"tokens_in":600,"tokens_out":662,"duration_ms":6842,"temperature":1.0,"reasoning_tokens":579,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:32:26.612813+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a control out-of-distribution set from Stable Diffusion 2.1 or Latent Diffusion using the same CLIP Interrogator, BLIP, and title-augmented prompt pipeline used for the SD3.5m dataset, and evaluate the five detectors at their locked thresholds. If these control images show the same large drop in balanced accuracy and recall as SD3.5m, the degradation is caused by the prompt pipeline rather than by the generator architecture; if they stay near in-distribution levels, the generator itself is the cause.","supporting_citations":[{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Supplies the Latent Diffusion generator family used for in-distribution training data."},{"cited_title":"Scaling rectified flow transformers for high-resolution image synthesis","cited_arxiv_id":null,"evidence_quote":"Defines the out-of-distribution generator, Stable Diffusion 3.5 Medium, a Diffusion Transformer-based model."},{"cited_title":"pharmapsychotic/clip-interrogator.https://github","cited_arxiv_id":null,"evidence_quote":"Provides CLIP Interrogator, the reverse-prompting tool used to construct the prompt-aligned SD3.5m dataset."},{"cited_title":"BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation","cited_arxiv_id":null,"evidence_quote":"Supplies the BLIP-Large captioning model used inside the reverse-prompting pipeline."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP ViT-L/14 backbone and the embedding space used for detection and prompt alignment."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet-18 and ResNet-50 backbones evaluated as CNN baselines."},{"cited_title":"EfficientNet: Rethinking model scaling for convo- lutional neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the EfficientNet-B0 backbone evaluated as a CNN baseline."},{"cited_title":"A ConvNet for the 2020s","cited_arxiv_id":null,"evidence_quote":"Supplies the ConvNeXt-Base backbone, the best CNN detector in the study."}],"review_version":1}