{"id":"6d6f0ad6-0475-4a75-9fdb-237a32a9898b","arxiv_id":"2412.01512","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"ArtBrain introduces a 185k-image AI-art dataset and an AttentionConvNeXt model reported to classify style and source with F1 0.869 and 0.999 attribution accuracy, though the evaluation protocol undermines those numbers.","lead":"This paper presents ArtBrain, a toolkit that classifies artworks by style and attributes their source to a human artist or a specific AI image generator. It also introduces a new 185,000-image dataset of human and AI art and reports that its model outperforms human volunteers at detecting AI-generated art.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline F1 and attribution scores are not unbiased because the same 30,000-image split was used for both checkpoint selection and final evaluation, so the central accuracy claims are not supported as reported.","rationale":"The reader's weakest_assumption identifies exactly the most load-bearing flaw: using the test split as the validation split and then reporting metrics on that same split. This is not a stylistic quibble; it directly undermines the central numerical claims and the human-comparison conclusion. The paper explicitly documents the protocol in Sections 4.2 and 5, so there is no ambiguity. Because the checkpoint was selected by validation loss on the same images later used for evaluation, the reported F1 of 0.869 and attribution accuracy of 0.999 cannot be interpreted as unbiased estimates of generalization. This alone justifies rejecting the paper's central claims, even though the AI-ArtBench dataset and the open-source ArtBrain toolkit have independent value. My stress-test pass did not find a different, stronger concern; the test-as-validation issue is sufficient and is the natural target for a decisive re-run. The proposed concrete test is a standard train/validation/test re-split, which would settle whether the reported numbers survive an honest evaluation protocol. I therefore agree with the reader's verdict and recommend no change.","tokens_in":16608,"tokens_out":3069,"duration_ms":29127,"concrete_test":"Retrain AttentionConvNeXt using the released code and dataset, but hold out a new validation set (e.g., 5,000 images randomly drawn from the current train split), select the best epoch by validation loss, and then evaluate on the original 30,000-image test set. Recompute the overall F1 (Table 2) and attribution accuracy (Table 3). If the test-set F1 falls materially below 0.869 or attribution below 0.999, the reported values were inflated by test-set-based model selection; if they remain essentially unchanged, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that AttentionConvNeXt achieves an F1-score of 0.869 and attribution accuracy of 0.999, and that it outperforms humans. These numbers are load-bearing for the conclusion. Section 4.2 states: 'The testing set was used as the Validation set in the training,' and the best epoch (epoch 15) was selected by lowest validation loss on that set. Section 5 then states: 'The same set used for validation is used as the test set.' This means the reported metrics measure performance on the very images used to choose the checkpoint, not on an independent holdout. Selecting a model on the evaluation set typically inflates measured performance because the model has been tuned to that set's loss landscape. The magnitude of inflation is unknown, but it violates the standard assumption that the test split provides an unbiased estimate of generalization. The Turing test comparison inherits this problem: the AI's 98% accuracy is computed on the same test-as-validation set, so the claim 'AI diffusion-generated artworks are better identified and described by CNNs than by humans' rests on a non-independent evaluation. The dataset and open-source toolkit are useful contributions, but the central numerical claims, and the human-comparison conclusion, are not reliable as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AI-ArtBench, a large dataset of 185,015 artistic images (125,015 AI-generated by Latent Diffusion and Stable Diffusion, 60,000 human artworks), and proposes AttentionConvNeXt, a CNN architecture combining ConvNeXt with a multi-level attention module, for classifying art style and attributing the generative source. The authors also present ArtBrain, a web-based toolkit, and an Artistic Turing Test with 50 participants. The paper claims an F1-score of 0.869 for the 30-class style/source classification, an attribution accuracy of 0.999, and that the CNN outperforms humans (98% vs. about 54%) at distinguishing AI-generated from human art. The dataset and code are publicly released.","tokens_in":16882,"tokens_out":3627,"duration_ms":32421,"significance":"If the reported numbers were reliable, this would be a useful contribution: it is one of the few large-scale datasets for AI-art detection and attribution, the toolkit is practical, and the architectural idea of combining low/mid/high-level features with squeeze-and-excitation across feature blocks is interesting. The open release of the dataset and model code is a clear strength. However, the central empirical claims are undermined by the evaluation protocol, which uses the test set for model selection, and the human-comparison conclusion rests on that biased evaluation. The paper's own robustness tests also raise questions about the practical utility of the model for real-world digitized or photographed artworks.","major_comments":[{"comment":"The evaluation is invalid because the test set is used for model selection. Section 4.2 states 'The testing set was used as the Validation set in the training' and the best epoch (epoch 15) was chosen by lowest validation loss on that set; Section 5 then states 'The same set used for validation is used as the test set.' Selecting the checkpoint on the same images that later produce the reported F1 (0.869) and attribution accuracy (0.999) biases those numbers upward in an unknown amount, and the comparison with human performance in Section 5.5 inherits this bias. The authors must retrain with a proper train/validation/test split and report metrics on a genuinely held-out test set.","section":"§4.2 and §5"},{"comment":"The abstract reports human accuracy of approximately 58% and AI accuracy of around 99%, but Table 5 reports overall human accuracy of 53.8% and ArtBrain accuracy of 98% on the same set. These numbers should be reconciled; the discrepancy could indicate an error in the abstract or in the table, and in either case the reader cannot determine which value is authoritative.","section":"Abstract vs. Table 5"},{"comment":"The contrast and re-photography tests show severe prediction instability: changing contrast by ±100% flips the top prediction for 'Starry Night' from Post-Impressionism-Human to other classes, and capturing a photo of the same image on a 2K screen changes the top prediction from 84% Post-Impressionism-Human to 73% Expressionism-Human. These results are acknowledged as limitations, but they undercut the claim that ArtBrain is a 'ready-to-use' detector for real-world art, where scanning and photographing are common. The conclusion that CNNs are better than humans at identifying AI art should be tempered by this operational fragility.","section":"§5.4"},{"comment":"The Artistic Turing Test is underpowered and its comparison is biased. Only 50 participants and 50 images were used, and the AI's 98% accuracy is computed on the same test-as-validation set that was used to select the model, while the human accuracy is on the same images but without any model-selection bias. The claim that 'AI diffusion-generated artworks are better identified and described by CNNs than by humans' is not supported by the reported protocol; a fair comparison requires an unbiased test set and a larger, more balanced participant sample.","section":"§5.5"}],"minor_comments":[{"comment":"Please fix the typo 'Intel Xenon' to 'Intel Xeon' in the hardware description.","section":"§4.1"},{"comment":"The phrase 'resting the practical effectiveness' should read 'testing the practical effectiveness'.","section":"§5.5"},{"comment":"The table header says 'F1-Score' but the reported values are attribution accuracies; please rename the column to 'Accuracy' or clarify whether F1 is actually meant.","section":"Table 3"},{"comment":"The reference to 'CIFAKE[?]' is unresolved and should be completed or removed.","section":"§2.3"},{"comment":"Consider adding a column description for 'ArtBench' as a baseline; the current table does not indicate whether the ArtBench column refers to a pretrained model or the original dataset's reported score.","section":"§5.1 and Table 2"},{"comment":"The visual difference between the preprocessed and original image is not clear from the figure; adding a legend or a quantitative description of preprocessing (e.g., normalization values) would help.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The core evaluation flaw is fixable by re-running experiments with a proper validation split, so I do not recommend reject at this stage. However, the authors must also address the abstract/table inconsistency and be honest about the model's sensitivity to image transformations when claiming practical utility. The self-citations (FM-G-CAM, ParallelXNet) are disclosed and used appropriately; the concern about source fingerprints is worth raising but does not by itself invalidate the approach."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is a useful dataset-and-toolkit contribution wrapped around an evaluation design that cuts its own legs off. The test set was used as the validation set, and the best epoch was selected on it; the reported F1/attribution numbers are therefore not independent estimates.\n\nWhat is genuinely good: AI-ArtBench is a sizable, publicly released dataset (185k images, 10 styles, 3 sources) with open code and an explainability tool. The combined style-plus-source label design is a real addition, and the attention extension to ConvNeXt is a sensible, if incremental, idea with a plausible ablation. The Artistic Turing Test, while small (n=50), is the kind of human-benchmark attempt the field needs.\n\nThe soft spots are serious, though mostly concentrated in one place. Section 4.2 says 'The testing set was used as the Validation set in the training,' and Section 5 repeats that the test/validation set is the same. Selecting the epoch with the lowest validation loss on the same images that are later scored makes the headline 0.869 F1 and 0.999 attribution accuracy biased upward, possibly substantially. There is also an internal inconsistency: the abstract says humans 58% and AI 99%, while Table 5 gives 53.8% and 98%. The contrast and re-photography tests in Section 5.4 show the model is brittle to simple image manipulations; the authors acknowledge this in the limitations, which is honest, but it cuts against the 'ready-to-use toolkit' framing.\n\nOne more thing: the near-perfect attribution accuracy is likely driven by generator-specific fingerprints (the authors themselves cite DE-FAKE). That is an interesting finding, but it means the results say less about 'AI art detection' in general and more about these two generators versus the ArtBench human set. The dataset does not cover other diffusion models, and the Turing test used only Standard Diffusion images.\n\nIs the central argument salvageable? The dataset and toolkit are salvageable and worth keeping. The numbers need to be re-generated with a proper holdout, and the human-comparison conclusion needs to be qualified. The paper's own limitations section is candid, and it does not overclaim in the conclusion—it explicitly narrows the claim to diffusion-generated art and CNNs.\n\nFor whom: researchers building datasets or tools for synthetic-art attribution; the dataset will likely be reused. The evaluation flaw is common enough that a referee can point to it precisely.\n\nRecommendation: send it to peer review, not desk-reject. The dataset and toolkit deserve scrutiny and refinement, and the flaw is fixable with a re-run on a clean split. It is a borderline case, but the resource value tips it over the line. I would want the revised version to report both the biased and corrected numbers.","headline":"Useful dataset and toolkit, but the headline accuracy numbers are not trustworthy because the test set was used as the validation set for checkpoint selection.","tokens_in":17415,"tokens_out":2316,"would_cite":false,"duration_ms":18962,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a convolutional network, trained on a new 185,015-image dataset of human and diffusion-generated art, can classify art style and source with F1 0.869 and attribute the generating model with 0.999 accuracy…","keywords":["AI art detection","attribution","AI-ArtBench dataset","diffusion models","AttentionConvNeXt","explainable AI","Artistic Turing Test","convolutional neural networks"],"falsifier":"Run the trained AttentionConvNeXt on a fresh 10,000-image set of art from the same ten styles and same three sources that was never used for checkpoint selection or early stopping; if the 30-class F1 drops substantially below 0.869 or the attribution accuracy falls below 0.999, the reported generalization is not as strong as claimed. A second check is to feed images from a diffusion model outside the training set, such as SDXL or Midjourney, and see whether attribution accuracy stays near ceiling or collapses.","tokens_in":16428,"feed_emoji":"🎨","tokens_out":7816,"duration_ms":61514,"temperature":0.7,"pith_summary":"This paper aims to give researchers and the public a working way to tell AI-generated artwork apart from human-made art and to name the model that produced it. To that end, it introduces AI-ArtBench, a dataset of 185,015 images in ten art styles: 125,015 images generated by two diffusion models (Latent Diffusion and Stable Diffusion) plus 60,000 human artworks. It then proposes AttentionConvNeXt, a convolutional network that reports an F1-score of 0.869 for jointly classifying style and source, and an attribution accuracy of 0.999 for identifying the generating model. In an Artistic Turing Test with 50 participants, humans identified AI images with roughly 54–58% accuracy while the model scored 98% on the same images, leading the authors to claim that diffusion-generated artworks are better identified and described by CNNs than by humans.","feed_headline":"New model spots AI art and names its source 99.9% of the time","feed_subtitle":"Trained on 185,015 artworks, AttentionConvNeXt also beats human viewers in a Turing test: 98% versus 54%.","key_machinery":"The load-bearing object is AttentionConvNeXt: a ConvNeXt CNN whose low- and mid-level feature blocks are frozen after ImageNet pretraining, while its high-level blocks are fine-tuned and an attention module (built from squeeze-and-excitation blocks) concatenates the feature maps from all three depths, computes per-channel importance via two fully connected layers, and reweights the concatenated maps before a two-layer classifier. This design is what lets the model use low-, mid-, and high-level visual features together; the ablation shows it adds about 3% F1 over plain ConvNeXt with transfer learning, and the attribution head (the same classifier's source labels) is what yields the near-perfect 0.999 source attribution.","core_discovery":"The central claim is that a single CNN, AttentionConvNeXt, can both tell whether an artwork is human-made or machine-generated and attribute it to the specific diffusion model that produced it. On the AI-ArtBench test split, the authors report an F1-score of 0.869 for the 30-way classification of style plus source, and an attribution accuracy of 0.999 across the three sources (human, Latent Diffusion, Stable Diffusion). Presented in good faith, the paper's conclusion goes further: because the model reached 98% on the same 50 images on which human participants averaged around 54% in the Artistic Turing Test, the authors state that AI diffusion-generated artworks are better identified and described by CNNs than by humans. The attribution result is interpreted as evidence that diffusion models leave a detectable statistical fingerprint in their outputs, extending earlier attribution work from general images to fine art.","pith_inferences":["Because the same images were used for validation and test, the headline F1 and attribution numbers are likely optimistic; a properly separated hold-out would be expected to show lower performance.","The contrast and photo-replication experiments suggest the detector is sensitive to low-level image transforms; this implies that in real use, simple edits could fool attribution, so robustness to perturbations is a natural next test.","If attribution fingerprints are model-specific, the method may transfer to newer diffusion models (e.g., SDXL or Midjourney) only if those models share traceable artifacts; the paper's own tests on one Midjourney piece hint at this but do not prove it.","The human-model comparison should be read with the small participant pool in mind (50 respondents, mostly self-reported non-experts); a larger, more diverse sample could change the human baseline."],"forward_implications":["If the numbers hold, ArtBrain gives competition organizers, galleries, and platforms a practical tool to screen submissions for AI generation and to name the likely model (Latent vs Stable Diffusion).","The near-perfect attribution accuracy suggests image generators leave identifiable statistical fingerprints, so attribution is feasible even when metadata is stripped.","The dataset's size (185,015 images across 10 styles) provides a benchmark for future detectors and generators, filling the gap the paper identifies in available AI-art datasets.","The Turing Test result implies that automated detection is substantially more reliable than human judgment on diffusion-art images, supporting the paper's conclusion.","The FM-G-CAM heatmaps add multi-class explanations, so users can see which visual structures drove a style/source decision rather than just a label."],"supporting_citations":[{"why":"Supplies the Latent Diffusion and Stable Diffusion models used to generate the 125,015 AI art images in AI-ArtBench.","marker":"[2]"},{"why":"Provides the ConvNeXt backbone that AttentionConvNeXt adapts with frozen low/mid blocks and a trainable high block.","marker":"[16]"},{"why":"Supplies the 60,000 human artworks in AI-ArtBench and the ArtBench baseline for human-style F1 comparisons.","marker":"[28]"},{"why":"Provides FM-G-CAM, the multi-class saliency method used for the explainable heatmaps in ArtBrain.","marker":"[34]"},{"why":"Supplies the squeeze-and-excitation channel-reweighting mechanism that the attention module is built from.","marker":"[46]"},{"why":"Defines the Turing Test concept that the Artistic Turing Test adapts to AI art identification.","marker":"[50]"},{"why":"Earlier Turing-style test using brush strokes, which this study extends to full diffusion-generated artworks.","marker":"[51]"},{"why":"Introduces the idea of detectable fingerprints in fake images that the attribution results build on.","marker":"[52]"}],"fun_headline_variants":["CNN beats humans at spotting AI art: 98% vs 54%","AI art detector names its source model with 99.9% accuracy","ArtBrain traces AI art to its generator 99.9%","New model identifies AI art and its specific AI source","ArtBrain: explainable AI art attribution outperforms humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 10,000-image test split gives an unbiased estimate of how the model will do on new art, even though the same split was used to pick the best training epoch and to report accuracy.","fun_headline_variants_meta":{"raw":{"variants":["CNN beats humans at spotting AI art: 98% vs 54%","AI art detector names its source model with 99.9% accuracy","ArtBrain traces AI art to its generator 99.9%","New model identifies AI art and its specific AI source","ArtBrain: explainable AI art attribution outperforms humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000506,"raw_usage":{"total_tokens":2484,"prompt_tokens":975,"completion_tokens":1509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1420}},"tokens_in":591,"tokens_out":1509,"duration_ms":12364,"temperature":1.0,"reasoning_tokens":1420,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:12.785112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained AttentionConvNeXt on a fresh 10,000-image set of art from the same ten styles and same three sources that was never used for checkpoint selection or early stopping; if the 30-class F1 drops substantially below 0.869 or the attribution accuracy falls below 0.999, the reported generalization is not as strong as claimed. A second check is to feed images from a diffusion model outside the training set, such as SDXL or Midjourney, and see whether attribution accuracy stays near ceiling or collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 60,000 human artworks in AI-ArtBench and the ArtBench baseline for human-style F1 comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Turing Test concept that the Artistic Turing Test adapts to AI art identification."},{"cited_title":"What Is Human?","cited_arxiv_id":null,"evidence_quote":"Earlier Turing-style test using brush strokes, which this study extends to full diffusion-generated artworks."}],"review_version":1}