{"id":"245b6ea9-7bd9-48c2-b351-4927d42c35f5","arxiv_id":"2412.16860","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"CNNs trained only on diffusion-generated synthetic medical images achieved 72-91% accuracy on real test images across three domains, but no comparison to models trained on real data was performed.","lead":"The authors trained a diffusion model to generate synthetic brain MRI, blood cancer, and COVID CT images, then trained eight CNN classifiers on the synthetic images and tested them on real images. The best classifiers reached 72 to 91 percent accuracy, but the study never compares these models against CNNs trained on real data, so its main promise about reducing patient-data dependence is not demonstrated.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic training labels are never defined: §4.3 first denies using a conditional diffusion model, then claims label conditioning, leaving per-class generation counts in Table 1 unsupported.","rationale":"The reader's weakest-assumption pinpoints the same load-bearing issue I identify: the synthetic images' class labels are never operationally defined. The manuscript's internal contradiction in §4.3—first denying conditional diffusion, then asserting that labels were used during training and generation—directly undermines the validity of every reported accuracy in Table 2. If the diffusion model was truly unconditional, there is no mechanism to assign the per-class counts shown in Table 1, and the CNN training described in §4.5 would be conducted with labels of unknown provenance. The accuracy numbers, while plausible, cannot then support the conclusion that synthetic data can replace real labeled data, because the labels themselves are part of what must be reproduced. The additional absence of a real-data-trained baseline, noted by the reader, is a separate but reinforcing problem: even with perfectly labeled synthetic images, the paper never tests the 'comparably to original datasets' claim it advertises. I therefore agree with the reader's assessment. The report has some positive elements—the LIME analysis is a reasonable interpretability step, and the multi-dataset setup is appropriate—but those do not repair the missing label mechanism. The verdict should remain REJECT, unchanged from the reader's determination.","tokens_in":16573,"tokens_out":5056,"duration_ms":47948,"concrete_test":"Inspect the diffusion-model training and sampling code to determine whether the loss in Eq. (7) receives a class embedding or whether separate per-class diffusion models were trained. If neither mechanism exists, the labels used for CNN training in §4.5 are undefined and the central claim fails. As a behavioral check, train a classifier on the original 20% sampled real images and apply it to the generated images; compare its predicted labels with the labels used for CNN training. Agreement near chance would confirm that the synthetic training labels are unreliable, whereas high agreement would indicate a conditioning mechanism that still needs to be reconciled with §4.3's denial before the comparative claim can be evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that diffusion-generated synthetic data can train CNNs comparably to original data—presupposes that each synthetic image carries the correct class label. The manuscript never establishes this. In §4.3 it says the study 'does not utilize a conditional diffusion model (CDM)' but later asserts that the architecture 'leverages the information from labeled data (images and corresponding labels)' and that generation is 'conditioned on the provided labels.' These statements are incompatible. Table 1 reports exact per-class counts of generated images, yet no protocol explains how a class is assigned during sampling; §4.4 hedges that labels are inherited 'if the DM architecture incorporates label conditioning,' which the same paper has denied. If the model was unconditional, the CNN training signal in §4.5 is built on labels of unknown provenance, so the accuracy numbers in Table 2 cannot be interpreted as evidence for the claim. If instead one model per class or a label-conditioned model was used, that implementation is absent from the text and from any released code or data. Separately, even with perfect labels, the advertised comparison to CNNs trained on original data is untested: no model in §5.4 was ever trained on the original 20% sample or the full real data, so the headline conclusion about reducing reliance on patient-specific data remains unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether synthetic medical images generated by a diffusion model can be used to train CNN classifiers for three domains: brain tumor MRI, acute lymphoblastic leukemia microscopy, and SARS-CoV-2 CT scans. A diffusion model is trained on a stratified 20% sample of each real dataset; synthetic datasets of 1,000–1,700 images per class are generated; eight CNN architectures are trained on the synthetic data and evaluated on the untouched 80% real split. The authors report test accuracies up to 78.24%, 86.46%, and 91.38%, respectively, and present LIME explanations for the best models. The central claim, stated in the abstract, is that diffusion-generated samples can help CNNs perform comparably to models trained on original datasets, thereby reducing reliance on patient-specific data. As detailed below, this comparative claim is not tested by the experiments, and the label-assignment protocol for the synthetic images is internally inconsistent.","tokens_in":16767,"tokens_out":5380,"duration_ms":46152,"significance":"If the results were properly supported, the paper would provide a useful proof-of-concept that diffusion-model-synthesized medical images can carry enough class-discriminative information to train CNNs, with an evaluation against a real external holdout that is not circular. The use of multiple medical domains, multiple architectures, and a real-image test set are genuine strengths. However, the paper does not establish the advertised comparison with models trained on original data, and the provenance of the synthetic labels—a precondition for interpreting any of the reported accuracies—is not defined. These are not presentation issues; they are load-bearing gaps in the experimental design. The paper also does not release code, data, or trained models, and it contains unresolved placeholders, so the experiments cannot be independently checked. The significance of the contribution therefore cannot be assessed at this stage.","major_comments":[{"comment":"","section":"§4.3, §4.4, Table 1"},{"comment":"","section":"Abstract, §5.4, §5.6"},{"comment":"","section":"§4.5, §5.4"},{"comment":"","section":"§4.5, §4.6, Table 2"}],"minor_comments":[{"comment":"","section":"§4.3"},{"comment":"","section":"Eq. (10), Eq. (11)"},{"comment":"","section":"Eq. (14)"},{"comment":"","section":"§5.4"},{"comment":"","section":"§4.7"},{"comment":"","section":"§5.3, Table 1"},{"comment":"","section":"References"},{"comment":"","section":"§4.1, Data Availability"}],"recommendation":"reject","confidential_remarks":"The manuscript is an early preprint with unresolved placeholders and an untested central hypothesis. The most serious issues—the absence of any baseline trained on original data and the contradictory description of label conditioning—require new experiments, not just textual revision. The external real-data holdout is a good design choice, and the paper's multi-domain, multi-architecture scope is appropriate, but the reported numbers cannot be interpreted without a properly defined synthetic-label protocol and a real-data baseline. I would encourage the authors to address these gaps and resubmit a substantially revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the short version of arXiv:2412.16860: it reports credible-looking accuracy numbers for CNNs trained on diffusion-generated synthetic medical images, but the paper's advertised claim—that synthetic-trained CNNs compare to real-data-trained CNNs—is never tested. No CNN is ever trained on the original data. The authors list that comparison as future work in §5.6. The central conclusion is therefore unsupported.\n\nWhat the paper does well: it uses three public datasets, eight pretrained CNN architectures, a stratified 20% sample for diffusion training, per-class synthetic generation, and evaluation on an untouched 80% real holdout. That evaluation is not circular. The LIME analysis on the best models is a reasonable extra. The literature review is broad and correctly situates the work against GANs and earlier diffusion results. So the pipeline is legitimate; the execution is another matter.\n\nSoft spots, in proportion. The most serious is label provenance. §4.3 says the study \"does not utilize a conditional diffusion model (CDM),\" then two paragraphs later says the architecture \"leverages the information from labeled data\" and is \"conditioned on the provided labels.\" §4.4 hedges that labels are inherited \"if the DM architecture incorporates label conditioning.\" These cannot all be true. Table 1 reports exact per-class generation counts, but no protocol explains how a class label is assigned during sampling. If the model was unconditional, the CNN training signal is built on labels of unknown provenance. If a conditional model or per-class models were used, that implementation is missing from the text and from any released code or data. This is a load-bearing gap, not a typo.\n\nSecond, even with perfect labels, the headline comparison is untested. No model is trained on the original data, so Table 2's numbers—ResNet-50 at 78.24% on COVID CT, DenseNet-121 at 91.38% on leukemia—are absolute accuracies, not evidence of comparability. Third, the manuscript is unfinished: §4.3 literally contains a placeholder \"[insert specific architecture reference here]\". There are corrupted equations (Eq. 3, 10) and no code or data release. Fourth, the numbers have no confidence intervals, and the best architecture per dataset is selected post-hoc from eight candidates, which biases the top-line results upward.\n\nMy verdict: this is a useful cautionary example of evaluation design, not a publishable paper. It needs an answer to the conditioning contradiction, a real-data baseline, and a completed manuscript. I would not cite it. My recommendation to an editor: desk-reject or return for major revision; do not accept as-is. A serious workshop reader might find it useful as a discussion piece about what synthetic-data training claims actually require.\n\nBest.","headline":"Plausible numbers, untested headline: no CNN is ever trained on real data, and the synthetic labels' provenance is never established.","tokens_in":17382,"tokens_out":3338,"would_cite":false,"duration_ms":26968,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Trained only on diffusion-generated images, CNNs classify real medical images at 78–91% accuracy across three domains.","keywords":["diffusion models","synthetic medical images","CNN training on synthetic data","data scarcity","brain tumor MRI","leukemia","COVID-19 CT","explainable AI"],"falsifier":"Take the trained diffusion model, generate synthetic images for each declared class, and check whether images from the same class are more similar to each other than to images from other classes; if the classes are not separable, the reported CNN accuracies cannot be caused by class-specific content.","tokens_in":16336,"feed_emoji":"🩻","tokens_out":4819,"duration_ms":41137,"temperature":0.7,"pith_summary":"This paper asks whether synthetic medical images produced by a diffusion model can replace real, patient-specific images when training convolutional neural networks (CNNs) for diagnosis. It trains a diffusion model on a 20% sample of each of three public datasets—brain tumor MRI, leukemia blood smears, and COVID-19 CT scans—generates thousands of synthetic images per class, and then trains eight pretrained CNN architectures on those synthetic images alone. Evaluated on the unseen 80% of the original real images, the best models reach 78.24% accuracy on COVID CT, 86.46% on brain tumor MRI, and 91.38% on leukemia. The paper's point is that if this holds, patient-specific data may no longer be needed for CNN training in these tasks, easing privacy and scarcity constraints.","feed_headline":"Synthetic-only training hits 91% on real medical images","feed_subtitle":"Diffusion-generated MRI, CT, and blood-smear images taught CNNs to classify unseen real cases.","key_machinery":"The load-bearing object is the denoising diffusion probabilistic model (DDPM), trained by adding Gaussian noise to images in a forward Markov chain and learning to reverse that noising, so new images can be sampled from random noise. The paper trains one such model per domain on a stratified 20% sample, then uses the trained reverse process to generate large synthetic datasets with fixed per-class counts. Those synthetic datasets feed eight pretrained CNN architectures under 5-fold cross-validation, and LIME is applied to the top model in each domain to show which image regions drove predictions. The DDPM supplies the realism of the training data; the CNN supplies the classification; LIME supplies the interpretability.","core_discovery":"The central claim is that a diffusion model can generate synthetic medical images statistically similar enough to real ones that CNNs trained exclusively on them classify real, unseen images at useful accuracy. On the three test domains, no real training images were used: the synthetic sets were built by sampling 1000–1700 images per class from the trained diffusion model, and CNN training, validation, and early stopping all used synthetic data. The best per-domain results are attributed to ResNet-50 for COVID CT, VGG-19 for brain tumor MRI, and DenseNet-121 for leukemia, with precision, recall, and F1-score in the same range. LIME heatmaps are offered as evidence that the models attend to disease-relevant regions rather than to artifacts. The paper presents the outcome as demonstrating potential, not as a finished clinical solution.","pith_inferences":["My inference: the comparative claim that synthetic training matches real-data training is not tested in this paper, since no CNNs trained on the original real images are reported as a baseline.","My inference: a direct next step would be training identical CNN architectures on the original 80% real data and on synthetic data, then comparing test accuracy and LIME maps to see whether synthetic-only training truly closes the gap.","My inference: because the paper says it does not use a conditional diffusion model yet later says the architecture leverages labels, the most direct extension is to repeat the pipeline with an explicitly label-conditioned DDPM and verify that generated images track their declared classes.","My inference: if verified, the approach could extend beyond classification to segmentation and anomaly detection, but those tasks require pixel-level label fidelity that the current study does not evaluate."],"forward_implications":["If the central claim is correct, medical image classification pipelines can be built with zero real training images, leaving real data only for final evaluation.","Synthetic data could be shared openly without exposing patient-level information, since generated images are not recordings of any individual.","The 91.38% leukemia result suggests diffusion-generated data may be most useful where class differences are visually well defined, such as blood cell morphology.","Architecture choice matters on synthetic data: the best model differed by domain, so a single default CNN is not the right recommendation.","A practical route opens for rare diseases: a small sample of available images could be expanded into a large, balanced training set, mitigating class imbalance."],"supporting_citations":[{"why":"Supplies the DDPM architecture and simplified training objective that the paper adopts for synthetic medical image generation.","marker":"[35]"},{"why":"Shows latent diffusion models can generate brain imaging, supporting the use of diffusion for medical MRI generation.","marker":"[4]"},{"why":"Adapts diffusion with adversarial shortening for medical image translation, a precedent for diffusion-generated medical images.","marker":"[5]"},{"why":"Applies score-based diffusion to accelerated MRI, supporting the premise that diffusion models learn medical image distributions.","marker":"[6]"},{"why":"Uses diffusion models for medical image segmentation, cited to show the methods are established in medical imaging.","marker":"[7]"},{"why":"A comprehensive survey of diffusion models in medical imaging, cited as the landscape in which the proposed pipeline sits.","marker":"[59]"},{"why":"Frames data scarcity and bias in medical image analysis as the problem the method targets.","marker":"[8]"}],"fun_headline_variants":["Synthetic MRI, CT, and smears train CNNs to read real scans","Diffusion-made images teach CNNs to spot real tumors and viruses","No real data needed: synthetic images train CNNs for medical reads","Diffusion-generated medical images pass the real-world test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiment assumes every synthetic image carries the correct disease label, but the paper never explains how labels are assigned and says it does not use a conditional diffusion model.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic MRI, CT, and smears train CNNs to read real scans","Diffusion-made images teach CNNs to spot real tumors and viruses","No real data needed: synthetic images train CNNs for medical reads","Diffusion-generated medical images pass the real-world test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000707,"raw_usage":{"total_tokens":3164,"prompt_tokens":900,"completion_tokens":2264,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2189}},"tokens_in":516,"tokens_out":2264,"duration_ms":12873,"temperature":1.0,"reasoning_tokens":2189,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:14:25.238386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained diffusion model, generate synthetic images for each declared class, and check whether images from the same class are more similar to each other than to images from other classes; if the classes are not separable, the reported CNN accuracies cannot be caused by class-specific content.","supporting_citations":[{"cited_title":"Score-based diffusion models for accelerated mri","cited_arxiv_id":null,"evidence_quote":"Applies score-based diffusion to accelerated MRI, supporting the premise that diffusion models learn medical image distributions."},{"cited_title":"Diffusion models for implicit image segmentation ensembles","cited_arxiv_id":null,"evidence_quote":"Uses diffusion models for medical image segmentation, cited to show the methods are established in medical imaging."},{"cited_title":"Diffusion models in medical imaging: A comprehensive survey","cited_arxiv_id":null,"evidence_quote":"A comprehensive survey of diffusion models in medical imaging, cited as the landscape in which the proposed pipeline sits."}],"review_version":1}