{"id":"886f3ba5-2591-4987-9601-62c3cc3053a3","arxiv_id":"2501.06869","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A diffusion model pretrained on 3.5 million breast ultrasound images generates synthetic data that reportedly trains downstream classifiers to match real-data performance and beat radiologists in early breast cancer diagnosis.","lead":"Researchers trained a diffusion model on 3.5 million breast ultrasound images and used it to generate synthetic images for training breast cancer screening, diagnosis, and prognosis models. The paper reports that models trained on this synthetic data match or beat models trained on real data and outperform nine radiologists in detecting early-stage cancer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The synthetic-only claim hinges on the undisclosed data-cleaning step in §4.3.2; if that step filters generated images using a real-data-trained label oracle, the 'trained only on generated data' comparisons are confounded.","rationale":"The reader's weakest assumption already identifies the data-cleaning step as a place where a real-data-trained oracle could bake labels into the synthetic training set. I agree that this is the most load-bearing gap. The paper's headline contribution depends on a sharp contrast: downstream models trained solely on generated data outperform real-data-trained CLIP models and board-certified radiologists. For that contrast to be meaningful, the generated data must be the only conduit of task-specific label information. The data-cleaning step in §4.3.2 is a potential second conduit. Because the paper does not describe how 'incorrect labels' are detected, the reader cannot rule out that a real-labeled classifier selected the generated images that survived to training. If that happened, the comparison is no longer synthetic-only; it is real-data knowledge transferred through a filter, and the radiologist comparison and scaling-curve claims lose their force. I do not treat this as evidence of misconduct; it is an incomplete description in a paper whose central claim is unusually strong. The numerical inconsistencies catalogued by the reader (TNBC AUC 0.803 vs. 0.954, Baseline-CLIP BUSI AUC 0.893 vs. 0.876, 296 vs. 196 test lesions) reinforce the need for artifacts and precise reporting, but they are secondary to the cleaning-oracle question in terms of scientific validity. A single, well-specified ablation of the cleaning step, run on the same external test set, would settle whether the concern lands. Until that is provided, rejection is the appropriate verdict, and the reader's high-confidence REJECT remains unchanged.","tokens_in":19309,"tokens_out":3940,"duration_ms":41570,"concrete_test":"Require the authors to fully specify and test the data-cleaning step. Concretely: (1) Disclose the algorithm; if it uses any classifier trained on real labels, state its architecture, training data, and whether it ever saw the external test distributions. (2) Re-run the §2.4 DCIS experiment with cleaning disabled, and with an alternative unsupervised cleaning that uses only generation-internal signals (e.g., per-class likelihood thresholds or image-quality metrics) rather than a learned label oracle. (3) Report the downstream AUC for both variants on the same external test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BUS-DMs are trained on generated data without real labels for the target task, yet match or beat real-data-trained baselines and radiologists. Section 4.3.2 states that a data-cleaning algorithm removes 'generated images with incorrect labels (i.e., generated lesions that are inconsistent with the given conditions)' and that this cleaning produced a 'notable improvement' in downstream performance. The paper never specifies how 'incorrect' is determined. A common implementation is to run a classifier trained on real labeled data over the generated samples and discard those the classifier mislabels. If that is what was done, the labels used for the synthetic training set are not the diffusion model's condition labels; they are the predictions of a real-data-trained oracle. In the DCIS task (§2.4), the 34 real examples only drive LoRA adaptation, but a cleaning oracle trained on the broader real BUS-3.5M data or on the external training splits would transfer real label knowledge into the supposedly synthetic-only classifier, undermining the claim that BUS-DM learned the task from generated data alone. The same issue applies to the 1M-image diagnostic data in §2.5. If instead cleaning uses only the generative model's own likelihood or an unsupervised quality metric, the concern disappears. The paper also contains unresolved numerical contradictions (TNBC AUC 0.803 vs. 0.954 in Fig. 5a; Baseline-CLIP BUSI AUC 0.893 vs. 0.876; the test set described as 296 lesions even though 63+133=196), which prevent adjudication of the reported numbers without artifacts. But the load-bearing scientific concern is the cleaning oracle: if labels are filtered using real-data knowledge, the headline comparison is not a test of synthetic data at all.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BUSGen, a conditional diffusion model pretrained on 3.5 million breast ultrasound images, adapted via LoRA to generate task-specific data, and used to train downstream models (BUS-DMs) for screening, diagnosis, and prognosis tasks. The main claims are that BUS-DMs trained on generated data match or outperform real-data-trained CLIP-based baselines, that a BUS-DM trained on 1 million generated images reaches AUC 0.929 on BUSI, comparable to a model trained on 288,767 real images, that a generated-data-only DCIS classifier outperforms nine board-certified radiologists, and that CPSampling prevents patient-identifying replication. The evaluation includes external test sets from PUMCH and CICAMS, the public BUSI dataset, and a radiologist reader study.","tokens_in":19599,"tokens_out":8469,"duration_ms":84834,"significance":"If the claims held, this would be a substantial advance: it would suggest that synthetic ultrasound data alone can replace real collected data for training diagnostic classifiers, with external validation and a comparison against human readers. The strengths include the scale of pretraining data, the use of external test sets not used in generation, the explicit radiologist reader study, and the public demo. However, the central quantitative claims are currently undermined by unresolved contradictions between the main text and figure captions, and the key 'synthetic-only' claim depends on a data-cleaning step that is not specified. These issues prevent the results from being assessed in their present form.","major_comments":[{"comment":"The paper's core comparisons in Sections 2.4 and 2.5 depend on the assertion that BUS-DMs are trained only on generated data, but the data-cleaning step is described only as removing 'generated images with incorrect labels (i.e., generated lesions that are inconsistent with the given conditions)'. The manuscript never states how 'incorrect' is determined. If this filter is implemented with a classifier trained on real labeled data, then the labels of the synthetic training set are partly supplied by a real-data-trained oracle, and the comparisons against real-data-trained baselines are confounded. Please specify the cleaning algorithm completely, state whether any real-labeled data or a pretrained oracle is used in cleaning, and provide an ablation with and without cleaning. Without this information, the 'trained only on generated data' claim cannot be evaluated.","section":"4.3.2"},{"comment":"The reported results contain unresolved numerical contradictions that affect the headline claims. In Section 2.6 the BUS-DM TNBC AUC is reported as 0.803 (95% CI 0.740-0.880), while the Fig. 5a caption reports 0.954 (95% CI 0.932-0.983) for the same comparison. In Section 2.5 the Baseline-CLIP AUC on the BUSI test set is 0.893 (95% CI 0.865-0.918), while the Fig. 4e caption reports 0.876 (95% CI 0.849-0.903). In Section 4.4.2 the early-diagnosis test set is described as containing 296 lesions (63 benign and 133 DCIS), but 63 + 133 = 196. These discrepancies concern the main quantitative evidence and must be resolved before the results can be assessed.","section":"2.6/Fig. 5a; 2.5/Fig. 4e; 4.4.2"},{"comment":"The privacy claim is supported by a maximum cosine similarity of 0.896 between generated samples and their nearest neighbors in an unspecified feature space, together with two example image pairs. This does not establish that exact replication cannot occur, nor does it support the abstract's stronger statement about 'fully de-identified data sharing'. The similarity threshold depends on the choice of feature extractor and normalization, and no membership-inference or retrieval audit is reported. Please report the exact similarity measure, a calibrated threshold, and a membership-inference/nearest-neighbor audit, or temper the privacy claim accordingly.","section":"2.2/Fig. 2b"}],"minor_comments":[{"comment":"The Visual Turing Test should report each reader's exact count of generated images judged as real, along with confidence intervals; 'approximately 50%-75%' is imprecise.","section":"2.2/Fig. 2a"},{"comment":"The pretraining data are derived from 5,907 scanning videos; please clarify how frames were sampled or downsampled so that temporally correlated frames are not inflating the effective number of independent training examples.","section":"4.2.2"},{"comment":"Pathology labels are available for only 1,387 of 3,749 lesions, yet pathology is one of the conditioning variables in Equation (1); please specify how missing pathology labels are encoded during conditional training.","section":"4.2.3"},{"comment":"For lesion detection, the manuscript reports P-values for Average Precision, but Section 4.5 only describes DeLong's test for AUC; please state the statistical test used for AP comparisons.","section":"4.4.1"}],"recommendation":"reject","confidential_remarks":"The external evaluation design is a genuine strength, and I would not want this work to be dismissed on ideological grounds. However, the main quantitative claims currently conflict in places that cannot be dismissed as typographical, and Section 4.3.2 leaves open a direct label-leakage path for the 'synthetic-only' comparisons. If a resubmission specifies the cleaning algorithm, corrects the TNBC/BUSI/test-set numbers, and adds a no-cleaning ablation, the paper could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core empirical claim here is worth taking seriously: a diffusion model pretrained on 3.5M breast ultrasound images can generate training data that, on external test sets, matches or beats real collected data. The scaling study comparing synthetic vs. real data curves on the BUSI test set is genuinely new for this domain, and the nine-radiologist reader study for DCIS is a real contribution. The external test design (PUMCH, CICAMS, BUSI) is a strength, not a weakness.\n\nThe soft spots are serious. First, the unresolved numerical contradictions are load-bearing: the TNBC AUC is 0.803 in Section 2.6 but 0.954 in the Fig. 5a caption; the Baseline-CLIP BUSI AUC is 0.893 in Section 2.5 but 0.876 in the Fig. 4e caption; and the early-diagnosis test set is described as 296 lesions even though 63+133=196. These make it impossible to know which results are correct. Second, the data-cleaning step in Section 4.3.2 is described only as removing \"generated images with incorrect labels\" with a \"notable improvement\" in downstream performance. If that cleaning uses a classifier trained on real labels to filter generated samples, then the supposedly \"trained only on generated data\" comparisons are confounded; the paper must disclose the cleaning mechanism. Third, no code, weights, or datasets are released, and the privacy claim rests on a cosine-similarity check that is not a recognized privacy guarantee.\n\nThe underlying approach is sound and potentially significant. The authors have done a large amount of work, and the external evaluation design suggests they know what they are doing. But the current manuscript is not publishable as-is; the contradictions need reconciliation, the cleaning step needs full transparency, and artifacts should be released. If those are fixed, this could be a strong paper.\n\nMy recommendation: send it to peer review, but with a referee who will press hard on the cleaning oracle and the numeric inconsistencies. A serious editor should not desk-reject it, because the scientific question is important and the evidence is substantial—but neither should anyone trust the specific AUCs in this draft.","headline":"Ambitious synthetic-data scaling study for breast ultrasound, but the current draft has unresolved numerical contradictions and an undisclosed data-cleaning step that could confound the main claim; still worth a serious referee.","tokens_in":20339,"tokens_out":1876,"would_cite":false,"duration_ms":19810,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BUSGen, a generative model pretrained on 3.5 million breast ultrasound images, can produce task-specific synthetic images that train downstream models to match or exceed real-data-trained foundation models and to outperform radiologists…","keywords":["breast ultrasound","generative foundation model","diffusion model","synthetic data","few-shot adaptation","breast cancer early diagnosis","data scaling","privacy-preserving data sharing"],"falsifier":"A decisive check is to retrain the DCIS and diagnosis models on BUSGen-generated data with the data-cleaning filter disabled and evaluate on the same external test sets; if the AUCs drop to near chance or to baseline levels, the cleaning step, not the generative model, was carrying the label signal. A second check is to run the same generated-data pipeline on a completely new external ultrasound dataset from a different scanner generation and region.","tokens_in":19021,"feed_emoji":"🩺","tokens_out":7032,"duration_ms":58673,"temperature":0.7,"pith_summary":"This paper tries to establish that scarcity of breast ultrasound data need not block deep-learning clinical tools: a diffusion model pretrained on 3.5 million images can generate unlimited task-specific images, and models trained only on those generated images can match or beat models trained on real collected data. If true, this removes a major obstacle—privacy-restricted data collection—from breast ultrasound AI development. The strongest test is early diagnosis of ductal carcinoma in situ, where the generated-data model outperformed nine board-certified radiologists on average sensitivity. The paper also claims that generated data scales like real data, improves external generalization, and enables fully de-identified sharing.","feed_headline":"Synthetic ultrasound images train AI that beats radiologists","feed_subtitle":"For early breast cancer, synthetic-only training lifts sensitivity by 16.5%.","key_machinery":"The central object is BUSGen, a pixel-space conditional Denoising Diffusion Probabilistic Model with a lightweight U-Net of roughly 50 million parameters, pretrained on 3,518,495 breast ultrasound images conditioned on pathology labels, lesion bounding boxes, and device type. In adaptation, the pretrained weights are frozen and low-rank adapters (LoRA) are fine-tuned with strong augmentation plus a device-type augmentation that translates images across 18 scanner types; classifier-free guidance controls the generation, DPMSolver++ accelerates sampling, and CPSampling prevents the generator from replicating training images. A data-cleaning step then filters low-quality generated samples, and the surviving synthetic images are used to train the downstream models.","core_discovery":"The paper claims that a generative foundation model for breast ultrasound can produce realistic, task-specific synthetic data that is at least as effective as collected real-world data for training diagnostic models. On six tasks spanning screening, diagnosis, and prognosis, downstream models trained on BUSGen-generated data (BUS-DMs) outperformed Baseline-CLIP, a contrastive vision-language model pretrained on 1.65 million medical image-caption pairs and fine-tuned on real breast ultrasound data. In breast cancer early diagnosis (DCIS vs. benign), the synthetic-trained model exceeded all nine board-certified radiologists, achieving 16.5% higher average sensitivity at matched specificity. Scaling generated data to 1 million images produced an AUC of 0.929 on the BUSI test set, comparable to NYU-AI trained on 288,767 real images (AUC 0.927).","pith_inferences":["Editorial inference: If synthetic data trains models as effectively as real data, prospective evaluation of ultrasound AI could be moved largely into simulation, reserving real clinical images for final confirmation.","Editorial inference: The same few-shot LoRA adaptation recipe is a candidate for other imaging modalities with scarce labeled pathology, such as thyroid, liver, or musculoskeletal ultrasound.","Editorial inference: The reported gains depend on the data-cleaning filter; a cleaner test of the core claim would train the diagnostic model on generated data with label-based cleaning disabled."],"forward_implications":["Models trained solely on BUSGen-generated data outperformed a CLIP-based foundation model fine-tuned on real data in all six evaluated tasks.","At 1 million generated images, the diagnostic model matched a real-data model trained on 288,767 images on the BUSI test set, and scaling curves suggest more generated data would continue to improve performance.","In DCIS early diagnosis, the synthetic-trained model exceeded nine board-certified radiologists by 16.5% average sensitivity at the same specificity, and eight of nine readers improved when shown its predictions.","Generated data reduced shortcut learning from acquisition biases, so BUS-DMs showed smaller performance drops in external evaluations than real-data-trained baselines.","CPSampling-based generation avoids exact replication of training images, supporting fully de-identified data sharing."],"supporting_citations":[{"why":"Supplies the denoising diffusion probabilistic model that BUSGen's pretraining and generation build on.","marker":"[48]"},{"why":"Provides classifier-free guidance, used to condition generation on pathology, lesion boxes, and device type.","marker":"[53]"},{"why":"Defines the CLIP architecture and pretraining paradigm used for the Baseline-CLIP comparison model.","marker":"[49]"},{"why":"Provides low-rank adaptation (LoRA), which lets the frozen pretrained model be adapted to downstream tasks with few examples.","marker":"[54]"},{"why":"Supplies the public BUSI test set used for external diagnostic evaluation and scaling comparison.","marker":"[22]"},{"why":"Provides CPSampling, the sampling strategy used to prevent the generator from replicating training images.","marker":"[57]"},{"why":"Establishes shortcut learning and data-acquisition bias quantification, which the paper uses to show generated data improves generalization.","marker":"[29]"},{"why":"Bases the scaling-effect analysis of synthetic training data that the paper extends to medical imaging.","marker":"[50]"}],"fun_headline_variants":["Synthetic ultrasounds train AI that beats breast cancer experts","First generative model for breast ultrasound outperforms radiologists","AI trained on fake ultrasound images tops real-world data","Generative model creates realistic breast ultrasounds for better diagnostics","Breast ultrasound AI: synthetic data matches real imaging in screening"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that images generated from a few real examples preserve the class-relevant morphology of the target population, so a model trained only on generated data transfers to external patients; if that transfer fails, the reported AUCs and radiologist comparisons collapse.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic ultrasounds train AI that beats breast cancer experts","First generative model for breast ultrasound outperforms radiologists","AI trained on fake ultrasound images tops real-world data","Generative model creates realistic breast ultrasounds for better diagnostics","Breast ultrasound AI: synthetic data matches real imaging in screening"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1381,"prompt_tokens":940,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":361}},"tokens_in":556,"tokens_out":441,"duration_ms":4502,"temperature":1.0,"reasoning_tokens":361,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:51:03.861697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to retrain the DCIS and diagnosis models on BUSGen-generated data with the data-cleaning filter disabled and evaluate on the same external test sets; if the AUCs drop to near chance or to baseline levels, the cleaning step, not the generative model, was carrying the label signal. A second check is to run the same generated-data pipeline on a completely new external ultrasound dataset from a different scanner generation and region.","supporting_citations":[{"cited_title":"CPSample: Classifier Protected Sampling for Guarding Training Data During Diffusion","cited_arxiv_id":"2409.07025","evidence_quote":"Provides CPSampling, the sampling strategy used to prevent the generator from replicating training images."},{"cited_title":"NPJ Digital Medicine 7(1), 124 (2024)","cited_arxiv_id":null,"evidence_quote":"Establishes shortcut learning and data-acquisition bias quantification, which the paper uses to show generated data improves generalization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Bases the scaling-effect analysis of synthetic training data that the paper extends to medical imaging."}],"review_version":1}