{"id":"6303dc38-71ae-494d-96c2-59a0f16c88f2","arxiv_id":"2607.15416","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pooling abnormal-enriched public mammography datasets into a screening cohort's training set lowers screening AUC, and dataset-of-origin is almost perfectly predictable despite identical preprocessing.","lead":"This paper tests whether adding cancer-positive mammograms from two public research datasets improves AI screening performance on a real screening cohort from Newfoundland. It finds that every mix reduces accuracy, and that a model can identify which dataset an image came from almost perfectly, revealing stubborn dataset-specific patterns.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mixed-training AUC drop may be due to NLBSD positive dilution in balanced batches, not dataset-origin shortcuts; controlled exposure test needed.","rationale":"The reader's weakest_assumption—that the balanced-batch design allows external positives to dominate the positive pool, producing a dilution/curriculum effect—is the most load-bearing concern. The paper's empirical observation (performance drops on NLBSD when external positives are added) is likely correct, but the central interpretive claim (dataset-origin shortcuts cause the drop) rests on the implicit assumption that the model sees a representative sample of NLBSD positives during mixed training. That assumption is violated by the described sampler, making the drop explainable by a more parsimonious mechanism: reduced exposure to target-domain positives. The three-class diagnostic, while demonstrating dataset separability in the frozen encoder, does not resolve whether the linear head exploits origin features or merely underfits NLBSD positives. A controlled experiment that fixes NLBSD positive frequency per batch would distinguish these alternatives. Until such a test is run, the verdict should remain CONDITIONAL: the empirical finding stands, but the causal attribution is not yet established. I agree with the reader's diagnosis, and recommend no change to the CONDITIONAL verdict.","tokens_in":13758,"tokens_out":4606,"duration_ms":50973,"concrete_test":"Re-run Phase A with a batch sampler that holds NLBSD positive exposure constant: use exactly 20 NLBSD positives per batch (as in baseline) and supplement each batch with K external positives (e.g., K=20 or proportional to dataset sizes), so total positives per batch = 20 + K. Keep the frozen encoder, preprocessing, optimization, and NLBSD test set unchanged. Compare AUC-ROC on the NLBSD test set against the baseline 0.737 and the original mixed configurations. If AUC remains near 0.737 (within CI), the paper's conclusion is undercut; if it drops to ~0.62–0.64, the domain-shift explanation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes the AUC drop on the fixed NLBSD test set to dataset-origin shortcuts and domain shift. However, the balanced batch sampler described in §2.4 fixes only the total number of positives per batch (20), not the number of NLBSD positives. In mixed configurations (§2.5, Phase A), the positive pool combines NLBSD's 149 suspicious exams with thousands of external abnormal images (CBIS-DDSM: 1,644 cases; CMMD: 3,712 images). If the sampler draws uniformly from this combined pool, each batch contains, on average, very few NLBSD positives—possibly zero. Over 30 epochs, the linear probe may see NLBSD positives only rarely, so its decision boundary is learned almost entirely from external positives. The observed AUC drop (0.737 → 0.620–0.644) could then reflect simple lack of exposure to the target positive distribution rather than the model exploiting dataset-specific features. The monotonic degradation with added sources is also consistent with increasing dilution. The three-class origin diagnostic (§3.4) shows that the frozen encoder's features separate datasets, but it does not demonstrate that the trained head uses those features; it could simply fail to recognize NLBSD positives because they were under-sampled. Thus the paper's attribution to shortcut learning is not established without controlling for NLBSD positive exposure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper tests whether supplementing a real screening cohort (NLBSD) with biopsy-confirmed abnormal cases from CBIS-DDSM and CMMD improves a frozen EfficientNet-B5 linear-probe classifier for screening mammography. On a fixed NLBSD test set, the NLBSD-only model achieves AUC-ROC 0.737; adding external positives lowers AUC to 0.620–0.644, with Holm-corrected DeLong p < 0.05 and monotonic degradation as more sources are added. A three-class dataset-origin diagnostic separates the three datasets almost perfectly (macro-average AUC 0.9998). The authors conclude that naive pooling introduces domain shift and dataset-origin shortcuts that outweigh the benefit of additional positives. The main concern is that the mixed-training batches may contain very few NLBSD positives, so the observed decline could reflect dilution rather than domain shift, and the origin diagnostic does not directly show that the binary head uses origin features.","tokens_in":14008,"tokens_out":8379,"duration_ms":78230,"significance":"If the core finding survives a properly controlled exposure experiment, it is a useful and credible negative result: it documents that simply adding abnormal-enriched external data can harm screening-cohort performance, contrary to a common assumption. The study has clear strengths: fixed patient-level splits, identical preprocessing across datasets, a frozen mammography-pretrained encoder, bootstrap confidence intervals, Holm-corrected DeLong tests, and a common decision threshold. The origin-separability result is a clean demonstration that dataset-specific structure persists after identical preprocessing, which is relevant to dataset-pooling claims. There is no circularity: the origin diagnostic is an independent observation, not a quantity used to fit the binary model. However, the paper's central mechanism—shortcut learning—requires a controlled exposure test and direct evidence in the trained head.","major_comments":[{"comment":"The balanced batch sampler fixes only the total number of positives per batch (20), not the number drawn from NLBSD. In the mixed phases the positive pool contains roughly 149 NLBSD suspicious exams (after the 70/10/20 split, about 104 patients) plus thousands of CBIS-DDSM/CMMD abnormal images (1,644 cases; 3,712 images), while negatives are always NLBSD. If the 20 positives are sampled from this combined pool, the expected number of NLBSD positives per batch is very small, so the linear head may see few NLBSD positives over 30 epochs. The monotonic AUC drop (0.737→0.644→0.640→0.620) is then also consistent with a dilution/curriculum effect rather than the model learning dataset-origin shortcuts. Please (i) report the actual origin composition of positives per batch in each configuration, and (ii) rerun Phase A with a stratified sampler that holds NLBSD-positive exposure fixed, or subsam","section":"§2.4 and §2.5 (Phase A)"},{"comment":"The three-class origin diagnostic does not establish the mechanism named in the title. It shows that the frozen Mammo-CLIP features separate datasets almost perfectly, which is a property of the fixed encoder, not of the binary head trained in Phase A. A linear probe could also fail on NLBSD positives simply because they are under-represented in its batches (§2.4); near-perfect origin separability in feature space does not prove that the head uses those features. To support 'shortcut learning,' test the trained binary model directly: correlate binary errors with predicted origin, remove origin-discriminative directions, or compare against a matched-exposure model. Without such evidence, the diagnostic is an interesting observation but does not carry the causal load assigned to it.","section":"§3.4 and §4.1"},{"comment":"The Phase B comparison is not controlled because the test set changes across rows. The NLBSD-only row is evaluated on the NLBSD test set, whereas mixed rows are evaluated on NLBSD∪CMMD, NLBSD∪CBIS-DDSM, or the union. Lower AUC in the mixed rows (0.653, 0.651, 0.648 vs 0.737) therefore does not support the abstract's statement that 'no configuration surpassed the NLBSD-only model' under domain-matched evaluation; the NLBSD-only model would need to be scored on the same mixed test sets. This issue does not affect the primary Phase A conclusion, but the abstract overstates a supplementary result.","section":"Table 2, bottom block / §3.3"}],"minor_comments":[{"comment":"Dataset abbreviations are inconsistent: the table uses 'DDSM+' while the text uses 'CBIS-DDSM+'. Please standardize.","section":"Table 2"},{"comment":"Case-level results are described as 'reported as a supplementary analysis,' but no case-level table or appendix appears in the manuscript. Either include the numbers or remove the statement.","section":"§2.7"},{"comment":"The image-level DeLong p-values are computed on four correlated views per exam. The authors acknowledge this, but a patient-level clustered bootstrap or case-level DeLong would make the statistical claim more robust; please provide the case-level values.","section":"§3.2 and §2.7"},{"comment":"The per-image min-max normalization is applied independently to each image. This is a defensible choice, but a sentence explaining why a global or dataset-level normalization was avoided would help readers assess the 'identical preprocessing' claim.","section":"§2.2 / §2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a suitable fit for the journal's audience and the negative result is potentially valuable. The main blocker is the exposure confound: the shortcut-learning interpretation is not supported until the authors show the AUC decline persists when NLBSD-positive exposure per batch is held fixed. The mixed-domain comparison also needs rewording. If the controlled experiment is run and the decline persists, this would be a strong contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline: this paper reports a real negative result—adding abnormal-enriched public datasets to a screening cohort hurts screening AUC—but the attribution to dataset-origin shortcuts is not nailed down, because the training design doesn't control how often NLBSD positives appear in each batch.\n\nWhat's genuinely new: the specific combination of NLBSD with CBIS-DDSM and CMMD under a frozen Mammo-CLIP linear probe, and the dataset-origin diagnostic that separates the three sources almost perfectly. That diagnostic is a cheap, useful tool for detecting domain leakage. The paper is carefully written, uses patient-level splits, reports bootstrap CIs and DeLong tests with Holm correction, and is upfront about the correlated-view caveat.\n\nThe soft spot is the one the stress test flags. The balanced batch sampler fixes 20 positives per batch but draws them from the combined positive pool. NLBSD contributes 149 suspicious exams; CBIS-DDSM and CMMD contribute thousands of images. So in mixed configurations, each batch contains very few NLBSD positives—possibly zero. Over 30 epochs, the linear head mostly sees external positives and barely learns the target positive distribution. The AUC drop (0.737 → 0.62–0.64) could then be a simple dilution/exposure effect, not evidence of shortcut learning. The monotonic degradation as more sources are added is also what you'd expect from increasing dilution. The three-class origin experiment shows the frozen features are separable by source, but it doesn't show the trained head uses those features; a head that fails to recognize NLBSD positives because it rarely saw them would produce the same result. So the central claim is plausible but not established.\n\nTwo smaller issues: the Phase B mixed-domain comparisons evaluate on different test sets, so the \"none surpassed baseline\" line compares AUCs that aren't fully commensurate. And the code and exact preprocessing thresholds aren't released, which makes independent verification harder.\n\nWho gets value: anyone working on multi-dataset training for mammography or medical imaging more broadly. It's a cautionary empirical result worth knowing even if the mechanism is unresolved. The paper deserves a serious referee, but the authors should be asked to run a positive-exposure-controlled version (e.g., hold the number of NLBSD positives per batch fixed across all conditions) and to soften the causal language about shortcuts if the dilution effect explains the result. I'd send it to review, not desk reject.","headline":"A credible negative result for pooling abnormal-enriched mammography datasets with a screening cohort, but the attribution to dataset-origin shortcuts is undercut by an uncontrolled dilution effect in the training sampler.","tokens_in":14532,"tokens_out":2730,"would_cite":true,"duration_ms":27258,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding cancer-enriched external mammograms to a screening cohort's training data consistently lowered screening performance in every configuration tested.","keywords":["screening mammography","dataset mixing","domain shift","shortcut learning","dataset-origin classification","abnormal-enriched datasets","linear probe","Mammo-CLIP"],"falsifier":"Retrain the three mixed configurations while holding the number of NLBSD positives per batch fixed at the level of the NLBSD-only baseline (for example, 20 NLBSD positives per batch, with external positives added outside that quota). If the AUC on the fixed NLBSD test set recovers to roughly 0.737, dilution is the driver; if the AUC still drops to the 0.620–0.644 range, the dataset-origin shortcut explanation is supported.","tokens_in":13619,"feed_emoji":"📉","tokens_out":6220,"duration_ms":57306,"temperature":0.7,"pith_summary":"The paper tests a common assumption in medical AI: that supplementing a real screening dataset with biopsy-confirmed abnormal cases from public mammography datasets will improve detection. It finds the opposite. Using a frozen mammography-pretrained encoder with a linear probe on the Newfoundland and Labrador Breast Screening Dataset, the screening-only model reached 0.737 AUC, while every configuration that added external positives fell to 0.620–0.644, with the drop growing as more external sources were added. A three-class diagnostic that predicts each image's dataset of origin separated the datasets almost perfectly (macro AUC 0.9998) despite identical preprocessing, showing that dataset-specific signatures survive normalization. The authors conclude that naïve pooling introduces domain shift and dataset-origin shortcuts that outweigh the benefit of more positive examples, motivating domain-aware integration strategies.","feed_headline":"External cancer images cut screening AI accuracy in every test","feed_subtitle":"A screening-only model hit 0.737 AUC; adding CBIS-DDSM or CMMD positives dropped it to 0.620-0.644.","key_machinery":"The central mechanism is the dataset-origin classification diagnostic: a three-class experiment that relabels each image by its source dataset instead of its clinical label, run under the same frozen EfficientNet-B5 encoder, preprocessing, augmentation, and balanced batch sampling. It reveals whether dataset-specific structure remains after normalization, and here it separates the three datasets almost perfectly. The binary experiments use a frozen linear probe, so differences across training configurations cannot come from adapting the encoder—isolating the effect of dataset composition.","core_discovery":"Central claim: holding architecture, preprocessing, and training procedure constant, adding abnormal-enriched external positives to a real screening cohort does not improve screening performance and consistently degrades it. Evidence is a monotonic AUC drop from 0.737 (NLBSD-only) to 0.620 (both external sources added), significant after Holm correction in every mixed configuration. A three-class dataset-origin diagnostic separates NLBSD, CBIS-DDSM, and CMMD almost perfectly (macro AUC 0.9998) despite identical preprocessing, showing that dataset-specific signatures survive normalization and can act as shortcuts. The authors conclude that naïve pooling introduces domain shift that outweighs","pith_inferences":["The mixed-training batches always contain 20 positives, but external datasets supply far more positive images than NLBSD's 149 suspicious exams; the observed drop could partly reflect reduced exposure to NLBSD positives per batch rather than domain shift alone. A variant that fixes the NLBSD-positive quota per batch would separate dilution from shortcut learning.","The near-perfect origin separability suggests a cheap pre-merge diagnostic: train a small source classifier on any planned multi-dataset combination; near-perfect separability predicts negative transfer unless domain alignment is applied.","Because the encoder was frozen, fine-tuning may change the size or direction of the effect; testing the same mixing protocol with fine-tuning would reveal whether the shortcut phenomenon is robust beyond linear probes."],"forward_implications":["Training a screening-triage model on the screening cohort alone yields higher AUC than mixing in abnormal-enriched public datasets; adding more external sources makes the degradation larger.","Dataset-origin signatures survive identical DICOM handling, per-image normalization, cropping, and laterality alignment, so standard preprocessing does not neutralize domain shift.","A model can learn to identify an image's source almost perfectly, making dataset-origin shortcuts a plausible mechanism for negative transfer in medical imaging.","At a fixed threshold, mixed training drops sensitivity to 0.53–0.55, moving away from the high-sensitivity regime that screening triage requires.","Effective multi-dataset mammography training will likely need domain-aware strategies such as intensity harmonization, site-conditioned models, or domain adaptation rather than naïve pooling."],"fun_headline_variants":["External cancer images hurt mammography AI in every mix","Mammography AI AUC drops from 0.737 when external data added","Pooling abnormal-enriched datasets degrades screening AI performance","Dataset origin signatures act as shortcut, sinking mixed mammo AI","Extra positives from external sets fail to boost screening AI"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The conclusion that domain shift rather than a training-batch dilution effect causes the decline assumes that fixing the total number of positives per batch at 20 while letting external datasets contribute most of those positives is neutral; if reduced exposure to NLBSD positives alone explains the drop, the dataset-origin shortcut may not be the primary cause.","fun_headline_variants_meta":{"raw":{"variants":["External cancer images hurt mammography AI in every mix","Mammography AI AUC drops from 0.737 when external data added","Pooling abnormal-enriched datasets degrades screening AI performance","Dataset origin signatures act as shortcut, sinking mixed mammo AI","Extra positives from external sets fail to boost screening AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1366,"prompt_tokens":805,"completion_tokens":561,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":477}},"tokens_in":549,"tokens_out":561,"duration_ms":5456,"temperature":1.0,"reasoning_tokens":477,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:25:18.369056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the three mixed configurations while holding the number of NLBSD positives per batch fixed at the level of the NLBSD-only baseline (for example, 20 NLBSD positives per batch, with external positives added outside that quota). If the AUC on the fixed NLBSD test set recovers to roughly 0.737, dilution is the driver; if the AUC still drops to the 0.620–0.644 range, the dataset-origin shortcut explanation is supported.","supporting_citations":[],"review_version":1}