{"id":"06c52e4b-15c0-49c9-ac99-42d0f835ce07","arxiv_id":"2411.10004","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A text-to-image diffusion model for multimodal ophthalmic images is claimed to improve rare eye disease diagnosis via synthetic-data augmentation, but the evaluation likely leaks training data into the test sets.","lead":"EyeDiff is a text-to-image model trained on ophthalmic images across 14 modalities and 80 diseases, and the study tests whether its synthetic images can boost diagnosis of rare eye diseases when added to real training data. The paper reports improved AUROC and AUPR on several public datasets, but the evaluation contains a critical data-overlap flaw that undermines the central claim.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The downstream test sets appear to overlap with EyeDiff's training data (Table 1: OCTDL 2,064 in both; rare-disease 2,339 from Retinal Image Bank), so the reported AUROC gains may reflect train/test leakage rather than generalization.","rationale":"The reader's identification of a train/test overlap is well-supported by the paper's own Table 1 and Methods text. This is the most load-bearing concern because the central claim is about improving diagnosis through generated images, which only has scientific value if it reflects generalization beyond the generator's training distribution. The lack of any documented exclusion means the reported gains could be an artifact of data leakage. I considered other potential issues—such as the JSIEC significance inconsistency (text claims significant, Table 3 p=0.082) and the small minority-class sample sizes—but these are secondary; the overlap concern alone is sufficient to undermine the headline result. The proposed concrete check is straightforward and would decisively resolve whether the overlap exists and whether the improvements survive a clean split. Since my analysis supports the reader's REJECT verdict, no change to the verdict is needed.","tokens_in":13926,"tokens_out":3800,"duration_ms":38897,"concrete_test":"Obtain the image identifiers or perceptual hashes for the 2,064 OCTDL images and the 2,339 rare-disease images used in downstream validation and compare them against the full EyeDiff training set. If any overlap is found (or if the authors cannot rule it out), retrain EyeDiff after excluding all downstream test images from its training data, then recompute Table 3. A genuine benefit requires the reported AUROC and AUPR gains, especially for the ImageBank rare-disease task, to persist on a strictly disjoint test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EyeDiff-generated images improve diagnostic accuracy on real-world data, especially for rare diseases. This requires that the downstream evaluation be disjoint from EyeDiff's training distribution. The paper does not document any such split. In Table 1, OCTDL appears with the identical count (2,064 images) in both the EyeDiff training set and the downstream validation set, and the rare-disease downstream set (2,339 images) is described as 'collected from the Retinal Image Bank,' which is also EyeDiff's largest training source (22,941 images). No exclusion, hash comparison, temporal cut, or patient-level deduplication is described in Methods. If the same images (or near-duplicates) were used to train EyeDiff, the diffusion model can memorize them—especially for rare classes with as few as 3–7 examples—and generate synthetic images that are effectively test-set replicas. Augmenting the downstream training set with such replicas would inflate AUROC on the overlapping test set, producing the observed gains without any genuine improvement in synthesis quality or generalizability. The headline rare-disease result (AUROC 0.871→0.919, AUPR 0.439→0.530) is therefore not trustworthy as evidence for the paper's claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EyeDiff, a latent diffusion model fine-tuned on multimodal ophthalmic image–text pairs, and evaluates whether synthetic images generated by EyeDiff improve downstream retinal disease diagnosis. The authors report that augmenting original real images with EyeDiff-generated images improves AUROC and AUPR for common and rare diseases relative to a RETFound baseline and to traditional oversampling, with the headline rare-disease result an AUROC increase from 0.871 to 0.919 (Table 3, ImageBank). The manuscript also includes VQAScore alignment metrics and a small human evaluation of generated images.","tokens_in":14194,"tokens_out":4189,"duration_ms":43792,"significance":"If the diagnostic improvement claims were valid, EyeDiff would offer a practical tool for addressing class imbalance and data scarcity in ophthalmic imaging, and the multimodal text-to-image model itself could be a useful resource. The paper provides a substantial amount of empirical data and a clear experimental framework. However, the central claim is compromised by a likely train–test overlap: two of the downstream validation datasets, OCTDL and the Retinal Image Bank rare-disease set, also appear in EyeDiff's training data (Table 1). This overlap means the reported gains may reflect test-set memorization rather than genuine generalization of the synthetic augmentation. The manuscripts also contains internal inconsistencies between the narrative text and the tabulated statistical significance. Because the main contribution is the claimed diagnostic benefit, the significance of the work as presented is not established.","major_comments":[{"comment":"The downstream evaluation is not independent of EyeDiff's training data. Table 1 lists OCTDL with the identical image count (2,064) in both the EyeDiff Development set and the Downstream Validation set, and the rare-disease downstream set (2,339 images) is described in Methods as 'collected from the Retinal Image Bank between 2019 and 2023,' the same repository that contributes 22,941 training images. No exclusion, temporal cut, hash-based deduplication, or patient-level split is described. If the generative model memorized near-duplicates of the test images, especially for rare classes with as few as 3–7 examples (Table 4), the synthetic images could be test-set replicas, and the reported AUROC gains in Table 3 (e.g., OCTDL 0.982→0.996, ImageBank 0.871→0.919) would be inflated. This directly undermines the paper's central claim that EyeDiff-generated images improve diagnosis on real-world data.","section":"Table 1; Methods – Datasets"},{"comment":"The text states that for JSIEC and Retina 'these differences were statistically significant,' but Table 3 reports a p-value of 0.082 for JSIEC EyeDiff versus RETFound, which is not significant at the conventional 0.05 level. Similarly, the earlier statement that DR and glaucoma improvements were 'statistically significant' is contradicted by Table 3 for APTOS2019 (p=0.211) and MESSIDOR2 (p=0.213), where EyeDiff does not improve AUROC, and for PAPILA, where EyeDiff's overall AUROC (0.814) is numerically lower than baseline (0.820). The paper's abstract and conclusion claim that EyeDiff 'significantly improved' diagnosis accuracy, but the tabulated results support that claim only for a subset of datasets. This discrepancy is load-bearing because the central contribution is the claimed diagnostic benefit.","section":"Results – EyeDiff improves multi-class disease diagnosis; Table 3"},{"comment":"The rare-disease evaluation protocol is critically underspecified. The manuscript does not describe how the 2,339 rare-disease images were split into training and test sets for the downstream classifier, how many synthetic images were generated per class, what prompts were used for each of the 17 classes, or whether any real images in the test set were also used to condition generation. Without this information, the reported AUPR improvement from 0.439 to 0.530 (Table 3, ImageBank) cannot be attributed to the augmentation method; it could be the result of the generative model having been trained on the same images that later appeared in the test set. This is not a minor omission but a fundamental experimental control that must be reported.","section":"Methods – Datasets for downstream tasks; Rare diseases diagnosis"},{"comment":"The abstract characterizes the downstream datasets as 'ten multi-country external datasets,' but Table 1 shows that OCTDL and the Retinal Image Bank are used both for training EyeDiff and for downstream validation. These are therefore not external in the standard sense of being disjoint from the training distribution. The term 'external validation' is misleading and overstates the generalizability of the reported performance. At minimum, the authors should either exclude all training-source datasets from downstream evaluation or explicitly describe the train/test splits within those datasets and confirm that no image-level overlap exists.","section":"Abstract; Results"}],"minor_comments":[{"comment":"The footnote states 'The results displayed in the table are validated on the test set,' but no description of how the test set was constructed is provided anywhere in Methods; this should be specified for every downstream dataset, not just implied.","section":"Table 4 footnote"},{"comment":"Several minority-class improvements reported in the text are accompanied by non-significant p-values in Table 4, such as OCTDL retinal artery occlusion (p=0.135) and JSIEC Bietti crystalline dystrophy (p=0.231); the text should not describe all listed improvements as significant.","section":"Results – EyeDiff-generated images enhance disease classification in minority classes; Table 4"},{"comment":"The human evaluation of generated images was performed by only two graders, both of whom are co-authors (R.C. and X.C.), on a sample of 50 images. The Turing test result that graders mistook 62–66% of generated images for real is close to chance (50%) and should be interpreted cautiously; the author involvement and small sample should be acknowledged as limitations.","section":"Methods – Human evaluation"},{"comment":"The code link points to a general HuggingFace Diffusers example (dreambooth) rather than to the actual EyeDiff implementation; the paper should provide the specific training and generation code, the exact text prompts used, and the hyperparameters so that the experiments are reproducible.","section":"Data and code availability"},{"comment":"The keywords list contains a stray double comma ('Stable Diffusion, , Rare disease'); also, the paper consistently misspells 'RETFound' as 'RetFound' in Table 3 and elsewhere, which should be corrected.","section":"Keywords"}],"recommendation":"reject","confidential_remarks":"The core problem is the apparent train–test overlap in Table 1, which compromises the central diagnostic claim. A revision would require re-running all downstream experiments with a rigorously disjoint split, excluding OCTDL and Retinal Image Bank from EyeDiff training or using a temporally/patient-level separated subset, and then recomputing the tables. This is a substantial reworking of the manuscript's evidence base. I also note that the paper's own limitation paragraph does not acknowledge this issue, which is concerning. If the authors can provide a clean split and updated results, a future submission may be worth reconsidering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nI’ll give you the short version: EyeDiff is a real step forward in one narrow sense, but the evaluation that supports its headline claim is compromised by a data leak you should know about.\n\nWhat’s new: the authors are the first to train a latent diffusion text-to-image model on a large, multimodal ophthalmic corpus — about 42k images spanning 14 modalities and over 80 disease labels. That’s a solid engineering contribution. They then use the generator to synthesize images for minority classes and test whether augmenting a RETFound classifier with those images improves diagnosis across 10 datasets. The idea is sensible, and the comparison with naive oversampling is the right control. The VQAScore and human Turing test give at least some signal that the generated images are plausible.\n\nBut here’s the soft spot, and it’s load-bearing. Table 1 shows OCTDL with the identical count (2,064) in both the EyeDiff training set and the downstream validation set. The rare-disease downstream set (2,339 images) is described as collected from the Retinal Image Bank, which is also EyeDiff’s largest training source (22,941 images). No split, exclusion, or patient-level deduplication is described anywhere in Methods. That means the diffusion model may have memorized the very images used to measure downstream benefit — particularly for rare classes with as few as 3–7 training examples. In that setting, the reported AUROC jump on rare diseases (0.871→0.919, AUPR 0.439→0.530) is not trustworthy evidence of generalizable augmentation.\n\nThere are also internal inconsistencies in the significance claims. In Table 3, the JSIEC EyeDiff p-value is 0.082, but the text says the improvement was statistically significant. APTOS and MESSIDOR show no benefit, and the paper doesn’t frame them as null results. The hand-crafted prompt dictionary isn’t released, so the exact generation protocol isn’t reproducible.\n\nIf the authors redo the evaluation with strictly disjoint data and release the prompt templates, the concept could still be useful. As it stands, the central claim is not supported.\n\nI’d send this to peer review — a good referee will force the fix — but I wouldn’t take the current numbers at face value. Worth a reading-group discussion on training/test leakage.\n\nBest.","headline":"Worth refereeing for the idea, but the headline result is undermined by an OCTDL/Retinal Image Bank train-test overlap.","tokens_in":14716,"tokens_out":2747,"would_cite":false,"duration_ms":28911,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding synthetic images from a text-to-image diffusion model significantly improves automated diagnosis of common and rare retinal diseases, with the largest gains on a 17-class rare-disease benchmark.","keywords":["text-to-image generation","latent diffusion model","multimodal retinal imaging","rare eye disease","data augmentation","class imbalance","synthetic medical images","retinal disease diagnosis"],"falsifier":"Retrain EyeDiff with OCTDL and the Retinal Image Bank held out from training (or hold out the specific test images), rerun the augmentation protocol on OCTDL and the rare-disease set, and check whether the AUROC gains over baseline and oversampling persist; if they disappear, the claimed benefit is an artifact of training/test overlap. A complementary check is to measure nearest-neighbor distances between generated images and the downstream test images to detect memorization.","tokens_in":13749,"feed_emoji":"👁","tokens_out":5789,"duration_ms":53526,"temperature":0.7,"pith_summary":"The paper introduces EyeDiff, a text-to-image diffusion model trained on 42,048 multimodal ophthalmic images with text descriptions covering 14 imaging modalities and over 80 ocular diseases. The authors aim to show that adding EyeDiff-generated images to real training images improves automated diagnosis of both common and rare retinal diseases, and does so more effectively than traditional random oversampling. On a 17-class rare-disease task, the reported AUROC rises from 0.871 to 0.919 and AUPR from 0.439 to 0.530 when synthetic images are added. If the claim holds, it would reduce the need for large annotated collections of rare eye disease images and enable sharing of training data without exposing real patient photos.","feed_headline":"Synthetic eye images lift rare-disease diagnosis AUROC to 0.92","feed_subtitle":"Text-to-image diffusion output added to real scans beats oversampling and the RETFound baseline.","key_machinery":"EyeDiff is a latent diffusion model built on Stable Diffusion v1-5, where text prompts are encoded by a CLIP text encoder and injected into a time-conditional UNet through cross-attention, and the UNet denoises latents produced by a variational autoencoder to synthesize images. The generated images are then used as additional training examples alongside real images; the downstream classifier is a Vision Transformer initialized with RETFound weights that fuses real and synthetic embeddings through an attention-based feature fusion module before softmax classification. The load-bearing step is the text-conditioned generation of class-specific synthetic images for minority classes.","core_discovery":"On the paper's own terms, the discovery is that a text-conditioned latent diffusion model can generate multimodal ophthalmic images whose lesion features align closely enough with natural-language prompts that, when fused with real images, they substantially improve a foundation model's classification of underrepresented and rare retinal diseases across ten external benchmarks. The authors report that the synthetic-image augmentation outperforms both the RETFound baseline and random oversampling, with statistically significant AUROC gains on most tasks and the largest gains on the rarest classes. The explanation offered is that generated images provide diverse, class-specific visual features for minority classes rather than merely duplicating existing examples.","pith_inferences":["A strict test would retrain EyeDiff with OCTDL and the Retinal Image Bank excluded from training; the current numbers may overstate generalization because those datasets appear in both training and validation.","If the improvement survives that split, a natural follow-up is to measure feature diversity of generated versus oversampled minority images to isolate why synthetic data helps.","The approach likely extends to other medical imaging domains with text-rich image archives, provided the text prompts capture the relevant lesion morphology, an aspect the authors note is still simplified.","A head-to-head comparison with GAN-based augmentation under identical held-out conditions would clarify whether diffusion-model generation, rather than any synthetic data, drives the gain."],"forward_implications":["Synthetic images generated from text prompts can partially replace real annotated data for training classifiers of rare eye diseases.","The benefit over random oversampling indicates that the value of synthetic data lies not only in class rebalancing but in the diversity or fidelity of the generated examples.","Because synthetic images do not belong to real patients, the approach could lower privacy barriers to sharing ophthalmic training data across centers.","The augmentation protocol transfers across multiple imaging modalities and datasets from different countries, pointing to a general strategy for multimodal retinal disease diagnosis.","The largest AUROC improvements are on the rarest classes, such as optic nerve hypoplasia rising from 0.701 to 0.774, where data scarcity is most severe."],"supporting_citations":[{"why":"Provides the RETFound foundation model that serves as the baseline and initialization for all downstream classifiers.","marker":"20"},{"why":"Supplies the latent diffusion architecture (Stable Diffusion) on which EyeDiff is built.","marker":"21"},{"why":"Supports the premise that synthetic images from text-to-image models can make strong visual representation learners.","marker":"22"},{"why":"One of the large training datasets (EyePACS) used to train EyeDiff on diabetic retinopathy fundus images.","marker":"23"},{"why":"The OCTDL dataset used both in EyeDiff training and as a downstream validation set, central to the overlap concern.","marker":"24"},{"why":"Provides the VQAScore used to quantify alignment between generated images and text prompts.","marker":"31"}],"fun_headline_variants":["Synthetic eye images sharpen rare disease diagnosis","AI-generated eye images improve rare disease screening","Diffusion-generated eye images boost rare disease detection","Text-to-image synthesis aids rare eye disease diagnosis","Generated eye images enhance rare disease classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The downstream evaluation assumes that the datasets used to test EyeDiff-generated augmentation are disjoint from the data EyeDiff was trained on, but OCTDL appears in both the training set and the downstream validation set with the same image count, and the rare-disease validation set is drawn from the Retinal Image Bank that also trained EyeDiff.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic eye images sharpen rare disease diagnosis","AI-generated eye images improve rare disease screening","Diffusion-generated eye images boost rare disease detection","Text-to-image synthesis aids rare eye disease diagnosis","Generated eye images enhance rare disease classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2851,"prompt_tokens":875,"completion_tokens":1976,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":1909}},"tokens_in":491,"tokens_out":1976,"duration_ms":15958,"temperature":1.0,"reasoning_tokens":1909,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:04:29.673784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain EyeDiff with OCTDL and the Retinal Image Bank held out from training (or hold out the specific test images), rerun the augmentation protocol on OCTDL and the rare-disease set, and check whether the AUROC gains over baseline and oversampling persist; if they disappear, the claimed benefit is an artifact of training/test overlap. A complementary check is to measure nearest-neighbor distances between generated images and the downstream test images to detect memorization.","supporting_citations":[{"cited_title":"A foundation model for generalizable disease detection from retinal images","cited_arxiv_id":null,"evidence_quote":"Provides the RETFound foundation model that serves as the baseline and initialization for all downstream classifiers."},{"cited_title":"High-Resolution Image Synthesis with Latent Diffusion Models","cited_arxiv_id":null,"evidence_quote":"Supplies the latent diffusion architecture (Stable Diffusion) on which EyeDiff is built."},{"cited_title":"Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs","cited_arxiv_id":null,"evidence_quote":"One of the large training datasets (EyePACS) used to train EyeDiff on diabetic retinopathy fundus images."},{"cited_title":"OCTDL: Optical Coherence Tomography Dataset for Image-Based Deep Learning Methods","cited_arxiv_id":null,"evidence_quote":"The OCTDL dataset used both in EyeDiff training and as a downstream validation set, central to the overlap concern."}],"review_version":1}