{"id":"d5815898-b15a-46d1-ae25-e5d3fdfe0ea9","arxiv_id":"2506.03186","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MobileNetV2 and NASNetMobile, pretrained on ImageNet and fine-tuned on a three-class subset of RFMiD, reach 90.8% and 89.5% accuracy in classifying normal, diabetic retinopathy, and macular hole fundus images.","lead":"This paper tests two lightweight pretrained networks, MobileNetV2 and NASNetMobile, on a subset of retinal fundus images and reports about 90% accuracy for three classes: normal, diabetic retinopathy, and macular hole. It is a routine benchmark for anyone asking whether small mobile-friendly models can support eye disease screening.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90.8% headline accuracy depends on a 1,451-image subset of the multi-label RFMiD whose exclusion rules and split procedure are not described; without this information the central estimate could be biased by label ambiguity or patient leakage.","rationale":"The paper's central claim is that two lightweight pretrained CNNs reach about 90% accuracy on a three-class retinal disease classification task using RFMiD. That claim requires that the data used for the test set be a valid, leakage-free sample of the intended screening population. The manuscript does not support this: RFMiD is multi-label, yet Section III-A describes only class counts and a random split, omitting the exclusion rules for multi-disease images and any patient-level grouping. The numerical mismatch between the stated 3,200-image dataset and the 1,451 images in Table I reinforces that a substantial, undescribed selection step occurred. Because the test set has only 362 samples, the reported accuracy has a wide confidence interval and the 1.3% gap between the two models is plausibly noise. These are not internal contradictions, but they make the headline result unverifiable from the manuscript alone. The reader's weakest_assumption identified exactly this issue, so I agree. A controlled reproduction with explicit subsetting and repeated splits would settle whether the concern lands. The verdict CONDITIONAL already reflects this uncertainty, so no adjustment is needed.","tokens_in":6634,"tokens_out":4282,"duration_ms":50644,"concrete_test":"Reconstruct the subset from RFMiD metadata by applying an explicit exclusion rule (e.g., retain images whose label set is exactly {Normal}, {DR}, or {MH}); then repeat training with the same hyperparameters under (a) the authors' random split if provided, (b) stratified 5-fold cross-validation, and (c) a patient-aware split when patient IDs are available. Report mean accuracy, 95% CI, and the McNemar p-value for MobileNetV2 vs NASNetMobile; if accuracy drops below ~85% or the gap becomes non-significant, the central claim as stated is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section IV-B claim 90.8% accuracy on RFMiD, but Table I totals 1,451 images (1,089 train + 362 test), not the 3,200 images described in Section III-A. The paper never states how the three-class subset was carved out of the 46-class multi-label dataset, e.g., whether images containing DR plus other retinal pathologies were discarded, assigned a single label, or kept with overlapping labels. It also reports a single 'randomly divided' split (Section III-A) with no patient-level stratification. If co-occurring pathologies remain, the Normal/DR/MH labels are ambiguous; if same-patient images appear in both train and test, the reported accuracy is inflated. With only 362 test images, the 90.8% vs 89.5% difference is within the confidence interval, so the 'outperforming' claim is fragile. The paper provides no code, seeds, or per-class details beyond Table I, so the central estimate cannot be independently reproduced from the manuscript.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper fine-tunes two ImageNet-pretrained lightweight CNNs, MobileNetV2 and NASNetMobile, on a three-class subset (Normal, Diabetic Retinopathy, Macular Hole) of the RFMiD fundus image dataset and reports classification performance. The authors report that MobileNetV2 achieves 90.8% accuracy, NASNetMobile 89.5% accuracy, and that MobileNetV2 outperforms all compared methods in a literature comparison. The stated contribution is a demonstration that lightweight transfer-learned CNNs can classify these retinal conditions with high accuracy while remaining computationally efficient.","tokens_in":6815,"tokens_out":2210,"duration_ms":27391,"significance":"If the central accuracy estimate is valid, the paper provides a useful but modest data point: two off-the-shelf lightweight architectures, with transfer learning and augmentation, reach roughly 90% accuracy on a three-class subset of a public fundus dataset. The work is empirical and externally validated on a held-out test set, so there is no circularity in the main claim. However, the paper's strengths are undercut by insufficient description of the dataset subsetting procedure, a single unstratified random split without uncertainty quantification, and a comparison table that mixes different protocols. The contribution is appropriate for a workshop-level or applied journal venue, but the current manuscript does not yet support its headline claims as rigorously as needed.","major_comments":[{"comment":"The dataset description is internally inconsistent: the text states that RFMiD consists of 3,200 fundus images, but Table I lists only 401+376+312 = 1,089 training images and 134+124+104 = 362 test images, totaling 1,451 images. The paper never explains how the three-class subset was carved out of the 46-class multi-label dataset, including whether images containing more than one disease were discarded, assigned a single label, or kept with overlapping labels, and how 'Normal' was defined. This is load-bearing because label ambiguity directly affects the measured accuracy; without an explicit exclusion rule, the reported 90.8% figure cannot be interpreted or reproduced.","section":"Section III-A and Table I"},{"comment":"The evaluation rests on a single randomized split with no patient-level stratification, no repeated runs, and no confidence intervals or significance tests. With 362 test images, the 1.3% accuracy difference between MobileNetV2 (90.8%) and NASNetMobile (89.5%) is within the margin of error; the claim that MobileNetV2 'outperformed' NASNetMobile is therefore not statistically supported. The authors should report multiple seeds, confidence intervals, or at least a McNemar test, and should clarify whether the split was performed at the patient level to avoid leakage from same-patient images appearing in both training and test sets.","section":"Section III-A and Section IV-B"},{"comment":"The comparison in Table III is not apples-to-apples. The cited methods use different class sets (e.g., multi-label classification, binary classification, different disease subsets), different dataset splits, and possibly different numbers of classes; a direct comparison of raw accuracy numbers under these conditions is not meaningful. The sentence 'MobileNetV2 outperformed all compared models' is therefore unsupported by the evidence presented. The authors should either restrict the comparison to methods evaluated on the same three-class protocol or explicitly state the protocol differences and refrain from claiming superiority.","section":"Section IV-C and Table III"},{"comment":"The paper reports only macro-level accuracy, recall, precision, and F1-score, without per-class metrics or the numeric confusion matrix. Given the class sizes in Table I (e.g., MH has only 312 training images), per-class performance is essential to judge whether the model is clinically plausible or merely exploiting class imbalance. Figure 3 shows confusion matrices and curves, but the subfigures are not labeled or described in enough detail to extract the per-class numbers; the authors should provide a numeric confusion matrix and per-class precision/recall for both models.","section":"Section IV-B and Figure 3"}],"minor_comments":[{"comment":"The paper inconsistently refers to 'MobileNet' and 'MobileNetV2'; please settle on one name and use it consistently throughout.","section":"Abstract and Section III-D"},{"comment":"The architecture description states that MobileNet starts with 'fully convolutional layers with 32 filters and includes 19 residual bottleneck layers,' which describes MobileNetV2 rather than the original MobileNet; please verify the description against the cited reference.","section":"Section III-C"},{"comment":"The heading 'Experment Setup and Training Options' contains a typo; please change to 'Experiment Setup and Training Options.'","section":"Section IV-A"},{"comment":"Some related-work entries (e.g., references [17] and [18]) appear to be self-citations that are not clearly related to retinal disease classification; please either connect them to the narrative or remove them.","section":"Section II"},{"comment":"The paper states that the data was 'randomly divided to ensure a balanced distribution' but Table I shows that the Normal class is larger than the DR and MH classes; please clarify whether any stratification or class balancing was performed.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a straightforward empirical application of two well-known architectures to a public dataset. The central finding is plausible but is currently supported by underspecified data curation and a statistically fragile comparison. The issues identified are fixable within the scope of the paper, so I do not recommend rejection, but the authors need to provide the dataset subsetting details, patient-level split information, uncertainty quantification, and a more careful comparison before the claims can be accepted. The paper may be more suitable for a venue that accepts applied incremental results; the current level of methodological detail is below what typically appears in leading journals in this field."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, this is a standard fine-tuning benchmark: MobileNetV2 and NASNetMobile pretrained on ImageNet, retrained on a three-class subset of RFMiD (Normal/DR/MH), with data augmentation. No new method, no new mechanism, no new dataset. Second, the central claim — 90.8% vs 89.5% — is plausible but not well supported, for reasons that are fixable.\n\nWhat the paper does well: the writing is clear, the choice of lightweight models is sensible for a deployment story, and the authors report accuracy, recall, precision, F1, and confusion matrices. Using RFMiD is a plus because it's public. The related work is adequate, though it already contains many similar CNN-plus-transfer-learning studies.\n\nThe big soft spot is the dataset description. The abstract says 3,200 fundus images, but Table I totals 1,451. The paper never says how the three-class subset was carved from the 46-class multi-label RFMiD. Were images with co-occurring pathologies discarded or assigned one label? Was the train/test split done per patient or per image? This matters: if same-patient images leaked across the split, accuracy is inflated; if multi-disease images remain, the labels are ambiguous. The stress-test note is right about this. Also, there is a single random split with no confidence intervals, no repeated runs, no statistical test. With 362 test images, the 1.3-point gap between MobileNetV2 and NASNetMobile is roughly half a standard error, so 'outperforming' is overclaiming. Table III compares against methods with different class sets and protocols; the claim that MobileNetV2 'outperformed all compared models' is not backed by a controlled comparison. No code or seeds are provided, so independent verification is impossible.\n\nI don't think there's anything dishonest here; it's a reporting problem. The methodology is standard and the numbers are the kind you'd expect from fine-tuning on a clean subset. But as written, the headline number is not a trustworthy benchmark.\n\nFor a reader collecting baselines for lightweight models on fundus images, this is a data point to be aware of, not a result to build on. It deserves a referee only if the authors can fix the reporting: specify subset exclusion rules, do patient-level stratified splits, run repeated trials with confidence intervals, and release code. I'd send it to review with that condition, but I wouldn't lose sleep if it were desk rejected on novelty. My recommendation: treat as a conditional accept for review, not a reject for fraud.","headline":"Routine transfer-learning benchmark whose headline accuracy is plausible but rests on an underspecified dataset subset and a single split; the 90.8% vs 89.5% gap is within noise.","tokens_in":7367,"tokens_out":2747,"would_cite":false,"duration_ms":33746,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that two lightweight pretrained CNNs, MobileNetV2 and NASNetMobile, classify normal, diabetic-retinopathy, and macular-hole fundus images with 90.8% and 89.5% accuracy, respectively, on a three-class subset of the RFMiD…","keywords":["MobileNetV2","NASNetMobile","retinal disease classification","fundus images","RFMiD dataset","transfer learning","diabetic retinopathy","macular hole"],"falsifier":"Compute the same MobileNetV2 and NASNetMobile accuracies on a version of the RFMiD three-class subset that removes images with multiple simultaneous diseases and splits by patient ID rather than by image; if accuracy falls well below 90%, the claimed screening-level performance does not survive.","tokens_in":6439,"feed_emoji":"👁️","tokens_out":4010,"duration_ms":42586,"temperature":0.7,"pith_summary":"This paper seeks to establish that off-the-shelf lightweight convolutional networks can serve as a practical screening tool for two sight-threatening retinal diseases. It trains two ImageNet-pretrained mobile architectures, MobileNetV2 and NASNetMobile, on a three-class subset of the RFMiD fundus-image dataset containing normal retinas, diabetic retinopathy, and macular hole. The claimed result is that MobileNetV2 reaches 90.8% accuracy and NASNetMobile 89.5%, with recall, precision, and F1-score all near 90%. If true, this means high-accuracy retinal triage does not require heavy, compute-intensive models, which matters for deployment in low-resource clinics.","feed_headline":"MobileNetV2 hits 90.8% accuracy on three-class retinal screening","feed_subtitle":"Lightweight pretrained CNNs classify normal, diabetic retinopathy, and macular hole fundus images with roughly nine-in-ten accuracy.","key_machinery":"The argument is carried by two pretrained lightweight CNN architectures, MobileNetV2 and NASNetMobile, adapted by removing their original fully connected heads and adding a Global Average Pooling layer, a Flatten layer, a 1024-unit Dense layer with ReLU, a 30% dropout, and a three-way Softmax. MobileNetV2's depthwise-separable bottleneck blocks and NASNetMobile's Normal and Reduction Cells with depthwise-separable convolutions are what keep parameter counts low, while transfer learning from ImageNet supplies the visual features. Preprocessing, consisting of resizing to 224x224, min-max normalization, and augmentation with rotation, shifts, shear, zoom, and horizontal flip, is what lets the small dataset support 30 epochs of fine-tuning with the Adam optimizer.","core_discovery":"The central claim is that a fine-tuned MobileNetV2 classifies a three-class subset of RFMiD with 90.8% accuracy, edging out NASNetMobile at 89.5% and beating published baselines including EfficientNetB3 (90%), a 20-layer CNN (90.34%), and Xception (90.78%) on the same dataset. The paper also claims both lightweight models keep recall, precision, and F1-scores between 89% and 91%, indicating that no single class is being sacrificed for overall accuracy. The discovery, on the paper's own terms, is that transfer-learned mobile CNNs give near-90% accuracy with far lower computational cost than larger models, providing a foundation for AI-assisted ophthalmic diagnosis.","pith_inferences":["A natural test the paper leaves implicit is splitting RFMiD by patient rather than by image; if images from the same patient leak across train and test partitions, the reported accuracies could overstate real-world screening performance.","The paper does not report class-wise sensitivity, so a follow-up analysis should check whether DR and MH recall differ sharply; if they do, the overall accuracy could hide a screening failure for one disease.","Because RFMiD is originally a 46-class multi-label dataset, a cleaned re-derivation of the three-class subset that specifies how multi-disease images were excluded would let others independently verify the 90.8% figure.","Since both models are pretrained on ImageNet, the method should plausibly transfer to other fundus cameras and disease taxonomies, but that transferability remains untested here."],"forward_implications":["If MobileNetV2's 90.8% accuracy holds, a lightweight model can serve as a screening triage for normal, diabetic-retinopathy, and macular-hole fundus images on modest hardware.","Both models' balanced recall and precision around 90% imply that neither diabetic retinopathy nor macular hole is systematically missed in this three-class setup.","The comparison results imply that resource-efficient models can match or exceed heavier published methods on the RFMiD three-class task.","The method's train/test split and augmentation protocol, as described, would form the basis for extending the approach to more disease classes and larger datasets."],"supporting_citations":[{"why":"Supplies the RFMiD fundus-image dataset and the normal, diabetic-retinopathy, and macular-hole labels used for training and testing.","marker":"[16]"},{"why":"Provides the EfficientNetB3 baseline at 90% accuracy that MobileNetV2 must beat in the comparison table.","marker":"[20]"},{"why":"Provides the semi-supervised GAN baseline at 87% accuracy on the same task, anchoring the lower end of the comparison.","marker":"[2]"},{"why":"Supplies the ViLReF vision-language baseline at 84.82% accuracy used as a comparison point in Table III.","marker":"[21]"},{"why":"Supplies the 20-layer CNN baseline at 90.34% accuracy that the paper claims MobileNetV2 outperforms.","marker":"[22]"},{"why":"Supplies the Xception baseline at 90.78% accuracy, the closest prior result that MobileNetV2 is claimed to surpass.","marker":"[23]"}],"fun_headline_variants":["Lightweight MobileNetV2 tops 90% for retinal disease triage","MobileNetV2 tops 90.8% on retinal disease screening","Three-class retinal diagnosis hits 90.8% via MobileNetV2","MobileNetV2 edges out NASNetMobile in retinal screening"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies are trustworthy only if the RFMiD ground-truth labels are correct and if the random three-class split keeps every patient's images entirely in one partition without leaking multi-disease examples.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight MobileNetV2 tops 90% for retinal disease triage","MobileNetV2 tops 90.8% on retinal disease screening","Three-class retinal diagnosis hits 90.8% via MobileNetV2","MobileNetV2 edges out NASNetMobile in retinal screening"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000601,"raw_usage":{"total_tokens":2781,"prompt_tokens":894,"completion_tokens":1887,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":1808}},"tokens_in":510,"tokens_out":1887,"duration_ms":14903,"temperature":1.0,"reasoning_tokens":1808,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:18:31.757153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the same MobileNetV2 and NASNetMobile accuracies on a version of the RFMiD three-class subset that removes images with multiple simultaneous diseases and splits by patient ID rather than by image; if accuracy falls well below 90%, the claimed screening-level performance does not survive.","supporting_citations":[{"cited_title":"Retinal fundus multi -disease image dataset (rfmid): A dataset for multi-disease detection research,","cited_arxiv_id":null,"evidence_quote":"Supplies the RFMiD fundus-image dataset and the normal, diabetic-retinopathy, and macular-hole labels used for training and testing."},{"cited_title":"Multi -label classification of fundus images with efficientnet,","cited_arxiv_id":null,"evidence_quote":"Provides the EfficientNetB3 baseline at 90% accuracy that MobileNetV2 must beat in the comparison table."},{"cited_title":"Classification of multiple retinal disorders from enhanced fundus images using semi -supervised GAN,","cited_arxiv_id":null,"evidence_quote":"Provides the semi-supervised GAN baseline at 87% accuracy on the same task, anchoring the lower end of the comparison."},{"cited_title":"A deep learning framework for the early detection of multi-retinal diseases,","cited_arxiv_id":null,"evidence_quote":"Supplies the Xception baseline at 90.78% accuracy, the closest prior result that MobileNetV2 is claimed to surpass."}],"review_version":1}