{"id":"a48537a7-7958-447c-ba6e-67c00d463f7f","arxiv_id":"2507.09731","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A comparison of pre-trained CNNs on noisy X-ray fracture detection finds VGG16 far more robust than ResNet50 or EfficientNet, suggesting a complexity-robustness tradeoff.","lead":"This paper compares three deep learning models for detecting bone fractures in X-rays while progressively adding simulated noise to the images. Its main finding is that the simpler VGG16 model degrades much more gracefully than ResNet50 or EfficientNet, which could influence model choice for clinics with lower-quality imaging equipment.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II contradicts the paper's own critical-failure definition and clean-accuracy tradeoff, leaving the VGG16-optimality conclusion unsupported.","rationale":"The reader's verdict was CONDITIONAL, citing the unspecified noise schedule, inconsistent failure thresholds, and lack of reproducibility. My strongest concern is narrower and more internal: the one quantitative table that supports the headline claim is contradicted by the paper's own definitions and text. This is not a generalization concern about real-world noise; it is a correctness issue in the reported evidence. I partially agree with the reader because they did mention 'critical failure threshold applied inconsistently' in their rationale, but their stated weakest assumption was about noise-model fidelity and clinical transfer. If the internal inconsistencies are confirmed, a revision that simply adds code and seeds would not suffice; the categorical failure labels and the clean-accuracy tradeoff would need to be corrected or the conclusion re-stated. If corrected data still show VGG16 with the most gradual degradation, the claim could be resurrected; as published, the central conclusion is not supported. REJECT reflects that the current evidence is self-contradictory, while remaining open to a properly documented resubmission.","tokens_in":10776,"tokens_out":7400,"duration_ms":78501,"concrete_test":"Independently recompute from the published Table II: (1) Does the listed EfficientNet critical failure point (75.69% to 52.77%) satisfy the stated '>40% accuracy drop between consecutive noise levels' rule? (2) Does ResNet50's clean accuracy (95.06%) exceed VGG16's (95.66%)? If either answer is no, the paper's own table contradicts the text. Then, if the authors supply the exact noise-injection code, parameters (distribution, sigma/rate, pixel-range scaling) and per-level metrics for every model, re-derive the critical failure points and the 0.001 Gaussian 'Functional/Failed' labels; the central robustness ranking should be re-evaluated against that corrected table.","verdict_should_be":"REJECT","load_bearing_attack":"The conclusion that VGG16 is the robust choice hinges entirely on the categorical labels in Table II (Functional/Failed, Catastrophic/Graceful), but that table is internally inconsistent. Section III.F.1 defines critical failure as a >40% accuracy drop between consecutive noise levels; Table II assigns EfficientNet a critical failure point of 0.0001 with a 75.69%→52.77% drop (only 30.3% relative), while ResNet50 is assigned 5×10−5 with a 45.1% drop, and the surrounding text says thresholds 'emerged somewhere between the noise levels of 0.001 - 0.0005.' The central tradeoff statement in Section IV — that ResNet and EfficientNet 'showed better performance on clean data' — is false for ResNet50: Table II reports ResNet50 at 95.06% clean accuracy versus VGG16 at 95.66%. Finally, Simple CNN is labeled 'No catastrophic failure' but listed as 'Failed' at 0.001 Gaussian, and ResNet50 and EfficientNet both report exactly 47.04% at that level. Because these labels are the evidence for VGG16's superiority, the reported data do not support the paper's central claim as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies how three pre-trained ImageNet classifiers (ResNet50, VGG16, and EfficientNetV2-S) and a simple CNN baseline degrade on a bone-fracture X-ray classification task when Gaussian, Poisson, or mixed Gaussian-Poisson noise is added to the test images. The authors fine-tune each model on the 'Bone Fracture Multi-Region X-ray Data' dataset, progressively add noise, and track accuracy, AUC, and F1-score. They define 'critical failure' as a greater than 40% accuracy drop between consecutive noise levels, summarize the results in Table II, and conclude that VGG16 is the optimal architecture for low-quality imaging because it degrades gracefully, while more complex models fail catastrophically. The paper also proposes a general methodology for evaluating AI robustness under simulated equipment degradation.","tokens_in":10939,"tokens_out":4882,"duration_ms":51526,"significance":"The paper addresses a practically important question: whether complex pre-trained networks remain usable for fracture detection when X-ray equipment quality is poor. Its strengths are that the experiments are direct accuracy measurements on a publicly available dataset, the Simple CNN baseline is a useful control, and the authors explicitly acknowledge dataset-specific and noise-model limitations. There is no circularity in the main results, which are direct measurements rather than outcomes forced by construction. However, the central claim that VGG16 is optimal currently rests on categorical labels in Table II that are internally inconsistent with the stated failure definition, and on a complexity-robustness tradeoff that is contradicted by the reported clean-data accuracies. If the data presentation and the noise protocol are corrected, the framework would provide modest, reproducible evidence; as written, the main conclusion is not supported.","major_comments":[{"comment":"The critical-failure labels in Table II are inconsistent with the definition given in Section III.F.1. The definition requires a >40% accuracy drop between consecutive noise levels, but EfficientNet's listed critical failure point of 0.0001 corresponds to 75.69% to 52.77%, a relative drop of about 30%, not >40%. Additionally, the text places thresholds 'somewhere between the noise levels of 0.001 - 0.0005,' whereas both listed points (0.0001 and 5e-5) are below that interval. Since the 'catastrophic vs. graceful' labels are the direct evidence for VGG16's superiority, this inconsistency must be resolved before the main claim can be assessed.","section":"III.F.1 and Table II"},{"comment":"The noise augmentation schedule is never specified. The manuscript does not report the noise amplitudes, the standard deviation values for Gaussian noise, the Poisson scaling, the number of progressive noise levels, or the ordering of levels; Section III.F.1 gives only a vague reference to levels 'between 0.001 - 0.0005.' Without this schedule, the experiment cannot be reproduced and the critical-failure comparisons in Table II cannot be verified. This is a load-bearing methodological omission because the entire conclusion depends on where and how suddenly each model's accuracy drops.","section":"III.D"},{"comment":"The stated complexity-robustness tradeoff is not supported by the reported clean-data accuracies. Section IV claims that 'the more sophisticated networks, ResNet and EfficientNet, showed better performance on clean data than the other two networks,' but Table II lists ResNet50 at 95.06% clean accuracy, below VGG16's 95.66% and only slightly above Simple CNN's 92.09%. Only EfficientNet (98.62%) actually outperforms all other models on clean data. The conclusion that complexity explains brittleness should either be revised or supported with a characterization that separates EfficientNet from ResNet50.","section":"Section IV and Table II"},{"comment":"The conclusion states that VGG16 is 'the optimal choice of architecture for building a robust bone fracture classification system that can hold up to lower-quality medical imaging,' but the study uses a single dataset, a single binary classification task, and three pre-trained architectures with no repeated runs, confidence intervals, or external validation. The authors themselves note that the results 'are specific to this dataset and may not generalize.' The conclusion should be explicitly scoped to these experimental conditions, or additional evidence (e.g., a second X-ray dataset or real noisy radiographs) is needed to justify the broader optimality claim.","section":"VI Conclusion and Study Limitations"}],"minor_comments":[{"comment":"The phrase 'Medical Imagings' in the first sentence is ungrammatical and should read 'Medical imaging.'","section":"Abstract"},{"comment":"There is a contradiction in the training protocol: Section III.B says the validation set is used to determine when to stop training via early stopping, while Section III.C says all models are trained for five epochs. The authors should clarify which procedure was actually followed.","section":"III.B and III.C"},{"comment":"In the discussion of quantum noise, 'a poison distribution' should be 'a Poisson distribution.'","section":"III.D"},{"comment":"The sentence 'Rotation is not be specifically applied' contains a grammatical error and should read 'Rotation is not specifically applied.'","section":"III.B"},{"comment":"The x-axis of Figure 2 is described as log-scale but the axis label is missing; adding explicit labels for the noise levels and units would improve interpretability.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a well-executed empirical study with limited novelty than a full archival contribution; its main value is as a robustness comparison on a public dataset. The decisive issue is not novelty but internal consistency: the data table and the stated failure definition must be reconciled, and the noise schedule must be reported before the conclusions can be evaluated. If the corrections are made and the claims are scoped more carefully, the paper could be acceptable for an applied venue; I would encourage the editor to require the authors to release the noise schedule and, if possible, the code or data artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick read for you. The paper is an empirical comparison of three pre-trained CNNs (ResNet50, VGG16, EfficientNetV2-S) plus a simple CNN baseline on bone fracture X-rays under progressive Gaussian, Poisson, and mixed noise. The novel bit is the head-to-head robustness comparison on a medical imaging dataset; that specific comparison isn't in the cited literature. The honest limitations paragraph is a plus.\n\nWhat it does well: clear research question, sensible choice of noise models (Poisson for quantum mottle, Gaussian for electronic), and a simple baseline that helps interpret the pre-trained results. The qualitative finding that VGG16 degrades more gracefully than ResNet or EfficientNet is plausible and could be useful for deployment in low-resource settings.\n\nBut the quantitative support is not there. Table II contradicts the paper's own critical-failure definition: Section III.F.1 defines critical failure as a >40% accuracy drop between consecutive noise levels, yet EfficientNet is listed as failing at 0.0001 with a 30.3% drop (75.69 to 52.77), and ResNet50's drop at 5e-5 is 45.1%. The text also says thresholds 'emerged somewhere between 0.001 - 0.0005,' which doesn't match either assigned value. The complexity-robustness tradeoff claim is misstated for ResNet50: its clean accuracy (95.06%) is below VGG16's (95.66%), so the sentence claiming the more sophisticated networks showed better performance on clean data is false for one of the two. Simple CNN is labeled 'No catastrophic failure' but 'Failed' at 0.001 Gaussian with 50.00% accuracy. The noise schedule itself is never specified; 'between 0.001 - 0.0005' is not a schedule. No error bars, no repeated runs, no code or seeds.\n\nThese are addressable in a revision. The core idea is worth testing; the VGG16 result may well replicate. But as written, the categorical labels in Table II are the load-bearing evidence and they don't hold up. I'd send this to peer review with a strong request for exact noise parameters, consistent threshold application, and ideally a second dataset or real noisy images. It's a borderline paper: the topic matters, the experiment is cheap to reproduce, and the current flaws are fixable. A competent referee could turn this into a useful empirical note.\n\nWould I cite it? Not until the numbers are fixed. Would I bring it to reading group? Maybe, as an example of how a good question can be undermined by reporting inconsistencies.","headline":"Plausible VGG16-robustness finding, but Table II's internal inconsistencies undercut the paper's central claim as written.","tokens_in":11518,"tokens_out":2942,"would_cite":false,"duration_ms":31031,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that under simulated real-world X-ray noise, VGG16 is the only pre-trained model among ResNet50, EfficientNetV2-S, and itself that degrades gracefully, and that this makes it the most reliable architecture for…","keywords":["bone fracture detection","transfer learning","pre-trained CNNs","noise robustness","medical imaging","X-ray classification","Poisson-Gaussian noise","low-resource healthcare"],"falsifier":"Take a set of radiographs acquired on an old or low-cost X-ray machine, add the same progressive Poisson-Gaussian noise to clean copies of the same anatomy, and compare which model keeps clinical accuracy on the real noisy images; if ResNet50 or EfficientNetV2-S outranks VGG16 there, the complexity-robustness claim is false.","tokens_in":10518,"feed_emoji":"🦴","tokens_out":6579,"duration_ms":70275,"temperature":0.7,"pith_summary":"The paper sets out to answer whether pre-trained deep-learning fracture classifiers can be trusted when they receive degraded X-ray images, the kind produced by older or low-cost equipment common in resource-limited clinics. It fine-tunes three pre-trained networks (ResNet50, VGG16, EfficientNetV2-S) on a public bone-fracture dataset and progressively corrupts test images with Poisson, Gaussian, and mixed noise that mimic quantum and electronic noise. The results it reports point to a complexity-robustness tradeoff: the more sophisticated networks score highest on clean images but collapse under noise, while the simpler VGG16 degrades gradually and remains clinically functional the longest. If correct, this means deployment choices for medical AI should weigh robustness under realistic input degradation as heavily as clean benchmark accuracy.","feed_headline":"VGG16 is the model that holds up when X-rays get noisy","feed_subtitle":"Degradation tests on fracture X-rays show complex networks fail fast while VGG16 keeps working.","key_machinery":"The load-bearing mechanism is a progressive noise-augmentation protocol that places each test image on a ladder of increasing Poisson noise (modeling quantum mottle), Gaussian noise (modeling electronic noise), and a Poisson-Gaussian combination, then tracks accuracy, AUC, and F1 after each step. The central comparison object is the degradation curve and the critical failure point, defined as a drop of more than 40% accuracy between consecutive noise levels, which separates catastrophic from graceful failure.","core_discovery":"The central claim is that model complexity is a liability under realistic imaging noise. On clean images ResNet50 reaches 98.62% accuracy and EfficientNetV2-S 95.06%, slightly edging out VGG16's 95.66%, but under progressive Gaussian noise ResNet and EfficientNet cross the paper's catastrophic-failure threshold early and drop to chance-level performance, while VGG16 holds 83.79% accuracy at a noise level where the others have already failed. The same ranking repeats under Poisson and mixed noise, making VGG16 the only pre-trained architecture that never fails catastrophically. The paper interprets this as overfitting to the clean, ImageNet-like feature distribution: skip connections and compound scaling help on clean data but create feature representations that break when the input distribution shifts.","pith_inferences":["Beyond the paper, the architecture ranking may not survive exposure to the structural noise, blur, and contrast variation that real low-quality machines also add; a natural next experiment is to repeat the ladder with radiographs from actual older devices.","Beyond the paper, fine-tuning ResNet50 and EfficientNetV2-S with noise augmentation could close the robustness gap, testing whether the brittleness is intrinsic to the architectures or an artifact of clean-only fine-tuning.","Beyond the paper, a practical evaluation metric suggested by this work would be the area under the degradation curve, letting buyers compare models by how much usable accuracy remains across equipment-quality levels."],"forward_implications":["In low-resource imaging settings, a VGG16-based fracture classifier is preferable to ResNet50 or EfficientNetV2-S despite its lower clean-data accuracy.","Robustness reporting should include degradation curves and critical failure points, since clean accuracy alone conceals the models' failure modes.","Because Gaussian noise proved the most disruptive, efforts to improve robustness in X-ray AI should prioritize electronic noise in the acquisition chain over quantum noise.","The observed precision-recall collapse to zero implies that as images degrade, complex models may silently stop detecting fractures, a safety-relevant warning for clinical deployment."],"supporting_citations":[{"why":"Defines the ResNet50 architecture whose brittleness under noise is the paper's key negative comparison.","marker":"[42]"},{"why":"Defines the VGG16 architecture that the paper identifies as the robust choice.","marker":"[43]"},{"why":"Defines the EfficientNetV2-S architecture, the other complex model that fails catastrophically.","marker":"[44]"},{"why":"Provides the Bone Fracture Multi-Region X-ray dataset used for fine-tuning and evaluation.","marker":"[45]"},{"why":"Supplies the claim that X-ray radiographs are most prone to Poisson and Gaussian noise, justifying the noise models.","marker":"[12]"},{"why":"Supports the mixed Poisson-Gaussian noise model used for low-dose X-ray degradation.","marker":"[52]"},{"why":"Determines the ImageNet normalization parameters used in preprocessing all three pre-trained models.","marker":"[47]"},{"why":"Provides a basis for modeling progressive noise in medical images during the augmentation protocol.","marker":"[18]"}],"fun_headline_variants":["Simpler VGG16 beats deep nets on noisy X-rays","Noisy X-rays? VGG16 stays accurate while ResNet and EfficientNet collapse","VGG16 survives noise that breaks ResNet and EfficientNet","Under X-ray noise, simple VGG16 outlasts fancier nets","Fracture detection: VGG16 robust to noise, ResNet50 and EfficientNet fail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking depends on the assumption that artificially noising clean X-ray images matches what real aging or low-cost X-ray machines do to images; if real degradation differs, VGG16's advantage might not appear in a clinic.","fun_headline_variants_meta":{"raw":{"variants":["Simpler VGG16 beats deep nets on noisy X-rays","Noisy X-rays? VGG16 stays accurate while ResNet and EfficientNet collapse","VGG16 survives noise that breaks ResNet and EfficientNet","Under X-ray noise, simple VGG16 outlasts fancier nets","Fracture detection: VGG16 robust to noise, ResNet50 and EfficientNet fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1373,"prompt_tokens":888,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":384}},"tokens_in":504,"tokens_out":485,"duration_ms":4950,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:48:15.966602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of radiographs acquired on an old or low-cost X-ray machine, add the same progressive Poisson-Gaussian noise to clean copies of the same anatomy, and compare which model keeps clinical accuracy on the real noisy images; if ResNet50 or EfficientNetV2-S outranks VGG16 there, the complexity-robustness claim is false.","supporting_citations":[{"cited_title":"Efficientnetv2: Smaller models and faster training,","cited_arxiv_id":null,"evidence_quote":"Defines the EfficientNetV2-S architecture, the other complex model that fails catastrophically."},{"cited_title":"Bone fracture multi-region x-ray data,","cited_arxiv_id":null,"evidence_quote":"Provides the Bone Fracture Multi-Region X-ray dataset used for fine-tuning and evaluation."},{"cited_title":"Noise issues prevailing in various types of medical images,","cited_arxiv_id":null,"evidence_quote":"Supplies the claim that X-ray radiographs are most prone to Poisson and Gaussian noise, justifying the noise models."},{"cited_title":"Poisson-gaussian noise analysis and estimation for low-dose x-ray images in the nsct domain,","cited_arxiv_id":null,"evidence_quote":"Supports the mixed Poisson-Gaussian noise model used for low-dose X-ray degradation."},{"cited_title":"ImageNet pre-trained models with batch normalization","cited_arxiv_id":"1612.01452","evidence_quote":"Determines the ImageNet normalization parameters used in preprocessing all three pre-trained models."},{"cited_title":"A method for modeling noise in medical images,","cited_arxiv_id":null,"evidence_quote":"Provides a basis for modeling progressive noise in medical images during the augmentation protocol."}],"review_version":1}