{"id":"f3b7cb2c-ad8a-4851-bb4a-e6d6f4e21c56","arxiv_id":"2501.16249","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A weighted average of MobileNetV2 and NASNetMobile is reported to reach 98.63% accuracy on the Kermany pediatric pneumonia dataset, though the ensemble weights were optimized on the test set.","lead":"This paper combines two lightweight pretrained neural networks, MobileNetV2 and NASNetMobile, into a weighted-average ensemble for detecting pneumonia in children's chest X-rays. The authors report 98.63% accuracy on the Kermany dataset, but the ensemble weights were tuned on the same test set used to report that accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 98.63% is obtained by grid-searching ensemble weights on the test set (Section III.J), so the headline accuracy is a test-set-selected maximum, not an unbiased estimate; this invalidates the claimed margin over individual models.","rationale":"The single most load-bearing condition for the central claim is that the ensemble weight is chosen without using test labels. The manuscript text confirms the opposite: Section III.J says the maximum accuracy is computed by evaluating the WAE model on the test dataset, and Section IV reports the 0.45/0.55 weights chosen by that procedure. The reported gain over the best base model is only 9 images out of 586, and a grid search over roughly 200 weights on those same 586 test labels can manufacture exactly this kind of margin. A clean validation split or nested cross-validation would settle the question. I find the reader's weakest assumption to be correct; in fact, the paper's own methodology section makes it more than an assumption. The undocumented baselines in Figure 7 are a further weakness, but the test-set weight selection alone is sufficient to invalidate the superiority claim as stated. Because the reader already recommended REJECT and my concern supports that outcome, the verdict is unchanged.","tokens_in":18037,"tokens_out":7110,"duration_ms":68641,"concrete_test":"Reproduce the pipeline with a proper 70/15/15 split: train MobileNetV2 and NASNetMobile on the 70% training split, grid-search the ensemble weight only on the 15% validation split, freeze the selected weight, and evaluate the fixed-weight ensemble once on the untouched 15% test split. Report test accuracy for MobileNetV2, NASNetMobile, and the fixed-weight WAE, along with the accuracy-versus-weight curve on both validation and test. If the validation-selected weight does not outperform both base models on the untouched test split by a nontrivial margin (at least 5 of 586 images), the reported 98.63% is a test-set selection artifact rather than evidence of a superior model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the 98.63% test accuracy. Section III.J states that the ensemble weights are chosen by a grid search over increments of 0.005 and that `the maximum accuracy is computed by evaluating the performance of the WAE model on test dataset.` Section IV reports the resulting 0.45/0.55 weights. Thus, the test labels are used to select the ensemble weights, and the reported accuracy is the maximum of roughly 200 accuracy estimates on the same 586 images rather than an unbiased evaluation of a fixed model. The Discussion in Section V says weights were determined `based on validation performance,` but no validation split, random seed, or code is provided to support that claim. The claimed improvement is small: WAE 98.63% versus MobileNetV2 97.10% is 9 images out of 586, and versus NASNetMobile is 14 images. A one-dimensional grid search over test accuracy can shift several boundary predictions and can be enough to produce exactly this margin. Without an independent weight-selection set, the claim that WAE outperforms its base models is not established, and the comparisons in Figure 7 inherit the same protocol problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a weighted average ensemble (WAE) of two fine-tuned lightweight CNNs, MobileNetV2 and NASNetMobile, for binary pneumonia classification on the Kermany pediatric chest X-ray dataset. After fine-tuning MobileNetV2, NASNetMobile, and EfficientNetB0, the authors form an ensemble with weights (0.45, 0.55) and report 98.63% accuracy, 98.66% weighted precision, 98.63% recall, and 98.64% F1 on a 586-image test split, along with favorable comparisons against larger CNN baselines. The paper also describes image preprocessing, augmentation, transfer learning, and a grid-search procedure for determining the ensemble weights.","tokens_in":18269,"tokens_out":4854,"duration_ms":44777,"significance":"If the evaluation protocol were sound, a lightweight ensemble with accuracy in the high 90s on the well-known Kermany benchmark would be practically useful for resource-constrained pneumonia screening, and the paper's focus on lightweight models is well motivated. The manuscript gives a clear pipeline description, covers relevant related work, and reports class-level metrics. However, the central evaluation is compromised: the ensemble weights are selected by maximizing accuracy on the same test set used for the final evaluation, so the headline numbers are selection artifacts rather than unbiased performance estimates. This affects the main claim, the comparison against the base models, and the comparison against larger architectures in Figure 7.","major_comments":[{"comment":"The weight-selection procedure uses the test set directly. Section III.J states, 'The maximum accuracy is computed by evaluating the performance of the WAE model on test dataset,' and describes a grid search over weights in increments of 0.005 with the sum constrained to one (Eqs. 2 and 3). The 98.63% accuracy reported in Section IV is therefore the maximum of roughly 200 accuracy values computed on the same 586 test images, not an independent evaluation of a fixed model. This biases the headline number upward and invalidates the comparisons in Tables IV and V and Figure 7. The Discussion's claim in Section V that weights were 'determined based on validation performance' is not supported by the methodology, which describes only a 90/10 train/test split in Section III.A. The authors should select weights on a held-out validation set and report test accuracy for the resulting fixed ensemble.","section":"III.J, IV, V"},{"comment":"The reported advantage over the individual models is small in absolute terms. The WAE misclassifies 8 of 586 test images, MobileNetV2 misclassifies 17, and NASNetMobile misclassifies 22, so the claimed improvements are 9 and 14 images, respectively. With about 200 weight combinations evaluated on the test set, a gain of this size is exactly what test-set selection can produce. No confidence intervals, repeated-split results, or alternative weight-selection runs are provided to show that the margin is stable.","section":"IV, Tables IV-V"},{"comment":"The baseline comparisons are under-specified. Section III.A describes only the overall 90/10 split, and no validation split is defined. Section IV does not describe how ResNet50, InceptionV3, DenseNet201, MobileNetV3Large, or EfficientNetB2 were trained or evaluated for Figure 7, and the text refers to EfficientNetB2 while Table III reports EfficientNetB0 at 72.18%. The 72.18% value is surprisingly low for a transfer-learned EfficientNetB0 on this dataset and is not explained. Without the training and evaluation protocol for these baselines, the claim that the WAE outperforms state-of-the-art architectures cannot be assessed.","section":"III.A, IV, Figure 7"}],"minor_comments":[{"comment":"The Shear Range is given as 0.05 but the Description column says 'intensity of 0.1'; the two values should be reconciled.","section":"Table I"},{"comment":"The dataset split is described as 90% training and 10% testing, and the numbers 5,270 and 586 are consistent with that split, but the paper does not state whether the split was stratified; given the class imbalance (4,273 pneumonia versus 1,583 normal), this should be clarified.","section":"III.A"},{"comment":"The affiliation list contains two entries numbered 3; the second should be renumbered.","section":"Affiliations"},{"comment":"The term 'weighted average' is used both for the ensemble combination in Eq. (2) and for the class-support-weighted metrics in Eq. (8); this double use is potentially confusing and should be disambiguated.","section":"III.J and Section IV"},{"comment":"No code, random seed, or data split file is provided, so the exact train/test split, augmentation, and fine-tuning procedure cannot be reproduced from the manuscript alone.","section":"Reproducibility"}],"recommendation":"reject","confidential_remarks":"The main concern is a methodological leak that is stated explicitly in Section III.J: the ensemble weights are chosen by grid search on the test set, and the reported accuracy is the maximum over that grid. This is not a matter of reinterpreting ambiguous text. I would be willing to reconsider a revised version that selects weights on a validation split, reports fixed-ensemble test accuracy, adds confidence intervals or repeated-split results, and documents the baseline training protocol. As submitted, the central quantitative claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nThe headline 98.63% accuracy in this paper is a selection artifact: the authors chose the ensemble weights by grid-searching on the test set. Section III.J says the maximum accuracy is computed by evaluating the WAE model on the test dataset, and that is exactly what they report. The Discussion later claims the weights were based on validation performance, but no validation split or code is provided. So the claimed margin over the individual models—9 images out of 586 over MobileNetV2—is not a trustworthy estimate.\n\nThat said, the paper is not without merit. It is clearly written, uses the standard Kermany dataset, and provides confusion matrices, ROC curves, and class-wise metrics. The specific weighted combination of MobileNetV2 and NASNetMobile appears to be new in the literature, though it is a routine application of a standard ensemble idea. The motivation for lightweight deployment is reasonable.\n\nThe soft spots are serious. Besides the test-set weight selection, the comparisons against ResNet50, InceptionV3, DenseNet201, etc. in Figure 7 have no experimental details; we just get a bar chart. The EfficientNetB0 result (72.18%) is far below what that architecture typically achieves on this dataset, which suggests under-training and an unfair comparison. The reported 98.63% is a maximum over roughly 200 weight pairs, so the improvement over the base models is not statistically meaningful.\n\nAs it stands, this is a low-novelty empirical study with an invalid evaluation protocol. A reader looking for a baseline might note the architecture combination, but the numbers should not be cited. If the authors redo the experiment with a proper validation split for weight selection and provide code, the paper could become a minor incremental contribution—still not exciting, but honest.\n\nI would not send this to peer review in its current form; desk reject is appropriate. If it gets resubmitted after a major revision with a valid evaluation, it might be refereeable, but I would not hold my breath.\n\nBest,","headline":"The headline 98.63% accuracy is a test-set selection artifact because the ensemble weights were grid-searched on the test set, so the claimed gains over the base models are not credible.","tokens_in":18852,"tokens_out":3728,"would_cite":false,"duration_ms":32524,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a weighted average of MobileNetV2 and NASNetMobile detects pneumonia in pediatric chest X-rays with 98.63% accuracy, beating both base models and larger architectures on the Kermany test set.","keywords":["pneumonia detection","chest X-ray classification","weighted average ensemble","MobileNetV2","NASNetMobile","transfer learning","Kermany dataset","lightweight CNN"],"falsifier":"Retrain the two models with the same split and protocol, but choose the weights on a held-out validation partition (or via cross-validation) before touching the 586-image test set. If the accuracy on the untouched test set falls materially below 98.63%, or if the optimal weights move far from 0.45/0.55, the headline is a selection artifact. A second check is to run the fixed 0.45/0.55 ensemble on independent pediatric chest X-rays from another hospital and compare accuracy and AUC.","tokens_in":17862,"feed_emoji":"🩻","tokens_out":10338,"duration_ms":84500,"temperature":0.7,"pith_summary":"This paper tries to establish that a weighted average of two lightweight pretrained convolutional networks, MobileNetV2 and NASNetMobile, classifies pediatric chest X-rays as pneumonia or normal with 98.63% accuracy on the Kermany dataset's 586-image test split. The authors fine-tune three lightweight models, set aside EfficientNetB0 after it trails at 72.18%, and combine the top two by grid-searching their weights at 0.005 increments to find the best blend. They report that the ensemble beats each base model and also heavier architectures such as ResNet50, InceptionV3, and DenseNet201, while keeping the total parameter count near 8.8 million. The contribution is an empirical recipe rather than a new theory: pick two efficient backbones with partly different inductive biases, fine-tune only the newly added classification layers, and average their predictions with optimized weights.","feed_headline":"Two small CNNs, averaged, hit 98.63% on pneumonia X-rays","feed_subtitle":"Weighted MobileNetV2 + NASNetMobile beats ResNet50 and DenseNet201 while staying light enough for mobile use.","key_machinery":"The carrying mechanism is the weighted-average ensemble formula $\\hat{y}_{\\text{ensemble}} = \\sum_i \\omega_i \\hat{y}_i$ with $\\sum_i \\omega_i = 1$, where $\\hat{y}_i$ is a base model's predicted probability and $\\omega_i$ the grid-searched weight; the authors test weight pairs in increments of 0.005 and report the optimum at $\\omega_{\\text{MobileNetV2}} = 0.45$, $\\omega_{\\text{NASNetMobile}} = 0.55$. Both base models use a transfer-learning scaffold: ImageNet-initialized weights, frozen early layers, and a task head of GlobalAveragePooling2D, Dropout, BatchNormalization, ReLU, and a sigmoid output. MobileNetV2 contributes depthwise separable convolutions with inverted residuals; NASNetMobile contributes cells found by neural architecture search. The mechanism does its work through error complementarity: the two networks disagree on a subset of images, and the weighted average moves enough of those disagreements to the correct side, cutting test error from 2.90% (MobileNetV2) or 3.75% (NASNetMobile) to 1.37%.","core_discovery":"On the Kermany pediatric chest X-ray benchmark, the weighted average ensemble (WAE) of NASNetMobile (weight 0.55) and MobileNetV2 (weight 0.45) reaches 98.63% accuracy, 98.66% weighted precision, 98.63% weighted recall, and 98.64% weighted F1 on the 586-image test set, with an AUC of 0.9977. In class-level terms it achieves 99.52% precision and 98.58% recall for pneumonia and 96.41% precision and 98.77% recall for normal, correcting 417 of 423 pneumonia images and 161 of 163 normal images. The paper presents this as an improvement over MobileNetV2 alone (97.10%), NASNetMobile alone (96.25%), and heavier architectures including ResNet50 (93.34%), InceptionV3 (94.71%), and DenseNet201 (97.78%). The authors attribute the gain to complementary feature extraction: MobileNetV2's inverted-residual depthwise separable convolutions capture fine detail cheaply, while NASNetMobile's search-derived cells pick up patterns suited to medical textures, so their weighted average cancels a share of the individual errors.","pith_inferences":["Because the optimal weights were grid-searched on the same 586-image test set that produced the headline accuracy, the published 98.63% is a selection result; fixing the weights before seeing the test labels would give the unbiased estimate.","The diversity between the two architectures may matter less than the averaging itself; comparing the reported ensemble against an ensemble of several MobileNetV2 checkpoints would isolate whether architecture diversity or ensembling drives the gain.","The 0.45/0.55 weights are fitted to one dataset and are unlikely to transfer unchanged to another hospital; a per-site calibration step would be a natural deployment requirement.","With only 586 test images, each image is roughly 0.17 percentage points of accuracy, so the difference between the reported 98.63% and a 97% result is about eight images; small test-set noise should be kept in mind when comparing models."],"forward_implications":["The full ensemble stays under roughly 8.8 million base parameters, so the paired models can run on the same resource-constrained hardware as a single small CNN while reporting higher accuracy.","On the Kermany benchmark, the weighted pair makes heavy backbones such as ResNet50, InceptionV3, and DenseNet201 unnecessary when compute is limited, since the paper reports it outperforms them.","The methodology is a reusable template for other small binary medical imaging datasets: fine-tune several lightweight pretrained CNNs, keep the two best, and grid-search their blend.","The ensemble's improvement on the NORMAL class recall (from about 95.71% for MobileNetV2 to 98.77%) means it reduces false alarms relative to the stronger base model, which matters in screening workflows."],"supporting_citations":[{"why":"Supplies the Kermany chest X-ray collection used for training, augmentation, and the 586-image test split.","marker":"[14]"},{"why":"The Cell paper documenting the pediatric pneumonia dataset and its image-based deep learning provenance.","marker":"[15]"},{"why":"Defines MobileNetV2, the inverted-residual/depthwise-separable architecture that is one of the two ensemble members.","marker":"[39]"},{"why":"Defines NASNet, whose mobile variant is the other ensemble member and the source of the searched-cell features.","marker":"[57]"},{"why":"The nearest published weighted-classifier ensemble on the same dataset; its 98.43% result is the comparison the paper's 98.63% is meant to surpass.","marker":"[33]"},{"why":"A prior ensemble of five pretrained models on Kermany (96.4% accuracy) that motivates combining diverse CNNs rather than relying on one architecture.","marker":"[32]"}],"fun_headline_variants":["Lightweight duo beats heavy nets for pneumonia detection","98.63% accuracy: tiny CNN ensemble tops ResNet50","MobileNetV2 + NASNetMobile ensemble: 98.63% on chest X-rays","Pneumonia detection hits 98.63% with lightweight ensemble","Small CNN pair outshines ResNet50 on pneumonia X-rays"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that choosing the ensemble weights by maximizing accuracy on the same 586-image test set used for the final report does not inflate the headline 98.63%; if the weights had been fixed before seeing the test labels, the reported gap over the base models could shrink.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight duo beats heavy nets for pneumonia detection","98.63% accuracy: tiny CNN ensemble tops ResNet50","MobileNetV2 + NASNetMobile ensemble: 98.63% on chest X-rays","Pneumonia detection hits 98.63% with lightweight ensemble","Small CNN pair outshines ResNet50 on pneumonia X-rays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2542,"prompt_tokens":1003,"completion_tokens":1539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1446}},"tokens_in":619,"tokens_out":1539,"duration_ms":9466,"temperature":1.0,"reasoning_tokens":1446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:36:26.295048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the two models with the same split and protocol, but choose the weights on a held-out validation partition (or via cross-validation) before touching the 586-image test set. If the accuracy on the untouched test set falls materially below 98.63%, or if the optimal weights move far from 0.45/0.55, the headline is a selection artifact. A second check is to run the fixed 0.45/0.55 ensemble on independent pediatric chest X-rays from another hospital and compare accuracy and AUC.","supporting_citations":[{"cited_title":"UltraLightSqueezeNet: A Deep Learning Architecture for Malaria Classification with up to 54x fewer trainable parameters for resource constrained devices","cited_arxiv_id":"2501.14172","evidence_quote":"Supplies the Kermany chest X-ray collection used for training, augmentation, and the 586-image test split."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Cell paper documenting the pediatric pneumonia dataset and its image-based deep learning provenance."},{"cited_title":"Light- weight hybrid convolutional network for liver tumor segmentation","cited_arxiv_id":null,"evidence_quote":"Defines MobileNetV2, the inverted-residual/depthwise-separable architecture that is one of the two ensemble members."},{"cited_title":"EfficientNet: Rethinking model scaling for convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Defines NASNet, whose mobile variant is the other ensemble member and the source of the searched-cell features."},{"cited_title":"A Novel Transfer Learning Based Approach for Pneumonia Detection in Chest X-ray Images","cited_arxiv_id":null,"evidence_quote":"The nearest published weighted-classifier ensemble on the same dataset; its 98.43% result is the comparison the paper's 98.63% is meant to surpass."},{"cited_title":"A Deep Feature Learning Model for Pneumonia Detection Applying a Combination of MRMR Feature Selection and Machine Learning Models","cited_arxiv_id":null,"evidence_quote":"A prior ensemble of five pretrained models on Kermany (96.4% accuracy) that motivates combining diverse CNNs rather than relying on one architecture."}],"review_version":1}