{"id":"cbb655c7-ee03-4b6e-9d5a-b6cb5086f1b1","arxiv_id":"2505.16033","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"VGG19 and Xception classify 21 leaf-disease classes across six Bangladeshi crops with roughly 99% accuracy on a public dataset, with GradCAM-family heatmaps used for explanation.","lead":"This paper tests six standard image-classification networks, plus a small CNN, on a 12,786-image dataset of diseased and healthy leaves from six Bangladeshi crops. VGG19 and Xception reach about 98.9% and 98.7% accuracy, and heatmap methods are used to show where the models look.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline accuracy claim may overstate generalization because the reported numbers come from the best of five random 80/10/10 splits without stratification, deduplication, or error bars; a single re-run with a stratified and deduplicated split would test whether VGG19's ~98.9% accuracy is a…","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the unverified i.i.d./leak-free random split in Section 4.1. I agree with the CONDITIONAL verdict. The paper is a standard transfer-learning benchmark application with a public dataset, and the reported accuracies are plausible for a 21-class leaf-disease task with strong color/texture cues and black backgrounds. The main risk is not the method but the evaluation protocol. Two concrete problems are visible in the manuscript: (1) the split is described only as an 80:10:10 random partition, with no deduplication or grouping by plant/field, despite the dataset being field photographs potentially containing multiple images of the same leaf; this makes the claim of 98.9% test accuracy a claim about the specific random split unless the authors establish that no near-duplicate images cross the boundary. (2) The paper explicitly states that each model runs five times and 'the optimal results recorded'; reporting best-of-five without variance is a selection bias that inflates the headline numbers. The internal ResNet50V2 inconsistency (Table 2 says 0.97417, text says 98.41%) is a symptom of careless reporting but is not the central argument's weak point. The XAI portion is qualitative and does not bear on the accuracy claim. Because the dataset is public and the models are standard, a targeted re-run with a deduplicated, stratified split and mean-variance reporting would settle whether the central claim generalizes; hence CONDITIONAL rather than REJECT. I also note the reviewers' instruction to treat manuscript text as evidence: Section 5.1's 'optimal results recorded' is an explicit admission that only the best run is reported, which I have incorporated directly into the concern rather than treating it as an artifact.","tokens_in":8371,"tokens_out":1983,"duration_ms":14723,"concrete_test":"Re-run VGG19 and Xception on the same Mendeley dataset after (1) computing perceptual hashes or embedding-based near-duplicate detection to identify images of the same leaf/plant, (2) partitioning so all near-duplicate images fall in the same fold, and (3) using a stratified 80/10/10 split by class. If test accuracy stays within roughly 1 point of 98.9%, the leak concern is mitigated; if it drops materially, the reported accuracy was inflated by leakage. Additionally, report the mean and standard deviation of accuracy across the five runs; if the best-run accuracy exceeds the mean by more than ~0.5 point, the headline number should be replaced by the mean.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that fine-tuned VGG19 reaches 98.90% test accuracy on a 21-class Bangladeshi leaf dataset (Section 5.2, Table 2). This claim depends on the evaluation protocol described in Section 4.1: an 80:10:10 random split of 12,786 images. The load-bearing weakness is that this protocol does not rule out image-level leakage, because the dataset (Mendeley, Hasan et al.) was collected by photographing leaves in fields, and the paper gives no evidence that images of the same leaf or the same plant were kept entirely within one split. If near-duplicate images straddle train and test, the reported 98.9% partly measures memorization of near-duplicate backgrounds, not generalization to new leaves. A second, related issue is that Section 5.1 says each model runs five times with 'the optimal results recorded'; reporting only the best run, without mean/standard deviation, means the headline number is the maximum of five correlated draws, so the expected published accuracy is inflated relative to a typical run even under a leak-free split. The ResNet50V2 inconsistency (0.97417 in Table 2 vs. 98.41% in the text) reinforces that the accuracy reporting is not fully checked. The central empirical claim would be much more convincing if a single split held out entire plants, or at least if image-level near-duplicates were removed before splitting, and if mean and standard deviation across the five runs were reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a deep learning pipeline for classifying 21 leaf disease classes across six Bangladeshi plant species. It uses a publicly available Mendeley dataset of 12,786 images, applies HSV-based background removal and resizing to 128×128, and evaluates a custom CNN plus six transfer learning models (VGG16, VGG19, MobileNetV2, InceptionV3, ResNet50V2, Xception) on an 80:10:10 train/validation/test split. VGG19 is reported as the best model with 98.90% accuracy, followed by Xception at 98.66%. The paper also applies five XAI methods (GradCAM, GradCAM++, LayerCAM, ScoreCAM, FasterScoreCAM) and qualitatively compares the resulting heatmaps, claiming that the combination of high accuracy and explainability provides a transparent tool for farmers.","tokens_in":8635,"tokens_out":4434,"duration_ms":34504,"significance":"If the reported results hold, the paper offers a useful empirical benchmark showing that standard pretrained networks can separate these 21 leaf disease classes nearly perfectly, and it provides a qualitative comparison of five XAI techniques across multiple models. The use of a real field-collected public dataset and the breadth of models/XAI methods are strengths. However, the significance is tempered by the evaluation protocol: the reported numbers are the best of five runs without error bars, there is no evidence that near-duplicate images do not straddle the train/test split, and there is a material inconsistency in the ResNet50V2 accuracy between the table and the text. The XAI analysis is entirely qualitative, with no quantitative evaluation or user study, so the claims about farmer-facing transparency are not directly supported.","major_comments":[{"comment":"The text states that 'Each model runs five times to ensure robustness, with the optimal results recorded for analysis' (§5.1). This means the reported accuracies, including the headline 98.90% for VGG19, are the maximum over five random-split runs rather than the expected performance. The paper provides no mean, standard deviation, or confidence intervals across the five runs, and the abstract and conclusion present this maximum as 'the' accuracy. This selection protocol inflates the reported numbers relative to a typical run and prevents the reader from assessing stability. Please report the mean and standard deviation (or all five runs) for each model, and rephrase the claims accordingly.","section":"§5.1, Table 2"},{"comment":"The dataset is described as collected from different agricultural fields (§3), and the split in §4.1 is a simple random 80:10:10 partition without stratification by plant or disease and without removal of near-duplicate images. Because the same leaf or plant may appear in multiple photographs, images of the same object could straddle the training and test sets. In that case, the reported 98.90% accuracy would partly reflect memorization of near-duplicate images rather than generalization to new leaves. The authors should either perform a split that holds out entire plants or fields, or demonstrate (e.g., by similarity analysis) that no near-duplicates cross the split.","section":"§4.1, §3 Dataset"},{"comment":"There is a material inconsistency in the reported ResNet50V2 result: Table 2 lists an accuracy of 0.97417 (97.42%), while the text states that 'Resnet50V2 also perform well, achieving an accuracy of 98.41%'. The difference of about one percentage point is too large to be a rounding artifact. The VGG16 text value (98.35%) is consistent with Table 2 (0.98356), but the ResNet50V2 discrepancy indicates that the numerical results have not been carefully cross-checked. Please correct the value and verify all other numbers in the table against the text.","section":"§5.2, Table 2"}],"minor_comments":[{"comment":"The preprocessing description says 'The final phase consists of eliminating the leaf (foreground) utilizing the mask,' but the procedure and Figure 3 indicate the background is removed, not the leaf. The wording should be corrected to 'eliminating the background.'","section":"§4.1, Figure 3"},{"comment":"The description of the CNN architecture gives filter counts but omits kernel sizes, strides, padding, and dropout rates. Please add these details for reproducibility.","section":"§4.2"},{"comment":"The phrase 'demonstrate quick convergence, with all models attaining higher accuracy at an early stage' is imprecise; consider 'high accuracy' instead of 'higher accuracy.'","section":"§5.2"},{"comment":"'Summary of the reviewd work' should be 'reviewed work'; the same typo appears in the body text.","section":"Table 1 caption"},{"comment":"'GradCAM is a pivotal XAI technique' contains a typo; 'an pivotal' should be 'a pivotal.'","section":"§6"},{"comment":"The paper does not state whether the code, trained models, or exact split indices are publicly available; including them would strengthen reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a conventional empirical benchmark. The most important issue is the evaluation protocol: best-of-five run selection without error bars is a central weakness, and the potential for image-level leakage is plausible given the field-collected dataset. Both are fixable within the manuscript's scope. The ResNet50V2 inconsistency suggests the numerical results need a careful audit. I do not see a circularity problem or an authorship conflict that affects the scientific claims; the self-citation in the references is background and not load-bearing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fair read: this is a competent but unremarkable application paper. The genuinely new thing is a set of benchmark numbers — VGG19 at 98.90%, Xception at 98.66% — on a public 21-class Bangladeshi leaf dataset, plus a qualitative comparison of five XAI methods. That is a useful reference point for anyone working on crop-disease screening in South Asia, and it should be easy to reproduce because the dataset is public and the models are standard.\n\nThe paper does a few things well. It uses a public dataset and describes the preprocessing (HSV background removal, 128x128 resizing) clearly enough to follow. The XAI comparison across multiple models is a reasonable way to check whether the models attend to disease regions, and the claim that GradCAM-family methods produce consistent activations is believable. The literature review is adequate, and the self-citation is not an issue here — it is background on VQA, not on the classification claim.\n\nThe soft spots are real but not disqualifying. First, Section 5.1 says each model runs five times and 'the optimal results recorded.' That means the headline 98.90% is the maximum of five correlated draws, not the expected accuracy. No standard deviations or confidence intervals are reported, so we cannot tell how stable that number is. Second, the 80/10/10 split is not stratified by plant/disease and the paper gives no evidence that near-duplicate images of the same leaf are kept within one split. If the same leaf is photographed multiple times and straddles train/test, the accuracy is partly memorization. The authors should at least run a plant-level or deduplicated split. Third, there is an internal inconsistency: Table 2 lists ResNet50V2 as 0.97417 (97.42%) while the text says 98.41%. It is a minor error, but it signals the accuracy reporting was not carefully proofchecked.\n\nThe XAI section is illustrative rather than evidential — no quantitative evaluation or human study — but the paper is honest about that. The claim that this is a novel gap is slightly overstated; prior Bangladeshi leaf work exists, as the authors themselves cite. Still, this is a solid benchmark paper, not a methodological breakthrough.\n\nWho is it for? Readers who want a quick reference on how standard pretrained models perform on this public dataset. It deserves a serious referee: the data is public, the method is standard, and the numbers are checkable. I would accept it for review but require (1) mean and standard deviation across runs, (2) a stratified and deduplicated split, or a justification of why leakage is unlikely, and (3) fixing the ResNet50V2 discrepancy. For peer review, it's a conditional accept.","headline":"Standard transfer-learning benchmark that is a useful dataset reference, but the headline 98.9% is the best of five runs and the evaluation protocol needs cleanup before those numbers are taken at face value.","tokens_in":9231,"tokens_out":2361,"would_cite":false,"duration_ms":19351,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned VGG19 identifies Bangladeshi leaf diseases with 98.90% accuracy.","keywords":["leaf disease detection","transfer learning","VGG19","Xception","explainable AI","GradCAM","Bangladeshi crops","image classification"],"falsifier":"Compute a stratified or deduplicated split of the 12,786 images, grouping by leaf identity or by collection field, retrain VGG19 under the same hyperparameters, and compare test accuracy; if accuracy drops well below 98.90%, the original claim overstates generalization to new leaves.","tokens_in":8112,"feed_emoji":"🌿","tokens_out":5225,"duration_ms":39792,"temperature":0.7,"pith_summary":"The paper claims that fine-tuned transfer learning models can classify 21 leaf diseases across six Bangladeshi crops with near-99% test accuracy, with VGG19 reaching 98.90% and Xception 98.66%. It also claims that five explainable-AI methods consistently highlight the same disease-relevant leaf regions, supporting model transparency for farmers. If correct, this would provide an accessible, expert-free screening tool for a country where agriculture is central to food security and expert plant pathologists are scarce. The evaluation, however, rests on an 80/10/10 random split of a single field-collected dataset, which the paper treats as representative of new leaves.","feed_headline":"VGG19 hits 98.9% accuracy on Bangladeshi leaf disease images","feed_subtitle":"Fine-tuned models separate 21 diseases across six crops; XAI heatmaps show where each model looks.","key_machinery":"The mechanism is transfer learning, reusing convolutional backbones pretrained on a large general image corpus and fine-tuning them on the 21-class Bangladeshi leaf dataset, with a custom head of flatten, dense layers of 1024 and 512 units, dropout, and a softmax output. The argument is carried by the accuracy and F1-scores of VGG19 and Xception, and by the heatmaps from five post-hoc explanation methods that localize the model's attention to leaf regions.","core_discovery":"On the paper's own terms, the central discovery is that standard pretrained CNN architectures, fine-tuned on a 12,786-image Bangladeshi leaf dataset, separate 21 disease and healthy classes with high fidelity, and that the best model, VGG19, reaches 98.90% accuracy with precision, recall, and F1-score of 0.99. The XAI analysis is claimed to show that the models focus on consistent, disease-relevant regions of the leaf rather than the black background, which the paper presents as evidence that the system is transparent enough for farmers to understand and act on its predictions.","pith_inferences":["The accuracy claim is vulnerable to train/test leakage because the random split does not group by plant, field, or individual leaf; if the same leaf appears in both sets, the 98.90% value would reflect image memorization rather than generalization, and a field-stratified or deduplicated evaluation would likely lower it.","The claim that farmers can use the XAI heatmaps is plausible but untested with actual farmers; a usability study with non-expert users would be the natural next test.","Because all images are background-removed single leaves, the model may be learning a shortcut based on leaf shape or color rather than disease-specific texture; testing on photos of whole plants or natural field conditions would reveal this.","The near-99% accuracy matches a pattern seen in other leaf-disease datasets, suggesting the classification task may be easier than real-world diagnosis; interpreting the score as a field-ready diagnostic rate would overstate readiness."],"forward_implications":["A smartphone-based leaf disease screening tool for Bangladeshi farmers is technically feasible using off-the-shelf pretrained networks, without requiring on-site expert diagnosis.","The 21-class, 12,786-image dataset becomes a benchmark for future automated plant disease detection work focused on Bangladesh.","The consistent XAI heatmaps across methods suggest the models rely on genuine disease symptoms, so the explanations could be used to build farmer-facing interfaces that show which part of the leaf is diseased.","If the accuracy holds under stricter evaluation, the approach could be extended to additional Bangladeshi crops and to real-time edge devices.","The performance gap between the plain CNN (92.17%) and the fine-tuned models (up to 98.90%) quantifies the value of pretrained weights for this domain."],"supporting_citations":[{"why":"The Mendeley Data record that supplies the dataset used for training and testing.","marker":"[13]"},{"why":"The paper describing the dataset's collection and composition: six crops, 21 classes, black-background field photos.","marker":"[14]"},{"why":"The general account of CNN and transfer learning in medical imaging that justifies using pretrained weights and fine-tuning.","marker":"[15]"},{"why":"A prior Bangladeshi mulberry leaf disease study using CNN and SHAP, providing a local baseline for disease-classification work.","marker":"[10]"},{"why":"A prior Bangladeshi study detecting grape and potato leaf diseases with CNN, EfficientNet, and ResNet, against which the paper positions its contribution.","marker":"[7]"}],"fun_headline_variants":["VGG19 tops 98.9% in Bangladeshi leaf disease test","Transfer learning spots 21 leaf diseases with 99% F1","XAI heatmaps show where models detect leaf disease","Fine-tuned CNN hits 98.9% on Bangladeshi leaf diseases","Deep learning IDs 21 leaf diseases across 6 crops"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the randomly split test images are genuinely new, unseen leaves; if images of the same leaf or same field are shared between training and test sets, the reported 98.90% accuracy would measure image memorization rather than the ability to classify a leaf never seen before.","fun_headline_variants_meta":{"raw":{"variants":["VGG19 tops 98.9% in Bangladeshi leaf disease test","Transfer learning spots 21 leaf diseases with 99% F1","XAI heatmaps show where models detect leaf disease","Fine-tuned CNN hits 98.9% on Bangladeshi leaf diseases","Deep learning IDs 21 leaf diseases across 6 crops"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2778,"prompt_tokens":932,"completion_tokens":1846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1756}},"tokens_in":548,"tokens_out":1846,"duration_ms":11160,"temperature":1.0,"reasoning_tokens":1756,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:07:18.485770+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute a stratified or deduplicated split of the 12,786 images, grouping by leaf identity or by collection field, retrain VGG19 under the same hyperparameters, and compare test accuracy; if accuracy drops well below 98.90%, the original claim overstates generalization to new leaves.","supporting_citations":[{"cited_title":"Plant leaf freshness and disease detection dataset from bangladesh, 2024","cited_arxiv_id":null,"evidence_quote":"The Mendeley Data record that supplies the dataset used for training and testing."},{"cited_title":"Comprehensive smart smartphone image dataset for plant leaf disease detection and freshness assessment from bangladesh vegetable fields.Data in Brief, 56:110775, 2024","cited_arxiv_id":null,"evidence_quote":"The paper describing the dataset's collection and composition: six crops, 21 classes, black-background field photos."},{"cited_title":"A study of cnn and transfer learning in medical imaging: Advantages, challenges, future scope.Sustainability, 15(7):5930, 2023","cited_arxiv_id":null,"evidence_quote":"The general account of CNN and transfer learning in medical imaging that justifies using pretrained weights and fine-tuning."},{"cited_title":"Explainable deep learning model for automatic mulberry leaf disease classification.Frontiers in Plant Science, 14:1175515, 2023","cited_arxiv_id":null,"evidence_quote":"A prior Bangladeshi mulberry leaf disease study using CNN and SHAP, providing a local baseline for disease-classification work."},{"cited_title":"Comparative analysis of cnn, efficientnet and resnet for grape and potato leaves disease prediction: A deep learning approach","cited_arxiv_id":null,"evidence_quote":"A prior Bangladeshi study detecting grape and potato leaf diseases with CNN, EfficientNet, and ResNet, against which the paper positions its contribution."}],"review_version":1}