{"id":"ea0ba1b4-1885-4c84-b22c-2f5bdf82dc77","arxiv_id":"2505.16039","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"On a 37-class brain MRI dataset, a Vision Transformer achieves 94.39 percent accuracy, outperforming four transfer learning CNN models.","lead":"This paper compares a Vision Transformer against four pretrained CNN models for classifying 37 types of brain disease from MRI scans collected in Bangladesh. It reports that the Vision Transformer reaches 94.39 percent accuracy and uses gradient-based explanation maps on one of the CNN models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SMOTE and augmentation are applied before the split (Section 3.2), so the test set is not genuinely unseen; the 94.39% ViT accuracy is inflated and the model ranking is unverified. A clean split-before-augmentation evaluation is required.","rationale":"The reader's weakest assumption is correct and is the load-bearing point: the evaluation protocol destroys the train/test separation. The paper states the order explicitly in Section 3.2, so this is not an interpretive stretch. I agree with the reader that best-of-five reporting compounds the problem, though the leak alone is sufficient to invalidate the headline numbers. I also note the paper offers no code or detailed run-to-run results, and the XAI section applies explanations only to ResNet50V2, so the abstract's 'integration of XAI' claim is not demonstrated for the ViT model that is the paper's focus; these are secondary. Because the central accuracy comparison is used in the abstract, conclusion, and Table 2 comparison against prior work, the paper cannot be accepted as is. The recommended verdict remains REJECT, so no change to the reader's verdict is needed.","tokens_in":8695,"tokens_out":4940,"duration_ms":39743,"concrete_test":"Split the original 5,285 MRI images into train/val/test (80/10/10) first; apply SMOTE and all augmentation only to the training partition; then retrain ViT and the four transfer-learning models with the same hyperparameters and report mean and standard deviation across all five runs for each model. If ViT's test accuracy on the clean unseen test set is materially below 94.39%, or if ResNet50V2 or another model matches or exceeds it, the central claim is rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 4.2, Table 1) is that ViT reaches 94.39% accuracy, outperforming ResNet50V2 and other transfer-learning models. That claim requires the test set to contain only unseen images. The protocol violates this: Section 3.2 applies SMOTE and a TensorFlow augmentation pipeline (resize, flip, rotation, zoom) to the full dataset before 'finally, the dataset was split into training, validation, and test sets with an 80:10:10 ratio.' Consequently, augmented copies or SMOTE-generated synthetic samples of a given original image can appear in both training and test splits, making the test accuracy a measure of memorization or near-duplicate retrieval rather than generalization. Section 4.1 also reports 'the optimal results chosen' from five runs, further inflating the point estimate and masking variance. These flaws directly undermine the abstract's '94.39%' and the conclusion that ViT 'significantly outperformed' CNNs. The comparison might survive in direction, but the specific numbers and the magnitude of the gap are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares a Vision Transformer (ViT) trained from scratch with four ImageNet-pretrained transfer-learning models (VGG16, VGG19, ResNet50V2, MobileNetV2) on a 37-class brain MRI dataset collected in Bangladesh, reporting that ViT reaches 94.39% accuracy and that five gradient-based XAI methods provide interpretability. The central empirical claims are supported by Table 1, Figure 6, and qualitative XAI visualizations in Figure 7. The paper concludes that ViT significantly outperforms the CNN baselines and that XAI improves model transparency.","tokens_in":8852,"tokens_out":6501,"duration_ms":52673,"significance":"If the evaluation were sound, the study would be a useful benchmark for a local clinical MRI dataset and a practical comparison of ViT against transfer learning under class imbalance, with visual explanations from five XAI methods. The use of a publicly available dataset and the explicit comparative design are strengths, as is the attempt to include interpretability. However, the experimental protocol contains a data-leakage flaw and a best-run selection procedure that invalidate the reported accuracy numbers; without code, checkpoints, or corrected experiments, the main comparative claim is not verifiable in its current form.","major_comments":[{"comment":"SMOTE and the TensorFlow augmentation pipeline (resizing, random flip, rotation, zoom) are applied to the full dataset before the 80:10:10 train/validation/test split, as stated in the final sentence of Section 3.2. This creates data leakage: augmented copies or SMOTE-generated synthetic versions of the same original image can appear in both the training partition and the test partition. Consequently, the test accuracies in Table 1, including the headline 94.39% for ViT, do not measure generalization to unseen scans; they partly measure recognition of near-duplicate images. The split must be performed first, with SMOTE fit and augmentation applied only to the training partition, before any accuracy claim can be accepted.","section":"Section 3.2"},{"comment":"Section 4.1 states that each model was executed five times 'with the optimal results chosen for analysis.' Selecting the best of five runs inflates the reported point estimate and discards variance information, and Table 1 reports no standard deviations, confidence intervals, or significance tests. The statement in Section 4.2 that ViT 'significantly outperformed' the CNN models is therefore not statistically supported. The authors should report the mean and standard deviation over runs, or otherwise justify a non-selected evaluation protocol.","section":"Section 4.1"},{"comment":"The abstract and conclusion credit the integration of XAI methods with enhancing model transparency, but Section 5 applies the five XAI techniques only to ResNet50V2, not to the ViT model that is the paper's main contribution, and Figure 7 provides only qualitative heatmaps. No quantitative evaluation of explanation quality (e.g., localization, faithfulness, or expert assessment) is presented, so the claim that XAI improves reliability or transparency is not demonstrated. This should be restated as a qualitative exploration or supported by an appropriate evaluation.","section":"Section 5 and Abstract"}],"minor_comments":[{"comment":"The paragraph beginning 'This study utilizes a range of pre-trained models...' is duplicated, and the architecture description repeats 'The task is effectively modified for brain disease' in consecutive sentences; one copy should be removed.","section":"Section 3.3"},{"comment":"The diagram label '16 Patches' is inconsistent with the stated input size of 128x128 and patch size of 64x64, which yields four patches; please correct the figure or the text.","section":"Figure 5"},{"comment":"The caption contains the typo 'Gradinat Based XAI' and should read 'Gradient Based XAI'; similarly, Figure 6 uses the misspelling 'Accurcay'.","section":"Figure 7 caption"},{"comment":"The phrase 'aver aging the complete dataset' contains a spacing error and should read 'averaging'.","section":"Section 3.2"},{"comment":"Reference [2] is formatted inconsistently with the other references (author name in all caps, thesis title not in a standard style), and the model name appears variously as 'MobileNetV2' and 'MobilenetV2' throughout the text and Table 1.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the arXiv version of an IEEE ICCIT 2024 conference paper with the same DOI. If this is submitted as a new journal article, the authors should disclose the relationship to that publication and identify the novel contributions beyond the conference version. The absence of code, trained models, or a detailed protocol also makes it difficult for reviewers to verify any corrected experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a standard empirical comparison, and the headline number is not trustworthy. The preprocessing pipeline applies SMOTE and augmentation before the split, so the test set is not clean. The best-of-five reporting makes it worse. The paper does have some merits: it applies ViT to a publicly available 37-class Bangladesh MRI dataset for the first time, and the XAI section, though narrow, is clearly described. But the central claim of 94.39% accuracy is unsupported.\n\nSection 3.2 describes SMOTE and a TensorFlow augmentation pipeline applied to the full dataset, then an 80:10:10 split. That means augmented or synthetic copies of the same image can appear in both training and test sets. The test accuracy becomes a measure of near-duplicate retrieval, not generalization. Section 4.1 reports each model run five times 'with the optimal results chosen,' which further inflates the point estimate and hides variance. Together these flaws directly undermine the abstract's 94.39% and the conclusion that ViT significantly outperformed the CNNs.\n\nWhat is genuinely new: applying ViT to this specific dataset and comparing it against transfer learning baselines on the same data. The comparison with prior work (Brima et al. 0.87, Ahmed et al. 0.92) is useful context. The XAI part is honest about its limitation: Section 5 explains why ViT is not suitable for gradient-based CAM and instead applies five XAI methods to ResNet50V2. That is a reasonable choice, though it makes the title's promise of explaining ViT somewhat misleading.\n\nThe XAI analysis is narrow, showing only one model and a few example images. There is no code or hyperparameter release beyond what is in the text. The ViT hyperparameters are listed, which is nice, but there is no seed control or variance reporting. The comparison might survive in direction, but the magnitude of the gap is unverified.\n\nWho is this for? A reader interested in a quick benchmark of ViT versus transfer learning on a specific South Asian MRI dataset might use this as a pointer, but not as evidence. It deserves a serious referee because the topic is relevant and the dataset is real, but it needs a corrected experimental setup: split before any augmentation or synthetic oversampling, report all runs with variance, and ideally release code. I'd send it back for major revision rather than reject outright if the authors can redo the evaluation, but the current version's central numeric claim should not be cited.","headline":"Routine ViT comparison on a Bangladesh MRI dataset, undone by a leakage-prone preprocessing pipeline and best-of-five reporting.","tokens_in":9493,"tokens_out":1952,"would_cite":false,"duration_ms":15182,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a Vision Transformer reaches 94.39% accuracy on a 37-class brain MRI dataset, beating four transfer-learning CNN baselines, and applies five explainability methods to the strongest CNN.","keywords":["Vision Transformer","Transfer Learning","Explainable AI","Brain Disease Detection","MRI Classification","SMOTE","GradCAM","37-class classification"],"falsifier":"Run the same five models with SMOTE and augmentation applied only after an 80:10:10 split of the original images, holding all hyperparameters fixed, and compare test accuracies; if ViT's 94.39% lead over ResNet50V2 shrinks or disappears, the original ranking was an artifact of preprocessing order.","tokens_in":8433,"feed_emoji":"🧠","tokens_out":10788,"duration_ms":83859,"temperature":0.7,"pith_summary":"This paper sets out to test whether a Vision Transformer (ViT) can beat standard transfer-learning convolutional networks when classifying brain diseases from MRI images. On a 37-class Bangladeshi brain MRI dataset, the authors report that a ViT reaches 94.39% accuracy, ahead of ResNet50V2's 91.41% and the other pretrained CNN baselines. The study also applies five gradient-based explainability methods to show which brain regions drive the best transfer-learning model's decisions. If the ranking holds under careful evaluation, transformer-based models would be a practical alternative for computer-assisted brain disease diagnosis in settings with limited local imaging data.","feed_headline":"Vision Transformer tops brain MRI models at 94.39%","feed_subtitle":"On a 37-class Bangladeshi benchmark, self-attention beat four pretrained CNN baselines by about six accuracy points.","key_machinery":"The carrying mechanism is the contrast between two learning regimes. ViT splits each 128x128 MRI into 64x64 patches, projects the patches into a 64-dimensional embedding space, adds positional embeddings, and runs them through a transformer encoder with eight layers and four attention heads per layer, ending in an MLP head; its self-attention can relate any patch to any other patch directly. The four transfer-learning baselines start from convolutional backbones pretrained on a large natural-image corpus, with a flatten layer, two dense layers (1024 and 512 units), dropout, and a softmax head, fine-tuned on the MRI data. The paper also uses SMOTE to balance the 37 classes and an augmentation pipeline (flip, rotation, zoom) to expand the training data, and it takes the reported numbers from a ten-percent held-out split.","core_discovery":"On the paper's own terms, the central claim is that a Vision Transformer trained for 400 epochs on a 37-class Bangladeshi brain MRI dataset reaches 94.39% accuracy, 96.51% precision, 96.44% recall, and a 96.38% F1 score, ahead of every transfer-learning CNN tested (ResNet50V2 91.41%, MobileNetV2 88.56%, VGG19 89.05%, VGG16 88.44%). The authors attribute the margin to the transformer's patch-based self-attention, which captures long-range relationships in the image that convolutional filters with local receptive fields miss. Because gradient-based explanation methods do not map cleanly onto the transformer's architecture, the paper applies GradCAM, GradCAM++, LayerCAM, ScoreCAM, and FasterScoreCAM to ResNet50V2 instead, and reports that the heatmaps consistently point to the same brain regions, which it reads as a sign the model is attending to clinically meaningful structures.","pith_inferences":["ViT was run for 400 epochs with a batch size of 256 while the CNN curves shown stop at about 80 epochs; an ablation that matches training budget across architectures would separate the transformer effect from the extra optimization time.","The explainability analysis covers only the best CNN (ResNet50V2); applying attention-based explanation methods to the ViT itself would test whether the transformer's higher accuracy comes from the same or different image regions.","The dataset's 37 classes include many closely related lesions, so a per-class confusion analysis, beyond overall accuracy, would show whether ViT's advantage comes from rare or morphologically similar classes."],"forward_implications":["If the reported ranking is correct, ViT should be the model of choice for this 37-class brain MRI task, with ResNet50V2 as the strongest CNN fallback.","The roughly three-percentage-point accuracy gap and four-point F1 gap between ViT and ResNet50V2 imply that long-range, patch-level context contributes real signal for distinguishing the 37 lesion classes.","The consistent XAI heatmaps across five gradient-based methods on ResNet50V2 imply that the best CNN model is making decisions from localized, reproducible image regions, which is a useful property for clinical review.","All five models clear 88% accuracy, so even if ViT is unavailable in a resource-constrained hospital, the transfer-learning baselines still provide a usable diagnostic aid on this dataset."],"supporting_citations":[{"why":"Supplies the 37-class Bangladeshi brain MRI dataset and the earlier ResNet50 transfer-learning baseline that this study extends and compares against.","marker":"[9]"},{"why":"Provides the public dataset access the experiments draw on.","marker":"[10]"},{"why":"Supplies the SMOTE algorithm used to balance the 37 classes before training.","marker":"[11]"},{"why":"Supplies the Vision Transformer classification architecture and configuration guidance used for the ViT model.","marker":"[13]"},{"why":"Provides a recent transfer-learning result on multi-class brain lesion classification that the comparison table uses as a prior benchmark.","marker":"[16]"}],"fun_headline_variants":["ViT edges out CNNs for brain disease MRI classification","Transformer tops CNN baselines on 37-class brain MRI benchmark","Self-attention wins: ViT hits 94.39% on brain MRI","Vision Transformer beats transfer learning in brain scan accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the test split containing only genuinely unseen images, but this pipeline applies SMOTE and augmentation before the 80:10:10 split, so versions of the same image can appear in both training and test data.","fun_headline_variants_meta":{"raw":{"variants":["ViT edges out CNNs for brain disease MRI classification","Transformer tops CNN baselines on 37-class brain MRI benchmark","Self-attention wins: ViT hits 94.39% on brain MRI","Vision Transformer beats transfer learning in brain scan accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1282,"prompt_tokens":975,"completion_tokens":307,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":236}},"tokens_in":591,"tokens_out":307,"duration_ms":3027,"temperature":1.0,"reasoning_tokens":236,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:06:53.433034+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same five models with SMOTE and augmentation applied only after an 80:10:10 split of the original images, holding all hyperparameters fixed, and compare test accuracies; if ViT's 94.39% lead over ResNet50V2 shrinks or disappears, the original ranking was an artifact of preprocessing order.","supporting_citations":[{"cited_title":"Deep Transfer Learning for Brain Magnetic Resonance Image Multi-class Classification","cited_arxiv_id":"2106.07333","evidence_quote":"Supplies the 37-class Bangladeshi brain MRI dataset and the earlier ResNet50 transfer-learning baseline that this study extends and compares against."},{"cited_title":"Smote: synthetic minority over-sampling technique.Journal of artificial intelligence research, 16:321–357, 2002","cited_arxiv_id":null,"evidence_quote":"Supplies the SMOTE algorithm used to balance the 37 classes before training."},{"cited_title":"Classification of brain tumor from magnetic resonance imaging using vision transformers ensembling.Current Oncology, 29(10):7498–7511, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the Vision Transformer classification architecture and configuration guidance used for the ViT model."},{"cited_title":"Multi-class brain lesion classification using deep transfer learning with mobilenetv3.IEEE Access, 2024","cited_arxiv_id":null,"evidence_quote":"Provides a recent transfer-learning result on multi-class brain lesion classification that the comparison table uses as a prior benchmark."}],"review_version":1}