{"id":"cdefb2f0-9e50-4bd0-b5f5-de0026a1eb89","arxiv_id":"2502.06939","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Swin-UNETR models trained on a large multi-site DWI stroke dataset reach high Dice scores, and a new evaluation framework exposes anatomical, morphological, and noise-dependent performance variation.","lead":"This paper trains transformer-based models to automatically outline stroke lesions on diffusion-weighted brain MRI, using 3,563 labelled images and about 6,900 normal scans. It reports high Dice scores and introduces an evaluation framework that checks fairness across brain regions, lesion shapes, scanner noise, and false positives.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Early stopping selects the best epoch on the same validation set later used for reporting, making the claimed out-of-sample Dice a selected maximum and therefore optimistically biased.","rationale":"Both the reader and I agree the paper is not as strong as the headline suggests, but for different reasons. The reader focused on the possibility that machine-generated labels are biased and inflate Dice. I find this plausible but it is explicitly disclosed in the Limitations and is hard to settle without additional expert annotations. A more immediate and objectively verifiable flaw is the use of the validation set for early stopping and then reporting the best validation score as 'out-of-sample.' Section 2.3.3 says the best epoch is selected based on validation Dice, and Section 2.4.1 incorrectly denies that any model selection was performed. This is a classic selection-bias leak. The reported mean Dice of 0.8952 is therefore likely an overestimate, potentially by an amount that matters for the 0.85 threshold used to justify clinical preference. This can be quantified from existing training logs or by re-running training with a nested protocol. Since the flaw is fixable and the underlying dataset and engineering may still be valuable, a conditional acceptance requiring corrected evaluation is appropriate. I therefore recommend CONDITIONAL, aligning with the reader's verdict but for a different reason.","tokens_in":15247,"tokens_out":8868,"duration_ms":75460,"concrete_test":"For each of the five folds, compute the validation Dice at every epoch during the early-stopping window (the last epoch with joint Dice/HD improvement plus 150 further epochs). Compare the reported value (the maximum of this window) with (a) the Dice at the final epoch and (b) the mean Dice over the window. If the maximum exceeds the final epoch or window mean by more than, say, 0.005 Dice, the reported 0.8952 is inflated. A more definitive test: retrain SWIN-UNETR and SWIN-UNETR+Ctr without epoch selection on the validation set—e.g., stop at a fixed epoch or on a separate inner validation split—and report the resulting fold means. If the unbiased Dice falls below the 0.85 threshold, the central claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper reports 'out-of-sample' 5-fold Dice of 0.8952 for SWIN-UNETR (Section 3.1). Section 2.3.3 states that training is stopped when Dice and HD have not improved together for 150 epochs, and 'we selected the best model with the best Dice score between the last epoch in which both Dice and HD improved and 150 further epochs.' Thus the validation set is used to choose the epoch. Section 2.4.1 claims 'no hyperparameter tuning or model selection was performed during the cross-validation process,' but early stopping with epoch selection is exactly model selection on the validation set. Because the reported metrics are computed on the same validation set used for this selection, the reported Dice is the maximum of a noisy curve over a window of up to 150 epochs, not an unbiased estimate of out-of-sample performance. This biases the absolute performance upward and calls into question the crossing of the 0.85 Dice threshold (Discussion) that is a key part of the SOTA argument. The relative ordering of models may be less affected, but the headline absolute numbers—and consequently the 'state-of-the-art' claim—are not supported without re-evaluation. A secondary concern, flagged by the reader, is the label quality; however, the early stopping bias is a more direct, resolvable methodological flaw.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents a pipeline for automated ischaemic stroke lesion segmentation from diffusion-weighted imaging, built around the SWIN-UNETR vision transformer and trained on a large (n=3563) curated set of lesion images plus 6900 controls. The authors compare SWIN-UNETR variants against a U-Net baseline using 5-fold cross-validation, reporting an out-of-sample Dice of 0.8952 and HD of 2.6498 voxels for SWIN-UNETR, and propose a multi-axis evaluation framework covering anatomical equity, morphological variation, false-positive control, and noise robustness. They also introduce a Thresholded Average loss for training with negative images and provide open-source code and model weights.","tokens_in":15475,"tokens_out":3974,"duration_ms":35409,"significance":"If the reported results are valid, the paper would make a useful contribution: it introduces a very large multi-site DWI lesion dataset, an open-source training pipeline, a practical control-training loss, and an evaluation framework that goes beyond mean Dice to examine equity and robustness. These are strengths that the field would benefit from. However, the central claim of state-of-the-art performance is not supported by comparison with published methods or public benchmarks, and the reported absolute Dice figures are vulnerable to a model-selection bias in the cross-validation protocol. The paper's significance therefore depends on the outcome of the methodological fixes requested below.","major_comments":[{"comment":"The reported 5-fold Dice/HD values (Section 3.1) are not unbiased out-of-sample estimates. Section 2.3.3 states that training was stopped when Dice and HD had not improved together for 150 epochs and that the best model was selected by best Dice between the last epoch of joint improvement and 150 further epochs; Section 2.4.1 then claims 'no hyperparameter tuning or model selection was performed during the cross-validation process.' This is contradictory. Early stopping with epoch selection is a form of model selection, and because the reported metrics are computed on the same validation folds used for that selection, the reported Dice is a maximum over a window of candidate epochs, biased upward. Please re-evaluate with a separate held-out test set, or use nested cross-validation where the inner loop selects the epoch, and report the variance of the selection.","section":"Section 2.3.3 and 2.4.1"},{"comment":"The state-of-the-art claim is not supported by the evidence presented. The only comparison is against an internal U-Net without augmentation; no comparison is made with published DWI lesion segmentation methods or with public benchmarks such as ISLES. The reference to the 0.85 Dice threshold (Discussion, citing Liew et al., 2022) is not a direct method comparison. Please add results on a public benchmark (e.g., ISLES 2015/2022) or a table comparing with recent published Dice/HD on comparable data, and temper the 'state-of-the-art' wording until such evidence is provided.","section":"Section 3.1 and Discussion"},{"comment":"The reference standard is machine-generated and only manually edited. Section 2.1.5 describes an iterative process in which initial labels came from the zeta anomaly method, then a residual U-Net, then UNETR, with manual selection and modification. The Limitations state that dense manual segmentation is infeasible at this scale and that there is no mechanism to quantify the advantage of dense manual labels. This is a load-bearing issue because if residual errors in these iterative labels correlate with the errors of the trained models, the reported Dice, HD, and false-positive counts are inflated. Please provide an independent assessment on a random subset of, say, 50-100 images, with expert manual segmentations, and report inter-rater agreement and the agreement of the final model with that independent reference.","section":"Section 2.1.5"},{"comment":"Standard performance metrics are reported as fold means without any confidence intervals or variance. For example, the difference between SWIN-UNETR+Ctr (Dice 0.8915, HD 2.6636) and SWIN-UNETR (Dice 0.8952, HD 2.6498) may not be meaningful with only five folds, yet the Discussion treats these as distinguishing the two models. Please report per-fold results, standard deviations, and bootstrapped confidence intervals for all primary metrics, and avoid over-interpreting small differences without statistical support.","section":"Section 3.1"},{"comment":"The claim that 'no hyperparameter tuning or model selection was performed' is at odds with the methods. Section 2.2.2 states that augmentation probabilities were 'chosen empirically between 5, 10 and 20%'; Section 2.2.3 sets the focal loss focusing parameter to 2; Section 2.2.4 assigns a weight to the Thresholded Average loss 'in response to training performance'; and Section 2.3.3 fixes the early-stopping patience at 150 epochs based on prior tests. These are hyperparameter choices that affect all reported results. Please disclose all such choices and treat them as part of model selection when discussing generalizability, or remove the claim in Section 2.4.1.","section":"Section 2.4.1 and Sections 2.2.2-2.3.3"}],"minor_comments":[{"comment":"The phrase 'Hausdorff Distance in the 95% confidence interval' is ambiguous; standard practice is the 95th percentile of surface-to-surface distances (HD95). Please define the metric explicitly.","section":"Section 2.2.5"},{"comment":"The Thresholded Average loss is described in prose but no equation is given. Please provide the mathematical definition, including how the 0.5 threshold is applied and how the loss is averaged over the control set.","section":"Section 2.2.4"},{"comment":"There is a typo: 'Kruskall-Wallis' should be 'Kruskal-Wallis'. Also, the description of the balancing criterion (highest p-value and lowest variability in average volume and standard deviation) would benefit from an explicit equation or pseudocode.","section":"Section 2.3.2"},{"comment":"The bootstrap procedure used for the paired t-tests in Table 1 and Figure 6 is not fully specified. Please state how the 100-image subsamples are drawn, whether the resampling is paired across models, and how the t-statistic and p-value are computed from the bootstrap distribution.","section":"Sections 2.4.4 and 3.4"},{"comment":"The claim of 'the largest DWI dataset of delineated acute ischaemic stroke lesions known to us' should be placed in context of public resources such as ATLAS or ISLES. If the claim is intended as a strong comparative statement, please provide a citation or explicit comparison of sizes.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The dataset, code release, and the multi-axis evaluation framework are genuine strengths, and the control-training idea is worth publishing. The main obstacle is the early-stopping leakage, which is a direct methodological flaw that affects the central absolute numbers; it is fixable by re-running with a properly separated selection procedure. I would not reject on the label-quality issue alone, but the manuscript must present an independent manual-label check for a journal of this standard. The SOTA claim also needs external benchmarks before it can be accepted. I recommend major revision, not rejection, because the core direction is sound and the requested changes are within scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a substantial dataset contribution, and the four-part evaluation framework is genuinely useful, but the top-line 'state-of-the-art' claim is not supported as written. The internal comparison between SWIN-UNETR and a U-Net baseline is credible; the absolute numbers are not.\n\nWhat's new: the 3,563 curated DWI lesions plus 6,900 controls is a real resource; the Thresholded Average loss for control images is a simple, sensible idea that produces a large drop in false positives; and the evaluation framework (anatomy, morphology, noise, false positives) is a step beyond mean Dice. The open code and weights help.\n\nSoft spots. The early stopping protocol selects the best epoch using validation-set Dice (Section 2.3.3), and the reported numbers are computed on that same validation set. That makes the reported 0.8952 Dice a selected maximum over a window of up to 150 epochs, not an unbiased out-of-sample estimate. The claim in Section 2.4.1 that no model selection was performed is incorrect, and it matters for the 0.85 threshold crossing. The relative ordering of the three models is probably less affected since all use the same protocol, but the absolute SOTA claim needs re-evaluation on a genuinely held-out set or with nested cross-validation. Second, there is no comparison to published methods or public benchmarks; the ISLES exclusion is explained but still leaves 'state-of-the-art' unsubstantiated. Third, the ground truth is machine-generated and manually edited, with no independent measure of label quality; the limitations section says this openly, but the risk of correlated errors inflating Dice is real.\n\nMinor: fold means are reported without confidence intervals; a paired bootstrap or per-fold intervals would help. The KCH external set is used only for distribution agreement, not as a performance check, which is fine but should be stated more clearly as validation of generalizability only in distributional terms.\n\nWho this is for: stroke imaging researchers and medical ML groups wanting a large DWI segmentation dataset and a more informative evaluation template. It deserves a serious referee; the dataset scarcity and the framework justify the time. I'd send it to review with a strong request to fix the early stopping/reporting issue and add an external benchmark comparison.","headline":"A valuable dataset and evaluation framework with a plausible internal comparison, but the headline SOTA Dice is optimistically biased by epoch selection on the validation set and the absence of external benchmarks.","tokens_in":16092,"tokens_out":1746,"would_cite":true,"duration_ms":15269,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A SWIN-UNETR trained on 3,563 curated DWI scans achieves state-of-the-art stroke lesion segmentation, and adding lesion-free controls cuts false positives more than tenfold.","keywords":["ischemic stroke","diffusion-weighted imaging","lesion segmentation","vision transformer","SWIN-UNETR","false positive control","medical image segmentation","evaluation framework"],"falsifier":"Manually and densely segment a random sample of roughly 200 of the 3,563 lesions, then compare SWIN-UNETR predictions against those dense manual labels instead of the curated semi-automated labels; if the mean Dice drops below 0.85 or voxel-level errors align with the original label errors, the central claim of state-of-the-art fidelity is not supported.","tokens_in":15018,"feed_emoji":"🧠","tokens_out":11498,"duration_ms":90691,"temperature":0.7,"pith_summary":"This paper claims that a vision-transformer-based segmentation model, trained on 3,563 expert-curated diffusion-weighted MRI images of acute ischaemic stroke, outperforms U-Net baselines and reaches state-of-the-art out-of-sample accuracy, with a mean Dice of 0.8952 and a Hausdorff distance of 2.6498 voxels. It further claims that adding 6,900 lesion-free control images to training, combined with a Thresholded Average loss that penalises false positives, cuts average false-positive voxels per control image from 126.80 to 3.31 without sacrificing segmentation fidelity. The paper also introduces a four-part evaluation framework that checks performance across brain anatomy, lesion morphology, image noise, and false-positive rates, and reports that the transformer models show no significant anatomical inequity, unlike the U-Net. If these claims hold, automated DWI lesion mapping could be reliable enough for clinical and research use on heterogeneous, real-world scanning data.","feed_headline":"Vision transformer outperforms U-Net on stroke lesions","feed_subtitle":"On 3,563 scans it hits 0.895 overlap accuracy; adding healthy controls cuts false positives from 127 to 3 voxels.","key_machinery":"The central machinery is the SWIN-UNETR architecture—a U-Net with Swin-transformer stages that capture long-range spatial dependencies—which the paper identifies as the main source of the accuracy gain over the U-Net baseline. Around it sit a data augmentation suite covering intensity shifts, geometric transforms, and DWI-typical noise; a fold-balancing scheme that splits the dataset by lesion phenotype and volume; a combined Dice-plus-Focal training loss; and, for the control-trained variant, a Thresholded Average loss that penalises any voxel with probability above 0.5 in lesion-free images. The evaluation framework uses voxel-wise general linear models, a low-dimensional UMAP embedding of lesion morphology, parametric noise corruption, and false-positive counting on held-out controls.","core_discovery":"The paper reports that a SWIN-UNETR vision transformer, trained with data augmentation and a statistically balanced five-fold split on 3,563 curated DWI lesion images, achieves state-of-the-art out-of-sample segmentation with a mean Dice of 0.8952 and a Hausdorff distance of 2.6498 voxels, surpassing the U-Net baseline at a Dice of 0.8408. It further reports that training a second SWIN-UNETR with 6,900 DWI-negative control images and a Thresholded Average loss reduces average false-positive voxels per control from 126.80 to 3.31, makes the model significantly more resilient to Rician and bias-field noise, and does not degrade standard segmentation metrics. The accompanying evaluation framework, using voxel-wise statistical mapping, low-dimensional morphological embedding, parametric noise corruption, and control-image false-positive counting, shows that the transformer models are equitable across brain anatomy while the U-Net performs significantly better within middle-cerebral-artery territory.","pith_inferences":["If the semi-automated labels contain residual error that correlates with model error, the paper's absolute Dice values may overstate true accuracy, while its between-model comparisons are likely to remain informative.","Because the U-Net baseline was trained without the augmentation and balancing scheme, part of the observed gap may reflect training recipe rather than the Transformer architecture itself; a U-Net trained with the same recipe would isolate the architectural contribution.","The open-source release of code and weights invites independent replication on other DWI datasets, including non-UK populations where scanner and protocol differences are even larger than those captured here."],"forward_implications":["DWI lesion segmentation at clinical scale is achievable: both transformer models clear the 0.85 Dice threshold that the literature associates with expert-level performance.","Reporting only mean Dice is insufficient: the new anatomical, morphological, noise, and false-positive calibrations expose differences among models that average metrics hide.","Adding DWI-negative controls with a false-positive-penalising loss is a cheap, replicable way to cut artefact-driven over-segmentation by more than an order of magnitude.","Balanced five-fold cross-validation on a large, multi-site routine dataset provides more dependable out-of-sample performance estimates than small held-out test sets."],"supporting_citations":[{"why":"Supplies the Swin-transformer U-Net architecture that carries the segmentation claim.","marker":"[31]"},{"why":"Provides the lesion phenotype taxonomy used to balance the cross-validation folds.","marker":"[8]"},{"why":"Contributes the initial labelled stroke dataset on which the curation chain is built.","marker":"[21]"},{"why":"Generates the initial unsupervised lesion masks that experts curated into ground truth.","marker":"[22]"},{"why":"Defines the Dice loss term used in training.","marker":"[37]"},{"why":"Defines the Focal loss term that handles extreme class imbalance.","marker":"[38]"},{"why":"Provides the augmentation and model-implementation library used throughout training.","marker":"[28]"},{"why":"Supplies the UMAP embedding used for morphological performance evaluation.","marker":"[45]"},{"why":"Sets the Dice threshold above which automated segmentation is held to outperform human raters.","marker":"[48]"}],"fun_headline_variants":["Vision transformer outperforms U-Net on stroke lesion Dice","Adding healthy controls cuts stroke lesion false positives 98%","SWIN-UNETR achieves 0.895 Dice on stroke lesions","Stroke segmentation: transformer model more equitable across brain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the semi-automated ground-truth labels being accurate enough for training and evaluation; if the machine-generated labels that experts curated carry residual errors that correlate with the models' errors, the reported Dice, equity, and false-positive numbers would all be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Vision transformer outperforms U-Net on stroke lesion Dice","Adding healthy controls cuts stroke lesion false positives 98%","SWIN-UNETR achieves 0.895 Dice on stroke lesions","Stroke segmentation: transformer model more equitable across brain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000557,"raw_usage":{"total_tokens":2655,"prompt_tokens":953,"completion_tokens":1702,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":1633}},"tokens_in":569,"tokens_out":1702,"duration_ms":12036,"temperature":1.0,"reasoning_tokens":1633,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:17:38.340082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually and densely segment a random sample of roughly 200 of the 3,563 lesions, then compare SWIN-UNETR predictions against those dense manual labels instead of the curated semi-automated labels; if the mean Dice drops below 0.85 or voxel-level errors align with the original label errors, the central claim of state-of-the-art fidelity is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the lesion phenotype taxonomy used to balance the cross-validation folds."},{"cited_title":"& Nachev, P","cited_arxiv_id":null,"evidence_quote":"Contributes the initial labelled stroke dataset on which the curation chain is built."},{"cited_title":"& Nachev, P","cited_arxiv_id":null,"evidence_quote":"Generates the initial unsupervised lesion masks that experts curated into ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sets the Dice threshold above which automated segmentation is held to outperform human raters."}],"review_version":1}