REVIEW 4 major objections 4 minor 26 references
Pre- and Post-Treatment Glioma Segmentation with the Medical Imaging Segmentation Toolkit
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Postprocessing improves average Dice on several glioma classes yet ranks below the unprocessed baseline under the BraTS scoring protocol.
desk verdict A useful caution about rank-based evaluation, but the paper's own cross-validation results undercut its claim of consistent baseline strength, and the rank differences are noise-level. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the postprocessing module of the Medical Imaging Segmentation Toolkit (MIST): a strategy-based system in which each postprocessing pipeline is declared as a JSON configuration of composable transforms, including small-object removal, small-object replacement with another label, extraction of the largest connected component, and morphological hole filling and closing, each applicable globally or to a specific segmentation class. The matching evaluative mechanism is the BraTS ranking protocol, which turns each patient's Dice and HD95 across six classes into per-patient average ranks and then into a global average rank, making consistency across patients and metrics the decisive criterion rather than any single pooled mean.
What would settle it
Run a paired Wilcoxon signed-rank test or bootstrap confidence interval on the per-patient global ranks comparing the baseline with Strategy 3 on the validation set; if the 0.02-point rank gap is not significant or favors Strategy 3, the paper's conclusion that postprocessing fails to improve rankings is unsupported.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a well-tuned postprocessing pipeline can lift average Dice and HD95 on specific tumor subregions, yet under the BraTS ranking protocol it does not improve, and can worsen, a model's standing. Strategy 3, which replaces small objects in the ET and RC classes with SNFH and then removes small SNFH objects, achieves the best average Dice on SNFH, ET, TC, and WT and the best average HD95 on ET and TC, but it receives the worst average global rank in both the five-fold cross-validation (2.5631) and the validation set (2.0126). The unprocessed baseline ranks best on validation (1.9918) and is chosen as the final challenge entry because it is consistently strong, while Strategy 3 helps a subset of outlier cases and degrades a larger number of patients. The paper's broad claim is that aggregate metric gains can coexist with worse per-patient ranking, so consistency across the cohort should be weighed alongside mean scores.
Load-bearing premise
The conclusion depends on the BraTS rank-based protocol being the right arbiter of segmentation quality, and on the small validation-set rank gaps between the baseline and the postprocessed strategies being stable and meaningful, even though no significance test is reported.
Editorial extensions
If this is right
- For BraTS 2025, the authors submit the baseline model with no postprocessing as their final entry.
- A class-specific pipeline like Strategy 3 can improve average Dice and HD95 on enhancing tumor and whole-tumor classes while still ranking last because it harms more patient cases than it helps.
- A targeted, single-class fix (Strategy 1) ranks best in cross-validation, indicating that narrow postprocessing can be safer than broad multi-class pipelines.
- Rank-based evaluation rewards consistency across patients and metrics, so teams that optimize only mean metrics may pick a strategy that performs worse on the leaderboard.
Reading between the lines
- If the challenge scoring pooled mean Dice and HD95 across patients instead of averaging per-patient ranks, Strategy 3 would likely be the recommended entry, so the practical recommendation is tied to the choice of ranking protocol.
- The same JSON-declared postprocessing strategies could be applied to other segmentation tasks whose base models are less heavily trained or less regularized; in those settings the headroom for postprocessing may be larger than the near-zero headroom observed here.
- A paired significance test over per-patient ranks, or a bootstrap interval, would tell whether the baseline's advantage is a real effect or within noise; the paper does not report one.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes the Medical Imaging Segmentation Toolkit (MIST) and evaluates three postprocessing strategies for the BraTS 2025 pre- and post-treatment glioma segmentation challenge. The authors train a Pocket nnU-Net with deep supervision and residual blocks on the BraTS training data, produce baseline predictions via five-fold cross-validation with test-time augmentation and ensembling, and then apply three postprocessing strategies composed of small-object removal, largest-component retention, hole filling, and class-specific label replacement. They compare baseline and strategies under the BraTS ranking protocol on the validation set and the cross-validation data. They report that Strategy 3 improves average Dice/HD95 for several classes but achieves the worst average global rank, while Strategy 1 is best in cross-validation and the unprocessed baseline is best on the validation set. Based on this, they elect to submit the baseline model as their final entry.
Significance. The paper provides a useful, fully described modular postprocessing framework and a cautionary demonstration that mean metric improvements can fail to improve rank-based evaluation. The training and inference details are specific enough to reproduce, and the open-source toolkit is a real contribution. The experimental design is empirical and does not contain circular reasoning; rankings are computed with a protocol close to the official BraTS ranking. However, the central conclusion that postprocessing 'does not translate into better rankings' rests on rank differences smaller than 0.1 on a 1–4 scale and is not accompanied by any uncertainty or significance analysis. Moreover, the claim of 'consistent strength' of the baseline is not supported by the cross-validation table, where Strategy 1 ranks first. These issues currently limit the strength of the conclusions.
major comments (4)
- [Section 4, Tables 3a–3b] The central decision claim, 'the consistent strength of the baseline across all evaluation settings' (Section 4), is contradicted by Table 3a: in five-fold cross-validation the baseline has the highest (worst) average global rank among the four strategies (2.4917), behind Strategy 1 (2.4703) and Strategy 2 (2.4749). Only the validation set (Table 3b) shows the baseline first. Please report per-fold rankings to show whether the baseline is consistently first, or explicitly acknowledge that the cross-validation evidence favors Strategy 1 and justify why the validation set is decisive for the final submission.
- [Tables 3a and 3b, Section 2.3] The rank differences underpinning the main conclusion are very small (e.g., 1.9918 versus 1.9955 on the validation set, and 2.4703 versus 2.4917 in cross-validation) on a 1–4 scale, and no uncertainty is reported. Because the ranking is per patient, metric, and class and 'ties are resolved in favor of the earlier-listed strategies,' the ordering may be sensitive to tie-breaking and to a few cases. Please add at least per-patient average rank distributions or a paired significance test (e.g., Wilcoxon signed-rank) to establish that the baseline-first ordering is not within the noise floor.
- [Section 4] The statement that Strategy 3 'degrades performance on a larger number of patients' and 'harms more predictions than it helps' is not supported by any patient-level counts in the manuscript. The only counts provided are for Strategy 1 on the RC class (improves 589, leaves 2079, degrades 207 for Dice). Please report the corresponding patient-level counts for Strategy 3 on the classes it modifies, or revise the claim to reflect the evidence that is actually presented.
- [Section 2.2] The size thresholds defining Strategies 1–3 (RC/ET = 100 voxels, SNFH = 64 voxels) are given without sensitivity analysis. Because the rank ordering in Tables 3a–3b could depend on these hand-selected values, it is unclear whether the conclusion that postprocessing does not improve ranking survives small threshold changes. Please add a brief sensitivity study, for instance varying the RC threshold in Strategy 1 over a plausible range and reporting the resulting average ranks.
minor comments (4)
- [Figure 2] The x-axis is not labeled; please add epoch labels and clarify which fold is plotted, since the text references drops at epochs 1,000 and 6,000.
- [References [3] and [4]] References [3] and [4] appear to cite the same paper (Segmentation labels for the pre-operative scans of the TCGA-LGG collection); please merge them or differentiate them appropriately.
- [Section 4] The paper states that MIST is available on GitHub or PyPI but does not provide a URL or version identifier; adding these would aid reproducibility.
- [Table 1] The caption notes that ties are resolved in favor of the earlier-listed strategies, but this tie-breaking rule is not discussed in the text; its potential effect on the reported rankings should be acknowledged.
Circularity Check
No significant circularity: the central claims rest on external BraTS evaluation and independent ranking, with only minor non-load-bearing self-citation.
full rationale
The paper's central claims are empirical and self-contained against an external benchmark. The baseline model is trained on BraTS training data and evaluated with the BraTS ranking protocol; the postprocessing strategies are defined as concrete transforms (small-object removal thresholds, largest-component retention, hole filling) applied to the model's own predictions. No prediction quantity is defined in terms of the result it is said to predict, and no fitted parameter is renamed as a forecast. The choice to submit the baseline is justified by measured global ranks (e.g., Table 3b), not by a quantity that was constructed to produce that rank. The self-citations to the authors' MIST and PocketNet work (refs 7 and 8) support the toolkit and architecture but are not load-bearing for the validity of the evaluation: the Dice and HD95 results would stand even if those references were removed, since they are computed on held-out data according to the BraTS protocol. A caveat noted in the manuscript and in the surrounding discussion is that the rank differences separating baseline from Strategy 1 on the validation set (1.9918 vs. 1.9955) are extremely small and are reported without significance tests or confidence intervals; this is a statistical robustness concern, not circular reasoning. Similarly, the claim that Strategy 3 'harms more predictions than it helps' is an inference from rank averages and per-case counts, not a definitional equivalence. Accordingly, no circular step meeting the evidentiary standard can be quoted, and the score reflects at most the presence of minor, non-load-bearing self-citation rather than any reduction of the derivation to its inputs.
Assumptions & free parameters
free parameters (3)
- RC size threshold =
100 voxels
- ET size threshold =
100 voxels
- SNFH size threshold =
64 voxels
assumptions (5)
- domain assumption BraTS ground truth segmentations are accurate and the public challenge labels are a valid reference.
- domain assumption The Dice score and 95th percentile Hausdorff distance are appropriate segmentation quality metrics.
- domain assumption The official BraTS ranking protocol, which averages per-patient ranks across metrics and classes, is the correct criterion for 'better' segmentation.
- domain assumption The five-fold cross-validation on the training set is representative of the validation and test distributions.
- domain assumption The postprocessing transforms (connected components, hole filling, morphological closing) are implemented correctly in MIST.
Cite this review
Pith. "Pith review of Pre- and Post-Treatment Glioma Segmentation with the Medical Imaging Segmentation Toolkit." pith.science (2026). https://pith.science/paper/IO3SS7L6
@misc{pith2026250719626,
author = {Pith},
title = {Pith review of: Pre- and Post-Treatment Glioma Segmentation with the Medical Imaging Segmentation Toolkit},
year = {2026},
howpublished = {\url{https://pith.science/paper/IO3SS7L6}},
note = {Machine review of arXiv:2507.19626}
}
read the original abstract
Medical image segmentation continues to advance rapidly, yet rigorous comparison between methods remains challenging due to a lack of standardized and customizable tooling. In this work, we present the current state of the Medical Imaging Segmentation Toolkit (MIST), with a particular focus on its flexible and modular postprocessing framework designed for the BraTS 2025 pre- and post-treatment glioma segmentation challenge. Since its debut in the 2024 BraTS adult glioma post-treatment segmentation challenge, MIST's postprocessing module has been significantly extended to support a wide range of transforms, including removal or replacement of small objects, extraction of the largest connected components, and morphological operations such as hole filling and closing. These transforms can be composed into user-defined strategies, enabling fine-grained control over the final segmentation output. We evaluate three such strategies - ranging from simple small-object removal to more complex, class-specific pipelines - and rank their performance using the BraTS ranking protocol. Our results highlight how MIST facilitates rapid experimentation and targeted refinement, ultimately producing high-quality segmentations for the BraTS 2025 challenge. MIST remains open source and extensible, supporting reproducible and scalable research in medical image segmentation.
Figures
Reference graph
Works this paper leans on
-
[1]
Nature communications 13(1), 4128 (2022)
Antonelli, M., Reinke, A., Bakas, S., Farahani, K., Kopp-Schneider, A., Landman, B.A., Litjens, G., Menze, B., Ronneberger, O., Summers, R.M., et al.: The medical segmentation decathlon. Nature communications 13(1), 4128 (2022)
work page 2022
-
[2]
arXiv preprint arXiv:2107.02314 (2021)
Baid, U., Ghodasara, S., Mohan, S., Bilello, M., Calabrese, E., Colak, E., Farahani, K., Kalpathy-Cramer, J., Kitamura, F.C., Pati, S., et al.: The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classifica- tion. arXiv preprint arXiv:2107.02314 (2021)
arXiv 2021
-
[4]
The cancer imaging archive (2017)
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J., Freymann, J., Farahani, K., Davatzikos, C.: Segmentation labels for the pre-operative scans of the tcga-lgg collection. The cancer imaging archive (2017)
work page 2017
-
[5]
Scientific data 4(1), 1–13 (2017)
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S., Freymann, J.B., Farahani, K., Davatzikos, C.: Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. Scientific data 4(1), 1–13 (2017)
2017
-
[6]
In: European conference on computer vision
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin- unet: Unet-like pure transformer for medical image segmentation. In: European conference on computer vision. pp. 205–218. Springer (2022)
2022
-
[7]
IEEE transactions on medical imaging 42(4), 1172–1184 (2022)
Celaya, A., Actor, J.A., Muthusivarajan, R., Gates, E., Chung, C., Schellingerhout, D., Riviere, B., Fuentes, D.: Pocketnet: A smaller neural network for medical image analysis. IEEE transactions on medical imaging 42(4), 1172–1184 (2022)
work page 2022
-
[8]
arXiv preprint arXiv:2407.21343 (2024)
Celaya, A., Lim, E., Glenn, R., Mi, B., Balsells, A., Schellingerhout, D., Netherton, T., Chung, C., Riviere, B., Fuentes, D.: Mist: A simple and scalable end-to-end 3d medical imaging segmentation framework. arXiv preprint arXiv:2407.21343 (2024)
arXiv 2024
-
[9]
arXiv preprint arXiv:2302.03868 (2023)
Celaya, A., Riviere, B., Fuentes, D.: A generalized surface loss for reduc- ing the Hausdorff distance in medical imaging segmentation. arXiv preprint arXiv:2302.03868 (2023)
arXiv 2023
Show all 26 references
-
[10]
arXiv preprint arXiv:2102.04306 (2021)
Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)
2021 arXiv
-
[11]
Medical image analysis 76, 102311 (2022)
Chen, S., Gamechi, Z.S., Dubost, F., van Tulder, G., de Bruijne, M.: An end-to-end approach to segmentation in medical images with cnn and posterior-crf. Medical image analysis 76, 102311 (2022)
2022
-
[12]
In: International conference on medical image computing and computer-assisted intervention
C ¸ i¸ cek,¨O., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. pp. 424–432. Springer (2016)
2016
-
[13]
In: International MICCAI brainlesion workshop
Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: International MICCAI brainlesion workshop. pp. 272–284. Springer (2021)
2021
-
[14]
Nature methods 18(2), 203–211 (2021)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
2021
-
[15]
In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 6th International Workshop, BrainLes 2020, Held in 10 A
Isensee, F., J¨ ager, P.F., Full, P.M., Vollmuth, P., Maier-Hein, K.H.: nnU-Net for brain tumor segmentation. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 6th International Workshop, BrainLes 2020, Held in 10 A. Celaya et al. Conjunction wi...
2021
-
[16]
arXiv preprint arXiv:2404.09556 (2024)
Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., Jaeger, P.F.: nnu-net revisited: A call for rigorous validation in 3d medical image segmen- tation. arXiv preprint arXiv:2404.09556 (2024)
2024 arXiv
-
[17]
Nature machine intelligence 5(7), 799–810 (2023)
Karargyris, A., Umeton, R., Sheller, M.J., Aristizabal, A., George, J., Wuest, A., Pati, S., Kassem, H., Zenk, M., Baid, U., et al.: Federated benchmarking of medi- cal artificial intelligence with MedPerf. Nature machine intelligence 5(7), 799–810 (2023)
2023
-
[18]
Medical image analysis 67, 101851 (2021)
Kervadec, H., Bouchtiba, J., Desrosiers, C., Granger, E., Dolz, J., Ayed, I.B.: Boundary loss for highly unbalanced segmentation. Medical image analysis 67, 101851 (2021)
2021
-
[19]
IEEE Access 9, 157035–157042 (2021)
Kim, J., Kang, S.: Model-agnostic post-processing based on recursive feedback for medical image segmentation. IEEE Access 9, 157035–157042 (2021)
2021
-
[20]
arXiv preprint arXiv:1608.03983 (2016)
Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
2016 arXiv
-
[21]
IEEE transactions on medical imaging 34(10), 1993–2024 (2014)
Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging 34(10), 1993–2024 (2014)
2014
-
[22]
Scientific Reports 14(1), 20988 (2024)
Patel, N., Celaya, A., Eltaher, M., Glenn, R., Savannah, K.B., Brock, K.K., Sanchez, J.I., Calderone, T.L., Cleere, D., Elsaiey, A., et al.: Training robust t1- weighted magnetic resonance imaging liver segmentation models using ensembles of datasets with different contrast pr...
2024
-
[23]
In: International Conference on Medical image computing and computer-assisted intervention
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedi- cal image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
2015
-
[24]
In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition
Tang, Y., Yang, D., Li, W., Roth, H.R., Landman, B., Xu, D., Nath, V., Hatamizadeh, A.: Self-supervised pre-training of swin transformers for 3d med- ical image analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 20730–20740 (2022)
2022
-
[25]
Advances in neural information pro- cessing systems 30 (2017)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in neural information pro- cessing systems 30 (2017)
2017
-
[26]
arXiv preprint arXiv:2405.18368 (2024)
de Verdier, M.C., Saluja, R., Gagnon, L., LaBella, D., Baid, U., Tahon, N.H., Foltyn-Dumitru, M., Zhang, J., Alafif, M., Baig, S., et al.: The 2024 brain tumor segmentation (BraTS) challenge: Glioma segmentation on post-treatment MRI. arXiv preprint arXiv:2405.18368 (2024)
2024 arXiv
-
[27]
arXiv preprint arXiv:2109.03201 (2021)
Zhou, H.Y., Guo, J., Zhang, Y., Yu, L., Wang, L., Yu, Y.: nnformer: Interleaved transformer for volumetric segmentation. arXiv preprint arXiv:2109.03201 (2021)
2021 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.