REVIEW 4 major objections 5 minor 1 cited by
Automatic brain tumor segmentation in 2D intra-operative ultrasound images using magnetic resonance imaging tumor annotations
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read MRI tumor annotations can replace ultrasound tumor annotations for training brain-tumor segmentation models.
desk verdict Useful, reproducible iUS segmentation work, but the headline 'substitute' claim rests on an underpowered non-significant difference and a test-derived cutoff; needs an equivalence test or independent cohort. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the registration-pseudo-label pipeline: rigid registration software transfers MRI tumor annotations into the space of the unannotated ultrasound volumes, producing pseudo-labels that are sliced along three perpendicular directions into 2D tumor-containing images. The segmentation models are trained with a standardized self-configuring deep learning framework in its 2D configuration, chosen for its automatic pre-processing, hyperparameters, and post-processing. A tumor area cut-off (200 $mm^{2}$) is applied to discard slices whose pseudo-labeled tumor is very small, on the rationale that these slices are most likely to suffer from registration mismatch and are hardest to learn from. This pipeline is what lets the paper convert 180 MRI-annotated volumes into training data without any manual ultrasound labeling.
What would settle it
Take a set of ultrasound volumes with both manual expert tumor annotations and registered MRI-derived pseudo-labels, and compute per-slice Dice overlap between the two label sources. If a substantial fraction of slices, especially small ones, have near-zero overlap, then the pseudo-labels are too noisy to be called substitutes; alternatively, a training comparison on a larger multi-expert test set in which MRI-only labels produce significantly lower Dice than manual ultrasound labels would refute the equivalence claim.
Extended reading notes
Core claim
The paper's central claim is that tumor annotations drawn on pre-operative MRI scans, transferred to intra-operative ultrasound images by rigid registration, can serve as training labels for a deep learning segmentation model in place of manual iUS annotations. In the head-to-head experiment, the MRI-only model, the iUS-only model, and the combined model had Dice scores of 0.58, 0.59, and 0.62 respectively, with no statistically significant pairwise differences (P > 0.0085, Bonferroni) and very small effect sizes. The best model matched an expert neurosurgeon's performance on tumors larger than 200 $mm^{2}$, while all models struggled on small tumors. The authors also report that filtering out the smallest tumor slices (below about 200 $mm^{2}$) significantly improved performance, which they attribute to registration inaccuracy and class imbalance hurting the network when small, poorly aligned labels are included.
Load-bearing premise
Rigid registration transfers MRI tumor outlines onto the corresponding ultrasound images accurately enough for those transferred outlines to be trustworthy training targets; the paper's own finding that small tumors harm training suggests this premise holds best for larger tumors.
Editorial extensions
If this is right
- Clinicians and researchers can build iUS segmentation models from MRI-annotated retrospective data, avoiding the bottleneck of manual ultrasound annotation.
- Combining MRI-derived labels with iUS labels does not hurt performance, so MRI labels can be used to expand existing small iUS datasets.
- Filtering training slices by tumor area improves model performance, suggesting a quality-over-quantity strategy for pseudo-labeled data.
- Models reach expert-level Dice on large tumors (above 200 mm^2), indicating clinical usefulness for localizing bulk tumor, though not for small residues.
Reading between the lines
- If registration accuracy improves through affine or nonlinear methods, the small-tumor performance gap may close, since the paper's own results implicate misaligned small pseudo-labels as a noise source.
- The label-substitution idea may transfer to other modality pairs where annotations are scarce in one imaging modality but abundant in another, e.g., CT-to-ultrasound or histology-to-MRI.
- The tumor area cut-off finding suggests that active-learning or confidence-based filtering of pseudo-labels could further improve training efficiency beyond a fixed size threshold.
- A larger multi-expert test cohort would be needed to confirm that the statistical equivalence holds beyond the six-patient test set.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes training a 2D nnU-Net for brain tumor segmentation in intra-operative ultrasound (iUS) using tumor annotations transferred from pre-operative MRI via rigid registration, instead of requiring manual iUS annotations. The authors compare models trained on MRI-derived pseudo-labels, manual iUS labels, a combination of both, and an expert annotator, reporting Dice, precision, recall, FNR, and positive-prediction rates. They find no statistically significant differences between the MRI-label model, the iUS-label model, the combined model, and the expert annotator on a six-patient test set, and conclude that MRI tumor annotations can be used as a substitute for iUS annotations. A secondary finding is that increasing the tumor-area cutoff in the training set improves performance, with 200 mm2 selected as the best cutoff.
Significance. If the central claim is valid, the practical value is high: it would let researchers train iUS segmentation models using the much larger supply of MRI-annotated images, reducing reliance on scarce expert-drawn iUS labels. The paper has concrete strengths: it uses public datasets (RESECT, RESECT-SEG, CuRIOUS-SEG, ReMIND), shares the trained models and code, applies clustered regression to account for within-patient correlation, corrects for multiple comparisons, and openly acknowledges the small test set. However, the central 'substitute' claim is currently supported only by a failure to reject the null in an underpowered six-patient test set, and the cutoff selection procedure uses the evaluation set. These are load-bearing issues that need to be addressed before the conclusion can be accepted.
major comments (4)
- [Section 3.2, Table 2, and Section 4] The central claim that MRI annotations can substitute for iUS annotations rests entirely on non-significant p-values (e.g., p=0.823 for MRI 200 vs US 200) and a small Cohen's d (0.005) on a test set of only 6 patients. A non-significant difference in an underpowered sample is not evidence of equivalence; the paper itself acknowledges the n=6 limitation in the Discussion. To support the 'substitute' claim, the authors need to report a pre-specified non-inferiority margin (e.g., based on inter-observer variability or a clinically acceptable Dice difference), an equivalence or non-inferiority test, and a confidence interval for the difference in Dice. Without this, the abstract's conclusion overstates what the data show.
- [Sections 2.4.1 and 3.1] The tumor-area cutoff of 200 mm2 was selected based on Experiment 1, whose test set consists of all 29 iUS-annotated volumes, including the 6 patients later used as the Experiment 2 test set. The same 6 patients are therefore used both to choose the cutoff and to evaluate the models with that cutoff. This makes the claim that 'a tumor area cut-off around 200 mm2 provides the best results' partly self-confirmatory with respect to the Experiment 2 test patients. The authors should either select the cutoff on a separate validation set (e.g., cross-validation within the MRI-annotated data only) or explicitly analyze the sensitivity of the Experiment 2 conclusions to the cutoff choice.
- [Section 2.2] The registration that transfers MRI annotations to iUS images is validated only by visual inspection ('All registrations were visually inspected to ensure adequate alignment'). Because the quality of these pseudo-labels is the central premise of the method, the paper needs a quantitative assessment of registration accuracy, for example by measuring overlap or surface distance between registered MRI annotations and available manual iUS annotations, or by reporting landmark/error statistics on a subset. The finding that excluding small tumors improves training performance is consistent with registration errors disproportionately affecting small structures, so without quantitative registration validation the 'substitute' claim is not fully supported.
- [Section 3.2 and Table 1] The comparison to 'inter-observer variability' is based on a single additional annotator (the first author with neurosurgeon adjustment), not on a multi-observer study. The term 'inter-observer variability' normally implies agreement statistics between multiple independent raters. The authors should either rename this comparison (e.g., 'comparison to an expert annotator') or, if claiming inter-observer variability, include multiple annotators and report pairwise agreement metrics.
minor comments (5)
- [Abstract and Conclusion] The abstract and conclusion state that MRI annotations 'can be used as a substitute' without qualifying the statistical strength; this should be tempered to reflect the equivalence-test limitation (e.g., 'no significant difference was found' rather than 'can be used as a substitute').
- [Figure 2] The box plots for nine models are visually crowded; using distinct colors or a small-multiple layout would make the trends across cutoff values easier to read.
- [Section 2.1] The relationship between the 29 iUS-annotated patients and the 180 MRI-annotated cases should be stated more explicitly: it should be clear whether any patients appear in both groups, because Experiment 1 uses all 29 iUS-annotated volumes for testing while Experiment 2 uses only 6 of them.
- [Section 2.4.2] The sentence 'For the MRI+US 200 model, 8 3D images from the MRI annotated data were excluded from the training set due to poor image quality' does not specify whether these exclusions were made before or after the cutoff selection, and whether the same exclusions apply to the MRI 200 model in Experiment 2. Please clarify.
- [Table 2] The table reports p-values and Cohen's d but does not report confidence intervals for the pairwise Dice differences; adding these would directly support the authors' equivalence interpretation.
Circularity Check
No derivation-level circularity; the only mild circular step is choosing the 200 mm2 tumor-area cutoff on the same test set later used to evaluate it, making the cutoff-improvement result partly self-confirmatory.
-
fitted input called prediction
[Section 2.4.1, Section 2.4.2, Table 1]
"All models were evaluated on all tumor-containing slices of the 29 3D volumes of the iUS annotated data resulting in 14 107 test slices. ... The 23 patients from RESECT were used for training, and the 6 patients from the test set of the CuRIOUS-SEG challenge were used as a test set, resulting in 2259 test 2D slices. ... we found that a tumor area cut-off value around 200 mm2 seems to provide the best and most stable results ... This value was therefore used in the second experiment."
The 200 mm2 cutoff is selected by ranking the nine cutoff models on all 29 iUS volumes, which include the same 6 CuRIOUS-SEG patients later designated as the Experiment 2 test set. The reported finding that removing small tumors improves results is therefore a restatement of the ranking that defined the cutoff, and the subsequent MRI-vs-US label comparison is evaluated on patients that contributed to that selection. This is test-set selection bias rather than an equation-level equivalence, so it is a mild, partly self-confirmatory step; the central label-origin comparison still has independent empirical content.
full rationale
The paper is an empirical study rather than a derivation chain: MRI annotations are registered to iUS and used as training labels, and the resulting nnU-Net models are compared on manual iUS annotations. I found no step where a prediction is equivalent to its input by construction. There is no self-definitional equation, no fitted parameter renamed as a prediction in the central claim, no uniqueness theorem imported from the authors, and no ansatz smuggled in via citation. The use of RESECT, RESECT-SEG, and Raidionics involves author overlap, but these are public datasets and an open software tool used as experimental inputs, not unverified load-bearing theorems; the empirical result would stand or fall on the measured Dice scores independently of those citations. The main weakness is statistical, not circular: the 'MRI annotations can substitute for iUS annotations' conclusion rests on a non-significant difference on only 6 test patients, and the paper itself acknowledges that 'the small patient sample (n=6) makes the performance highly sensitive to the test patients.' The one genuine self-confirmatory element is the tumor-area cutoff selection in Experiment 1, which is chosen using the same patients later used in Experiment 2; this is selection bias and mildly circular for the cutoff-improvement claim, but it does not reduce the central label-origin comparison to its inputs. Overall circularity is therefore low.
Assumptions & free parameters
free parameters (1)
- Tumor area cut-off (200 mm2) =
200 mm2
assumptions (4)
- domain assumption Rigid registration adequately aligns pre-operative MRI tumor annotations with intra-operative ultrasound images.
- domain assumption Tumor annotations in MRI, including Raidionics auto-segmentations, are accurate enough to serve as training labels.
- domain assumption nnU-Net's self-configuration yields appropriate pre-processing, hyperparameters and post-processing for each training set.
- domain assumption The CuRIOUS-SEG ground truth annotations are accurate reference labels for evaluation.
Cite this review
Pith. "Pith review of Automatic brain tumor segmentation in 2D intra-operative ultrasound images using magnetic resonance imaging tumor annotations." pith.science (2026). https://pith.science/paper/I5MSCNM2
@misc{pith2026241114017,
author = {Pith},
title = {Pith review of: Automatic brain tumor segmentation in 2D intra-operative ultrasound images using magnetic resonance imaging tumor annotations},
year = {2026},
howpublished = {\url{https://pith.science/paper/I5MSCNM2}},
note = {Machine review of arXiv:2411.14017}
}
abstract
Automatic segmentation of brain tumors in intra-operative ultrasound (iUS) images could facilitate localization of tumor tissue during resection surgery. The lack of large annotated datasets limits the current models performances. In this paper, we investigated the use of tumor annotations in magnetic resonance imaging (MRI) scans, which are more accessible than annotations in iUS images, for training of deep learning models for iUS brain tumor segmentation. We used 180 annotated MRI scans with corresponding unannotated iUS images, and 29 annotated iUS images. Image registration was performed to transfer the MRI annotations to the corresponding iUS images before training the nnU-Net model with different configurations of the data and label origins. The results showed no significant difference in Dice score for a model trained with only MRI annotated tumors compared to models trained with only iUS annotations and both, and to expert annotations, indicating that MRI tumor annotations can be used as a substitute for iUS tumor annotations to train a deep learning model for automatic brain tumor segmentation in iUS images. The best model obtained an average Dice score of $0.62\pm0.31$, compared to $0.67\pm0.25$ for an expert neurosurgeon, where the performance on larger tumors were similar, but lower for the models on smaller tumors. In addition, the results showed that removing smaller tumors from the training sets improved the results. The main models are available here: https://github.com/mathildefaanes/us_brain_tumor_segmentation/tree/main
Forward citations
Cited by 1 Pith paper
-
Real-Time Brain Tumor Detection in Intraoperative Ultrasound Using YOLO11: From Model Training to Deployment in the Operating Room
A YOLO11-based detector identifies brain tumors in intraoperative ultrasound in real time and was qualitatively validated in 15 consecutive surgeries.
Reference graph
Works this paper leans on
-
[1]
Clinical and Translational Oncology 19(8):931–944
Delgado-L´ opez PD, Corrales-Garc ´ ıa EM, Martino J, Lastra-Aras E, Due˜ nas-Polo MT (2017) Diffuse low-grade glioma: a review on the new molecular classification, natural history and current management strategies. Clinical and Translational Oncology 19(8):931–944. https://doi.org/10.1007/s12094-017-1631-4
-
[2]
Advanced Biomedical Research 4:4
Kheirollahi M, Dashti S, Khalaj Z, Nazemroaia F, Mahzouni P (2015) Brain tumors: Special characters for research and banking. Advanced Biomedical Research 4:4. https://doi.org/10.4103/2277-9175.148261 11
-
[3]
Sanai N, Berger MS (2008) GLIOMA EXTENT OF RESECTION AND ITS IMPACT ON PATIENT OUTCOME. Neurosurgery 62(4):753. https://doi.org/ 10.1227/01.neu.0000318159.21731.cf
arXiv 2008
-
[4]
Journal of Neuroimaging 27(1):5–15
Sastry R, Bi WL, Pieper S, Frisken S, Kapur T, Wells III W, Golby AJ (2017) Applications of Ultrasound in the Resection of Brain Tumors. Journal of Neuroimaging 27(1):5–15. https://doi.org/10.1111/jon.12382
-
[5]
Journal of Neuro-Oncology 167(3):387–396
Cepeda S, Garc ´ ıa-Garc ´ ıa S, Arrese I, Sarabia R (2024) Non-navigated 2D intra- operative ultrasound: An unsophisticated surgical tool to achieve high standards of care in glioma surgery. Journal of Neuro-Oncology 167(3):387–396. https: //doi.org/10.1007/s11060-024-04614-5
-
[6]
phdthesis, Universit´ e Grenoble Alpes [2020-....] ; Vanderbilt university (Nashville, Tennessee)
Carton FX (2021) Image segmentation and registration using machine learning for brain shift compensation in image-guided neurosurgery. phdthesis, Universit´ e Grenoble Alpes [2020-....] ; Vanderbilt university (Nashville, Tennessee)
work page 2021
-
[7]
Springer Nature Switzerland, Cham, https://doi.org/10.1007/978-3-031-27324-7
Xiao Y, Yang G, Song S (eds) (2023) Lesion Segmentation in Surgical and Diag- nostic Applications: MICCAI 2022 Challenges, CuRIOUS 2022, KiPA 2022 and MELA 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18–22, 2022, Proceedings, Lecture Notes in Computer Science, vol 13648. Springer Nature Switzerland, Cham, https://doi.org/10.1007/978-3...
-
[8]
In: Xiao Y, Yang G, Song S (eds) Lesion Segmen- tation in Surgical and Diagnostic Applications
Qayyum A, Mazher M, Niederer S, Razzak I (2023) Segmentation of Intra- operative Ultrasound Using Self-supervised Learning Based 3D-ResUnet Model with Deep Supervision. In: Xiao Y, Yang G, Song S (eds) Lesion Segmen- tation in Surgical and Diagnostic Applications. Springer Nature Switzerland, Cham, Lecture Notes in Computer Science, pp 55–62, https://doi....
work page 2023
Show all 24 references
-
[9]
Medical Physics 44(7):3875–3882
Xiao Y, Fortin M, Unsg ˚ ard G, Rivaz H, Reinertsen I (2017) REtroSpective Eval- uation of Cerebral Tumors (RESECT): A clinical database of pre-operative MRI and intra-operative ultrasound in low-grade glioma surgeries. Medical Physics 44(7):3875–3882. https://doi.org/10.1002/mp.12268
2017 doi
-
[10]
https://doi.org/10.48550/ arXiv.2207.07494, arXiv:2207.07494 [physics]
Behboodi B, Carton FX, Chabanas M, De Ribaupierre S, Solheim O, Munkvold BKR, Rivaz H, Xiao Y, Reinertsen I (2022) RESECT-SEG: Open access annota- tions of intra-operative brain tumor ultrasound images. https://doi.org/10.48550/ arXiv.2207.07494, arXiv:2207.07494 [physics]
-
[11]
Medical Image Computing and Computer Assisted Intervention – MICCAI 2024
Dorent R, Torio E, Haouchine N, Galvin C, Frisken S, Golby A, Kapur T, Wells WM (2024) Patient-specific real-time segmentation in trackerless brain ultra- sound. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024
2024
-
[12]
https://doi.org/10.1101/2023.09.14.23295596, pages: 2023.09.14.23295596
Juvekar P, Dorent R, Kogl F, Torio E, Barr C, Rigolo L, Galvin C, Jowkar N, Kazi A, Haouchine N, Cheema H, Navab N, Pieper S, Wells WM, Bi WL, 12 Golby A, Frisken S, Kapur T (2023) ReMIND: The Brain Resection Multi- modal Imaging Database. https://doi.org/10.1101/2023.09.14.23...
2023 doi
-
[13]
Scientific Reports 13(1):15570
Bouget D, Alsinan D, Gaitan V, Helland RH, Pedersen A, Solheim O, Reinertsen I (2023) Raidionics: an open software for pre- and postoperative central ner- vous system tumor segmentation and standardized reporting. Scientific Reports 13(1):15570. https://doi.org/10.1038/s41598-...
2023 doi
-
[14]
https://curious2022.grand-challenge.org/
CuRIOUS 2022 (2022) Brain shift with Intraoperative Ultrasound - Segmentation tasks - Grand Challenge. https://curious2022.grand-challenge.org/
2022
-
[15]
Neurosurgery 47(6):1373–1379; discussion 1379–1380
Gronningsaeter A, Kleven A, Ommedal S, Aarseth TE, Lie T, Lindseth F, Langø T, Unsg ˚ ard G (2000) SonoWand, an ultrasound-based neuronavigation system. Neurosurgery 47(6):1373–1379; discussion 1379–1380
2000
-
[16]
International Journal of Computer Assisted Radiology and Surgery 11(4):505–519
Askeland C, Solberg OV, Bakeng JBL, Reinertsen I, Tangen GA, Hofstad EF, Iversen DH, V ˚ apenstad C, Selbekk T, Langø T, Hernes TAN, Olav Leira H, Unsg ˚ ard G, Lindseth F (2016) CustusX: an open-source research platform for image-guided therapy. International Journal of Compu...
2016 doi
-
[17]
https://www.kreftregisteret.no/en/The-Registries/clinical-registries/ Quality-registry-for-brain-tumours/
Cancer Registry of Norway (2023) Norwegian registry of brain and spinal cord tumours. https://www.kreftregisteret.no/en/The-Registries/clinical-registries/ Quality-registry-for-brain-tumours/
2023
-
[18]
https://www.imfusion.com/ products/imfusion-suite
ImFusion (2018) ImFusion - ImFusion Suite. https://www.imfusion.com/ products/imfusion-suite
2018
-
[19]
In: Crimi A, Bakas S (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries
Isensee F, J¨ ager PF, Full PM, Vollmuth P, Maier-Hein KH (2021) nnU-Net for Brain Tumor Segmentation. In: Crimi A, Bakas S (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Springer International Publishing, Cham, Lecture Notes in Computer Sc...
2021 doi
-
[20]
Magnetic Resonance Imaging 30(9):1323–1341
Fedorov A, Beichel R, Kalpathy-Cramer J, Finet J, Fillion-Robin JC, Pujol S, Bauer C, Jennings D, Fennessy F, Sonka M, Buatti J, Aylward S, Miller JV, Pieper S, Kikinis R (2012) 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magnetic Resonance I...
2012 doi
-
[21]
Routledge, https://doi.org/https://doi.org/10.4324/9780203771587 13
Cohen J (1988) Statistical Power Analysis for the Behavioral Sciences (2nd ed. Routledge, https://doi.org/https://doi.org/10.4324/9780203771587 13
1988 doi
-
[22]
Journal of Modern Applied Statistical Methods https://doi.org/https://doi.org/10.56801/10.56801/v8.i.452
Sawilowsky S (2009) New effect size rules of thumb. Journal of Modern Applied Statistical Methods https://doi.org/https://doi.org/10.56801/10.56801/v8.i.452
2009 doi
-
[23]
Medical Image Analysis 96:103202
Jiao J, Zhou J, Li X, Xia M, Huang Y, Huang L, Wang N, Zhang X, Zhou S, Wang Y, Guo Y (2024) USFM: A universal ultrasound foundation model generalized to tasks and organs towards label efficient image analysis. Medical Image Analysis 96:103202. https://doi.org/10.1016/j.media....
2024
-
[24]
Acta Neurochirurgica 166(1):317
Weld A, Dixon L, Anichini G, Patel N, Nimer A, Dyck M, O’Neill K, Lim A, Giannarou S, Camp S (2024) Challenges with segmenting intraoperative ultra- sound for brain tumours. Acta Neurochirurgica 166(1):317. https://doi.org/10. 1007/s00701-024-06179-8 14
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.