More unlabelled data or label more data? A study on semi-supervised laparoscopic image segmentation

Bongjin Koo; Brian Davidson; Crispin Schneider; Danail Stoyanov; Maria R. Robu; Matthew J. Clarkson; Stijn van Laarhoven; Yipeng Hu; Yunguan Fu

REVIEW 3 major objections 4 minor 13 references

More unlabelled data or label more data? A study on semi-supervised laparoscopic image segmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mean-teacher semi-supervised training improves laparoscopic liver segmentation, and extra unlabelled frames can substitute for some expert labels.

desk verdict Useful data-planning study whose headline significance is unsupported by its own 13-fold design; the reported p-values require per-frame pooling. read the letter →

arxiv 1908.08035 v1 pith:RF2IVGKT submitted 2019-08-20 eess.IV cs.CVcs.LGstat.ML

Yunguan Fu , Maria R. Robu , Bongjin Koo , Crispin Schneider , Stijn van Laarhoven , Danail Stoyanov , Brian Davidson , Matthew J. Clarkson

show 1 more author

Yipeng Hu

This is my paper · ORCID

classification eess.IVcs.CVcs.LGstat.ML

keywords semi-supervisedlearningmeanteacherlaparoscopicimagesegmentationlivermedicaldataefficiencyDicescoreHausdorffdistance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how a clinical team should spend its data-collection effort: label more laparoscopic video frames, collect more unlabelled frames, or both. Using a mean-teacher semi-supervised network for liver segmentation, it reports higher Dice scores and Hausdorff distances than the supervised baseline, and it isolates that some of the gain comes from the training strategy alone rather than from the extra unlabelled data. It also shows that a model trained on half the labelled frames plus all available unlabelled frames outperforms a fully supervised model, giving a concrete example of the labelled-versus-unlabelled trade-off.

What carries the argument

The carrier of the argument is the mean-teacher training loop: a student network is trained with a supervised Dice loss on labelled frames plus a consistency loss between student and teacher predictions on all frames, while the teacher weights are an exponential moving average of the student weights. The paper adds random affine transformations as the input noise, applying one transformation to the student and a composed second transformation to the teacher, so the consistency target encourages invariance to spatial perturbations. This mechanism lets the same network and loss operate with any mix of labelled and unlabelled frames, and the controlled sweeps over labelled and unlabelled subset sizes are what separate the effect of extra data from the effect of the training strategy.

What would settle it

Re-run the Wilcoxon signed-rank test on the 13 per-patient median Dice scores instead of per-frame scores; if the difference is no longer below 0.05, the reported significance is an artefact of frame correlation.

Watch

Extended reading notes

Core claim

The central claim is that, on real laparoscopic liver video from 13 patients, the mean-teacher training strategy yields more accurate segmentation than the supervised baseline and that unlabelled frames can substitute for some labels. With all labels and all unlabelled frames, the median Dice score rose from 0.9594 (supervised) to 0.9646 (mean teacher) and median Hausdorff distance fell from 91.61 to 81.49 pixels, with reported Wilcoxon p-values below 0.001. A mean-teacher model using only 50% of the labels plus all unlabelled frames reached a median Dice of 0.9611, above the fully supervised 0.9594. The paper argues that part of this improvement is attributable to the semi-supervised training procedure itself, because mean teacher with no unlabelled data also generally outperformed supervised training.

Load-bearing premise

The statistical significance claims rest on treating each video frame as an independent observation in the Wilcoxon tests, even though frames from the same patient are highly correlated, so the reported p-values are likely over-optimistic.

Editorial extensions

If this is right

Because mean teacher improved segmentation even with zero unlabelled frames, comparisons between supervised and semi-supervised models should control for changes in network architecture and training schedule.
Adding unlabelled frames can make a half-labelled model beat a fully labelled one, so data-planning decisions in clinical imaging should be based on the relative cost of labelling versus acquisition.
The effect of unlabelled data is not monotonic in this dataset (e.g., MT(2%) fell from 0.9259 to 0.9202 with 6.25% unlabelled frames), so practical deployments need to validate at the specific operating point.
The leave-one-patient-out results on real surgical video support the feasibility of using these models for computer-assisted liver resection, if the reported statistical significance holds under a correct analysis unit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the training-strategy effect generalizes, then part of the reported superiority of many semi-supervised medical segmentation methods may come from a stronger training setup rather than from the semi-supervised mechanism; the same decomposition should be applied to other methods.
The non-monotonic response to unlabelled data suggests frame diversity, not sheer volume, is the active ingredient; a testable extension would stratify unlabelled frames by temporal spacing or procedure phase.
Because the reported Wilcoxon tests pool per-frame predictions, a per-patient analysis could shift the conclusions; future studies should pre-specify the analysis unit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a useful empirical study with a real data-planning question, but the headline significance claim as reported is statistically impossible under its own design. The Wilcoxon p-values in Table 1 (9.27e-28 and 4.40e-07) cannot be produced by a 13-pair leave-one-patient-out signed-rank test; the minimum two-sided p for 13 pairs is about 0.00024. So the tests must be pooling per-frame predictions and treating frames from the same video as independent, which the authors themselves acknowledge to be correlated (Sec 3.3) with "omitted inter-patient variation" (Sec 5). This means the "statistically significant higher accuracy" claim currently rests on a broken independence assumption. The median Dice difference (0.9646 vs 0.9594) is small but plausible; the problem is the p-value, not necessarily the direction.

What is genuinely new and useful: the decomposition of the mean-teacher improvement into a training-strategy effect (MT with 0% unlabelled vs supervised) and an unlabelled-data effect is a good idea, and the quantitative trade-off (MT(50%) with 100% unlabelled beats SL(100%)) is exactly the kind of evidence a clinical data-planning decision wants. The evaluation is on real laparoscopic video from 13 patients, held out by patient, with a sensible baseline. The paper is honest about its limitations.

Soft spots in proportion: The significance issue is load-bearing for the "significantly better" framing, but the raw medians can stand on their own once re-analysed with patient-level bootstrap or a mixed model. The single-training-run curves for the data-quantity sweeps mean the non-monotonic patterns (e.g., 6.25% unlabelled hurting MT(2%)) are likely noise; that is a minor issue relative to the p-value problem. The claim about being "first" to present such quantitative evidence is not especially important.

Bottom line: worth a serious referee, but only after the authors redo the statistics with the patient as the unit and report per-fold numbers. If they do that, the paper's practical message will likely survive. I'd bring it to reading group for the methodology discussion, but I wouldn't cite the significance claims as they stand.

Referee Report

3 major / 4 minor

Summary. The manuscript studies semi-supervised segmentation of the liver in laparoscopic video frames using the mean teacher paradigm, with a 13-fold leave-one-patient-out design on data from thirteen patients. It compares the supervised baseline (SL) with mean teacher (MT) models under varying amounts of labelled and unlabelled training data, reports Dice and 95th-percentile Hausdorff distances, and claims a significantly higher accuracy for MT, an improvement attributable partly to the training strategy rather than only to added unlabelled data, and a quantitative trade-off in which more unlabelled data can substitute for additional labels.

Significance. If the statistical claims survive re-analysis, this is a useful empirical contribution: real patient data, a medically relevant segmentation task, and an experimental design that varies labelled and unlabelled data in a controlled way. The 13-fold leave-one-patient-out evaluation is the right overall structure, and the attempt to decompose the mean teacher gain into training-strategy and data-addition components is methodologically valuable. However, the current significance tests violate the stated experimental design, so the main quantitative conclusions are not yet supported. The strength of the application and the practical relevance of the trade-off question justify revision rather than rejection.

major comments (3)

[Sec. 3.3, Table 1] The reported Wilcoxon p-values are impossible under the stated 13-fold leave-one-patient-out design. With 13 paired observations, the smallest possible two-sided signed-rank p-value is 2/2^13 ≈ 2.44e-4, yet Table 1 reports p = 9.27e-28 for Dice and p = 4.40e-07 for Hausdorff distance. These values can only arise if individual frames, which are highly correlated within each patient's video, are pooled as independent observations. The manuscript itself acknowledges high intra-patient correlation and omitted inter-patient variation in Sec. 3.3 and Sec. 5. Because the significance tests are the only quantitative support for the headline 'significantly higher accuracy' claim and for the subsequent trade-off discussion, the authors must re-analyse the data with the patient as the statistical unit (e.g., Wilcoxon on 13 per-fold summary scores, or a mixed-effects model on per-frame metrics with patient as a random effect), and report the resulting test statistics, effect sizes, and confidence intervals.
[Sec. 4, Fig. 5 and trade-off claim] The statement that 'using 100% unlabelled data, MT (50%) reached a Dice score of 0.9611 which was higher than SL (100%), 0.9594, depicting a scenario in which more unlabelled data achieve a comparable performance as adding labels would' is a central practical conclusion, but it is based on a single point estimate with no measure of uncertainty and no statistical test. The per-fold variability of these medians must be reported (e.g., boxplots or confidence intervals across the 13 folds), and the comparison between MT(50%)+100% unlabelled and SL(100%) should be tested using the correct statistical unit. Without this, the trade-off conclusion is not quantitatively supported.
[Sec. 4, 'Mean Teacher with Different Labelled Data Set Sizes'] The claim that MT 'consistently outperformed' SL across different labelled-data sizes is supported visually in Fig. 4 but no statistical tests are provided for these paired comparisons. Given the multiple labelled-data fractions and the correlation among frames, the authors should either supply formal paired tests at the patient level or explicitly state that these differences are descriptive only. As written, the 'consistent' claim outruns the evidence.

minor comments (4)

[Fig. 4 and Fig. 5] The x-axis tick labels such as '2 5 10%' and '2 5 100%' appear to omit percent signs after 2 and 5; they should read 2%, 25%, 100% (and similarly for the other panel).
[Sec. 3.2] The hyperparameter values are described as 'configured empirically without extensive tuning,' but no sensitivity analysis is given for lambda, alpha, or the L2 weight. A brief statement of how sensitive the main conclusions are to these choices would strengthen the study.
[Sec. 3.3] The manuscript does not state whether the random sampling of labelled-data fractions was repeated with multiple seeds. Since the fractions are small (e.g., 2% of 2,209 images), a single random draw could materially affect the results; reporting the seed or repeated draws would improve reproducibility.
[Sec. 4] The reported Hausdorff distance is given in pixels with a range of physical conversions (1.5 to 6.0 mm per 100 pixels), but the dependence on object-to-camera distance is not incorporated into the evaluation. A brief remark on how this affects interpretation would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical hold-out comparison with fixed hyperparameters; self-citations are baselines, not load-bearing premises.

full rationale

The paper is an empirical evaluation, not a derivation, so there is no chain in which an output is equivalent to an input by construction. The central comparison (mean teacher vs supervised baseline, Table 1) is measured on held-out patient folds, with hyperparameters fixed before evaluation; the mean-teacher-minus-unlabelled ablation is a genuine intervention rather than a fitted prediction. Self-citations (e.g., [6] as the prior supervised method) function as comparison baselines, not as load-bearing justification for the observed Dice differences. The paper's limitations about inter-patient correlation and unlabelled-data correlation are acknowledged statistical caveats, not evidence of circularity: a Wilcoxon assumption violation is a validity concern, not a definitional reduction. No equation is fitted to the target metric and then renamed as a prediction, and no uniqueness or ansatz is imported through self-citation. Hence no circular step is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claims are empirical; no new theory is derived. The main load-bearing choices are the hand-set hyperparameters of the mean teacher method and the statistical assumptions behind the significance tests.

free parameters (4)

unsupervised loss weight lambda = 0.1 * exp(-5 * max(1 - S/1000, 0)^2)
Hand-chosen schedule balancing supervised and consistency losses; reported as empirically configured without extensive tuning.
EMA decay alpha = 0.99 during ramp-up, 0.999 afterwards
Smoothing factor for teacher weights, fixed by hand.
L2 regularization weight = 1e-5
Fixed for all experiments; chosen without extensive tuning.
learning rate = 1e-4
Adam optimizer learning rate, fixed by experiment design.

assumptions (3)

domain assumption Student and teacher predictions are compared after random affine warps; this perturbation is a valid proxy for anatomical variation and does not bias the consistency loss.
The mean teacher mechanism assumes consistency targets from a noisy teacher are meaningful; the paper proposes affine transformations as the noise without validating that this captures clinically relevant variation.
domain assumption Frames from the same patient are independent for the Wilcoxon tests.
The reported p-values imply per-frame comparisons, but video frames from one patient are temporally correlated; this assumption is likely violated and would inflate significance.
domain assumption Leave-one-patient-out cross-validation controls for inter-patient variation, isolating data-quantity effects.
Sec 3.3 explicitly simplifies by excluding inter-patient variation; this is only an approximation because training folds include frames from other patients in the same type of procedure.

how reviews work

0 comments

Cite this review

Pith. "Pith review of More unlabelled data or label more data? A study on semi-supervised laparoscopic image segmentation." pith.science (2026). https://pith.science/paper/RF2IVGKT

@misc{pith2026190808035,
  author       = {Pith},
  title        = {Pith review of: More unlabelled data or label more data? A study on semi-supervised laparoscopic image segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RF2IVGKT}},
  note         = {Machine review of arXiv:1908.08035}
}

read the original abstract

Improving a semi-supervised image segmentation task has the option of adding more unlabelled images, labelling the unlabelled images or combining both, as neither image acquisition nor expert labelling can be considered trivial in most clinical applications. With a laparoscopic liver image segmentation application, we investigate the performance impact by altering the quantities of labelled and unlabelled training data, using a semi-supervised segmentation algorithm based on the mean teacher learning paradigm. We first report a significantly higher segmentation accuracy, compared with supervised learning. Interestingly, this comparison reveals that the training strategy adopted in the semi-supervised algorithm is also responsible for this observed improvement, in addition to the added unlabelled data. We then compare different combinations of labelled and unlabelled data set sizes for training semi-supervised segmentation networks, to provide a quantitative example of the practically useful trade-off between the two data planning strategies in this surgical guidance application.

Figures

Figures reproduced from arXiv: 1908.08035 by the authors.

**Figure 1.** U-Net architecture with multi-scale inputs (depth = 3). Student Model Noise1 Teacher Model Noise2 Pred. of Labeled Pred. of Unlabeled Pred. of Labeled Pred. of Unlabeled Label Supervised Loss Unsupervised Loss Gradient Update EMA Update Labeled Image Unlabeled Image Labeled Image Unlabeled Image [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Mean Teacher Structure faster training. The details of the network are illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Two examples with ground truth (green) and the predictions of the supervised model (blue) and mean teacher trained with all labelled and unlabelled data (orange). 2 5 10% 2 5 100% 0.93 0.94 0.95 0.96 SL MT Labelled Data Dice Score 2 5 10% 2 5 100% 80 90 100 110 120 130 140 SL MT Labelled Data Hausdorff Distance [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Comparison between supervised model and mean teacher with different quantities of labels. Mean Teacher with Different Labelled Data Set Sizes The median Dice scores for the MT models, trained with all available unlabelled data and different quantities of labelled dat…

**Figure 5.** Figure 5: Comparison between supervised model (SL) and mean teacher (MT) with different quantities of unlabelled data. The quantity of labelled data used is indicated in brackets. 5 Conclusion The quantified differences showed in this work, such as the improvement due to more l…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[1]

arXiv preprint arXiv:1810.07842 (2018)

Abraham, N., Khan, N.M.: A novel focal tversky loss function with improved attention u-net for lesion segmentation. arXiv preprint arXiv:1810.07842 (2018)

arXiv 2018
[2]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Bai, W., Oktay, O., Sinclair, M., Suzuki, H., Rajchl, M., Tarroni, G., Glocker, B., King, A., Matthews, P.M., Rueckert, D.: Semi-supervised learning for network-based cardiac mr image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 253--260. Springer (2017)

work page 2017
[3]

Medical image analysis 54, 280--296 (2019)

Cheplygina, V., de Bruijne, M., Pluim, J.P.: Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis 54, 280--296 (2019)

work page 2019
[4]

International journal of computer assisted radiology and surgery 10(3), 301--316 (2015)

Clarkson, M.J., Zombori, G., Thompson, S., Totz, J., Song, Y., Espak, M., Johnsen, S., Hawkes, D., Ourselin, S.: The niftk software platform for image-guided interventions: platform overview and niftylink messaging. International journal of computer assisted radiology and surgery 10(3), 301--316 (2015)

work page 2015
[5]

arXiv preprint arXiv:1903.01248 (2019)

Cui, W., Liu, Y., Li, Y., Guo, M., Li, Y., Li, X., Wang, T., Zeng, X., Ye, C.: Semi-supervised brain lesion segmentation with an adapted mean teacher model. arXiv preprint arXiv:1903.01248 (2019)

arXiv 2019
[6]

In: Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling

Gibson, E., Robu, M.R., Thompson, S., Edwards, P.E., Schneider, C., Gurusamy, K., Davidson, B., Hawkes, D.J., Barratt, D.C., Clarkson, M.J.: Deep residual networks for automatic segmentation of laparoscopic videos of the liver. In: Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling. vol. 10135, p. 101351M. International Soc...

work page 2017
[7]

In: Workshop on Challenges in Representation Learning, ICML

Lee, D.H.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML. vol. 3, p. 2 (2013)

work page 2013
[8]

In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp

Perone, C.S., Cohen-Adad, J.: Deep semi-supervised segmentation with weight-averaged consistency targets. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 12--19. Springer (2018)

work page 2018

Show all 13 references

[9]

In: International Conference on Medical image computing and computer-assisted intervention

Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234--241. Springer (2015)

2015
[10]

In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp

Sudre, C.H., Li, W., Vercauteren, T., Ourselin, S., Cardoso, M.J.: Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 240--248. Sp...

2017
[11]

In: Advances in neural information processing systems

Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In: Advances in neural information processing systems. pp. 1195--1204 (2017)

2017
[12]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[13]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

arXiv preprint arXiv:1810.07842 (2018)

Abraham, N., Khan, N.M.: A novel focal tversky loss function with improved attention u-net for lesion segmentation. arXiv preprint arXiv:1810.07842 (2018)

arXiv 2018

[2] [2]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Bai, W., Oktay, O., Sinclair, M., Suzuki, H., Rajchl, M., Tarroni, G., Glocker, B., King, A., Matthews, P.M., Rueckert, D.: Semi-supervised learning for network-based cardiac mr image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 253--260. Springer (2017)

work page 2017

[3] [3]

Medical image analysis 54, 280--296 (2019)

Cheplygina, V., de Bruijne, M., Pluim, J.P.: Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis 54, 280--296 (2019)

work page 2019

[4] [4]

International journal of computer assisted radiology and surgery 10(3), 301--316 (2015)

Clarkson, M.J., Zombori, G., Thompson, S., Totz, J., Song, Y., Espak, M., Johnsen, S., Hawkes, D., Ourselin, S.: The niftk software platform for image-guided interventions: platform overview and niftylink messaging. International journal of computer assisted radiology and surgery 10(3), 301--316 (2015)

work page 2015

[5] [5]

arXiv preprint arXiv:1903.01248 (2019)

Cui, W., Liu, Y., Li, Y., Guo, M., Li, Y., Li, X., Wang, T., Zeng, X., Ye, C.: Semi-supervised brain lesion segmentation with an adapted mean teacher model. arXiv preprint arXiv:1903.01248 (2019)

arXiv 2019

[6] [6]

In: Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling

Gibson, E., Robu, M.R., Thompson, S., Edwards, P.E., Schneider, C., Gurusamy, K., Davidson, B., Hawkes, D.J., Barratt, D.C., Clarkson, M.J.: Deep residual networks for automatic segmentation of laparoscopic videos of the liver. In: Medical Imaging 2017: Image-Guided Procedures, Robotic Interventions, and Modeling. vol. 10135, p. 101351M. International Soc...

work page 2017

[7] [7]

In: Workshop on Challenges in Representation Learning, ICML

Lee, D.H.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on Challenges in Representation Learning, ICML. vol. 3, p. 2 (2013)

work page 2013

[8] [8]

In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp

Perone, C.S., Cohen-Adad, J.: Deep semi-supervised segmentation with weight-averaged consistency targets. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 12--19. Springer (2018)

work page 2018