REVIEW 3 major objections 6 minor 23 references
MLV$^2$-Net: Rater-Based Majority-Label Voting for Consistent Meningeal Lymphatic Vessel Segmentation
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Rater-aware segmentation of brain lymphatic vessels reaches human-level agreement and reproduces age-related volume growth.
desk verdict First automatic MLV segmentation with a clean rater-conditioning idea, but the headline Dice is inflated by choosing the voting weight on the same four-image test set and the gain over nnU-Net is within noise. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the combination of rater encoding, rater-specific output labels, and weighted majority-label voting. The rater encoding is a zero-centered one-hot vector (for four raters, two extra input channels) concatenated to the MRI volume without per-channel normalization, so the network can condition on which expert it is imitating. The rater-specific labels form the Cartesian product of the four raters with the three anatomical segments (anterior, middle, posterior), giving twelve foreground classes plus background; this forces one shared network to generate each rater's annotation style. At inference, the foreground-class votes are counted with a multiplicative weight $w_{\mathrm{fg}}=3$, so a voxel is foreground if at most two of the four rater-specific predictions call it background, with ties going to the lower label index. The same vote tallies provide a rater-based uncertainty map. A side theorem, derived from the confusion-matrix definition of Dice, converts any reported Dice value into upper and lower bounds on predicted relative volume, which the paper uses to attach error boundaries to volume measurements.
What would settle it
Run the frozen MLV2-Net on an independent set of at least ten new FLAIR images, segmented jointly by raters who had no part in the original annotation. If the mean foreground Dice is below about $0.75$ or no longer exceeds the equal-weight majority vote ($w_{\mathrm{fg}}=1$), the claimed benefit of foreground weighting and rater-based ensembling does not generalize. As a complementary check, manually trace MLVs on a subset of the 22 raw scans and test whether the age-volume association survives manual measurement; if it disappears, the downstream replication is an artifact of the model's foreground bias.
Extended reading notes
Core claim
The central claim is that inter-rater variability does not have to be averaged away; it can be learned and exploited. The paper augments nnU-Net, a widely used U-Net-based segmentation model with automated hyperparameter selection, by adding a zero-centered one-hot rater identity as extra input channels and replacing the usual foreground label with rater-specific labels for each of the four experts, so one model can explicitly predict four plausible segmentation styles. At inference, a weighted majority-label vote over those rater-specific maps, with foreground weight $w_{\mathrm{fg}}=3$, produces the final segmentation while the vote disagreement yields an uncertainty map. On the held-out consensus test set, MLV2-Net reaches a foreground Dice of $0.806 \pm 0.030$, slightly above the best plain nnU-Net variant; its agreement with human experts ($\kappa = 0.79/0.82$) matches or exceeds the experts' agreement with each other ($\kappa = 0.73/0.79$); and its predicted volumes on unannotated scans reproduce the previously reported positive relation between age and MLV volume at $p < 0.05$. The paper also proves that, given only the Dice coefficient, the predicted volume relative to ground truth is bounded between $\frac{2}{2-\mathrm{DSC}}-1$ and $\frac{2}{\mathrm{DSC}}-1$.
Load-bearing premise
The method's accuracy figure rests on the assumption that a panel of expert raters jointly annotating just four images produces a true reference standard; because those same four images were also used to select the foreground voting weight $w_{\mathrm{fg}}=3$, the reported Dice of $0.806$ could be optimistic for new scans.
Editorial extensions
If this is right
- Clinical researchers can compute MLV volumes automatically from FLAIR MRI across large cohorts, with provable error bounds on those volumes, instead of relying on slow manual tracing.
- A single trained model can output several plausible segmentations matching individual expert styles plus an uncertainty map, so ambiguous regions can be flagged for human review.
- Rater-aware training worked with only about 30 annotated scans, suggesting that the method can handle new tasks where detailed manual labels are scarce.
- Reproducing the known age-volume association on previously unannotated scans gives indirect evidence that the automatic segmentation preserves a clinically meaningful biological signal.
Reading between the lines
- The same recipe, rater identity as extra input channels and rater-specific output labels merged by a weighted vote, should transfer to other segmentation tasks with multiple disagreeing annotators, since it changes only the input and output structure of the base model; testing it on existing multi-rater datasets would show whether $w_{\mathrm{fg}}$ must be retuned per task.
- The observation that moderate foreground weighting best matches human consensus raises a general question about how vote aggregation should balance sensitivity against specificity for thin, low-contrast structures; a comparative study across structures with different prevalence could reveal whether the optimal weight tracks structure prevalence.
- The volume-bound theorem is a direct consequence of Dice's definition rather than an empirical discovery, so its practical usefulness hinges on how tightly ensemble Dice is concentrated across images; on a four-image test set the bound $[0.67, 1.49]$ for relative volume is too wide to detect moderate volume differences.
- If the age-volume replication survives larger and more balanced cohorts, automated MLV volumetry could become a screening instrument for glymphatic clearance in aging and neurodegeneration; the present 22-scan group is too small to establish that by itself.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MLV$^2$-Net, a rater-aware extension of nnU-Net for segmenting meningeal lymphatic vessels (MLVs) in 3D FLAIR MRI. The model adds a zero-centered rater encoding to the input and rater-specific output labels, then aggregates the rater-conditioned predictions with a weighted majority-label vote. The authors report a foreground Dice of 0.806 on a four-image consensus test set, Fleiss kappa values close to or above human inter-rater reliability, a volume-error bound derived from Dice, and an age-volume association on 22 unlabeled scans. The core design idea, keeping nnU-Net intact while conditioning on raters, is simple and potentially transferable, but the evaluation protocol raises concerns about the headline accuracy estimate.
Significance. If the evaluation were clean, this would be a useful practical contribution: it is the first automatic MLV segmentation method, it preserves nnU-Net's self-configuration pipeline, and the rater-conditioned predictions provide a natural uncertainty map. The volume bound in Theorem 1 is correctly derived and holds for any segmentation model, and the age-association replication is a sensible external check. The main weakness is that the aggregation weight wfg is selected on the same consensus set used to report the headline Dice, so the 0.806 figure is not an unbiased estimate, and the margin over nnU-Net 3D Fullres is within the noise of only four test images. The rater-based uncertainty and the algebraic bound are genuine strengths, but they do not by themselves resolve the evaluation-protocol problem.
major comments (3)
- [§3.4, Tables 2 and 3] The foreground weight wfg is chosen by maximizing Dice on the consensus set, and the same set is then used in Table 2 to report the headline result (wfg=3, foreground Dice=0.806). This makes the headline a selected maximum rather than an independent test estimate. With only four consensus images and per-image standard deviations of roughly 0.03, the difference between MLV$^2$-Net (0.806±0.030) and nnU-Net 3D Fullres (0.787±0.046) is within one standard deviation and is not statistically established. Please use a selection procedure that does not touch the consensus set, for example fixing wfg by cross-validation on the training set or using a nested/leave-one-image-out scheme, and report paired per-image Dice values with confidence intervals for the comparison to the strongest baseline.
- [§3.3, Table 1] The consensus reference is created by all four raters jointly annotating only four images after the training annotations were completed. The manuscript calls this the 'reference standard for consensus decision-making' but provides no evidence about the reliability or repeatability of this joint annotation process. With n=4, the mean±SD values in Tables 2 and 3 are fragile, and the ranking of methods (e.g., 0.806 vs. 0.787) is presented with more certainty than the effective sample size supports. Please state the effective sample size explicitly, discuss the sensitivity of the conclusions to the consensus-reference construction, and avoid presenting the small margin over nnU-Net as a stable property of the model.
- [§3.2] The claim that MLV$^2$-Net 'closely replicates' or even exceeds human inter-rater reliability is based on Fleiss kappa from only two IRR images, with no confidence intervals. A single-model kappa higher than the human kappa is not inherently desirable: it may indicate that the rater-conditioned outputs are less diverse than the human raters, which would make the rater-based uncertainty map an underestimate of true inter-rater variability. Please report per-image kappa values, confidence intervals, and a quantitative voxel-level comparison between model disagreement and human disagreement, rather than relying on the qualitative inspection of Figure 5.
minor comments (6)
- [§2.5] The proof of Theorem 1 is correct, but 'derieve' is a typo, and the normalization of the confusion matrix to the ground-truth volume should be stated explicitly before Equation (1).
- [Table 2] In the submitted layout it is unclear which numeric columns belong to cross-validation and which belong to the consensus set; please use explicit row-group labels so that the two evaluation settings cannot be confused.
- [§3.5, Figure 8] The age-volume comparison is a reasonable sanity check, but it should be described as such; with n=4 vs. n=18 it cannot by itself validate voxel-level segmentation accuracy.
- [Figures 5 and 7] Please add legends or clearer captions explaining what each dot and pixel value represents; the current captions are too terse for the reader to interpret the scatter plots.
- [Data and Code Availability] The code is only promised for future release; please include the exact nnU-Net configuration, the rater-encoding preprocessing steps, and the inference/tie-breaking implementation in the supplement so that the reported results are reproducible.
- [Abstract and title] The notation is inconsistent: the title uses 'MLV$^2$-Net' while the abstract and body text write 'ML V2-Net'; please unify the notation.
Circularity Check
No circular derivation: Theorem 1 is an algebraic identity, the age-association check is external, and the central method is not defined in terms of its reported accuracy. The only concern is in-sample wfg selection on the consensus set, which is an evaluation-bias issue rather than circularity.
full rationale
The claimed derivation chain is not circular. Theorem 1 derives volume bounds directly from the definitions of Dice and relative volume (Eqs. 1-3); it involves no fitted parameters and no imported uniqueness or ansatz. The downstream age-volume result is a replication of an external finding (Albayram et al., 2022) on 22 unlabeled scans and is not used to fit the model. The rater-aware training and majority-voting scheme are defined independently of the evaluation metric. The one notable risk is that the foreground weight wfg is selected by comparing Dice on the four-image consensus set (Table 3) and the final Dice 0.806 is reported on that same set (Table 2), so the headline number is an in-sample, selection-biased estimate rather than an independent test; the 0.019 margin over nnU-Net is within the reported SD. However, this is a methodological evaluation concern, not a case where a prediction is equivalent to its input by construction: no equation forces Dice to equal the wfg-selection criterion, and the core contributions (rater conditioning, uncertainty, volume bounds) retain independent content. Hence no specific circular step is identified.
Assumptions & free parameters
free parameters (1)
- Foreground weight wfg =
3
assumptions (3)
- domain assumption Consensus annotations by four raters on the 4 held-out images are a valid reference standard.
- domain assumption The age-related association of MLV volume from Albayram et al. (2022) is correct.
- ad hoc to paper Weighted majority voting with a moderate foreground bias mimics human consensus.
Cite this review
Pith. "Pith review of MLV$^2$-Net: Rater-Based Majority-Label Voting for Consistent Meningeal Lymphatic Vessel Segmentation." pith.science (2026). https://pith.science/paper/GZCFP5XZ
@misc{pith2026241108537,
author = {Pith},
title = {Pith review of: MLV$^2$-Net: Rater-Based Majority-Label Voting for Consistent Meningeal Lymphatic Vessel Segmentation},
year = {2026},
howpublished = {\url{https://pith.science/paper/GZCFP5XZ}},
note = {Machine review of arXiv:2411.08537}
}
abstract
Meningeal lymphatic vessels (MLVs) are responsible for the drainage of waste products from the human brain. An impairment in their functionality has been associated with aging as well as brain disorders like multiple sclerosis and Alzheimer's disease. However, MLVs have only recently been described for the first time in magnetic resonance imaging (MRI), and their ramified structure renders manual segmentation particularly difficult. Further, as there is no consistent notion of their appearance, human-annotated MLV structures contain a high inter-rater variability that most automatic segmentation methods cannot take into account. In this work, we propose a new rater-aware training scheme for the popular nnU-Net model, and we explore rater-based ensembling strategies for accurate and consistent segmentation of MLVs. This enables us to boost nnU-Net's performance while obtaining explicit predictions in different annotation styles and a rater-based uncertainty estimation. Our final model, MLV$^2$-Net, achieves a Dice similarity coefficient of 0.806 with respect to the human reference standard. The model further matches the human inter-rater reliability and replicates age-related associations with MLV volume.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Non-invasive mr imaging of human brain lymphatic networks with connections to cervical lymph nodes
Mehmet Sait Albayram, Garrett Smith, Fatih Tufan, Ibrahim Sacit Tuna, Mehmet Bostanc kl o g lu, Michael Zile, and Onder Albayram. Non-invasive mr imaging of human brain lymphatic networks with connections to cervical lymph nodes. Nature communications, 13 0 (1): 0 203, 2022
work page 2022
-
[2]
Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M
Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A. Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M. Summers, Bram van Ginneken, Michel Bilello, Patrick Bilic, Patrick F. Christ, Richard K. G. Do, Marc J. Gollub, Stephan H. Heckers, Henkjan Huisman, William R. Jarnagin, Maureen K. McHugo, ...
work page 2022
-
[3]
Sabuncu, John Guttag, and Adrian V
Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R. Sabuncu, John Guttag, and Adrian V. Dalca. Universeg: Universal medical image segmentation. International Conference on Computer Vision, 2023
work page 2023
-
[4]
Alberto Del Guerra , Salleh Ahmad, Mihai Avram, Nicola Belcari, Arne Berneking, Laura Biagi, Maria Giuseppina Bisogni, Felix Brandl, Jorge Cabello, Niccolò Camarlinghi, Piergiorgio Cerello, Chang-Hoon Choi, Silvia Coli, Sabrina Colpo, Julien Fleury, Vito Gagliardi, Giuseppe Giraudo, Karsten Heekeren, Wolfram Kawohl, Theodora Kostou, Jean-Luc Lefaucheur, C...
work page 2018
-
[5]
Impaired meningeal lymphatic drainage in patients with idiopathic parkinson’s disease
Xue-Bing Ding, Xin-Xin Wang, Dan-Hao Xia, Han Liu, Hai-Yan Tian, Yu Fu, Yong-Kang Chen, Chi Qin, Jiu-Qi Wang, Zhi Xiang, Zhong-Xian Zhang, Qin-Chen Cao, Wei Wang, Jia-Yi Li, Erxi Wu, Bei-Sha Tang, Ming-Ming Ma, Jun-Fang Teng, and Xue-Jing Wang. Impaired meningeal lymphatic drainage in patients with idiopathic parkinson’s disease. Nature Medicine, 27 0 (3)...
work page 2021
-
[6]
Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76 0 (5): 0 378–382, November 1971
work page 1971
-
[7]
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050--1059, New York, New York, USA, 20--22 Jun ...
work page 2016
-
[8]
James R. Goodman, Zachariah O. Adham, Randall L. Woltjer, Amanda W. Lund, and Jeffrey J. Iliff. Characterization of dural sinus-associated lymphatic vasculature in human alzheimer’s dementia subjects. Brain, Behavior, and Immunity, 73: 0 34–40, October 2018
work page 2018
Show all 23 references
-
[9]
Uctnet: Uncertainty-guided cnn-transformer hybrid networks for medical image segmentation
Xiayu Guo, Xian Lin, Xin Yang, Li Yu, Kwang-Ting Cheng, and Zengqiang Yan. Uctnet: Uncertainty-guided cnn-transformer hybrid networks for medical image segmentation. Pattern Recognition, 152: 0 110491, August 2024
2024
-
[10]
Unetr: Transformers for 3d medical image segmentation
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574--584, 2022
2022
-
[11]
Iliff, Minghuan Wang, Yonghong Liao, Benjamin A
Jeffrey J. Iliff, Minghuan Wang, Yonghong Liao, Benjamin A. Plogg, Weiguo Peng, Georg A. Gundersen, Helene Benveniste, G. Edward Vates, Rashid Deane, Steven A. Goldman, Erlend A. Nagelhus, and Maiken Nedergaard. A paravascular pathway facilitates csf flow through the brain par...
2012
-
[12]
Jaeger, Simon A
Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18 0 (2): 0 203--211, 2 2021
2021
-
[13]
Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F. Jäger. nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation . In proceedings of Medical Image Computing and Computer Assisted Intervent...
2024
-
[14]
Simon A. A. Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R. Ledsam, Klaus H. Maier-Hein, S. M. Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. A probabilistic u-net for segmentation of ambiguous images. In Proceedings of the 32nd Internatio...
2018
-
[15]
Richard Landis and Gary G
J. Richard Landis and Gary G. Koch. The measurement of observer agreement for categorical data. Biometrics, 33 0 (1): 0 159, March 1977
1977
-
[16]
Keyes, Jacob D
Antoine Louveau, Igor Smirnov, Timothy J. Keyes, Jacob D. Eccles, Sherin J. Rouhani, J. David Peske, Noel C. Derecki, David Castle, James W. Mandell, Kevin S. Lee, Tajie H. Harris, and Jonathan Kipnis. Structural and functional features of central nervous system lymphatic vess...
2015
-
[17]
Dong, Kenneth E
Antoine Louveau, Jasmin Herz, Maria Nordheim Alme, Andrea Francesca Salvador, Michael Q. Dong, Kenneth E. Viar, S. Grace Herod, James Knopp, Joshua C. Setliff, Alexander L. Lupi, Sandro Da Mesquita, Elizabeth L. Frost, Alban Gaultier, Tajie H. Harris, Rui Cao, Song Hu, John R....
2018
-
[18]
D-lema: Deep learning ensembles from multiple annotations - application to skin lesion segmentation
Zahra Mirikharaji, Kumar Abhishek, Saeed Izadi, and Ghassan Hamarneh. D-lema: Deep learning ensembles from multiple annotations - application to skin lesion segmentation. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1837--1846, 2021
2021
-
[19]
Fast free-form deformation using graphics processing units
Marc Modat, Gerard R Ridgway, Zeike A Taylor, Manja Lehmann, Josephine Barnes, David J Hawkes, Nick C Fox, and S \'e bastien Ourselin. Fast free-form deformation using graphics processing units. Comput Methods Programs Biomed, 98 0 (3): 0 278--284, October 2009
2009
-
[20]
U- Net : Convolutional Networks for Biomedical Image Segmentation
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- Net : Convolutional Networks for Biomedical Image Segmentation . In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2...
2015
-
[21]
Warfield, K.H
S.K. Warfield, K.H. Zou, and W.M. Wells. Simultaneous truth and performance level estimation (staple): An algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging, 23 0 (7): 0 903–921, July 2004
2004
-
[22]
Meyer, Maurice Pradella, Daniel Hinck, Alexander W
Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth. Totalsegmentator: Robust segmentation of 104 anatomic structures in ct image...
2023
-
[23]
Alexander
Le Zhang, Ryutaro Tanno, Moucheng Xu, Yawen Huang, Kevin Bronik, Chen Jin, Joseph Jacob, Yefeng Zheng, Ling Shao, Olga Ciccarelli, Frederik Barkhof, and Daniel C. Alexander. Learning from multiple annotators for medical image segmentation. Pattern Recognition, 138: 0 109400, 2023
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.