{"id":"d52f0a0c-e0c8-4534-aa2b-b526b2531fe1","arxiv_id":"2411.08537","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"MLV2-Net, a rater-aware nnU-Net with weighted majority voting, segments meningeal lymphatic vessels from 3D FLAIR MRI at a Dice score of 0.806.","lead":"Researchers built an AI model that automatically traces the brain's meningeal lymphatic vessels, tiny drainage channels visible in MRI. The model learns from several experts' differing annotation styles, reports uncertainty, and claims to match expert-level performance in segmenting these vessels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline Dice 0.806 is selected by tuning wfg on the same 4-image consensus set, so the reported accuracy is optimistically biased and the improvement over nnU-Net is within sampling noise.","rationale":"After reading the paper, the main quantitative claim is the DSC=0.806 on the human reference standard. The weakest part of the evidence for that claim is the evaluation protocol: the consensus set has only 4 images, and the aggregation weight wfg was explicitly chosen to maximize performance on that same set (Section 3.4, Table 3). This is selection on the test set; the reported 0.806 is a selected maximum over wfg, so it is optimistically biased even if the consensus annotations are perfect. The bias is material because the gain over the vanilla nnU-Net baseline (0.787±0.046) is small relative to the variance across 4 images; no significance test is reported. I do not see a fatal flaw in the method itself: the rater-aware conditioning is a reasonable way to expose annotation style, the weighted majority vote is transparent, and the volume-bound theorem is correct and clearly derived. The downstream age-association result is suggestive but based on n=4 older adults and n=18 young adults, so it is a secondary validation. The correct remedy is to fix the evaluation protocol: choose wfg on a validation split (or via nested cross-validation) and then evaluate once on the consensus set, ideally with a paired test against the baseline. Until then, the central claim should be considered conditional, consistent with the reader's verdict. The reader's weakest assumption focused on the reliability of the consensus reference; I partially agree, but the more concrete and actionable defect is the test-set tuning, which is independent of whether the joint annotation is a good reference.","tokens_in":11021,"tokens_out":6975,"duration_ms":59441,"concrete_test":"Select wfg on a separate validation split before touching the consensus set (e.g., use 5-fold cross-validation on the 27 training scans, choosing wfg by average Dice on the held-out folds). Then, with wfg fixed, evaluate once on the 4 consensus images and compare to nnU-Net 3D Fullres using a paired permutation test across the 4 images. If the pre-selected wfg yields Dice not exceeding 0.787 by more than the across-image SD, the headline accuracy claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline accuracy of MLV2-Net is not an unbiased estimate: the only hyperparameter of the proposed aggregation, the foreground weight wfg, is set to the value that maximizes Dice on the same 4-image consensus set that is used to report the 0.806 result. Table 3 shows wfg swept over 1, 2, 3, 4 on the consensus set; the reported model uses the argmax (wfg=3). Evaluating on the set used for selection inflates the expected performance, and with n=4 images the variance is large. The difference to vanilla nnU-Net 3D Fullres on the same set (0.787 ± 0.046) is 0.019, far below the across-image SD, so the claimed boost is not statistically demonstrated. Moreover, the consensus reference was produced by all four raters jointly annotating only 4 images after the training annotations were completed; its status as a 'human reference standard' is assumed, not validated, and the small n makes the reported mean ± SD fragile. Together, the central claim 'achieves Dice 0.806' rests on a test-set-tuned hyperparameter and effectively 4 test images. The rater-aware uncertainty and the volume-bound theorem are useful contributions, but they do not rescue the headline from this evaluation protocol concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MLV$^2$-Net, a rater-aware extension of nnU-Net for segmenting meningeal lymphatic vessels (MLVs) in 3D FLAIR MRI. The model adds a zero-centered rater encoding to the input and rater-specific output labels, then aggregates the rater-conditioned predictions with a weighted majority-label vote. The authors report a foreground Dice of 0.806 on a four-image consensus test set, Fleiss kappa values close to or above human inter-rater reliability, a volume-error bound derived from Dice, and an age-volume association on 22 unlabeled scans. The core design idea, keeping nnU-Net intact while conditioning on raters, is simple and potentially transferable, but the evaluation protocol raises concerns about the headline accuracy estimate.","tokens_in":11275,"tokens_out":8298,"duration_ms":77718,"significance":"If the evaluation were clean, this would be a useful practical contribution: it is the first automatic MLV segmentation method, it preserves nnU-Net's self-configuration pipeline, and the rater-conditioned predictions provide a natural uncertainty map. The volume bound in Theorem 1 is correctly derived and holds for any segmentation model, and the age-association replication is a sensible external check. The main weakness is that the aggregation weight wfg is selected on the same consensus set used to report the headline Dice, so the 0.806 figure is not an unbiased estimate, and the margin over nnU-Net 3D Fullres is within the noise of only four test images. The rater-based uncertainty and the algebraic bound are genuine strengths, but they do not by themselves resolve the evaluation-protocol problem.","major_comments":[{"comment":"The foreground weight wfg is chosen by maximizing Dice on the consensus set, and the same set is then used in Table 2 to report the headline result (wfg=3, foreground Dice=0.806). This makes the headline a selected maximum rather than an independent test estimate. With only four consensus images and per-image standard deviations of roughly 0.03, the difference between MLV$^2$-Net (0.806±0.030) and nnU-Net 3D Fullres (0.787±0.046) is within one standard deviation and is not statistically established. Please use a selection procedure that does not touch the consensus set, for example fixing wfg by cross-validation on the training set or using a nested/leave-one-image-out scheme, and report paired per-image Dice values with confidence intervals for the comparison to the strongest baseline.","section":"§3.4, Tables 2 and 3"},{"comment":"The consensus reference is created by all four raters jointly annotating only four images after the training annotations were completed. The manuscript calls this the 'reference standard for consensus decision-making' but provides no evidence about the reliability or repeatability of this joint annotation process. With n=4, the mean±SD values in Tables 2 and 3 are fragile, and the ranking of methods (e.g., 0.806 vs. 0.787) is presented with more certainty than the effective sample size supports. Please state the effective sample size explicitly, discuss the sensitivity of the conclusions to the consensus-reference construction, and avoid presenting the small margin over nnU-Net as a stable property of the model.","section":"§3.3, Table 1"},{"comment":"The claim that MLV$^2$-Net 'closely replicates' or even exceeds human inter-rater reliability is based on Fleiss kappa from only two IRR images, with no confidence intervals. A single-model kappa higher than the human kappa is not inherently desirable: it may indicate that the rater-conditioned outputs are less diverse than the human raters, which would make the rater-based uncertainty map an underestimate of true inter-rater variability. Please report per-image kappa values, confidence intervals, and a quantitative voxel-level comparison between model disagreement and human disagreement, rather than relying on the qualitative inspection of Figure 5.","section":"§3.2"}],"minor_comments":[{"comment":"The proof of Theorem 1 is correct, but 'derieve' is a typo, and the normalization of the confusion matrix to the ground-truth volume should be stated explicitly before Equation (1).","section":"§2.5"},{"comment":"In the submitted layout it is unclear which numeric columns belong to cross-validation and which belong to the consensus set; please use explicit row-group labels so that the two evaluation settings cannot be confused.","section":"Table 2"},{"comment":"The age-volume comparison is a reasonable sanity check, but it should be described as such; with n=4 vs. n=18 it cannot by itself validate voxel-level segmentation accuracy.","section":"§3.5, Figure 8"},{"comment":"Please add legends or clearer captions explaining what each dot and pixel value represents; the current captions are too terse for the reader to interpret the scatter plots.","section":"Figures 5 and 7"},{"comment":"The code is only promised for future release; please include the exact nnU-Net configuration, the rater-encoding preprocessing steps, and the inference/tie-breaking implementation in the supplement so that the reported results are reproducible.","section":"Data and Code Availability"},{"comment":"The notation is inconsistent: the title uses 'MLV$^2$-Net' while the abstract and body text write 'ML V2-Net'; please unify the notation.","section":"Abstract and title"}],"recommendation":"major_revision","confidential_remarks":"The paper is well suited to the ML4H audience and the rater-conditioning idea is simple and potentially useful. In my view the evaluation protocol is the main obstacle: the selection of wfg on the consensus set and the effective sample size of four test images prevent the reported 0.806 from being treated as an unbiased accuracy estimate. If the authors can move the hyperparameter selection off the consensus set and temper the small-sample claims, the paper could become acceptable; I would not reject on the basis of the algebraic volume bound, which is correct."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first automatic segmentation method for meningeal lymphatic vessels from 3D FLAIR MRI, and the rater-conditioned nnU-Net scheme is a clean, reusable idea. The paper is worth engaging with. But the headline number is not as reliable as the abstract implies: the single hyperparameter of the voting scheme, wfg, is chosen by sweeping over the same four-image consensus set that produces the reported Dice, and the improvement over vanilla nnU-Net is within the across-image standard deviation.\n\nWhat is genuinely good: the task is new and clinically relevant; the authors assembled their own dataset with four expert raters and make code/models available; the rater-as-input/rater-as-output trick is simple and keeps nnU-Net intact; the uncertainty output and the post-hoc removal of a rater are useful properties; and the volume-bound theorem is correct. I checked the algebra; it is a direct identity from the confusion matrix, and the derived bounds can be practically useful when only Dice is reported. Credit is due for including several baselines (UniverSeg, registration propagation, standard nnU-Net) on a task where no prior automatic method exists.\n\nThe soft spot is exactly where the stress-test points. Table 3 sweeps wfg over 1..4 on the consensus set and then the paper reports the argmax (wfg=3) as the final accuracy. That is in-sample tuning on the evaluation set, and with n=4 images the variance is large. The 0.806 vs 0.787 for nnU-Net 3D is a 0.019 difference with SDs around 0.03–0.05; no paired test or confidence interval is given. So the 'boost' is not statistically demonstrated, even though the cross-validation numbers point in the same direction. The age-association replication is a reasonable external check but uses only four older subjects in the affected group; it is suggestive, not confirmatory. I do not think there is a circularity problem beyond the wfg selection—the theorem is not fitted, and the age result is not used to adjust the model.\n\nWho should read it: anyone working on segmentation with multiple annotators, and researchers studying the glymphatic system who want a first automatic MLV tool. The method section is clear enough to re-implement.\n\nRecommendation: send it to peer review, but require an unbiased evaluation: select wfg on a separate validation split or report the score averaged over wfg values, report image-level paired statistics, and ideally validate on an external dataset. With that change the central claim would be much safer. As it stands, I would read the 0.806 as an upper bound.","headline":"First automatic MLV segmentation with a clean rater-conditioning idea, but the headline Dice is inflated by choosing the voting weight on the same four-image test set and the gain over nnU-Net is within noise.","tokens_in":11840,"tokens_out":3519,"would_cite":true,"duration_ms":32407,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rater-aware segmentation of brain lymphatic vessels reaches human-level agreement and reproduces age-related volume growth.","keywords":["meningeal lymphatic vessels","glymphatic system","segmentation","inter-rater variability","rater-aware training","majority-label voting","FLAIR MRI","uncertainty estimation"],"falsifier":"Run the frozen MLV2-Net on an independent set of at least ten new FLAIR images, segmented jointly by raters who had no part in the original annotation. If the mean foreground Dice is below about $0.75$ or no longer exceeds the equal-weight majority vote ($w_{\\mathrm{fg}}=1$), the claimed benefit of foreground weighting and rater-based ensembling does not generalize. As a complementary check, manually trace MLVs on a subset of the 22 raw scans and test whether the age-volume association survives manual measurement; if it disappears, the downstream replication is an artifact of the model's foreground bias.","tokens_in":10826,"feed_emoji":"🧠","tokens_out":15318,"duration_ms":117933,"temperature":0.7,"pith_summary":"Meningeal lymphatic vessels (MLVs), the vessels that drain waste from the human brain, have only recently become visible in 3D FLAIR MRI (fluid-attenuated inversion recovery imaging), and experts disagree substantially about their exact contours. This paper tries to establish that automatic segmentation can still be accurate and consistent if the model is explicitly rater-aware: the network receives a code for which expert's style to imitate, is trained to produce a separate segmentation in each style, and then merges them by weighted majority-label voting. On a small consensus set of four jointly annotated images, the resulting MLV2-Net reaches a foreground Dice of $0.806 \\pm 0.030$, matches human inter-rater agreement, and reproduces the known age-related increase in MLV volume on 22 previously unannotated scans at $p < 0.05$. If these claims hold, researchers studying the brain's waste-drainage system could compute vessel volumes automatically, with per-voxel uncertainty derived from the raters' disagreements rather than from a single averaged ground truth.","feed_headline":"Brain lymphatic vessel AI matches human raters at Dice 0.806","feed_subtitle":"Rater-aware voting turns expert disagreements into uncertainty estimates and reproduces age-linked volume gains.","key_machinery":"The load-bearing object is the combination of rater encoding, rater-specific output labels, and weighted majority-label voting. The rater encoding is a zero-centered one-hot vector (for four raters, two extra input channels) concatenated to the MRI volume without per-channel normalization, so the network can condition on which expert it is imitating. The rater-specific labels form the Cartesian product of the four raters with the three anatomical segments (anterior, middle, posterior), giving twelve foreground classes plus background; this forces one shared network to generate each rater's annotation style. At inference, the foreground-class votes are counted with a multiplicative weight $w_{\\mathrm{fg}}=3$, so a voxel is foreground if at most two of the four rater-specific predictions call it background, with ties going to the lower label index. The same vote tallies provide a rater-based uncertainty map. A side theorem, derived from the confusion-matrix definition of Dice, converts any reported Dice value into upper and lower bounds on predicted relative volume, which the paper uses to attach error boundaries to volume measurements.","core_discovery":"The central claim is that inter-rater variability does not have to be averaged away; it can be learned and exploited. The paper augments nnU-Net, a widely used U-Net-based segmentation model with automated hyperparameter selection, by adding a zero-centered one-hot rater identity as extra input channels and replacing the usual foreground label with rater-specific labels for each of the four experts, so one model can explicitly predict four plausible segmentation styles. At inference, a weighted majority-label vote over those rater-specific maps, with foreground weight $w_{\\mathrm{fg}}=3$, produces the final segmentation while the vote disagreement yields an uncertainty map. On the held-out consensus test set, MLV2-Net reaches a foreground Dice of $0.806 \\pm 0.030$, slightly above the best plain nnU-Net variant; its agreement with human experts ($\\kappa = 0.79/0.82$) matches or exceeds the experts' agreement with each other ($\\kappa = 0.73/0.79$); and its predicted volumes on unannotated scans reproduce the previously reported positive relation between age and MLV volume at $p < 0.05$. The paper also proves that, given only the Dice coefficient, the predicted volume relative to ground truth is bounded between $\\frac{2}{2-\\mathrm{DSC}}-1$ and $\\frac{2}{\\mathrm{DSC}}-1$.","pith_inferences":["The same recipe, rater identity as extra input channels and rater-specific output labels merged by a weighted vote, should transfer to other segmentation tasks with multiple disagreeing annotators, since it changes only the input and output structure of the base model; testing it on existing multi-rater datasets would show whether $w_{\\mathrm{fg}}$ must be retuned per task.","The observation that moderate foreground weighting best matches human consensus raises a general question about how vote aggregation should balance sensitivity against specificity for thin, low-contrast structures; a comparative study across structures with different prevalence could reveal whether the optimal weight tracks structure prevalence.","The volume-bound theorem is a direct consequence of Dice's definition rather than an empirical discovery, so its practical usefulness hinges on how tightly ensemble Dice is concentrated across images; on a four-image test set the bound $[0.67, 1.49]$ for relative volume is too wide to detect moderate volume differences.","If the age-volume replication survives larger and more balanced cohorts, automated MLV volumetry could become a screening instrument for glymphatic clearance in aging and neurodegeneration; the present 22-scan group is too small to establish that by itself."],"forward_implications":["Clinical researchers can compute MLV volumes automatically from FLAIR MRI across large cohorts, with provable error bounds on those volumes, instead of relying on slow manual tracing.","A single trained model can output several plausible segmentations matching individual expert styles plus an uncertainty map, so ambiguous regions can be flagged for human review.","Rater-aware training worked with only about 30 annotated scans, suggesting that the method can handle new tasks where detailed manual labels are scarce.","Reproducing the known age-volume association on previously unannotated scans gives indirect evidence that the automatic segmentation preserves a clinically meaningful biological signal."],"supporting_citations":[{"why":"Supplies the nnU-Net base model whose architecture and hyperparameter selection MLV2-Net inherits.","marker":"Isensee et al., 2021"},{"why":"Supports the claim that best segmentation results are typically obtained with nnU-Net, justifying the base architecture.","marker":"Isensee et al., 2024"},{"why":"First describes MLVs in 3D FLAIR MRI and reports the age-volume association used as the downstream replication target.","marker":"Albayram et al., 2022"},{"why":"Provides the foundation-model baseline that the method competes against in the accuracy comparison.","marker":"Butoi et al., 2023"},{"why":"Provides the kappa statistic used to quantify inter-rater reliability and model-rater agreement.","marker":"Fleiss, 1971"},{"why":"Describes the trimodal PET-MR-EEG acquisition setup that produced all images in the study.","marker":"Del Guerra et al., 2018"},{"why":"Supplies the kappa thresholds used to classify the raters' agreement as substantial.","marker":"Landis and Koch, 1977"},{"why":"Establishes the problem of disagreement among annotators in segmentation ground truth and motivates rater-aware modeling.","marker":"Warfield et al., 2004"},{"why":"Provides the registration-based segmentation propagation used as a simpler baseline method.","marker":"Modat et al., 2009"}],"fun_headline_variants":["Rater-aware AI matches human experts on brain vessel segmentation","MLV2-Net: AI that learns from rater disagreement matches human-level Dice","Voting-based deep learning turns rater variability into uncertainty estimates","Brain lymphatic vessel AI: Dice 0.806, matches human rater consistency","AI matches human raters on brain vessel segmentation and age trends"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's accuracy figure rests on the assumption that a panel of expert raters jointly annotating just four images produces a true reference standard; because those same four images were also used to select the foreground voting weight $w_{\\mathrm{fg}}=3$, the reported Dice of $0.806$ could be optimistic for new scans.","fun_headline_variants_meta":{"raw":{"variants":["Rater-aware AI matches human experts on brain vessel segmentation","MLV2-Net: AI that learns from rater disagreement matches human-level Dice","Voting-based deep learning turns rater variability into uncertainty estimates","Brain lymphatic vessel AI: Dice 0.806, matches human rater consistency","AI matches human raters on brain vessel segmentation and age trends"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001048,"raw_usage":{"total_tokens":4453,"prompt_tokens":1041,"completion_tokens":3412,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":657,"completion_tokens_details":{"reasoning_tokens":3318}},"tokens_in":657,"tokens_out":3412,"duration_ms":22353,"temperature":1.0,"reasoning_tokens":3318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:31:08.255885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the frozen MLV2-Net on an independent set of at least ten new FLAIR images, segmented jointly by raters who had no part in the original annotation. If the mean foreground Dice is below about $0.75$ or no longer exceeds the equal-weight majority vote ($w_{\\mathrm{fg}}=1$), the claimed benefit of foreground weighting and rater-based ensembling does not generalize. As a complementary check, manually trace MLVs on a subset of the 22 raw scans and test whether the age-volume association survives manual measurement; if it disappears, the downstream replication is an artifact of the model's foreground bias.","supporting_citations":[{"cited_title":"Jaeger, Simon A","cited_arxiv_id":null,"evidence_quote":"Supplies the nnU-Net base model whose architecture and hyperparameter selection MLV2-Net inherits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the claim that best segmentation results are typically obtained with nnU-Net, justifying the base architecture."},{"cited_title":"Non-invasive mr imaging of human brain lymphatic networks with connections to cervical lymph nodes","cited_arxiv_id":null,"evidence_quote":"First describes MLVs in 3D FLAIR MRI and reports the age-volume association used as the downstream replication target."},{"cited_title":"Sabuncu, John Guttag, and Adrian V","cited_arxiv_id":null,"evidence_quote":"Provides the foundation-model baseline that the method competes against in the accuracy comparison."},{"cited_title":"Jon Shah, Christian Sorg, Giancarlo Sportelli, Michela Tosetti, Riccardo Trinchero, Christine Wyss, and Sibylle Ziegler","cited_arxiv_id":null,"evidence_quote":"Describes the trimodal PET-MR-EEG acquisition setup that produced all images in the study."},{"cited_title":"Fast free-form deformation using graphics processing units","cited_arxiv_id":null,"evidence_quote":"Provides the registration-based segmentation propagation used as a simpler baseline method."}],"review_version":1}