{"id":"e683e319-9674-4448-946a-48943441c6eb","arxiv_id":"1908.10219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A 3D U-Net trained on 9,149 low-resolution DTI scans segments the FMI and CST tracts in about 0.5 seconds with scan-rescan reproducibility higher than the tractography reference.","lead":"The authors trained a 3D U-Net on 9,149 low-resolution brain DTI scans to segment two white matter tracts directly, replacing slower tractography. On 80 subjects scanned twice, the network's scan-rescan agreement was higher than the reference tractography method, suggesting faster and more reliable longitudinal tract measurements.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'comparable performance' claim is evaluated against the same tractography-derived reference used as training labels, so the accuracy evidence is partly circular; independent validation is needed.","rationale":"The reader's identification of the reference standard as the weakest assumption is correct and the current CONDITIONAL verdict remains appropriate. The paper's reproducibility evidence is genuinely supportive: scan-rescan kappa and R2 values in Table 1 are strong, and the Bland-Altman plots show no consistent longitudinal bias. However, the load-bearing risk is that the same probabilistic tractography output is used both to train the network and to score its accuracy, so the reported Dice and diffusion-measure agreement cannot independently establish that the segmentations reflect true tract anatomy. The reproducibility advantage is plausibly inflated by the network's tendency to average over noisy labels, which increases stability without necessarily increasing validity. An independent expert-annotated subset, even at modest size, would break the circularity and test whether both the proposed method and the reference are comparably biased. The reader's condition of independent verification is therefore well justified, but this stress-test does not find a reason to move the verdict further toward rejection; the concern is addressable and the underlying reproducibility result is credible.","tokens_in":6829,"tokens_out":6891,"duration_ms":70634,"concrete_test":"Select a random subset of 20 subjects from D3test and have an expert independently annotate FMI and CST on the tractography images (semi-manual, as in Wasserthal et al. 2017). Compute Dice and volume bias for (a) the proposed 3D U-Net segmentations and (b) the de Groot reference standard against these independent labels. If the proposed method and the reference have similar Dice to the expert labels and no systematic volume bias, the 'comparable performance' claim holds. If both agree with each other but disagree with the expert labels to the same degree, the reference-standard limitation undermines the central claim and the method needs an independent calibration or a re-evaluation against a validated ground truth.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central conclusion in Section 4.2 and Section 5 ('comparable performance to the reference standard') rests on an evaluation in which the reference standard from Section 2.4 (de Groot et al. 2015 probabilistic tractography with tract-specific thresholds) is simultaneously the training target and the accuracy yardstick. As a result, the Dice values (FMI 0.66, CST 0.77) and the mean FA and MD agreement in Table 1 measure fidelity to the reference, not agreement with independent truth. Any systematic error in the reference, such as threshold choice, tractography parameters, or registration error, is inherited by the network and is invisible in the evaluation. The reproducibility comparison (kappa 0.74 vs 0.64 for FMI, 0.80 vs 0.72 for CST; R2 for volumes) is also not evidence of validity by itself: a CNN trained on noisy labels typically regresses toward the mean tract location, which inflates scan-rescan agreement even if the average location is biased. The paper explicitly acknowledges 'our low quality reference standard' in Section 5, but the clinical-applicability claim requires an independent anchor that the present comparison does not provide.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a 3D U-Net based method for direct segmentation of the forceps minor (FMI) and corticospinal tract (CST) from low-resolution 1.5T DTI data, trained on a large dataset of 9149 scans from the Rotterdam Study. The authors optimize input representation, loss function, and network architecture, and then evaluate segmentation accuracy (Dice coefficient) against a probabilistic tractography reference standard and reproducibility on a separate 80-subject test-retest cohort using Cohen's kappa, R2 of scan-rescan diffusion measures, and Bland-Altman analysis. They report Dice values of 0.66 (FMI) and 0.77 (CST), and higher scan-rescan reproducibility than the tractography reference (e.g., kappa 0.74 vs 0.64 for FMI, 0.80 vs 0.72 for CST), with tract-specific FA and MD consistent between scans. The authors conclude that the method is applicable to clinical practice and longitudinal studies of white matter microstructure.","tokens_in":7040,"tokens_out":4870,"duration_ms":49865,"significance":"The scale of the training set (9149 scans) and the explicit scan-rescan reproducibility evaluation on 80 subjects are clear strengths, as is the use of a train-independent test cohort of 1036 subjects. The reported speed (0.5 seconds per tract) relative to tractography is practically relevant. If the reproducibility and accuracy claims hold, the method could provide a fast and automated alternative for tract-specific diffusion measures in large epidemiological and clinical studies. The optimization experiments (input, loss, architecture) are useful for practitioners. However, the accuracy evidence is partly circular because the Dice reference is the same tractography output used as training labels, which limits the strength of the 'comparable performance' conclusion.","major_comments":[{"comment":"The central claim of 'comparable performance to the reference standard' is supported by Dice coefficients computed against the de Groot et al. (2015) probabilistic tractography reference (Section 2.4), which is simultaneously the training target. This evaluation is circular: the DC values (0.66 FMI, 0.77 CST) measure how well the network imitates the reference, not agreement with independent anatomical truth. The authors acknowledge the reference is of 'low quality' (Section 5), but do not discuss that any systematic error in the reference is inherited by the learned segmentation and is invisible in the reported accuracy metrics. To support the clinical applicability claim, the authors should either (a) explicitly reframe the accuracy claim as 'agreement with the reference standard' and avoid implying independent accuracy, or (b) provide an independent validation on at least a subsample, e.g., manual delineation on tractography-derived ROIs or comparison to a public high-quality dataset.","section":"Section 4.2 and Section 5"},{"comment":"The higher scan-rescan reproducibility of the proposed method relative to the reference (e.g., kappa 0.74 vs 0.64 for FMI, 0.80 vs 0.72 for CST) is interpreted as a strength. However, a CNN trained on noisy labels may regress toward a mean tract location, thereby reducing scan-to-scan variability at the cost of sensitivity to true anatomical variation. The paper does not address this bias-variance trade-off. Without an independent accuracy anchor, the reproducibility improvement cannot be unambiguously interpreted as a net clinical benefit. Please discuss this possibility, ideally with an analysis of the segmentation boundaries on the test-retest cohort (e.g., comparing disagreement patterns with the reference) to determine whether the higher kappa reflects true stability or smoothing toward the mean.","section":"Section 4.2 and Table 1"},{"comment":"The loss Lwip in Eq. (1) defines pi as 'the binarized prediction.' If pi is indeed a binary thresholded value, the loss is non-differentiable and no gradient flows to the network, which is incompatible with gradient-based training of a neural network. If pi is intended to be a continuous probability, the definition is incorrect and the equation should be updated. This is a reproducibility issue: readers cannot reproduce the training procedure without knowing the exact differentiable form of the loss and how gradients are computed.","section":"Section 2.5 and Section 3.1, Eq. (1)"}],"minor_comments":[{"comment":"The text states 'The R2 and K show that our method has better reproducibility than reference,' but the asterisk footnote indicates that only the kappa values and the CST volume difference reach statistical significance (p < .01). The R2 differences (e.g., FA R2 .91 vs .89 for FMI; .84 vs .83 for CST) are small and not tested. Please qualify the statement accordingly.","section":"Section 4.2 and Table 1"},{"comment":"It is stated that all input combinations gave similar performance, and the text reports the choice of tensor-only input. However, the actual numerical DC values and standard deviations for the input comparison are not given in the text or the figure. Please report them so that the similarity claim is quantifiable.","section":"Section 4.1 and Fig. 1"},{"comment":"The dataset decomposition (D1a, D1b, D2, D3) is difficult to follow in prose. A small table or schematic summarizing the sizes, roles, and subject independence of each set would improve clarity.","section":"Section 2.2"},{"comment":"The sentence 'with FLIRT we registered the MNI152 template and T1 weighted image to DTI space' is ambiguous about which image serves as the fixed and moving image. Please clarify the registration targets and the purpose of each registration.","section":"Section 2.3"},{"comment":"Reference [8] is cited as the arXiv preprint (Wasserthal et al., 2017). A peer-reviewed version of this work exists (Medical Image Analysis, 2019); citing the updated version would be more helpful to readers.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The circularity of the accuracy evaluation is the main substantive issue. It is a common pitfall in this literature, but it is load-bearing for the 'comparable performance' claim and the clinical applicability statement. Requiring the authors to either obtain an independent validation or explicitly weakened the claim would be appropriate. Also, the loss function definition in Eq. (1) should be corrected before publication, as it currently appears non-differentiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know this paper is a workmanlike empirical study of direct white-matter tract segmentation with a 3D U-Net on a large low-resolution DTI dataset. The genuinely new bits are scale (9,149 scans), the 1.5T low-res setting, and the scan-rescan reproducibility evaluation on 80 subjects. The network, the Lwip loss, and the tractography reference are all from earlier work, but the paper extends them in a direction that matters for clinical cohorts.\n\nWhat it does well: the reproducibility evidence is solid. Cohen's kappa for segmentations (0.74 FMI, 0.80 CST), R2 for FA/MD, and Bland-Altman plots all support the claim that the method is as reproducible as, or better than, the tractography reference. The dataset splits are careful, with an independent test cohort and exclusion of rescan sessions from training. The optimization experiments for input, loss, and architecture are reasonable, though none are surprises.\n\nThe soft spot is the one you'd expect: Dice accuracy is computed against the same probabilistic tractography reference used to create training labels. That makes the reported DC values (0.66 FMI, 0.77 CST) a fidelity check to the reference, not an independent anatomical validation. The paper explicitly acknowledges the 'low quality reference standard' in the Discussion, which is honest, but it does not break the circularity. The reproducibility advantage could in principle be inflated by regression to the mean of noisy labels. That doesn't sink the paper—reproducibility is a legitimate standalone claim—but it does mean the 'comparable performance' conclusion is not independently established. Lack of code/data and no deep-learning baseline on the same data make this worse, though both are addressable.\n\nWho does this help? Researchers running longitudinal DTI studies on large 1.5T cohorts who want fast, reproducible tract measures. It is not a conceptual breakthrough, but it is a careful, useful step.\n\nRecommendation: send it to peer review. It deserves referee time, and the circularity concern is fixable in revision (independent validation on a small set, comparison to tractography output, or at least an honest discussion of what the DC values can and cannot show).\n\nBest.","headline":"A careful empirical study of 3D U-Net tract segmentation on 9k low-res DTI scans with strong scan-rescan reproducibility evidence, but the accuracy claims rest on the same reference used to create labels.","tokens_in":7569,"tokens_out":2434,"would_cite":false,"duration_ms":24502,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 3D U-Net can segment white-matter tracts directly from low-resolution DTI with better scan-rescan reproducibility than the tractography reference.","keywords":["white matter tract segmentation","diffusion tensor imaging","3D U-Net","convolutional neural network","scan-rescan reproducibility","low-resolution DTI","fractional anisotropy","mean diffusivity"],"falsifier":"Take an independent cohort with manually annotated forceps minor and corticospinal tract segmentations and compare the trained 3D U-Net output against those annotations; if the Dice coefficients are substantially below the reported 0.66 and 0.77, the claim of comparable performance to a valid reference is not established.","tokens_in":6650,"feed_emoji":"🧠","tokens_out":10853,"duration_ms":90025,"temperature":0.7,"pith_summary":"The paper aims to show that a 3D U-Net can segment white-matter tracts directly from low-resolution diffusion tensor images, replacing a slower tractography pipeline. The method is trained and tested on a large clinical-scale dataset of 9,149 scans, and the authors evaluate both segmentation accuracy and scan-rescan reproducibility. They report that the learned segmentations agree with the tractography-based reference standard (Dice 0.66 for the forceps minor, 0.77 for the corticospinal tract) while achieving higher scan-rescan agreement than that reference, with Cohen's kappa rising from 0.64 to 0.74 for FMI and from 0.72 to 0.80 for CST. The diffusion measures computed inside the segmented tracts are consistent between repeated scans, and the authors conclude that the method can be used in clinical practice and in longitudinal analysis of white-matter microstructure.","feed_headline":"3D U-Net segments brain tracts more reproducibly than tractography","feed_subtitle":"Trained on 9,149 low-resolution DTI scans, it matches reference accuracy and runs in about half a second.","key_machinery":"The load-bearing mechanism is a 3D U-Net, a convolutional encoder-decoder network that maps a 3D input volume to a per-voxel segmentation. The input is the voxel-wise diffusion tensor elements restricted to a tract-specific bounding-box region of interest; the network uses batch normalization, PReLU activation, and a weighted inner-product loss with a tract weight of 3. During optimization the authors compared input combinations, loss functions and tract weights, the V-Net variant, and Adam versus Nadam optimizers, and selected the configuration with the best test Dice. The model outputs a binary segmentation of a specific tract in about 0.5 seconds, avoiding tractography reconstruction and storage.","core_discovery":"The central claim is that direct segmentation of white-matter tracts with a 3D U-Net trained on voxel-wise diffusion tensor elements is reproducible and performs comparably to the tractography-based reference standard. In the evaluation, the method segments a tract in about 0.5 seconds, achieves a mean Dice coefficient (segmentation overlap) of 0.66 for the forceps minor and 0.77 for the corticospinal tract against the reference, and shows higher scan-rescan agreement than the reference (kappa 0.74 vs 0.64 for FMI, 0.80 vs 0.72 for CST). Tract-specific fractional anisotropy and mean diffusivity values from the two scans show no significant paired differences, and the Bland-Altman plots show no consistent bias. The authors conclude that the method is applicable to clinical practice and to longitudinal analysis of white-matter microstructure.","pith_inferences":["A natural test is to retrain the same architecture on a small set of manually or independently validated tract annotations; if Dice against that ground truth is much lower than the reported 0.66 and 0.77, the method would be shown to reproduce the tractography pipeline rather than anatomical truth.","Because the input is only the diffusion tensor, the same trained model might transfer to other DTI acquisitions, but cross-scanner and cross-protocol generalization is not tested and needs a dedicated dataset.","The reported Dice scores compare the method with the reference standard that also generated the training labels, so accuracy and reproducibility cannot be fully separated from the reference's own quality.","The speed and reproducibility make it plausible to pool tract-specific measures across multiple cohorts, but harmonization of preprocessing and ROI definitions would be required first."],"forward_implications":["A single 3D U-Net can replace tractography-based pipelines for segmentation of the forceps minor and corticospinal tract in large cohorts, cutting per-subject runtime to about 0.5 seconds.","Tract-specific FA and MD derived from the learned segmentations are stable across repeated scans, so the method can be used to track white-matter changes in longitudinal studies.","Scan-rescan reproducibility reaches 'substantial' to 'almost perfect' agreement (kappa 0.74 and 0.80), a level commonly considered acceptable for clinical use.","The method works on low-resolution 1.5T clinical images, making it applicable to existing large population datasets."],"supporting_citations":[{"why":"Provides the probabilistic tractography and tract-specific threshold reference standard used to generate training labels and to evaluate accuracy and reproducibility.","marker":"[2]"},{"why":"Introduces the U-Net architecture on which the proposed segmentation model is built.","marker":"[6]"},{"why":"Introduces the V-Net architecture that is compared with 3D U-Net in the architecture optimization experiments.","marker":"[7]"},{"why":"Reports the only other deep-learning white matter tract segmentation method known to the authors and supplies the performance comparison in the discussion.","marker":"[8]"},{"why":"Describes the population-based cohort from which the 9,149 DTI scans were drawn.","marker":"[9]"},{"why":"Introduces the Nadam optimizer that was evaluated as an alternative to Adam in the optimization experiments.","marker":"[13]"},{"why":"Provides the weighted inner product loss function used to train the final model.","marker":"[14]"},{"why":"Introduces the Adam optimizer, which was selected for the final configuration based on test Dice.","marker":"[15]"},{"why":"Supplies the kappa agreement thresholds ('substantial' and 'almost perfect') used to interpret the reproducibility results.","marker":"[16]"}],"fun_headline_variants":["Fast 3D U-Net matches tractography accuracy, beats reproducibility","3D U-Net improves scan-rescan reproducibility for tract segmentation","9,149 scans train U-Net for clinical-grade tract segmentation","Half-second tract segmentation outperforms tractography reproducibility","U-Net rivals tractography with higher reproducibility on DTI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The tractography-based reference standard is assumed to be a correct enough map of each tract's true location, since it serves both as the training label and as the accuracy yardstick.","fun_headline_variants_meta":{"raw":{"variants":["Fast 3D U-Net matches tractography accuracy, beats reproducibility","3D U-Net improves scan-rescan reproducibility for tract segmentation","9,149 scans train U-Net for clinical-grade tract segmentation","Half-second tract segmentation outperforms tractography reproducibility","U-Net rivals tractography with higher reproducibility on DTI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00096,"raw_usage":{"total_tokens":4064,"prompt_tokens":896,"completion_tokens":3168,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":3081}},"tokens_in":512,"tokens_out":3168,"duration_ms":20355,"temperature":1.0,"reasoning_tokens":3081,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:06:49.953005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take an independent cohort with manually annotated forceps minor and corticospinal tract segmentations and compare the trained 3D U-Net output against those annotations; if the Dice coefficients are substantially below the reported 0.66 and 0.77, the claim of comparable performance to a valid reference is not established.","supporting_citations":[{"cited_title":"Alzheimer’s & Dementia 11(3), 321–330 (2015)","cited_arxiv_id":null,"evidence_quote":"Provides the probabilistic tractography and tract-specific threshold reference standard used to generate training labels and to evaluate accuracy and reproducibility."},{"cited_title":"In: 3D Vision (3 DV), 2016 Fourth International Conference on","cited_arxiv_id":null,"evidence_quote":"Introduces the V-Net architecture that is compared with 3D U-Net in the architecture optimization experiments."},{"cited_title":"Direct White Matter Bundle Segmentation using Stacked U-Nets","cited_arxiv_id":"1703.02036","evidence_quote":"Reports the only other deep-learning white matter tract segmentation method known to the authors and supplies the performance comparison in the discussion."},{"cited_title":", Franco, O.H., Goedegebure, A., Ikram, M.A., Klaver, C.C., Nijsten, T.E., Peeters, R.P., et al.: The rotterdam study: 2016 objectives and design update","cited_arxiv_id":null,"evidence_quote":"Describes the population-based cohort from which the 9,149 DTI scans were drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Nadam optimizer that was evaluated as an alternative to Adam in the optimization experiments."},{"cited_title":"Journal of Systemics, Cybernetics and Informati cs 8(1), 43–48 (2010)","cited_arxiv_id":null,"evidence_quote":"Provides the weighted inner product loss function used to train the final model."},{"cited_title":"biometrics pp","cited_arxiv_id":null,"evidence_quote":"Supplies the kappa agreement thresholds ('substantial' and 'almost perfect') used to interpret the reproducibility results."}],"review_version":1}