REVIEW 4 major objections 5 minor 12 references
Unstable Prompts, Unreliable Segmentations: A Challenge for Longitudinal Lesion Analysis
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The ULS23 segmentation model collapses when a lesion is displaced from the center of its input volume, and follow-up CT registration errors routinely cause that displacement, breaking lesion tracking.
desk verdict A solid, useful evaluation of ULS23's center assumption; the controlled displacement experiment carries the paper, but the real-data causal claim needs a direct per-lesion test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the ULS23 model, a 3D Residual Encoder UNet that disables default resampling and patching and processes fixed-size volumes of interest ($256\times256\times128$ voxels) with the assumption that a single target lesion is centered in each volume. This centered-lesion bias was reinforced during training on volumes cropped around lesion centroids and on non-exhaustively annotated data, which taught the model to treat peripheral lesions as background. The paper converts that implicit assumption into a measurable transfer function through a controlled displacement experiment, and pairs it with a center-proximity correspondence rule—each lesion ID is assigned to the segmented connected component closest to the volume center—which is the downstream mechanism that turns a segmentation miss into a tracking failure.
What would settle it
Compare Dice on follow-up lesions whose propagated centroid error is under 5 mm with the same lesions at baseline. If the drop persists when the lesion is effectively centered in the input crop, then appearance change, not misalignment, drives the real-world degradation—a result the controlled displacement experiment cannot provide because it reuses the same image content.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that ULS23's segmentation accuracy is governed by a centered-lesion assumption with a sharp tolerance: displacement of the lesion from the input-volume center by more than about 20 mm drives Dice to near zero, and even 10–15 mm displacements push the downstream center-proximity correspondence below 50% correct assignments. In the real longitudinal data, propagated centroids frequently exceed 10 mm of registration error, producing a statistically significant Dice decrease at follow-up relative to baseline ($p<0.001$) and a rise in clinically critical tracking errors—incorrect assignments plus false negatives—from 26.3% at baseline to 31.8% at follow-up. The paper interprets this as a fundamental limitation of applying single-time-point models to temporal analysis, since the failure cascade (registration error → off-center lesion → segmentation failure → correspondence failure) is built into the pipeline structure.
Load-bearing premise
The load-bearing premise for the real-data portion is that the follow-up performance drop is caused primarily by misalignment of the input crop around the lesion, rather than by treatment-induced changes in lesion size and appearance between scans; the paper acknowledges that it does not control for this confound.
Editorial extensions
If this is right
- ULS23 maintains high Dice for lesion-to-center displacements up to roughly 20 mm, then fails almost completely beyond that threshold in the controlled experiment.
- In follow-up CT scans, registration-based propagated centroids with errors above 10 mm are common enough to produce a significant overall Dice drop relative to baseline.
- Lesion correspondence by center proximity is only as reliable as the segmentation feeding it; when segmentation fails, incorrect assignments and false negatives rise to 31.8% of follow-up lesions.
- Cascading independent stages (registration, volume-of-interest segmentation, correspondence) compounds errors, so robust longitudinal tracking should move toward integrated models that process temporal context jointly.
Reading between the lines
- A practical extension the paper does not develop: the sharp 20 mm threshold could be turned into an automatic quality flag, so that any propagated centroid whose distance to the segmented component exceeds the threshold is routed to a human reader instead of trusted.
- The same centered-crop failure mode likely applies to interactive click-based segmentation tools and to any model that crops a volume around a user-provided coordinate, not only to registration-based longitudinal pipelines.
- The displacement experiment suggests a testable training fix: augmenting ULS23 with off-center volumes or conditioning on the centroid-to-lesion distance might flatten the threshold and restore robustness, although the paper argues the cascaded architecture would still remain fragile.
- A direct comparison of follow-up Dice for lesions with minimal registration error versus lesions with large error would separate the geometric mechanism from treatment-induced appearance change; the paper's real-data analysis does not fully control for that confound.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates the ULS23 baseline lesion segmentation model on a public longitudinal CT dataset of melanoma patients. It compares segmentation Dice and lesion correspondence outcomes between baseline and follow-up scans, showing a statistically significant follow-up Dice drop and a long-tailed distribution of registration errors for propagated lesion centroids. To explain this drop, the authors run a controlled experiment in which 30 of the best-segmented lesions are artificially displaced from the VOI center; they observe a sharp threshold around 20 mm beyond which segmentation fails and correspondence breaks down. The paper concludes that inter-scan registration errors violate ULS23's centered-lesion assumption and that this is the cause of the real-data follow-up degradation, motivating a shift toward end-to-end longitudinal models.
Significance. If the causal claim were fully supported, the paper would be a valuable cautionary result for longitudinal lesion tracking: it identifies a concrete mechanism (center-of-VOI sensitivity) in a widely used open model and evaluates it on an open dataset, which makes the work reproducible. The controlled displacement experiment is a clean, self-contained probe of the model's behavior and credibly demonstrates that large displacements destroy segmentation. However, the real-data attribution to registration error is not directly tested, and the selection of the 30 best-segmented lesions limits extrapolation to the full cohort. The contribution is therefore a solid mechanism demonstration with an overreaching causal interpretation in its current form.
major comments (4)
- [§3.2 (last paragraph; Fig. 5)] The sentence 'This controlled experiment confirms that the performance drop seen in real-world follow-up data (Figure 3) is caused by VOI misalignment' is not supported by the evidence presented. The real-data analysis in §3.1 never correlates per-lesion registration error with follow-up Dice, so it cannot separate VOI misalignment from the morphology and appearance confounds acknowledged in the Discussion. Because the displacement experiment uses only the 30 lesions with the highest Dice (Section 2.3), it excludes the cases where appearance changes are most likely to be severe and hence cannot estimate what fraction of the real-data drop is attributable to displacement. The authors should add a per-lesion or stratified analysis, such as plotting Dice change versus registration error or conditioning on lesions that actually failed, before claiming causation.
- [§3.2, Fig. 5a vs. Fig. 5b] The two panels of Figure 5 are in tension with the proposed failure cascade. Figure 5a shows that Dice remains high for displacements below about 20 mm, while Figure 5b shows correct assignments falling below 50% by 10 mm. This means that for moderate displacements, the center-proximity correspondence rule fails even though segmentation quality is largely preserved, so the correspondence breakdown is not, in those cases, a consequence of segmentation failure. The narrative that 'registration error leads to segmentation failure, which in turn causes correspondence failure' is only valid for large displacements; the moderate-displacement regime is dominated by the assignment rule itself. Please reconcile the two panels or revise the mechanism claim accordingly.
- [§3.1 / Fig. 3b] The Wilcoxon signed-rank test demonstrates a statistically significant Dice drop, but it does not control for lesion size change, split/merge events, or new lesion appearance, all of which are explicitly present in the dataset (Section 2.2). These factors can directly lower follow-up Dice independently of any registration error. The Discussion acknowledges this possibility, but the abstract and conclusion treat the registration-error connection as established. A stratified analysis with matched lesions of similar size change but different registration errors, or an adjustment for lesion volume change, is needed to support the causal interpretation.
- [§2.3, Eq. (1)] The correspondence rule in Eq. (1), which assigns the lesion ID to the segmented component whose centroid is closest to the VOI center, is the authors' own evaluation choice rather than a component of the ULS23 model. The paper nevertheless uses the resulting failure rates (Fig. 4) as evidence that segmentation unreliability causes correspondence failure. Because an alternative matching rule (e.g., largest component, or a learned matcher) would produce different correspondence outcomes, the reported correspondence failure rates are partly an artifact of this rule. The authors should evaluate at least one alternative rule or explicitly discuss the sensitivity of their conclusions to this choice.
minor comments (5)
- [§2.3] The text contains inconsistent spacing in '256× 256× 128 voxel'; please use consistent notation such as '256 × 256 × 128'.
- [Figures 3 and 4] The figure captions use 'Fig.3' and 'Fig.4' without a space; please use 'Fig. 3' and 'Fig. 4' for consistency.
- [§2.3] The definition of 'False Negative' says 'No component was segmented as s∗' but the earlier symbol definition uses s∗ for the assigned component; clarify whether a false negative means no component at all or no component selected by the assignment rule.
- [§5] The citation style '[10,5,9]' is unusual and the order is inconsistent with the reference numbering; please use a standard citation format and ensure the cited works include the relevant longitudinal methods.
- [§2.3] The criteria for selecting the '30 best-segmented lesions' are not fully specified; state whether this is based on baseline Dice, follow-up Dice, or an average, and report the range of those Dice values.
Circularity Check
No significant circularity: the evaluation uses an external public model and dataset, and the central measurements are direct experimental outputs rather than quantities derived from the paper's own assumptions.
full rationale
The paper's central claims are empirical measurements, not derivations that reduce to their inputs. The ULS23 model and the Longitudinal-CT dataset are external, publicly available artifacts (arXiv:2406.05231; fdat.uni-tuebingen.de), and the paper evaluates them rather than fitting parameters to predict its own conclusions. The centered-lesion assumption is a documented architectural property of ULS23 that serves as the experimental input, not as a result obtained from the experiments. The controlled displacement experiment directly measures segmentation Dice as a function of imposed VOI shifts; no fitted quantity is renamed as a prediction, and no equation in the paper encodes the sharp-threshold finding. The causal sentence in Section 3.2 ('This controlled experiment confirms that the performance drop seen in real-world follow-up data is caused by VOI misalignment') is an interpretive inference, and the paper itself explicitly acknowledges in the Discussion that 'changes in lesion morphology and appearance between scans... are confounding factors that could also contribute to the observed performance degradation.' That is a limitation in causal attribution, not a circular derivation. The correspondence rule s* = argmin over components of distance to VOI center is explicitly defined by the authors, and the paper openly states that the method is 'operationally fragile'; the observation that correspondence fails when segmentation fails is a stated property of the defined cascade, not a hidden equivalence. The self-citations to the ULS23 challenge and the longitudinal dataset refer to the external objects under study and to related work, not to an unverified uniqueness theorem or ansatz that the paper needs in order to force its result. Therefore the paper is self-contained as an evaluation study, and no circular step meets the evidentiary standard required for a positive finding.
Assumptions & free parameters
free parameters (1)
- Number of top-segmented lesions selected for displacement study =
30
assumptions (4)
- domain assumption Manual segmentations in the Longitudinal-CT dataset are accurate ground truth for lesion presence, location, and extent.
- domain assumption The propagated centroids provided in the dataset are representative of errors in clinically used registration-based propagation.
- domain assumption The ULS23 baseline model is representative of single-timepoint universal lesion segmentation models.
- domain assumption The center-proximity correspondence rule is a fair proxy for clinical lesion tracking.
Cite this review
Pith. "Pith review of Unstable Prompts, Unreliable Segmentations: A Challenge for Longitudinal Lesion Analysis." pith.science (2026). https://pith.science/paper/YDXYVUPO
@misc{pith2026250719230,
author = {Pith},
title = {Pith review of: Unstable Prompts, Unreliable Segmentations: A Challenge for Longitudinal Lesion Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/YDXYVUPO}},
note = {Machine review of arXiv:2507.19230}
}
read the original abstract
Longitudinal lesion analysis is crucial for oncological care, yet automated tools often struggle with temporal consistency. While universal lesion segmentation models have advanced, they are typically designed for single time points. This paper investigates the performance of the ULS23 segmentation model in a longitudinal context. Using a public clinical dataset of baseline and follow-up CT scans, we evaluated the model's ability to segment and track lesions over time. We identified two critical, interconnected failure modes: a sharp degradation in segmentation quality in follow-up cases due to inter-scan registration errors, and a subsequent breakdown of the lesion correspondence process. To systematically probe this vulnerability, we conducted a controlled experiment where we artificially displaced the input volume relative to the true lesion center. Our results demonstrate that the model's performance is highly dependent on its assumption of a centered lesion; segmentation accuracy collapses when the lesion is sufficiently displaced. These findings reveal a fundamental limitation of applying single-timepoint models to longitudinal data. We conclude that robust oncological tracking requires a paradigm shift away from cascading single-purpose tools towards integrated, end-to-end models inherently designed for temporal analysis.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
CA: a cancer journal for clinicians 74(3), 229--263 (2024)
Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R.L., Soerjomataram, I., Jemal, A.: Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 74(3), 229--263 (2024). doi:10.3322/caac.21834
-
[2]
European Journal of Cancer (Oxford, England: 1990) 45(2), 228--247 (Jan 2009)
Eisenhauer, E.A., Therasse, P., Bogaerts, J., Schwartz, L.H., Sargent, D., Ford, R., Dancey, J., Arbuck, S., Gwyther, S., Mooney, M., Rubinstein, L., Shankar, L., Dodd, L., Kaplan, R., Lacombe, D., Verweij, J.: New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). European Journal of Cancer (Oxford, England: 1990) 45(2...
-
[3]
doi:10.48550/arXiv.2406.05231, http://arxiv.org/abs/2406.05231, arXiv:2406.05231 [eess] version: 2
Grauw, M.J.J.d., Scholten, E.T., Smit, E.J., Rutten, M.J.C.M., Prokop, M., Ginneken, B.v., Hering, A.: The ULS23 Challenge : a Baseline Model and Benchmark Dataset for 3D Universal Lesion Segmentation in Computed Tomography (Jun 2024). doi:10.48550/arXiv.2406.05231, http://arxiv.org/abs/2406.05231, arXiv:2406.05231 [eess] version: 2
-
[4]
International Journal of Computer Assisted Radiology and Surgery 19(9), 1689--1697 (Sep 2024)
Hering, A., Westphal, M., Gerken, A., Almansour, H., Maurer, M., Geisler, B., Kohlbrandt, T., Eigentler, T., Amaral, T., Lessmann, N., Gatidis, S., Hahn, H., Nikolaou, K., Othman, A., Moltz, J., Peisen, F.: Improving assessment of lesions in longitudinal CT scans: a bi-institutional reader study on an AI -assisted registration and volumetric segmentation ...
-
[5]
Nature Methods 18(2), 203--211 (Feb 2021)
Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU - Net : a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203--211 (Feb 2021). doi:10.1038/s41592-020-01008-z, https://www.nature.com/articles/s41592-020-01008-z, publisher: Nature Publishing Group
-
[6]
Jacobs, C., Schreuder, A., van Riel, S.J., Scholten, E.T., Wittenberg, R., Wille, M.M.W., de Hoop, B., Sprengers, R., Mets, O.M., Geurts, B., Prokop, M., Schaefer-Prokop, C., van Ginneken, B.: Assisted versus Manual Interpretation of Low - Dose CT Scans for Lung Cancer Screening : Impact on Lung - RADS Agreement . Radiology. Imaging Cancer 3(5), e200160 (...
-
[7]
Küstner, T., Peisen, F., Gatidis, S., Wagner, A., Megne, O., Othman, A., Sanner, A., Loßau, T., Moltz, J.H., Kohlbrandt, T., Hering, A.: Longitudinal- CT (Mar 2025), https://fdat.uni-tuebingen.de/records/qwsry-7t837, publisher: University of Tübingen
work page 2025
-
[8]
Neuro-Oncology 23(9), 1560--1568 (Sep 2021)
Lu, S.L., Xiao, F.R., Cheng, J.C.H., Yang, W.C., Cheng, Y.H., Chang, Y.C., Lin, J.Y., Liang, C.H., Lu, J.T., Chen, Y.F., Hsu, F.M.: Randomized multi-reader evaluation of automated detection and segmentation of brain tumors in stereotactic radiosurgery with deep neural networks. Neuro-Oncology 23(9), 1560--1568 (Sep 2021). doi:10.1093/neuonc/noab071
Show all 12 references
-
[9]
doi:10.48550/arXiv.2502.20985, http://arxiv.org/abs/2502.20985, arXiv:2502.20985 [cs]
Rokuss, M., Kirchhoff, Y., Akbal, S., Kovacs, B., Roy, S., Ulrich, C., Wald, T., Rotkopf, L.T., Schlemmer, H.P., Maier-Hein, K.: LesionLocator : Zero - Shot Universal Tumor Segmentation and Tracking in 3D Whole - Body Imaging (Feb 2025). doi:10.48550/arXiv.2502.20985, http://a...
-
[10]
doi:10.48550/arXiv.2409.13416, http://arxiv.org/abs/2409.13416, arXiv:2409.13416 [eess]
Rokuss, M., Kirchhoff, Y., Roy, S., Kovacs, B., Ulrich, C., Wald, T., Zenk, M., Denner, S., Isensee, F., Vollmuth, P., Kleesiek, J., Maier-Hein, K.: Longitudinal Segmentation of MS Lesions via Temporal Difference Weighting (Sep 2024). doi:10.48550/arXiv.2409.13416, http://arxi...
-
[11]
, " * write output.state after.block = add.period write
ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
-
[12]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.