Pith. sign in

REVIEW 4 major objections 5 minor 12 references

Unstable Prompts, Unreliable Segmentations: A Challenge for Longitudinal Lesion Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read The ULS23 segmentation model collapses when a lesion is displaced from the center of its input volume, and follow-up CT registration errors routinely cause that displacement, breaking lesion tracking.

desk verdict A solid, useful evaluation of ULS23's center assumption; the controlled displacement experiment carries the paper, but the real-data causal claim needs a direct per-lesion test. read the letter →

arxiv 2507.19230 v1 pith:YDXYVUPO submitted 2025-07-25 eess.IV cs.CV

classification eess.IVcs.CV
keywords LongitudinalAnalysisLesionSegmentationComputedTomographyDeepLearningRegistrationErrorULS23CorrespondenceCentered-lesionassumption
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that ULS23, a general-purpose universal lesion segmentation model trained on single time-point CT scans, cannot be naively applied to longitudinal oncology workflows because it silently assumes the target lesion sits at the center of a fixed-size input volume. Using a public longitudinal melanoma CT dataset, the authors show that registration errors displace propagated lesion centroids in follow-up scans, and that this displacement produces a sharp drop in segmentation quality and a breakdown of the lesion correspondence step. A controlled experiment, in which the input volume is artificially shifted by known distances, isolates the mechanism: Dice remains high up to roughly 20 mm of displacement, then collapses to near zero beyond that threshold. The authors conclude that cascaded single-time-point tools compound errors and argue that robust tracking requires end-to-end temporal models.

What carries the argument

The load-bearing object is the ULS23 model, a 3D Residual Encoder UNet that disables default resampling and patching and processes fixed-size volumes of interest ($256\times256\times128$ voxels) with the assumption that a single target lesion is centered in each volume. This centered-lesion bias was reinforced during training on volumes cropped around lesion centroids and on non-exhaustively annotated data, which taught the model to treat peripheral lesions as background. The paper converts that implicit assumption into a measurable transfer function through a controlled displacement experiment, and pairs it with a center-proximity correspondence rule—each lesion ID is assigned to the segmented connected component closest to the volume center—which is the downstream mechanism that turns a segmentation miss into a tracking failure.

What would settle it

Compare Dice on follow-up lesions whose propagated centroid error is under 5 mm with the same lesions at baseline. If the drop persists when the lesion is effectively centered in the input crop, then appearance change, not misalignment, drives the real-world degradation—a result the controlled displacement experiment cannot provide because it reuses the same image content.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that ULS23's segmentation accuracy is governed by a centered-lesion assumption with a sharp tolerance: displacement of the lesion from the input-volume center by more than about 20 mm drives Dice to near zero, and even 10–15 mm displacements push the downstream center-proximity correspondence below 50% correct assignments. In the real longitudinal data, propagated centroids frequently exceed 10 mm of registration error, producing a statistically significant Dice decrease at follow-up relative to baseline ($p<0.001$) and a rise in clinically critical tracking errors—incorrect assignments plus false negatives—from 26.3% at baseline to 31.8% at follow-up. The paper interprets this as a fundamental limitation of applying single-time-point models to temporal analysis, since the failure cascade (registration error → off-center lesion → segmentation failure → correspondence failure) is built into the pipeline structure.

Load-bearing premise

The load-bearing premise for the real-data portion is that the follow-up performance drop is caused primarily by misalignment of the input crop around the lesion, rather than by treatment-induced changes in lesion size and appearance between scans; the paper acknowledges that it does not control for this confound.

Editorial extensions

If this is right

  • ULS23 maintains high Dice for lesion-to-center displacements up to roughly 20 mm, then fails almost completely beyond that threshold in the controlled experiment.
  • In follow-up CT scans, registration-based propagated centroids with errors above 10 mm are common enough to produce a significant overall Dice drop relative to baseline.
  • Lesion correspondence by center proximity is only as reliable as the segmentation feeding it; when segmentation fails, incorrect assignments and false negatives rise to 31.8% of follow-up lesions.
  • Cascading independent stages (registration, volume-of-interest segmentation, correspondence) compounds errors, so robust longitudinal tracking should move toward integrated models that process temporal context jointly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical extension the paper does not develop: the sharp 20 mm threshold could be turned into an automatic quality flag, so that any propagated centroid whose distance to the segmented component exceeds the threshold is routed to a human reader instead of trusted.
  • The same centered-crop failure mode likely applies to interactive click-based segmentation tools and to any model that crops a volume around a user-provided coordinate, not only to registration-based longitudinal pipelines.
  • The displacement experiment suggests a testable training fix: augmenting ULS23 with off-center volumes or conditioning on the centroid-to-lesion distance might flatten the threshold and restore robustness, although the paper argues the cascaded architecture would still remain fragile.
  • A direct comparison of follow-up Dice for lesions with minimal registration error versus lesions with large error would separate the geometric mechanism from treatment-induced appearance change; the paper's real-data analysis does not fully control for that confound.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates the ULS23 baseline lesion segmentation model on a public longitudinal CT dataset of melanoma patients. It compares segmentation Dice and lesion correspondence outcomes between baseline and follow-up scans, showing a statistically significant follow-up Dice drop and a long-tailed distribution of registration errors for propagated lesion centroids. To explain this drop, the authors run a controlled experiment in which 30 of the best-segmented lesions are artificially displaced from the VOI center; they observe a sharp threshold around 20 mm beyond which segmentation fails and correspondence breaks down. The paper concludes that inter-scan registration errors violate ULS23's centered-lesion assumption and that this is the cause of the real-data follow-up degradation, motivating a shift toward end-to-end longitudinal models.

Significance. If the causal claim were fully supported, the paper would be a valuable cautionary result for longitudinal lesion tracking: it identifies a concrete mechanism (center-of-VOI sensitivity) in a widely used open model and evaluates it on an open dataset, which makes the work reproducible. The controlled displacement experiment is a clean, self-contained probe of the model's behavior and credibly demonstrates that large displacements destroy segmentation. However, the real-data attribution to registration error is not directly tested, and the selection of the 30 best-segmented lesions limits extrapolation to the full cohort. The contribution is therefore a solid mechanism demonstration with an overreaching causal interpretation in its current form.

major comments (4)
  1. [§3.2 (last paragraph; Fig. 5)] The sentence 'This controlled experiment confirms that the performance drop seen in real-world follow-up data (Figure 3) is caused by VOI misalignment' is not supported by the evidence presented. The real-data analysis in §3.1 never correlates per-lesion registration error with follow-up Dice, so it cannot separate VOI misalignment from the morphology and appearance confounds acknowledged in the Discussion. Because the displacement experiment uses only the 30 lesions with the highest Dice (Section 2.3), it excludes the cases where appearance changes are most likely to be severe and hence cannot estimate what fraction of the real-data drop is attributable to displacement. The authors should add a per-lesion or stratified analysis, such as plotting Dice change versus registration error or conditioning on lesions that actually failed, before claiming causation.
  2. [§3.2, Fig. 5a vs. Fig. 5b] The two panels of Figure 5 are in tension with the proposed failure cascade. Figure 5a shows that Dice remains high for displacements below about 20 mm, while Figure 5b shows correct assignments falling below 50% by 10 mm. This means that for moderate displacements, the center-proximity correspondence rule fails even though segmentation quality is largely preserved, so the correspondence breakdown is not, in those cases, a consequence of segmentation failure. The narrative that 'registration error leads to segmentation failure, which in turn causes correspondence failure' is only valid for large displacements; the moderate-displacement regime is dominated by the assignment rule itself. Please reconcile the two panels or revise the mechanism claim accordingly.
  3. [§3.1 / Fig. 3b] The Wilcoxon signed-rank test demonstrates a statistically significant Dice drop, but it does not control for lesion size change, split/merge events, or new lesion appearance, all of which are explicitly present in the dataset (Section 2.2). These factors can directly lower follow-up Dice independently of any registration error. The Discussion acknowledges this possibility, but the abstract and conclusion treat the registration-error connection as established. A stratified analysis with matched lesions of similar size change but different registration errors, or an adjustment for lesion volume change, is needed to support the causal interpretation.
  4. [§2.3, Eq. (1)] The correspondence rule in Eq. (1), which assigns the lesion ID to the segmented component whose centroid is closest to the VOI center, is the authors' own evaluation choice rather than a component of the ULS23 model. The paper nevertheless uses the resulting failure rates (Fig. 4) as evidence that segmentation unreliability causes correspondence failure. Because an alternative matching rule (e.g., largest component, or a learned matcher) would produce different correspondence outcomes, the reported correspondence failure rates are partly an artifact of this rule. The authors should evaluate at least one alternative rule or explicitly discuss the sensitivity of their conclusions to this choice.
minor comments (5)
  1. [§2.3] The text contains inconsistent spacing in '256× 256× 128 voxel'; please use consistent notation such as '256 × 256 × 128'.
  2. [Figures 3 and 4] The figure captions use 'Fig.3' and 'Fig.4' without a space; please use 'Fig. 3' and 'Fig. 4' for consistency.
  3. [§2.3] The definition of 'False Negative' says 'No component was segmented as s∗' but the earlier symbol definition uses s∗ for the assigned component; clarify whether a false negative means no component at all or no component selected by the assignment rule.
  4. [§5] The citation style '[10,5,9]' is unusual and the order is inconsistent with the reference numbering; please use a standard citation format and ensure the cited works include the relevant longitudinal methods.
  5. [§2.3] The criteria for selecting the '30 best-segmented lesions' are not fully specified; state whether this is based on baseline Dice, follow-up Dice, or an average, and report the range of those Dice values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation uses an external public model and dataset, and the central measurements are direct experimental outputs rather than quantities derived from the paper's own assumptions.

full rationale

The paper's central claims are empirical measurements, not derivations that reduce to their inputs. The ULS23 model and the Longitudinal-CT dataset are external, publicly available artifacts (arXiv:2406.05231; fdat.uni-tuebingen.de), and the paper evaluates them rather than fitting parameters to predict its own conclusions. The centered-lesion assumption is a documented architectural property of ULS23 that serves as the experimental input, not as a result obtained from the experiments. The controlled displacement experiment directly measures segmentation Dice as a function of imposed VOI shifts; no fitted quantity is renamed as a prediction, and no equation in the paper encodes the sharp-threshold finding. The causal sentence in Section 3.2 ('This controlled experiment confirms that the performance drop seen in real-world follow-up data is caused by VOI misalignment') is an interpretive inference, and the paper itself explicitly acknowledges in the Discussion that 'changes in lesion morphology and appearance between scans... are confounding factors that could also contribute to the observed performance degradation.' That is a limitation in causal attribution, not a circular derivation. The correspondence rule s* = argmin over components of distance to VOI center is explicitly defined by the authors, and the paper openly states that the method is 'operationally fragile'; the observation that correspondence fails when segmentation fails is a stated property of the defined cascade, not a hidden equivalence. The self-citations to the ULS23 challenge and the longitudinal dataset refer to the external objects under study and to related work, not to an unverified uniqueness theorem or ansatz that the paper needs in order to force its result. Therefore the paper is self-contained as an evaluation study, and no circular step meets the evidentiary standard required for a positive finding.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper is empirical, so there are no invented entities or fitted equations. Its central claim rests on four domain assumptions, mainly the accuracy of manual annotations, the representativeness of propagated centroids, the generalizability from one model to a class of systems, and the validity of the center-proximity matching rule. The only hand-chosen number is the 30-lesion cutoff for the displacement experiment.

free parameters (1)
  • Number of top-segmented lesions selected for displacement study = 30
    The controlled experiment uses only the 30 lesions with the highest Dice from the longitudinal evaluation, chosen by hand to create a reliable reference set. This cutoff affects the statistical power and generalizability of the displacement results.
assumptions (4)
  • domain assumption Manual segmentations in the Longitudinal-CT dataset are accurate ground truth for lesion presence, location, and extent.
    All Dice and correspondence metrics are computed against these annotations; no assessment of annotation quality is provided.
  • domain assumption The propagated centroids provided in the dataset are representative of errors in clinically used registration-based propagation.
    The follow-up VOIs are centered on these propagated coordinates, so the observed failure modes depend on their accuracy distribution.
  • domain assumption The ULS23 baseline model is representative of single-timepoint universal lesion segmentation models.
    The paper generalizes its conclusion to a 'paradigm shift' for all cascaded single-purpose tools based on one model.
  • domain assumption The center-proximity correspondence rule is a fair proxy for clinical lesion tracking.
    The reported incorrect-assignment and false-negative rates are defined through this rule, not through clinician judgment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unstable Prompts, Unreliable Segmentations: A Challenge for Longitudinal Lesion Analysis." pith.science (2026). https://pith.science/paper/YDXYVUPO

@misc{pith2026250719230,
  author       = {Pith},
  title        = {Pith review of: Unstable Prompts, Unreliable Segmentations: A Challenge for Longitudinal Lesion Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDXYVUPO}},
  note         = {Machine review of arXiv:2507.19230}
}
read the original abstract

Longitudinal lesion analysis is crucial for oncological care, yet automated tools often struggle with temporal consistency. While universal lesion segmentation models have advanced, they are typically designed for single time points. This paper investigates the performance of the ULS23 segmentation model in a longitudinal context. Using a public clinical dataset of baseline and follow-up CT scans, we evaluated the model's ability to segment and track lesions over time. We identified two critical, interconnected failure modes: a sharp degradation in segmentation quality in follow-up cases due to inter-scan registration errors, and a subsequent breakdown of the lesion correspondence process. To systematically probe this vulnerability, we conducted a controlled experiment where we artificially displaced the input volume relative to the true lesion center. Our results demonstrate that the model's performance is highly dependent on its assumption of a centered lesion; segmentation accuracy collapses when the lesion is sufficiently displaced. These findings reveal a fundamental limitation of applying single-timepoint models to longitudinal data. We conclude that robust oncological tracking requires a paradigm shift away from cascading single-purpose tools towards integrated, end-to-end models inherently designed for temporal analysis.

Figures

Figures reproduced from arXiv: 2507.19230 by the authors.

Figure 1
Figure 1. Four examples from the public Longitudinal CT dataset, showing stable, [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Schematic overview of the two experimental setups: (a) evaluating ULS23 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The impact of registration error on longitudinal segmentation performance. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Lesion correspondence outcomes for (a) baseline and (b) follow-up scans. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Controlled experiment showing ULS23’s sensitivity to lesion displace [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 4 canonical work pages

  1. [1]

    CA: a cancer journal for clinicians 74(3), 229--263 (2024)

    Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R.L., Soerjomataram, I., Jemal, A.: Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 74(3), 229--263 (2024). doi:10.3322/caac.21834

  2. [2]

    European Journal of Cancer (Oxford, England: 1990) 45(2), 228--247 (Jan 2009)

    Eisenhauer, E.A., Therasse, P., Bogaerts, J., Schwartz, L.H., Sargent, D., Ford, R., Dancey, J., Arbuck, S., Gwyther, S., Mooney, M., Rubinstein, L., Shankar, L., Dodd, L., Kaplan, R., Lacombe, D., Verweij, J.: New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). European Journal of Cancer (Oxford, England: 1990) 45(2...

  3. [3]

    doi:10.48550/arXiv.2406.05231, http://arxiv.org/abs/2406.05231, arXiv:2406.05231 [eess] version: 2

    Grauw, M.J.J.d., Scholten, E.T., Smit, E.J., Rutten, M.J.C.M., Prokop, M., Ginneken, B.v., Hering, A.: The ULS23 Challenge : a Baseline Model and Benchmark Dataset for 3D Universal Lesion Segmentation in Computed Tomography (Jun 2024). doi:10.48550/arXiv.2406.05231, http://arxiv.org/abs/2406.05231, arXiv:2406.05231 [eess] version: 2

  4. [4]

    International Journal of Computer Assisted Radiology and Surgery 19(9), 1689--1697 (Sep 2024)

    Hering, A., Westphal, M., Gerken, A., Almansour, H., Maurer, M., Geisler, B., Kohlbrandt, T., Eigentler, T., Amaral, T., Lessmann, N., Gatidis, S., Hahn, H., Nikolaou, K., Othman, A., Moltz, J., Peisen, F.: Improving assessment of lesions in longitudinal CT scans: a bi-institutional reader study on an AI -assisted registration and volumetric segmentation ...

  5. [5]

    Nature Methods 18(2), 203--211 (Feb 2021)

    Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU - Net : a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203--211 (Feb 2021). doi:10.1038/s41592-020-01008-z, https://www.nature.com/articles/s41592-020-01008-z, publisher: Nature Publishing Group

  6. [6]

    Radiology

    Jacobs, C., Schreuder, A., van Riel, S.J., Scholten, E.T., Wittenberg, R., Wille, M.M.W., de Hoop, B., Sprengers, R., Mets, O.M., Geurts, B., Prokop, M., Schaefer-Prokop, C., van Ginneken, B.: Assisted versus Manual Interpretation of Low - Dose CT Scans for Lung Cancer Screening : Impact on Lung - RADS Agreement . Radiology. Imaging Cancer 3(5), e200160 (...

  7. [7]

    Küstner, T., Peisen, F., Gatidis, S., Wagner, A., Megne, O., Othman, A., Sanner, A., Loßau, T., Moltz, J.H., Kohlbrandt, T., Hering, A.: Longitudinal- CT (Mar 2025), https://fdat.uni-tuebingen.de/records/qwsry-7t837, publisher: University of Tübingen

  8. [8]

    Neuro-Oncology 23(9), 1560--1568 (Sep 2021)

    Lu, S.L., Xiao, F.R., Cheng, J.C.H., Yang, W.C., Cheng, Y.H., Chang, Y.C., Lin, J.Y., Liang, C.H., Lu, J.T., Chen, Y.F., Hsu, F.M.: Randomized multi-reader evaluation of automated detection and segmentation of brain tumors in stereotactic radiosurgery with deep neural networks. Neuro-Oncology 23(9), 1560--1568 (Sep 2021). doi:10.1093/neuonc/noab071

Show all 12 references
  1. [9]

    doi:10.48550/arXiv.2502.20985, http://arxiv.org/abs/2502.20985, arXiv:2502.20985 [cs]

    Rokuss, M., Kirchhoff, Y., Akbal, S., Kovacs, B., Roy, S., Ulrich, C., Wald, T., Rotkopf, L.T., Schlemmer, H.P., Maier-Hein, K.: LesionLocator : Zero - Shot Universal Tumor Segmentation and Tracking in 3D Whole - Body Imaging (Feb 2025). doi:10.48550/arXiv.2502.20985, http://a...

  2. [10]

    doi:10.48550/arXiv.2409.13416, http://arxiv.org/abs/2409.13416, arXiv:2409.13416 [eess]

    Rokuss, M., Kirchhoff, Y., Roy, S., Kovacs, B., Ulrich, C., Wald, T., Zenk, M., Denner, S., Isensee, F., Vollmuth, P., Kleesiek, J., Maier-Hein, K.: Longitudinal Segmentation of MS Lesions via Temporal Difference Weighting (Sep 2024). doi:10.48550/arXiv.2409.13416, http://arxi...

  3. [11]

    , " * write output.state after.block = add.period write

    ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...

  4. [12]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.