Promptable Longitudinal Lesion Segmentation in Whole-Body CT

Fabian Isensee; Klaus H. Maier-Hein; Maximilian Rokuss; Yannick Kirchhoff

REVIEW 3 major objections 4 minor 12 references

Promptable Longitudinal Lesion Segmentation in Whole-Body CT

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pretraining on synthetic longitudinal CT pairs raises promptable lesion-segmentation Dice by up to 6 points over from-scratch training.

desk verdict The '6 Dice' headline is real but mostly generic pretraining; the longitudinal-specific gain is about 0.6 Dice—still a solid challenge report. read the letter →

arxiv 2509.00613 v1 pith:I5PDJCUD submitted 2025-08-30 eess.IV cs.CV

Yannick Kirchhoff , Maximilian Rokuss , Fabian Isensee , Klaus H. Maier-Hein This is my paper

classification eess.IVcs.CV

keywords longitudinalCTlesiontrackingpromptablesegmentationsyntheticpretrainingwhole-bodyautoPET/CTIVinteractiveU-Net

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Building on a longitudinal segmentation framework called LongiSeg, this paper makes the case that lesion tracking in whole-body CT is best framed as a promptable segmentation problem: given a lesion's point and prior mask on baseline and follow-up scans, the model should segment that lesion in the follow-up in one forward pass. The authors' central claim is that their model can only exploit the longitudinal information if it is first pretrained on a large synthetic longitudinal CT dataset, where a fake baseline scan is generated from real follow-up volumes by anatomy-informed augmentation. In their five-fold cross-validation on the autoPET/CT IV Task 2 data, this pretraining raises the Dice score by up to 6 points compared to training from scratch, and the final model reaches 63.71 Dice. The finding matters because it suggests that the bottleneck for longitudinal lesion tracking is not architecture but the scarcity of real paired timepoints.

What carries the argument

The model is a ResEncL-anchored U-Net that takes in a channel-concatenation of aligned baseline and follow-up patches, the prior segmentation mask, and a point prompt rendered as a Gaussian blob rescaled to unit intensity at its center. Treating prompts and temporal input symmetrically as extra channels is what makes a single forward pass per lesion produce a tracked segmentation. The other load-bearing component is the synthetic longitudinal pretraining set: real CT volumes from 2625 patients, each modified by anatomy-informed augmentation to create a synthetic baseline, giving the model many plausible timepoint pairs to learn from before it ever sees the small real challenge dataset.

What would settle it

Take a held-out real longitudinal dataset with genuine lesion growth or shrinkage, train one model on the synthetic pretraining plus challenge data and another from scratch, and compare Dice on the held-out pairs. If the pretrained model does not beat the from-scratch model by the claimed margin, or loses to a cross-sectional baseline, the claim that synthetic pretraining unlocks longitudinal context is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a plain U-Net given both aligned timepoints and prompts as extra input channels fails to use the previous timepoint from the small challenge dataset alone; the best non-pretrained model is actually the cross-sectional one that sees only the current image plus the prior mask. Pretraining on the synthetic longitudinal dataset flips this: the same longitudinal architecture, initialized with weights learned on fake timepoint pairs, outperforms the cross-sectional solution by 0.6 Dice and the from-scratch longitudinal model by a larger margin. The full model, which additionally receives point prompts as Gaussian blobs and merges predictions

Load-bearing premise

The whole pretraining advantage rests on the synthetic longitudinal pairs—real CT volumes with a fake baseline created by data augmentation—being a faithful stand-in for true temporal change; if real follow-up scans differ in ways the augmentation never imitates, the 6-Dice benefit may vanish.

Editorial extensions

If this is right

If the claim holds, longitudinal lesion tracking in whole-body CT can be served by a single promptable forward pass per lesion, using both the baseline and follow-up prompt information without any propagation step.
The 'pretraining beats from-scratch' result implies that other longitudinal medical segmentation tasks with scarce paired data may benefit from the same synthetic-pair strategy rather than architectural changes.
The ensemble of five folds, where each fold is pretrained on the synthetic dataset with a large batch size, is the configuration the authors commit to for the challenge test set.
Because the non-pretrained longitudinal variant underperforms the cross-sectional one, the paper's ablations imply that longitudinal input is only useful after synthetic pretraining—no benefit is expected from adding timepoints to a from-scratch model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The synthetic-to-real gap is left unmeasured: the pretraining data's fake baselines may not resemble real treatment changes such as tumor growth or shrinkage, so the claimed 6-Dice advantage might shrink on real longitudinal pairs with strong temporal evolution; a natural next step would be external validation on another real longitudinal dataset.
The reported failure mode is boundary inaccuracy, and prompts are currently one-shot Gaussian blobs; adding iterative click refinement at inference, where the radiologist corrects a false boundary and the model re-segments, is a direct extension the paper does not explore.
Fold 4 collapsed during training, and the reported mean excludes it; ensembling hides this instability, so a single-fold deployment might not deliver the cross-validated performance.
The symmetric treatment of prompts and timepoints as input channels suggests the same architecture could ingest other conditioning signals, such as prior lesion volume or time between scans, without changing the model.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a competent, honest challenge write-up, but the headline claim overstates what the ablations show. If you cite it, cite the 0.6 Dice figure for longitudinal context, not the 6.

What's actually new: it extends LongiSeg with promptable inputs (point/mask via extra channels) and applies it to the autoPET/CT IV Task 2 longitudinal lesion tracking benchmark. It also tests a synthetic longitudinal pretraining dataset built from their earlier LesionLocator work. The combination is new for this benchmark, and the ablation table is genuinely informative: from-scratch training can't use the previous timepoint well, interactive-weight pretraining helps a bit, and synthetic longitudinal pretraining helps the most.

What I like: the paper is transparent. Full fold-level results are in the appendix, code is public, and they compare to relevant baselines including their own prior work. For a challenge report, that's above average.

The soft spot is the attribution. The abstract says 'pretraining substantially improves the ability to exploit longitudinal context, yielding up to 6 Dice points' compared to from-scratch. But the pretraining effect is mostly generic. Pretrained cross-sectional + mask gets 62.27 vs 56.64 for the same model without pretraining—a 5.6-point gain. Adding synthetic longitudinal data on top moves it to 62.89, a 0.62-point gain. Table 2 shows the same pattern. So the '6 Dice' is a large-scale pretraining effect, not a longitudinal-context effect. The stress-test note is right. The paper's Results section does acknowledge the 0.6, but the abstract and conclusion frame it as the 6, which is misleading.

Other issues are minor for a challenge report: Fold 4 was excluded from the mean after a training collapse (they report it in the appendix, but the exclusion rule wasn't pre-specified), there are no error bars, and hyperparameters were tuned on the same cross-validation they report. The synthetic-proxy question is real but secondary—given the marginal longitudinal gain is small, even a perfect proxy wouldn't change the main conclusion.

Who is this for? Anyone working on longitudinal segmentation or submitting to autoPET-type benchmarks. It's a useful, reproducible baseline. I'd send it to review, but ask the authors to reword the abstract, add per-fold results with spread, and clarify that the bulk of the gain is from pretraining scale, not longitudinal input.

Referee Report

3 major / 4 minor

Summary. The paper presents a promptable longitudinal lesion segmentation framework for autoPET/CT IV Task 2. It extends LongiSeg with point- and mask-prompt inputs via channel concatenation, and trains on the challenge's 300-patient longitudinal CT dataset, with additional large-scale pretraining on a synthetic longitudinal CT dataset created from LesionLocator. Five-fold cross-validation ablations compare input modalities, batch sizes, and pretraining strategies; the final submission is a five-fold ensemble. The central claim is that pretraining substantially improves the ability to exploit longitudinal context, with up to 6 Dice points improvement over from-scratch training. Code is provided publicly.

Significance. If the headline claim were supported, the paper would provide a practical and reproducible recipe for the autoPET Task 2 lesion-tracking benchmark, combining an established backbone, promptable inputs, and synthetic longitudinal pretraining. The evaluation is against external benchmark data (autoPET), so the result is not circular, and the release of code is a clear strength. The ablation table includes a pretrained cross-sectional control, which is exactly the right comparison unit for isolating longitudinal context; unfortunately, the paper does not interpret this control correctly, and the reported means omit one fold without a robustness analysis. With a corrected interpretation, the core finding is useful: large-scale pretraining gives most of the gain, while longitudinal pretraining adds a small but nonzero increment.

major comments (3)

[Abstract; Section 4, Table 1] The claim that pretraining 'substantially improves the ability to exploit longitudinal context, yielding up to 6 Dice points' conflates generic pretraining with longitudinal-context pretraining. In Table 1, 'Pretrained Cross Sectional + Mask' reaches 62.27 Dice versus 56.64 for 'Cross Sectional + Mask' (+5.63), while 'Pretrained Synth. Longitudinal Data' reaches 62.89 (+0.62 over the pretrained cross-sectional control). Table 2 shows the same pattern (61.20 vs. 61.94, +0.74). Thus the reported improvement relative to from-scratch training is almost entirely attributable to large-scale pretraining per se, not to longitudinal context. The abstract and conclusion should be revised to state this distinction explicitly and to report the matched longitudinal-vs-cross-sectional-pretraining comparison.
[Section 4, Table 1 footnote; Appendix Table 2] Fold 4 is excluded from all means in Table 1 because it collapsed during training with resulting Dice of 0 (Appendix Table 2). This selective exclusion materially changes the results: 'Longitudinal Batch Size 2' is 58.08 in Table 1 but 46.47 in Table 2, and the relative ranking of settings changes. No error bars or per-fold results are provided for the remaining folds. The paper should report means and standard deviations or confidence intervals over all folds, and clearly state whenever fold exclusion is applied. The headline 'up to 6 Dice' should be recomputed on the complete data or accompanied by the excluded-fold sensitivity analysis.
[Section 2.2; Section 4] The synthetic longitudinal pretraining dataset is generated by anatomy-informed augmentation of real CT volumes, but the paper provides no validation that these synthetic baseline–follow-up pairs resemble genuine longitudinal change (e.g., lesion evolution, treatment response, patient motion). The authors do note that without pretraining the model cannot learn longitudinal context from the challenge data alone, but the causal link between the synthetic pairs and longitudinal-context learning is not established. At minimum, the paper should present a distributional comparison or a control that trains on real longitudinal pairs without synthetic pretraining, to support the claim that the pretraining signal is longitudinal rather than merely a generic image-augmentation benefit.

minor comments (4)

[Section 2.2] Typo: 'syntetic' should be 'synthetic'.
[Appendix Table 2] The '*' marker on 'Longitudinal Batch Size 2*' is not explained; clarify that it denotes the fold-4 collapse or another condition.
[Section 3.2] The data-sampling paragraph states that 'both scans are randomly shifted by up to 4 voxels in each direction' but does not specify whether this shift is applied jointly to baseline and follow-up or independently; please clarify, since this affects the alignment assumption.
[Figure 1] The figure caption says 'Case 1' and 'Case 2' but includes patient IDs; consider stating which fold or split these cases come from, since fold exclusion makes the qualitative examples harder to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: central result is an empirical comparison on an external benchmark; the synthetic pretraining dataset is an independent artifact.

full rationale

The paper's derivation chain does not reduce to its inputs. The headline 'up to 6 Dice points' compares pretrained longitudinal models to models trained from scratch, and the ablations show most of that gain is attributable to generic pretraining rather than longitudinal context; this is a potential over-attribution in the narrative, not a circular construction. The synthetic longitudinal pretraining set is taken from prior work by the same authors (LesionLocator, ref [7]) and the LongiSeg framework from ref [8], but these are external artifacts tested on new challenge data; citing them is normal scientific reuse, not load-bearing self-citation that forces the result. No fitted parameter is relabeled as a prediction, no uniqueness theorem is imported, and no definition equates a derived quantity with an input. The evaluation is against the external autoPET Task 2 benchmark, so the central quantitative claims are self-contained and empirically falsifiable.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The paper's claims rest on the validity of the synthetic longitudinal pretraining data and on the stability of the CV protocol. No new physical entities are introduced; the only invented artifact is the synthetic data, which is a dataset rather than an entity.

free parameters (4)

Gaussian blob intensity normalization = unit intensity at center
Chosen after initial experiments showed gains over unit-volume normalization (Section 3.1).
Training batch size = 2 for from-scratch training, larger for pretraining
A batch size of 2 performed noticeably better than 4 in cross-validation (Section 4).
Random shift range = up to 4 voxels per direction
Data-augmentation setting used during training sampling (Section 3.2).
Fold exclusion rule = exclude Fold 4
Fold 4 was excluded from all means because training collapsed on it (Section 4, Table 1 footnote).

assumptions (3)

domain assumption The synthetic longitudinal dataset, made by anatomy-informed augmentation of real CT volumes, is a valid proxy for real longitudinal image pairs.
The entire pretraining claim depends on this transfer; invoked in Section 2.2 without validation against real longitudinal change.
domain assumption The provided lesion center locations correctly correspond between baseline and follow-up and are sufficient for aligning the scans.
Used in Sections 3.1 and 3.2; if centers are noisy, the channel-concatenation input is misaligned.
domain assumption The five-fold cross-validation on the 300-patient training set approximates test-set generalization.
No test-set scores are reported, so the CV protocol is the only evidence (Section 4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Promptable Longitudinal Lesion Segmentation in Whole-Body CT." pith.science (2026). https://pith.science/paper/I5PDJCUD

@misc{pith2026250900613,
  author       = {Pith},
  title        = {Pith review of: Promptable Longitudinal Lesion Segmentation in Whole-Body CT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5PDJCUD}},
  note         = {Machine review of arXiv:2509.00613}
}

read the original abstract

Accurate segmentation of lesions in longitudinal whole-body CT is essential for monitoring disease progression and treatment response. While automated methods benefit from incorporating longitudinal information, they remain limited in their ability to consistently track individual lesions across time. Task 2 of the autoPET/CT IV Challenge addresses this by providing lesion localizations and baseline delineations, framing the problem as longitudinal promptable segmentation. In this work, we extend the recently proposed LongiSeg framework with promptable capabilities, enabling lesion-specific tracking through point and mask interactions. To address the limited size of the provided training set, we leverage large-scale pretraining on a synthetic longitudinal CT dataset. Our experiments show that pretraining substantially improves the ability to exploit longitudinal context, yielding an improvement of up to 6 Dice points compared to models trained from scratch. These findings demonstrate the effectiveness of combining longitudinal context with interactive prompting for robust lesion tracking. Code is publicly available at https://github.com/MIC-DKFZ/LongiSeg/tree/autoPET.

Figures

Figures reproduced from arXiv: 2509.00613 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages

[1]

Nature methods 18(2), 203--211 (2021)

Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203--211 (2021)

2021
[2]

arXiv preprint arXiv:2503.08373 (2025)

Isensee, F., Rokuss, M., Kr \"a mer, L., Dinkelacker, S., Ravindran, A., Stritzke, F., Hamm, B., Wald, T., Langenberg, M., Ulrich, C., et al.: nninteractive: Redefining 3d promptable segmentation. arXiv preprint arXiv:2503.08373 (2025)

arXiv 2025
[3]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., Jaeger, P.F.: nnu-net revisited: A call for rigorous validation in 3d medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 488--498. Springer (2024)

work page 2024
[4]

In: Proceedings of the IEEE/CVF international conference on computer vision

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015--4026 (2023)

work page 2023
[5]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Kovacs, B., Netzer, N., Baumgartner, M., Eith, C., Bounias, D., Meinzer, C., J \"a ger, P.F., Zhang, K.S., Floca, R., Schrader, A., et al.: Anatomy-informed data augmentation for enhanced prostate cancer detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 531--540. Springer (2023)

work page 2023
[6]

https://fdat.uni-tuebingen.de/records/qwsry-7t837 (Mar 2025)

K\" u stner, T., Peisen, F., Gatidis, S., Wagner, A., Megne, O., Othman, A., Sanner, A., Lo au, T., Moltz, J.H., Kohlbrandt, T., Hering, A.: Longitudinal-ct. https://fdat.uni-tuebingen.de/records/qwsry-7t837 (Mar 2025). doi:10.57754/FDAT.qwsry-7t837, version v1, Published March 16, 2025

doi:10.57754/fdat.qwsry-7t837 2025
[7]

In: Proceedings of the Computer Vision and Pattern Recognition Conference

Rokuss, M., Kirchhoff, Y., Akbal, S., Kovacs, B., Roy, S., Ulrich, C., Wald, T., Rotkopf, L.T., Schlemmer, H.P., Maier-Hein, K.: Lesionlocator: Zero-shot universal tumor segmentation and tracking in 3d whole-body imaging. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 30872--30885 (2025)

work page 2025
[8]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Rokuss, M.R., Kirchhoff, Y., Roy, S., Kovacs, B., Ulrich, C., Wald, T., Zenk, M., Denner, S., Isensee, F., Vollmuth, P., et al.: Longitudinal segmentation of ms lesions via temporal difference weighting. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 64--74. Springer (2024)

work page 2024

Show all 12 references

[9]

In: European Conference on Computer Vision

Wang, H., Guo, S., Ye, J., Deng, Z., Cheng, J., Li, T., Chen, J., Su, Y., Huang, Z., Shen, Y., et al.: Sam-med3d: towards general-purpose segmentation models for volumetric medical images. In: European Conference on Computer Vision. pp. 51--67. Springer (2024)

2024
[10]

In: European Conference on Computer Vision

Wong, H.E., Rakic, M., Guttag, J., Dalca, A.V.: Scribbleprompt: fast and flexible interactive segmentation for any biomedical image. In: European Conference on Computer Vision. pp. 207--229. Springer (2024)

2024
[11]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[12]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Nature methods 18(2), 203--211 (2021)

Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203--211 (2021)

2021

[2] [2]

arXiv preprint arXiv:2503.08373 (2025)

Isensee, F., Rokuss, M., Kr \"a mer, L., Dinkelacker, S., Ravindran, A., Stritzke, F., Hamm, B., Wald, T., Langenberg, M., Ulrich, C., et al.: nninteractive: Redefining 3d promptable segmentation. arXiv preprint arXiv:2503.08373 (2025)

arXiv 2025

[3] [3]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K., Jaeger, P.F.: nnu-net revisited: A call for rigorous validation in 3d medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 488--498. Springer (2024)

work page 2024

[4] [4]

In: Proceedings of the IEEE/CVF international conference on computer vision

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 4015--4026 (2023)

work page 2023

[5] [5]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Kovacs, B., Netzer, N., Baumgartner, M., Eith, C., Bounias, D., Meinzer, C., J \"a ger, P.F., Zhang, K.S., Floca, R., Schrader, A., et al.: Anatomy-informed data augmentation for enhanced prostate cancer detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 531--540. Springer (2023)

work page 2023

[6] [6]

https://fdat.uni-tuebingen.de/records/qwsry-7t837 (Mar 2025)

K\" u stner, T., Peisen, F., Gatidis, S., Wagner, A., Megne, O., Othman, A., Sanner, A., Lo au, T., Moltz, J.H., Kohlbrandt, T., Hering, A.: Longitudinal-ct. https://fdat.uni-tuebingen.de/records/qwsry-7t837 (Mar 2025). doi:10.57754/FDAT.qwsry-7t837, version v1, Published March 16, 2025

doi:10.57754/fdat.qwsry-7t837 2025

[7] [7]

In: Proceedings of the Computer Vision and Pattern Recognition Conference

Rokuss, M., Kirchhoff, Y., Akbal, S., Kovacs, B., Roy, S., Ulrich, C., Wald, T., Rotkopf, L.T., Schlemmer, H.P., Maier-Hein, K.: Lesionlocator: Zero-shot universal tumor segmentation and tracking in 3d whole-body imaging. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 30872--30885 (2025)

work page 2025

[8] [8]

In: International Conference on Medical Image Computing and Computer-Assisted Intervention

Rokuss, M.R., Kirchhoff, Y., Roy, S., Kovacs, B., Ulrich, C., Wald, T., Zenk, M., Denner, S., Isensee, F., Vollmuth, P., et al.: Longitudinal segmentation of ms lesions via temporal difference weighting. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 64--74. Springer (2024)

work page 2024