REVIEW 4 major objections 6 minor 26 references
Spatio-Temporal Conditional Diffusion Models for Forecasting Future Multiple Sclerosis Lesion Masks Conditioned on Treatments
T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper presents a treatment-conditioned spatio-temporal diffusion model that generates future NET2 lesion masks for MS patients from baseline MRI and treatment arm, with counterfactual predictions across six treatments.
desk verdict A genuinely new treatment-conditioned generative forecasting task for MS lesion masks, with an evaluation that stops short of its own claims — the unquantified 15-slice crop is the main thing to fix. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Treatment-conditioned variational diffusion model with a ControlNet adapter. The diffusion UNet is pretrained with class embeddings of the treatment arm summed into time embeddings; a frozen copy then receives a trainable ControlNet whose zero-convolution connections inject baseline MRI conditioning. Three-dimensional volumes are reshaped into pseudo-2D slabs by folding the slice and channel dimensions, so high-resolution generation happens in voxel space without a separate VAE; classifier-free guidance and stochastic dropout of conditioning allow multiple plausible futures, which are ensembled at inference.
What would settle it
Measure the fraction of NET2 lesions outside the central 15-slice slab in the full-volume trial data, then re-score the model's masks against full-brain ground truth. If a non-negligible share of activity lies outside the slab, or if full-brain evaluation substantially lowers the regional, count, or activity metrics, the full-brain forecasting claim fails.
Extended reading notes
Core claim
On its own terms, the paper's central discovery is that future focal MS pathology can be generated, not just scored: a variational diffusion UNet is pretrained to model the distribution of future NET2 masks and then steered by a ControlNet adapter conditioned on the patient's baseline MRI and treatment arm to produce voxel-level NET2 masks at week 96. The model is stochastic by design, and ensembling many samples yields more reliable predictions; the same patient can be re-sampled under different treatment labels to produce counterfactual masks. Across test-set treatment arms, the generated masks beat Monte Carlo baselines that use only population prevalence, and downstream counts and activi
Load-bearing premise
The load-bearing premise is that nearly all NET2 lesion activity in these patients falls inside the central 15 axial slices the model sees; if meaningful activity occurs outside that slab, the model cannot forecast it and the reported metrics overstate full-brain performance.
Editorial extensions
If this is right
- Clinicians could present a patient with a set of plausible future lesion masks and counts under their actual treatment, grounding prognosis in images rather than group averages.
- Switching the treatment label at inference yields counterfactual future masks, making individual treatment-effect comparison visible, pending formal causal validation.
- The same generated masks feed downstream tasks such as lesion count estimation and binary activity classification without retraining the model for those tasks.
- Because the pipeline trains in voxel space from pseudo-2D slabs, it avoids a separately trained latent compressor, simplifying training and preserving spatial fidelity.
- The reported gains over population-level Monte Carlo baselines suggest the model captures patient-specific regional structure in lesion formation.
Reading between the lines
- A natural next experiment the paper only gestures at: quantify individual treatment effects by generating counterfactual masks for the same patient under all six arms and comparing the implied lesion counts to known arm-level efficacy; if differences collapse after controlling for baseline activity, the counterfactual contrast may be associative rather than causal.
- The 15-slice central crop bounds the claim: before calling the model whole-brain, one should measure how much NET2 activity lies outside that slab, since a clinically meaningful fraction would be systematically invisible to the model and to its evaluation metrics.
- Ensemble heatmaps could be repurposed as calibrated uncertainty maps, for example by thresholding high-probability regions to guide radiologist review; a testable check is whether the model's predictive variance across samples matches inter-reader or inter-scan variability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a two-stage conditional diffusion framework for forecasting future NET2 lesion masks in multiple sclerosis. Stage 1 pre-trains a treatment-conditioned diffusion UNet on future NET2 masks; Stage 2 trains a ControlNet that conditions generation on baseline MRI and treatment arm. The method is evaluated on 2131 samples from five RCTs using regional presence/absence accuracy, binary activity prediction, log-count MSE, and qualitative counterfactual generations across six treatment arms. The central claim is that the model accurately predicts voxel-level NET2 lesion masks in a treatment-aware manner.
Significance. If substantiated, this would be a valuable step toward personalized, image-based forecasting of focal MS pathology and treatment response. The strengths are the large multi-centre dataset from actual RCTs, the treatment-conditioned generative formulation, the use of stochastic ensembles to represent outcome variability, and the transparent comparison against population-level baselines. However, the current evaluation does not directly support the paper's strongest voxel-level claim, and one data-preprocessing choice (the 15-slice crop) is load-bearing and unquantified. The work is therefore promising but needs substantive revision before its claims are supported.
major comments (4)
- [Sec. 3 (Dataset and Implementation Details)] The central axial 15-slice crop is load-bearing but unquantified. The text states that 'the vast majority of patients only have NET2 lesion activity occurring within a 15 slice slab taken from the center of the full volume' and then crops all MRI to (15,256,256). Because this crop defines the target space, the conditioning inputs, and every label used in Tables 2-4, any NET2 lesion outside the slab cannot be predicted by construction. Please report the empirical exclusion rate from full-volume labels: the percentage of patients with any NET2 activity outside the slab, and the fraction of NET2 lesions outside the slab. If that fraction is non-negligible, either evaluate on full-brain labels or restrict the claims to the central slab. This concern is not addressed by adding a voxel-level Dice metric inside the slab.
- [Secs. 4.1 and 4.3 (Tables 2-4)] The abstract and Sec. 6 claim that the model 'accurately predicts NET2 lesion masks,' but no voxel- or lesion-level spatial overlap metric is reported. Table 2 is regional presence/absence for only two broad tissue classes; Table 3 is binary activity; Table 4 is log-count MSE. No Dice, lesion-wise detection rate, distance-based metric, or probabilistic overlap is given. Given the acknowledged stochasticity of exact lesion locations, please report at least one lesion-wise or distributional overlap metric (e.g., expected Dice over samples, lesion hit rate within a tolerance) or explicitly soften the claim to regional and count prediction rather than voxel-level mask prediction.
- [Sec. 4.3 and Sec. 3 (inference protocol)] The conversion from the diffusion model's raw output to lesion counts and activity labels is unspecified. The paper states that a morphological opening with a 1x5x5 kernel and connected component analysis are applied, but does not state how the generated continuous mask is binarized, what threshold is used, or how the ensemble of stochastic samples is aggregated (e.g., averaging, majority vote, or per-sample CCA). All quantitative downstream results depend on this binarization and ensemble rule. Please specify the full inference protocol and report sensitivity to the threshold.
- [Secs. 4.1-4.3 (baselines)] The population-level Monte Carlo and mean-count baselines are reasonable sanity checks, but they do not establish that the model exploits image information beyond treatment-group statistics. To support the 'accurately predicts' claim, compare against a simple image-based discriminative baseline trained on the same inputs (e.g., a UNet regressing future NET2 masks, or a classifier/counter regressor using w000 and w048 images plus treatment). If such a baseline is infeasible, say so explicitly and temper the wording from 'accurately predicts' to 'outperforms population-level statistics.'
minor comments (6)
- [Abstract and Sec. 4.4] The abstract uses 'causal' and 'counterfactual' language, while Sec. 4.4 states that 'a detailed quantitative analysis of treatment effects is left for future work' and presents only qualitative examples. Please soften the terminology to match the evidence.
- [Secs. 2 and 3] The conditioning image modalities are described inconsistently: the introduction mentions FLAIR, T2, and gadolinium-enhanced sequences, but the implementation description refers only to 'FLAIR-NET2 image pairs.' Clarify exactly which modalities are used as ControlNet conditioning inputs.
- [Sec. 3 (Implementation Details)] 'Self-attention at 162 and 82 resolutions' is ambiguous; write 16^2 and 8^2 or equivalent clear notation.
- [Table 1] Typo: 'T reatment' should be 'Treatment'.
- [Tables 2-4] No confidence intervals or significance tests are reported. Given the relatively small per-arm test sets, please add confidence intervals for the main metrics.
- [Sec. 3 (Dataset)] Preprocessing details are missing: how were images registered to a template, normalized, and handled for scanner variability? These choices affect the practical reproducibility of the results.
Circularity Check
No significant circularity; predictions are learned from held-out data and compared to independent baselines.
full rationale
The paper's derivation chain is a standard supervised learning pipeline. The diffusion model is trained to predict future NET2 lesion masks conditioned on baseline MRI and treatment arm, with an 80/10/10 train/validation/test split. The pretraining stage learns a prior over NET2 masks from w096 FLAIR-NET2 pairs; this is transfer learning, not a circular reduction, since the final model conditions on w000 and w048 images to predict w096 labels. Evaluation metrics (regional accuracy, binary activity, log-count MSE) are computed on a held-out test set and compared to population-level Monte Carlo baselines that do not use the model's outputs. The 15-slice central crop is a data-preprocessing choice based on observed lesion distribution; while it limits the anatomical scope and is not justified with an exclusion rate, it does not make the model's predictions equivalent to its inputs by construction. Self-citations (e.g., [5] for log-count MSE, future-work suggestions) are contextual rather than load-bearing; no uniqueness claim is imported from the authors' prior work. Thus no circular step is identifiable from the paper's text.
Assumptions & free parameters
free parameters (6)
- central slab depth =
15 axial slices
- NET2 lesion loss up-weight factor =
10
- conditioning dropout rates =
treatment dropped 10%; ce/cf dropped stochastically in Stage 2, rate unspecified
- morphological opening kernel =
1 x 5 x 5
- training subset balancing =
unspecified balanced subset of active and inactive patients
- ensemble size at inference =
100 for heatmaps; unspecified for Tables 3 and 4
assumptions (8)
- standard math Variational diffusion forward and reverse process with the VDM loss (Eqs. 1-2)
- standard math ControlNet zero-convolution adapter learns semantic conditioning without destabilizing the pretrained UNet (Eq. 3)
- standard math Classifier-free guidance with random dropout recovers the conditional distribution
- domain assumption Pseudo-2D slab representation preserves 3D lesion structure
- domain assumption Central 15-slice slab contains the clinically relevant NET2 activity
- domain assumption Treatment arm is a sufficient conditioning signal for counterfactual comparisons
- domain assumption SynthSeg white matter and cortex parcellations are reliable for regional labels
- domain assumption Ensembled stochastic samples reflect a clinically meaningful probability map
Cite this review
Pith. "Pith review of Spatio-Temporal Conditional Diffusion Models for Forecasting Future Multiple Sclerosis Lesion Masks Conditioned on Treatments." pith.science (2026). https://pith.science/paper/XBXO5CYF
@misc{pith2026250807006,
author = {Pith},
title = {Pith review of: Spatio-Temporal Conditional Diffusion Models for Forecasting Future Multiple Sclerosis Lesion Masks Conditioned on Treatments},
year = {2026},
howpublished = {\url{https://pith.science/paper/XBXO5CYF}},
note = {Machine review of arXiv:2508.07006}
}
read the original abstract
Image-based personalized medicine has the potential to transform healthcare, particularly for diseases that exhibit heterogeneous progression such as Multiple Sclerosis (MS). In this work, we introduce the first treatment-aware spatio-temporal diffusion model that is able to generate future masks demonstrating lesion evolution in MS. Our voxel-space approach incorporates multi-modal patient data, including MRI and treatment information, to forecast new and enlarging T2 (NET2) lesion masks at a future time point. Extensive experiments on a multi-centre dataset of 2131 patient 3D MRIs from randomized clinical trials for relapsing-remitting MS demonstrate that our generative model is able to accurately predict NET2 lesion masks for patients across six different treatments. Moreover, we demonstrate our model has the potential for real-world clinical applications through downstream tasks such as future lesion count and location estimation, binary lesion activity classification, and generating counterfactual future NET2 masks for several treatments with different efficacies. This work highlights the potential of causal, image-based generative models as powerful tools for advancing data-driven prognostics in MS.
Figures
Reference graph
Works this paper leans on
-
[1]
Medical Image Analysis 86, 102789 (May 2023)
Billot, B., Greve, D.N., Puonti, O., Thielscher, A., Van Leemput, K., Fischl, B., Dalca, A.V., Iglesias, J.E.: Synthseg: Segmentation of brain mri scans of any contrast and resolution without retraining. Medical Image Analysis 86, 102789 (May 2023). https://doi.org/10.1016/j.media.2023.102789, http://dx.doi. org/10.1016/j.media.2023.102789
-
[2]
Eprint arXiv:2105.05233 (2021)
Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. Eprint arXiv:2105.05233 (2021)
arXiv 2021
-
[3]
In: BrainLes@MICCAI (2017), https://api.semanticscholar.org/ CorpusID:3440295
Doyle, A., Elliott, C., Karimaghaloo, Z., Subbanna, N.K., Arnold, D.L., Ar- bel, T.: Lesion detection, segmentation and prediction in multiple sclerosis clin- ical trials. In: BrainLes@MICCAI (2017), https://api.semanticscholar.org/ CorpusID:3440295
work page 2017
-
[4]
Durso-Finley, J., Falet, J.P., Mehta, R., Arnold, D.L., Pawlowski, N., Arbel, T.: Improving image-based precision medicine with uncertainty-aware causal models (2023), https://arxiv.org/abs/2305.03829
work page Pith review arXiv 2023
-
[5]
In: Konukoglu, E., Menze, B., Venkataraman, A., Baumgartner, C., Dou, Q., Albarqouni, S
Durso-Finley, J., Falet, J.P., Nichyporuk, B., Douglas, A., Arbel, T.: Personalized prediction of future lesion activity and treatment effect in multiple sclerosis from baseline mri. In: Konukoglu, E., Menze, B., Venkataraman, A., Baumgartner, C., Dou, Q., Albarqouni, S. (eds.) Proceedings of The 5th International Conference on Medical Imaging with Deep L...
work page 2022
-
[6]
Favero, G.M., Saremi, P., Kaczmarek, E., Nichyporuk, B., Arbel, T.: Conditional diffusion models are medical image classifiers that provide explainability and un- certainty for free (2025),https://arxiv.org/abs/2502.03687
arXiv 2025
-
[7]
Friedrich, P., Wolleb, J., Bieder, F., Durrer, A., Cattin, P.C.: WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis, p. 11–21. Springer Nature Switzerland (Oct 2024).https://doi.org/10.1007/978-3-031-72744-3_ 2, http://dx.doi.org/10.1007/978-3-031-72744-3_2
-
[8]
The Annals of Applied Statistics 8 (06 2014)
Ge, T., Müller-Lenke, N., Bendfeldt, K., Nichols, T., Johnson, T.: Analysis of multiple sclerosis lesions via spatially varying coefficients. The Annals of Applied Statistics 8 (06 2014). https://doi.org/10.1214/14-AOAS718
Show all 26 references
-
[9]
The New England Journal of Medicine367(12), 1098–1107 (2012)
Gold, R., Kappos, L., Arnold, D.L., Bar-Or, A., Giovannoni, G., Selmaj, K., Tor- natore, C., Sweetser, M.T., Yang, M., Sheikh, S.I., Dawson, K.T., Investigators, D.S.: Placebo-controlled phase 3 study of oral BG-12 for relapsing multiple scle- rosis. The New England Journal of...
2012 doi
-
[10]
Eprint arXiv:2303.09556 (2024)
Hang, T., Gu, S., Li, C., Bao, J., Chen, D., Hu, H., Geng, X., Guo, B.: Efficient diffusion training via min-snr weighting strategy. Eprint arXiv:2303.09556 (2024)
2024 arXiv
-
[11]
The New England Journal of Medicine 376(3), 221–234 (2017)
Hauser, S.L., Bar-Or, A., Comi, G., Giovannoni, G., Hartung, H.P., Hemmer, B., Lublin, F., Montalban, X., Rammohan, K.W., Selmaj, K., Traboulsee, A., Wolin- sky, J.S., Arnold, D.L., Klingelschmitt, G., Masterman, D., Fontoura, P., Belachew, S., Chin, P., Mairon, N., Garren, H....
2017 doi
-
[12]
Eprint arXiv:2207.12598 (2022) 12 G
Ho, J., Salimans, T.: Classifier-free diffusion guidance. Eprint arXiv:2207.12598 (2022) 12 G. Favero et al
2022 arXiv
-
[13]
Eprint arXiv:2301.11093 (2023)
Hoogeboom, E., Heek, J., Salimans, T.: Simple diffusion: End-to-end diffusion for high resolution images. Eprint arXiv:2301.11093 (2023)
2023 arXiv
-
[14]
Scientific Reports 13, 7303 (2023)
Khader, F., Müller-Franzes, G., Tayebi Arasteh, S., et al.: Denoising diffusion prob- abilistic models for 3d medical image generation. Scientific Reports 13, 7303 (2023). https://doi.org/10.1038/s41598-023-34341-2, https://doi.org/10. 1038/s41598-023-34341-2
2023 doi
-
[15]
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision
Kim, J., Park, H.: Adaptive latent diffusion model for 3d medical image to image translation: Multi-modal magnetic resonance imaging study. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 7604–7613 (2024)
2024
-
[16]
Eprint arXiv:2107.00630 (2023)
Kingma, D.P., Salimans, T., Poole, B., Ho, J.: Variational diffusion models. Eprint arXiv:2107.00630 (2023)
2023 arXiv
-
[17]
org/10.1093/brain/122.7.1261, https://doi.org/10.1093/brain/122.7.1261
Lee, M.A., Smith, S., Palace, J., Narayanan, S., Silver, N., Minicucci, L., Fil- ippi, M., Miller, D.H., Arnold, D.L., Matthews, P.M.: Spatial mapping of t2 and gadolinium-enhancing t1 lesion volumes in multiple sclerosis: evidence for distinct mechanisms of lesion genesis? Br...
1999 doi
-
[18]
Annals of Neurology47(6), 707–717 (Jun 2000).https://doi
Lucchinetti, C., Brück, W., Parisi, J., Scheithauer, B., Rodriguez, M., Lassmann, H.: Heterogeneity of multiple sclerosis lesions: implications for the pathogenesis of demyelination. Annals of Neurology47(6), 707–717 (Jun 2000).https://doi. org/10.1002/1531-8249(200006)47:6<70...
-
[19]
Puglisi, L., Alexander, D.C., Ravì, D.: Enhancing spatiotemporal disease progres- sion models via latent diffusion and prior knowledge (2024),https://arxiv.org/ abs/2405.03328
2024 arXiv
-
[20]
Scientific Reports 15, 1208 (2025)
Rachmadi, M.F., Valdés-Hernández, M.C., Makin, S., et al.: Prediction of white matter hyperintensities evolution one-year post-stroke from a single- point brain mri and stroke lesions information. Scientific Reports 15, 1208 (2025). https://doi.org/10.1038/s41598-024-83128-6, ...
2025 doi
-
[21]
Medical Image Computing and Computer- Assisted Intervention (MICCAI) 9351, 234–241 (2015)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer- Assisted Intervention (MICCAI) 9351, 234–241 (2015). https://doi.org/10. 1007/978-3-319-24574-4_28
2015
-
[22]
2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) pp
Sepahvand,N.M.,Arnold,D.L.,Arbel,T.:Cnndetectionofnewandenlargingmul- tiple sclerosis lesions from longitudinal mri using subtraction images. 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI) pp. 127–130 (2020), https://api.semanticscholar.org/CorpusID:218895124
2020
-
[23]
Journal of Neu- rology 261(4), 773–783 (2014)
Vollmer, T.L., Sorensen, P.S., Selmaj, K., Zipp, F., Havrdova, E., Cohen, J.A., Sasson, N., Gilgun-Sherki, Y., Arnold, D.L., Group, B.S.: A randomized placebo- controlled phase III trial of oral laquinimod for multiple sclerosis. Journal of Neu- rology 261(4), 773–783 (2014). ...
2014 doi
-
[24]
arXiv preprint arXiv:2412.13059 (2024)
Wang, H., Liu, Z., Sun, K., Wang, X., Shen, D., Cui, Z.: 3d meddiffusion: A 3d medical diffusion model for controllable and high-quality medical image generation. arXiv preprint arXiv:2412.13059 (2024)
2024
-
[25]
Yoon, J.S., Zhang, C., Suk, H.I., Guo, J., Li, X.: SADM: Sequence-Aware Diffusion Model for Longitudinal Medical Image Generation, p. 388–400. Springer Nature Switzerland (2023). https://doi.org/10.1007/978-3-031-34048-2_30 , http:// dx.doi.org/10.1007/978-3-031-34048-2_30
2023 doi
-
[26]
Eprint arXiv:2302.05543 (2023)
Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. Eprint arXiv:2302.05543 (2023)
2023 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.