REVIEW 3 major objections 3 minor 36 references
Towards 3D Semantic Image Synthesis for Medical Imaging
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that 3D medical volumes generated from semantic maps by a latent diffusion model are close enough to real scans to serve as privacy-preserving training data.
desk verdict Med-LSDM is a new 3D latent semantic synthesis pipeline, but its headline numbers rest on an unreliable FID estimate and the augmentation claim is untested. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Med-LSDM couples a pre-trained 3D VQ-GAN, which compresses a 256x256x32 volume into a discrete latent code of size 64x64x8 with a 16384-entry codebook, with a semantic diffusion model that denoises Gaussian noise in that latent space conditioned on the semantic map. The conditioning is done by semantic diffusion decoder residual blocks that use SPADE-style spatially-adaptive normalization (modulating normalized feature maps with learned scale and shift from the semantic map). After denoising, the latent code is quantized against the frozen codebook and decoded to the voxel space, which is the step that carries the claim that 4x compression preserves clinically relevant structure.
What would settle it
Run the same segmentation network on real test volumes that have been passed through the VQ-GAN encoder and decoder without any diffusion; if the Dice score on these reconstructed volumes drops well below 0.715, then the 4x compression, not the diffusion model, is what limits anatomical fidelity, and the small-domain-gap result would not transfer to tasks needing finer detail.
Extended reading notes
Core claim
The paper's central claim is that running a denoising diffusion model inside the learned latent space of a 3D VQ-GAN, with the semantic map injected at multiple decoder stages via spatially-adaptive normalization, produces volumetric medical images whose distribution and anatomy match real data closely enough for downstream use. The authors report that segmentation performance on Med-LSDM-generated volumes (Dice 0.70964) is nearly identical to segmentation performance on real test volumes (Dice 0.71496), and far above the 2D slice-by-slice baseline (0.602). They position this as the first 3D semantic medical image synthesis in latent space, with the main remaining weakness being behavior in limited-data cross-modal settings like SynthRAD2023.
Load-bearing premise
The whole approach assumes that squeezing each volume down to a quarter of its original size keeps all clinically meaningful details; the paper shows only pictures to support this, so if fine structures get lost in that squeeze, the generated images will look smooth but cannot be trusted for clinical tasks.
Editorial extensions
If this is right
- If correct, researchers can train segmentation networks on synthetic volumes and expect test-time behavior close to real-data training, making augmentation viable when real labels are scarce.
- De-identified semantic maps become a shareable asset: only maps and synthetic volumes need leave an institution, reducing re-identification risk.
- 3D consistency across slices is preserved, so volumetric analyses (coronal and sagittal views) avoid the inter-slice artifacts of stacked 2D generation.
- On cross-modal tasks with small datasets, the method's advantage shrinks, so future work needs modality-aware conditioning or pretraining to close that gap.
Reading between the lines
- A direct test the paper leaves implicit: measure VQ-GAN reconstruction fidelity quantitatively (e.g., Dice between original and reconstructed real volumes); if that reconstruction Dice is much lower than 0.71, the latent space itself, not the diffusion, may be the bottleneck.
- The reported 3D-FID of 0.0054 is unusually low compared with typical FID scales, suggesting the Med3D feature distribution is narrow; a reader should compare models on the same feature extractor before treating absolute numbers as meaningful.
- The method can be extended to unconditional or weakly supervised settings by replacing the semantic map with a predicted or partial label, which would broaden the privacy-preserving claim beyond datasets that already have annotations.
- Another testable extension is an augmentation experiment where a segmentation model is trained on real plus synthetic volumes and compared against real-only training; the paper demonstrates equivalence of synthetic to real but does not directly demonstrate a gain from adding synthetic data.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Med-LSDM, a 3D latent semantic diffusion model for medical image synthesis. A 3D VQ-GAN is first trained to compress volumetric images into a discrete latent space; a semantic diffusion model is then trained in that latent space, using SPADE modules conditioned on semantic maps to guide denoising. The authors evaluate the model on AutoPET, SynthRAD2023, and Duke Breast datasets, comparing against a 2D segmentation-guided diffusion baseline (SegGuidedDiff). The central claims are that Med-LSDM achieves a very small distributional gap with real data (3D-FID 0.0054 on Duke Breast) and that the synthetic data are useful for data augmentation, based on a Dice similarity of 0.70964 between a real-trained segmentation network's output on synthetic images and the input masks, versus 0.71496 on real test images.
Significance. If the claims are substantiated, this would be a useful contribution to 3D medical image synthesis, offering a way to generate volumetric images from semantic maps in latent space, which could reduce computational cost and support privacy-preserving data sharing and augmentation. The architecture is clearly described, the two-stage training (VQ-GAN followed by latent diffusion) is sound in principle, and the paper includes a baseline comparison and qualitative volumetric visualizations. However, the central quantitative evidence for the 'small domain gap' and 'useful for augmentation' claims is not yet convincing: the 3D-FID estimate is prone to severe bias at the reported sample size, no actual augmentation experiment is performed, and the VQ-GAN reconstruction fidelity is only qualitatively assessed. These issues need to be addressed before the main conclusions can be accepted.
major comments (3)
- [Section 4.1 and Section 4.4, Table 3] The 3D-FID results, including the headline value of 0.0054 on Duke Breast, are not reliable as reported. The experimental setup states that the Duke Breast test set contains only 10 scans, and the synthetic images are generated from the semantic maps of that same test set. If the FID is computed on these 10 real and 10 synthetic volumes using high-dimensional Med3D features, the plug-in Fréchet distance estimate is strongly biased toward zero. The paper does not report the number of volumes used for the FID computation nor the feature dimensionality. The stark contrast between the 2D FID (193.620) and 3D-FID (0.0054) on the same data is consistent with this artifact. Please report the exact sample size, feature dimension, and ideally a bootstrap confidence interval or an unbiased estimator. Without this, the 'small domain gap' conclusion (abstract, Section 5) is not established.
- [Section 4.2 and Abstract] The abstract and Section 5 claim that the synthetic data are 'useful for data augmentation,' but no augmentation experiment is conducted. Table 2 reports Dice scores of a segmentation network trained on real Duke Breast data when evaluated on synthetic images. This is a mask-faithfulness check, not a test of whether adding synthetic images to a training set improves downstream task performance. To support the augmentation claim, the authors should train segmentation networks on (i) real data only and (ii) real plus synthetic data, and compare performance on a held-out real test set. This is a standard and necessary experiment for the stated practical utility.
- [Section 4.3] The assumption that the VQ-GAN latent space 'can effectively compress spatial information while preserving its correspondence with the pixel space' is load-bearing for the entire generation chain, but it is supported only by visual inspection (Figure 7). No quantitative reconstruction error is reported (e.g., PSNR/SSIM on held-out volumes, or segmentation Dice on reconstructed images). Given that a 4x compression is used, the authors should provide quantitative evidence that clinically relevant structures survive the compression step. Without this, the possibility that the diffusion model operates on an overly smoothed or anatomically altered latent space cannot be ruled out.
minor comments (3)
- [Throughout] There are numerous typos and grammatical errors, e.g., 'Trasnlation' (Section 2 heading), 'Econder' (Section 3.3), 'comparation' (Figure 6), 'advant a' (Section 4.4), 'Med-LSTM' (should be 'Med-LSDM', Section 2), and inconsistent capitalization. A careful proofread is needed.
- [Table 1] The Duke Breast row lists '3D Scans (Train/Test)' as '(822 + 90)/10'. The text explains that 822 unlabeled scans are used for VQ-GAN training and 90 labeled scans for the diffusion model, but this notation is confusing. Please separate the two training subsets explicitly.
- [Table 2 and Abstract] The abstract reports a Dice score of 0.71496 for real test data, while Table 2 reports 0.715. Please make the values consistent, and report them to the same number of significant digits in both places.
Circularity Check
No significant circularity: the central numerical claims are measured on generated outputs against independent benchmarks; no fitted constant is renamed as a prediction.
full rationale
The paper's derivation chain is empirical and self-contained. The VQ-GAN is trained by reconstruction (Eqs. 3-6), and the latent diffusion model is trained with the standard noise-prediction loss (Eq. 13); neither loss contains the evaluation quantities (3D-FID, Dice). The reported 3D-FID is computed with Med3D features pretrained on an external 3DSeg-8 dataset, and the Dice evaluation uses a segmentation network trained only on real Duke Breast training images and then applied to synthetic images generated from test-set semantic maps. Thus the target numbers are not built into the model's objective or into the conditioning masks by construction. The self-citations to the authors' earlier works (MedGAN [15], USIS [20]) appear only in the related-work discussion and are not load-bearing for the architecture or the evaluation. Concerns that n=10 test volumes make the 3D-FID estimate statistically unreliable, and that the data-augmentation utility claim is not directly tested by an augmentation experiment, are validity and support-strength issues rather than circularity, so they do not change the circularity score.
Assumptions & free parameters
free parameters (3)
- compression factor t =
4
- latent codebook size K and code dimension nz =
K=16384, nz=8
- number of slices per volume =
32
assumptions (5)
- standard math Standard DDPM forward and reverse Gaussian noise assumptions and the simplified L2 noise-prediction objective (Equations 10-13).
- domain assumption The VQ-GAN latent space preserves enough spatial and semantic information for the diffusion process to be meaningful.
- domain assumption A segmentation network trained on real Duke training volumes is a valid probe for distributional similarity of synthetic volumes.
- domain assumption TotalSegmentator-generated CT semantic maps are correct and useful for cross-modal training on SynthRAD2023.
- domain assumption Med3D-pretrained features provide a meaningful and stable 3D-FID measure on 10 test volumes.
Cite this review
Pith. "Pith review of Towards 3D Semantic Image Synthesis for Medical Imaging." pith.science (2026). https://pith.science/paper/KZU5DTHA
@misc{pith2026250700206,
author = {Pith},
title = {Pith review of: Towards 3D Semantic Image Synthesis for Medical Imaging},
year = {2026},
howpublished = {\url{https://pith.science/paper/KZU5DTHA}},
note = {Machine review of arXiv:2507.00206}
}
read the original abstract
In the medical domain, acquiring large datasets is challenging due to both accessibility issues and stringent privacy regulations. Consequently, data availability and privacy protection are major obstacles to applying machine learning in medical imaging. To address this, our study proposes the Med-LSDM (Latent Semantic Diffusion Model), which operates directly in the 3D domain and leverages de-identified semantic maps to generate synthetic data as a method of privacy preservation and data augmentation. Unlike many existing methods that focus on generating 2D slices, Med-LSDM is designed specifically for 3D semantic image synthesis, making it well-suited for applications requiring full volumetric data. Med-LSDM incorporates a guiding mechanism that controls the 3D image generation process by applying a diffusion model within the latent space of a pre-trained VQ-GAN. By operating in the compressed latent space, the model significantly reduces computational complexity while still preserving critical 3D spatial details. Our approach demonstrates strong performance in 3D semantic medical image synthesis, achieving a 3D-FID score of 0.0054 on the conditional Duke Breast dataset and similar Dice scores (0.70964) to those of real images (0.71496). These results demonstrate that the synthetic data from our model have a small domain gap with real data and are useful for data augmentation.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
Balasubramaniam .S, Prasanth Ap, Satheesh Kumar, and V . Kavitha. Medical image analysis based on deep learning approach for early diagnosis of diseases, 03 2024
work page 2024
-
[2]
van der Laak, Bram van Ginneken, and Clara I
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen A.W.M. van der Laak, Bram van Ginneken, and Clara I. Sánchez. A survey on deep learning in medical image analysis. Medical Image Analysis, 42:60–88, December 2017
work page 2017
-
[3]
Transformers in medical image analysis: A review, 2022
Kelei He, Chen Gan, Zhuoyuan Li, Islem Rekik, Zihao Yin, Wen Ji, Yang Gao, Qian Wang, Junfeng Zhang, and Dinggang Shen. Transformers in medical image analysis: A review, 2022
work page 2022
-
[4]
Deep learning techniques for medical image segmentation: Achievements and challenges
Pim Moeskops, Max A Viergever, Juan M Iñesta, and Josien P Pluim. Deep learning techniques for medical image segmentation: Achievements and challenges. Journal of Imaging Informatics in Medicine, 10:123–139, 2022
work page 2022
-
[5]
Andreea Roxana Luca, Tudor Florin Ursuleanu, and Liliana Gheorghe. Impact of quality, type and volume of data used by deep learning models in the analysis of medical images. Informatics in Medicine Unlocked, 29:100911, 2022
work page 2022
-
[6]
Luciano M Prevedello, Safwan S Halabi, George Shih, Carol C Wu, Marc D Kohli, Falgun H Chokshi, Bradley J Erickson, Jayashree Kalpathy-Cramer, Katherine P Andriole, and Adam E Flanders. Challenges related to artificial intelligence research in medical imaging and the importance of image analysis competitions. Radiology: Artificial Intelligence, 1(1):e180031, 2019
work page 2019
-
[7]
Denoising diffusion probabilistic models, 2020
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020
2020
-
[8]
Shifting machine learning for healthcare from development to deployment and from models to data
Angela Zhang, Lei Xing, James Zou, and Joseph C Wu. Shifting machine learning for healthcare from development to deployment and from models to data. Nature, 580(252-256), 2021
work page 2021
Show all 36 references
-
[9]
X. Chen, Y . Zhang, Y . Li, and C. Sun. The use of generative adversarial networks in medical image augmentation: A review. Neural Computing and Applications, 33:15227–15246, 2021
2021
-
[10]
Health insurance portability and accountability act of 1996, August 21, 1996
National Archives Office of the Federal Register and Records Administration. Health insurance portability and accountability act of 1996, August 21, 1996
1996
-
[11]
Segment anything in medical images
Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1), 2024
2024
-
[12]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks, 2018
2018
-
[13]
High-resolution image synthesis and semantic manipulation with conditional gans
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018
2018
-
[14]
Taming transformers for high-resolution image synthesis, 2021
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2021
2021
-
[15]
Medgan: Medical image translation using gans
Karim Armanious, Chenming Jiang, Marc Fischer, Thomas Küstner, Tobias Hepp, Konstantin Nikolaou, Sergios Gatidis, and Bin Yang. Medgan: Medical image translation using gans. Computerized Medical Imaging and Graphics, 79:101684, January 2020
2020
-
[16]
Dual diffusion implicit bridges for image-to-image translation, 2023
Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation, 2023
2023
-
[17]
Semantic image synthesis with spatially-adaptive normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. CoRR, abs/1903.07291, 2019
1903 arXiv
-
[18]
Sean: Image synthesis with semantic region-adaptive normalization
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region-adaptive normalization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2020
2020
-
[19]
Analyzing and improving the image quality of stylegan, 2020
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan, 2020. 14 A PREPRINT - S EPTEMBER 15, 2025
2020
-
[20]
Usis: Unsupervised semantic image synthesis, 2021
George Eskandar, Mohamed Abdelsamad, Karim Armanious, and Bin Yang. Usis: Unsupervised semantic image synthesis, 2021
2021
-
[21]
Semantic image synthesis via diffusion models, 2022
Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models, 2022
2022
-
[22]
High-resolution image synthesis with latent diffusion models, 2022
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2022
2022
-
[23]
Generative adversarial nets
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014
2014
-
[24]
Generative adversarial network in medical imaging: A review
Xin Yi, Ekta Walia, and Paul Babyn. Generative adversarial network in medical imaging: A review. Medical Image Analysis, 58:101552, December 2019
2019
-
[25]
Red-gan: Attacking class imbalance via conditioned generation
Ahmad B Qasim, Ivan Ezhov, Suprosanna Shit, Oliver Schoppe, Johannes C Paetzold, Anjany Sekuboyina, Florian Kofler, Jana Lipkova, Hongwei Li, and Bjoern Menze. Red-gan: Attacking class imbalance via conditioned generation. yet another perspective on medical image synthesis for...
2021
-
[26]
Mazurowski
Nicholas Konz, Yuwen Chen, Haoyu Dong, and Maciej A. Mazurowski. Anatomically-controllable medical image generation with segmentation-guided diffusion models, 2024
2024
-
[27]
Medical diffusion: Denoising diffusion probabilistic models for 3d medical image generation, 2023
Firas Khader, Gustav Mueller-Franzes, Soroosh Tayebi Arasteh, Tianyu Han, Christoph Haarburger, Maximilian Schulze-Hagen, Philipp Schad, Sandy Engelhardt, Bettina Baessler, Sebastian Foersch, Johannes Stegmaier, Christiane Kuhl, Sven Nebelung, Jakob Nikolas Kather, and Daniel ...
2023
-
[28]
Conditional diffusion models for semantic 3d brain mri synthesis
Zolnamar Dorjsembe, Hsing-Kuo Pao, Sodtavilan Odonchimed, and Furen Xiao. Conditional diffusion models for semantic 3d brain mri synthesis. IEEE Journal of Biomedical and Health Informatics, 28(7):4084–4093, July 2024
2024
-
[29]
Prajit Ramachandran, Barret Zoph, and Quoc V . Le. Searching for activation functions, 2017
2017
-
[30]
Rectified linear units improve restricted boltzmann machines
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010
2010
-
[31]
A whole-body fdg-pet/ct dataset with manually annotated tumor lesions
Sergios Gatidis, Tobias Hepp, Marcel Früh, Christian La Fougère, Konstantin Nikolaou, Christina Pfannenberg, Bernhard Schölkopf, Thomas Küstner, Clemens Cyran, and Daniel Rubin. A whole-body fdg-pet/ct dataset with manually annotated tumor lesions. Scientific Data, 9(1):601, 2022
2022
-
[32]
Adrian Thummerer, Erik van der Bijl, Arthur Galapon, Joost J. C. Verhoeff, Johannes A. Langendijk, Stefan Both, Cornelis (Nico) A. T. van den Berg, and Matteo Maspero. Synthrad2023 grand challenge dataset: Generating synthetic ct for radiotherapy. Medical Physics, 50(7):4664–4...
2023
-
[33]
Saha, M.R
A. Saha, M.R. Harowicz, and L.J. Grimm. A machine learning approach to radiogenomics of breast cancer: a study of 922 subjects and 529 dce-mri features. Journal of Magnetic Resonance Imaging, 49:508–516, 2018
2018
-
[34]
Meyer, Maurice Pradella, Daniel Hinck, Alexander W
Jakob Wasserthal, Hanns-Christian Breit, Manfred T. Meyer, Maurice Pradella, Daniel Hinck, Alexander W. Sauter, Tobias Heye, Daniel T. Boll, Joshy Cyriac, Shan Yang, Michael Bach, and Martin Segeroth. Totalsegmentator: Robust segmentation of 104 anatomic structures in ct image...
2023
-
[35]
Hierarchical amortized gan for 3d high resolution medical image synthesis
Li Sun, Junxiang Chen, Yanwu Xu, Mingming Gong, Ke Yu, and Kayhan Batmanghelich. Hierarchical amortized gan for 3d high resolution medical image synthesis. IEEE Journal of Biomedical and Health Informatics , 26(8):3966–3975, 2022
2022
-
[36]
Med3d: Transfer learning for 3d medical image analysis, 2019
Sihong Chen, Kai Ma, and Yefeng Zheng. Med3d: Transfer learning for 3d medical image analysis, 2019. 15 A PREPRINT - S EPTEMBER 15, 2025 Appendix A Number of model parameters of our architecture Table 4: Trainable Parameters of VQ-GAN Components Name Encoder Decoder 2D-Dis 3D-...
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.