Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single latent diffusion model can learn text and mask conditioning from unpaired histopathology datasets and, at inference, generate images guided by text, mask, or both.

desk verdict Useful augmentation paper with a solid single-modality story, but the advertised joint text+mask capability is never actually trained for, so the headline claim rests on an untested extrapolation. read the letter →

arxiv 2506.23440 v1 pith:6S2UCLZF submitted 2025-06-30 cs.CV

Mahesh Bhosale , Abdul Wasi , Yuanhao Zhai , Yunjie Tian , Samuel Border , Nan Xi , Pinaki Sarder , Junsong Yuan
show 2 more authors
David Doermann Xuan Gong
This is my paper · ORCID
classification cs.CV
keywords diffusionmodelshistopathologyimagesynthesisunpairedtextandmaskconditioninglatentclassifier-freeguidancenucleisegmentationdataaugmentationcellmasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes PathDiff, a diffusion-based method for synthesizing histopathology images when no public dataset contains the same image with both a diagnostic text report and a cell mask. It trains jointly on one text-image dataset and three mask-image datasets, replacing the missing modality in each batch with a null condition. The claimed result is that this produces a unified conditioning space, so the same network can generate from text alone, mask alone, or both, and that joint training improves fidelity, text-image alignment, and mask faithfulness compared with single-modality baselines. The authors further show that these synthetic images improve downstream nuclei segmentation and classification when added to real training data.

What carries the argument

The mechanism is a joint training loop over two unpaired dataset types: in each step a switch chooses a text-to-image or mask-to-image sample with probability psplit, the missing condition is set to a null token, and the denoising U-Net is trained with the latent diffusion noise-prediction objective on LT2I or LM2I. A frozen shared VAE encodes both domains into one latent z, text is embedded by PLIP and cross-attended in U-Net layers, and the mask is embedded by a shallow CNN and injected through duplicated encoder blocks with zero convolutions, following ControlNet. During sampling, the conditioning pair can be (null mask, text), (mask, null text), or (mask, text), and classifier-free guidance combines conditional and unconditional noise estimates. This is what carries the argument: no paired triplet is ever observed, so the null conditions and the shared VAE are what make a unified conditioning space possible.

What would settle it

Present PathDiff with a deliberately conflicting text-mask pair, such as text describing carcinoma paired with a benign-tissue layout mask, and inspect whether the generated image follows both conditions, one, or neither; a genuine unified conditioning space should resolve the conflict, while a model that only memorized null-paired conditions will likely drop one modality.

Watch

Extended reading notes

Core claim

On the paper's terms, the central discovery is that masks and diagnostic text do not need to be paired in the training data for a diffusion model to use both as conditions. By alternating batches between a text-to-image dataset and a mask-to-image dataset, with the absent modality replaced by a null token (an empty string for text and an invalid-label mask for masks), the model learns to embed both conditioning signals in one latent space. At inference, classifier-free guidance can combine the two conditions, and the paper reports that this joint training beats DiffMix, SDM, and ControlNet on FID/KID, faithfulness scores, and downstream CellViT segmentation and classification across three nuclei datasets. The paper also reports that even randomly paired text and masks yield better FID than ControlNet, and that using silver-standard masks with PathCap text improves FID and PLIP similarity further.

Load-bearing premise

A single shared VAE compresses images from the text-image dataset and the mask-image datasets into one consistent latent space, even though the datasets share no images and this consistency is not directly verified.

Editorial extensions

If this is right

  • PathDiff can produce histopathology images from text, mask, or both, so downstream segmentation and classification augmentation no longer requires paired text-mask annotations.
  • Joint training with a large text-to-image corpus improves mask-only generation on PanNuke, CoNIC, and MoNuSAC relative to DiffMix, SDM, and ControlNet.
  • Synthetic images added to real training data raise CellViT Dice, Jaccard, AJI, and F1 on all three datasets, and training on PathDiff synthetic data alone approaches the real-data baseline.
  • Random text-mask pairing still improves FID over ControlNet, and silver-standard masks improve it further, suggesting the unified conditioning space is at least partially modality-agnostic.
  • Scaling the augmentation set from 1x to 2.5x progressively improves PanNuke segmentation and classification before performance plateaus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the model never sees true (text, mask, image) triplets, the reported unified-sampling gains rest on silver-standard masks from DeepCMorph and random cross-dataset pairings; the more direct claim that a real paired annotation helps generation remains untested.
  • Joint training likely acts as implicit regularization or data augmentation: the text stream adds semantic diversity that improves mask-only fidelity and vice versa, a mechanism the paper does not isolate.
  • If the shared-latent premise holds, the same switch-based joint training could extend to other unpaired conditioning modalities, such as genomic or survival data, without architectural change.
  • The psplit ablation shows that 0.2 or 0.8 sometimes gives lower FID than 0.5 on one side of the split, so the benefit of balanced sampling depends on the metric; the paper's choice of psplit = 0.5 is reasonable but not universally dominant.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. PathDiff is a latent diffusion framework that trains a single denoising U-Net on two unpaired datasets: a text-image dataset (PathCap) and three mask-image datasets (PanNuke, CoNIC, MoNuSAC). At training time, each batch uses a mask condition with null text or a text condition with null mask. At inference, the model can be queried with text only, mask only, or both. The paper reports lower CLIP-FID/KID and higher faithfulness/PLIP scores than DiffMix, SDM, and ControlNet, and shows that augmenting CellViT training with PathDiff-generated images improves downstream segmentation and classification. The supplementary includes ablations (psplit, mask depth, VAE fine-tuning), domain-expert preference and Turing tests, and a permutation test for downstream significance.

Significance. If the unified text+mask conditioning is demonstrated, PathDiff would address a real data-scarcity problem in histopathology by combining complementary supervision from existing unpaired datasets. The paper ships code, and the supplementary is unusually thorough: it contains psplit and mask-depth ablations, VAE and U-Net checkpoint selection, in-domain CONCH-FID, domain-expert evaluation, and a permutation test. These are strengths. However, the central claim of true joint conditioning is not yet supported, and the empirical generation comparisons lack error bars; these issues need to be addressed before the paper can be accepted.

major comments (3)
  1. [Sec. 3.2, Algorithm 1, Eqs. (4)-(5), Sec. 4.3, Tab. 4] The model is never trained on samples with both conditions active: every training triplet has either cm = ∅m (for DT2I) or ct = ∅t (for DM2I). The unified sampling evaluation in Sec. 4.3 assesses only FID/KID/PLIP on random or silver-standard pairs; these metrics do not measure whether both conditions causally influence the output. The near-parity of PathDiff-R (FID 11.12) and PathDiff (FID 10.54) in Tab. 4 is consistent with a model that interpolates the two marginal conditionals. To substantiate the central claim, please report an intervention experiment — for example, fix a mask cm and vary the text ct, then measure the change in the generated image (e.g., via PLIP similarity to the text, or by training a text/mask-condition classifier on generated images and measuring its accuracy). Alternatively, train a reference model on a small set of true (text, mask, image) triplets to calibrate what 'both conditions' performance should look like.
  2. [Sec. 3.2] The shared-VAE premise is asserted but not verified. The text states that a shared encoder-decoder results in a unified latent representation 'assuming that the VAE can compress and reconstruct images from both datasets without significant loss,' yet only reconstruction metrics (LPIPS, SSIM, MSE) are reported in the supplementary. No evidence is given that the latent distributions of the T2I and M2I datasets are aligned. Please add a quantitative comparison of the latent spaces (e.g., MMD or FID computed on latent z0 for the two datasets, or a PCA/t-SNE visualization colored by dataset).
  3. [Tables 1-4] No error bars or confidence intervals are reported for any FID, KID, or PLIP value. The permutation test in the supplementary (Sec. 14) applies only to downstream Dice/F1, not to generation metrics. Since some reported margins are small (e.g., PanNuke train FID 6.94 vs 7.14 for SDM in Tab. 1), the claims of superiority are not statistically grounded. Please report standard deviations over multiple sampling runs (or at least the number of seeds) and, if feasible, a significance test for the main generation metric comparisons.
minor comments (5)
  1. [Sec. 3.2 / Fig. 2] The main text does not specify that the mask condition is a 6-channel input (cell-type mask plus instance-edge mask concatenated); this detail appears only in the supplementary (Sec. 12). Please state the mask depth and construction in the main text.
  2. [Algorithm 1] Line 3 refers to 'MT2I' but should read 'DM2I' for consistency with the dataset definitions.
  3. [Sec. 4.3] The term 'PLIP similarity scores' is used; consider clarifying that this is the cosine similarity between PLIP image and text embeddings.
  4. [Supplementary Sec. 8] The Turing test section contains an incomplete sentence: 'We ask to choose whether the given image looks' — please finish it. Also, '42.6' should be '42.6%'.
  5. [Tab. 5] The 'w/ PathDiff Synth' row shows notably lower performance than the baseline on PanNuke (e.g., Dice 0.7311 vs 0.7834), yet the text states that synthetic-only training is 'comparable to the baseline'. Please quantify this claim or discuss the gap.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the unpaired text+mask capability is an under-tested extrapolation rather than a definitional equivalence.

full rationale

PathDiff is an empirical systems paper with no derivation chain in which a predicted quantity reduces by construction to a fitted input. The joint training losses in Eqs. (4)-(5) and Algorithm 1 always set one condition to null, so the network is never directly trained on (cm, ct) with both conditions active; the unified text+mask sampling in Sec. 3.2 is therefore an extrapolation from the two marginal conditionals rather than a circular reduction. The paper explicitly flags the load-bearing shared-VAE assumption in Sec. 3.2 ('This approach assumes that the VAE can compress and reconstruct images from both datasets without significant loss'), which is an unverified premise but not a self-definitional one. The mask-faithfulness metric FS1 measures adherence to the mask signal the model was optimized to follow, yet all compared baselines share that same training objective, and the downstream CellViT segmentation/classification benchmarks use external real test data, so the comparisons retain independent content. The only self-referential element is the use of PLIP both as the frozen text-conditioning encoder and as the text-image alignment metric; because PLIP is not optimized by the diffusion loss, this is metric consistency rather than circularity. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameters renamed as predictions.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on the assumption that two unpaired datasets can share one VAE latent space; this is explicitly acknowledged in Sec. 3.2. Several hyperparameters (guidance scale, psplit, mask depth) are chosen by hand or ablated, and the null-condition probability is inherited from ControlNet without a stated value. No new entities are introduced.

free parameters (4)
  • classifier-free guidance scale w = 1.75
    Chosen for all sampling; no ablation for w is shown.
  • psplit = 0.5
    Probability of sampling from T2I dataset; ablated in supplementary (0.2/0.5/0.8), 0.5 chosen for main experiments.
  • mask depth = 6
    Uses both cell-type mask and instance-mask edges; ablated in supplementary Tab. 12, but main method text does not describe this input representation.
  • puncond = not specified
    Probability of unconditional training; set as in ControlNet [50] but the value is not stated in the paper.
assumptions (4)
  • domain assumption Shared VAE latent space is consistent across T2I and M2I datasets.
    Sec. 3.2 states that a shared VAE encoder-decoder is used, and notes that this assumes the VAE can compress both datasets without significant loss. Joint training depends on this.
  • domain assumption Text captions and masks are complementary and can be combined in a single conditioning space.
    Core premise of the method; no paired data exists to verify that text and mask conditions do not conflict.
  • domain assumption Silver-standard masks from DeepCMorph are accurate enough for evaluating unified conditioning.
    Used to create triplets for the unified sampling evaluation; the quality of these masks is not validated in the main paper.
  • domain assumption CellViT is a reliable auxiliary segmenter for measuring faithfulness.
    FS1 and FS2 rely on CellViT predictions; errors in the segmenter would propagate into the faithfulness scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions." pith.science (2026). https://pith.science/paper/6S2UCLZF

@misc{pith2026250623440,
  author       = {Pith},
  title        = {Pith review of: PathDiff: Histopathology Image Synthesis with Unpaired Text and Mask Conditions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6S2UCLZF}},
  note         = {Machine review of arXiv:2506.23440}
}
read the original abstract

Diffusion-based generative models have shown promise in synthesizing histopathology images to address data scarcity caused by privacy constraints. Diagnostic text reports provide high-level semantic descriptions, and masks offer fine-grained spatial structures essential for representing distinct morphological regions. However, public datasets lack paired text and mask data for the same histopathological images, limiting their joint use in image generation. This constraint restricts the ability to fully exploit the benefits of combining both modalities for enhanced control over semantics and spatial details. To overcome this, we propose PathDiff, a diffusion framework that effectively learns from unpaired mask-text data by integrating both modalities into a unified conditioning space. PathDiff allows precise control over structural and contextual features, generating high-quality, semantically accurate images. PathDiff also improves image fidelity, text-image alignment, and faithfulness, enhancing data augmentation for downstream tasks like nuclei segmentation and classification. Extensive experiments demonstrate its superiority over existing methods.

Figures

Figures reproduced from arXiv: 2506.23440 by the authors.

Figure 1
Figure 1. PathDiff is trained on unpaired datasets, integrating two conditional modalities —Text and Mask— to enable versatile image generation. Unlike unimodal conditional models, PathDiff can generate images conditioned on Text, Mask, or both, allowing greater control and adaptability in image synthesis. gravate this problem. Pathologists must start with low mag￾nification to assess tissue architecture and cellular arrange￾… view at source ↗
Figure 2
Figure 2. (a) PathDiff Training Framework: A training sample is drawn from either the Mask-to-Image (M2I) or the Text-to-Image (T2I) dataset, determined by the probability psplit, which controls the connection of the Switch. The missing condition in each case is set to ∅. When the Switch selects a sample from the M2I dataset, the Mask-to-Image loss LM2I is applied; when it selects from the T2I dataset, the Text-to-Image loss … view at source ↗
Figure 3
Figure 3. Synthetic images generated by PathDiff on Mask to Image datasets (a) PanNuke [12], (b) CoNIC [15], (c) MoNuSAC [46]. PathDiff closely follows the spatial structures defined by the cell type label map, producing synthetic images that closely resemble the spatial arrangement of real images. Method Condition Modality PanNuke CoNIC MoNuSAC Train Test Train Test Train Test (Train, Sample) FID ↓ KID ↓ FID ↓ KID ↓ FID ↓ KI… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Qualitative examples demonstrate the effectiveness of unifying guidance from random pairings of text and mask conditions. (a) Real images are shown with their respective text-based descriptions. (b) PathDiff generated images conditioned solely on text replicate the vis…
Figure 5
Figure 5. Figure 5: Qualitative examples demonstrate the effectiveness of unifying guidance from paired text and silver standard masks generated by [24] from PathCap dataset [43] (a) Real images and Text with silver masks (b) PathDiff generated images (c) ControlNet generated images. Data…
Figure 6
Figure 6. Figure 6: Comparison of segmentation and classification metrics on the PanNuke [12] dataset across augmentation scaling factors. The addition of PathDIff-generated synthetic data consistently increases downstream classification and segmentation performance. 1x uses one synthetic…
Figure 7
Figure 7. Figure 7: Qualitative comparison of synthetic images generated by PathDiff, DiffMix [34], SDM [47], and ControlNet [50] on the CoNIC [15], MoNuSAC [46], and PanNuke [12] datasets. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison of Pathdiff and ControlNet generated images on paired Silver standard masks and texts. psplit PanNuke PathCap Train Test Train Test FID ↓ KID ↓ FID ↓ KID ↓ FID ↓ KID ↓ PLIP ↑ FID ↓ KID ↓ PLIP ↑ psplit = 0.2 10.25 0.0672 11.99 0.0800 15.21 0.0846 …
Figure 9
Figure 9. Figure 9: Qualitative comparison of synthetic images generated by PathDiff and ControlNet [50] on the PathCap [43] dataset. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Both domain experts significantly preferred PathDiff [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Pairwise significance test, p < 0.05 indicates that PathDiff augmentation set helps improve downstream classification and segmentation tasks statistically significantly as compared to other methods. 15. Generation Performance on Hard Pathology Cases To assess how well…
Figure 12
Figure 12. Figure 12: Interactive web interface used for domain expert image preference experiment. [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Interactive web interface used for domain expert Turing test. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CytoDiff: AI-Driven Cytomorphology Image Synthesis for Medical Diagnostics

    cs.CV 2025-07 conditional novelty 4.0 of 10

    Adding 5,000 CytoDiff-generated synthetic white blood cell images per class is reported to improve ResNet-50 accuracy from 27% to 78% and CLIP accuracy from 62% to 77% on the Munich AML dataset.

Reference graph

Works this paper leans on

67 extracted references · 59 canonical work pages · cited by 1 Pith paper

  1. [1]

    Diffinfinite: Large mask-image synthesis via parallel random patch dif- fusion in histopathology

    Marco Aversa, Gabriel Nobis, Miriam H ¨agele, Kai Stand- voss, Mihaela Chirica, Roderick Murray-Smith, Ahmed Alaa, Lukas Ruff, Daniela Ivanova, Wojciech Samek, Frederick Klauschen, Bruno Sanguinetti, and Luis Oala. Diffinfinite: Large mask-image synthesis via parallel random patch dif- fusion in histopathology. In Thirty-seventh Conference on Neural Infor...

    work page 2023

  2. [2]

    Sutherland, Michael Arbel, and Arthur Gretton

    Mikołaj Bi´nkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018. 6, 13

    work page 2018

  3. [3]

    Butte, H

    S. Butte, H. Wang, A. Vakanski, and M. Xian. Enhanced sharp-gan for histopathology image synthesis. In Proceedings of IEEE International Symposium on Biomedical Imaging (ISBI), page 10.1109/isbi53787.2023.10230516, 2023. Epub 2023 Sep 1. 2

    arXiv 2023

  4. [4]

    Daroach, Savannah R

    Gagandeep B. Daroach, Savannah R. Duenweg, Michael Brehler, Allison K. Lowman, Kenneth A. Iczkowski, Ken- neth M. Jacobsohn, Josiah A. Yoder, and Peter S. LaViolette. Prostate cancer histology synthesis using stylegan latent space annotation. In Medical Image Computing and Computer As- sisted Intervention – MICCAI 2022, pages 398–408, Cham,

    work page 2022

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 12

    2009

  6. [6]

    Diffusion mod- els beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion mod- els beat gans on image synthesis. In Advances in Neural Information Processing Systems, pages 8780–8794. Curran Associates, Inc., 2021. 1

    work page 2021

  7. [7]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Proceedings of the 35th Interna- tional Conference on Neural Information Processing Systems, Red Hook, NY , USA, 2024. Curran Associates Inc. 2

    work page 2024

  8. [8]

    Dolezal, Rachelle Wolk, Hanna M

    James M. Dolezal, Rachelle Wolk, Hanna M. Hieromnimon, Frederick M. Howard, Andrew Srisuwananukorn, Dmitry Karpeyev, Siddhi Ramesh, Sara Kochanny, Jung Woo Kwon, Meghana Agni, et al. Deep learning generates synthetic can- cer histology for explainability and education. NPJ Precision Oncology, 7(1):49, 2023. 1

    work page 2023

Show all 67 references
  1. [9]

    The cancer genome atlas pan-cancer analysis project

    JN Cancer Genome Atlas Research Network et al. The cancer genome atlas pan-cancer analysis project. Nature Genetics, 45(10):1113–1120, 2013. 1, 3, 4, 17

    2013

  2. [10]

    Graham, Tom Ver- cauteren, and M

    Virginia Fernandez, Walter Hugo Lopez Pinaya, Pedro Borges, Petru-Daniel Tudosiu, Mark S. Graham, Tom Ver- cauteren, and M. Jorge Cardoso. Can segmentation models be trained with fully synthetically generated data? In Simu- lation and Synthesis in Medical Imaging: 7th Internat...

    2022

  3. [11]

    Multiple in- stance learning for digital pathology: A review of the state-of- the-art, limitations & future potential

    Michael Gadermayr and Maximilian Tschuchnig. Multiple in- stance learning for digital pathology: A review of the state-of- the-art, limitations & future potential. Computerized Medical Imaging and Graphics, 112:102337, 2024. 1

    2024

  4. [12]

    Pannuke: An open pan- cancer histology dataset for nuclei instance segmentation and classification

    Jevgenij Gamper, Navid Alemi Koohbanani, Ksenija Benet, Ali Khuram, and Nasir Rajpoot. Pannuke: An open pan- cancer histology dataset for nuclei instance segmentation and classification. In Digital Pathology, pages 11–19, Cham,

  5. [13]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680,

  6. [14]

    Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images

    Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Medical image anal- ysis, 58:101563, 2019. 7

    2019

  7. [15]

    Conic challenge: Pushing the frontiers of nuclear detection, segmentation, classification and counting

    Simon Graham, Quoc Dang Vu, Mostafa Jahanifar, Martin Weigert, Uwe Schmidt, Wenhua Zhang, Jun Zhang, Sen Yang, Jinxi Xiang, Xiyue Wang, et al. Conic challenge: Pushing the frontiers of nuclear detection, segmentation, classification and counting. Medical image analysis, 92:103...

    2024

  8. [16]

    Learned representation-guided diffusion models for large-image generation

    Alexandros Graikos, Srikar Yellapragada, Minh-Quan Le, Saarthak Kapse, Prateek Prasanna, Joel Saltz, and Dimitris Samaras. Learned representation-guided diffusion models for large-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  9. [17]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017. 6, 12, 13, 15

    2017

  10. [18]

    Classifier-free diffusion guidance, 2022

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. 2, 3, 4

    2022

  11. [19]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840–6851, 2020. 1, 2, 3

    2020

  12. [20]

    Cellvit: Vision transform- ers for precise cell segmentation and classification

    Fabian H¨orst, Moritz Rempe, Lukas Heine, Constantin Sei- bold, Julius Keyl, Giulia Baldini, Selma Ugurel, Jens Siveke, Barbara Gr¨unwald, Jan Egger, et al. Cellvit: Vision transform- ers for precise cell segmentation and classification. Medical Image Analysis, 94:103143, 2024. 7

    2024

  13. [21]

    A visual–language foundation model for pathology image analysis using medical twitter

    Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual–language foundation model for pathology image analysis using medical twitter. Nature Medicine, pages 1–10, 2023. 4, 6, 13, 15

    2023

  14. [22]

    Klanderman, and William J Rucklidge

    Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. Comparing images using the haus- dorff distance. IEEE Transactions on pattern analysis and machine intelligence, 15(9):850–863, 1993. 7

    1993

  15. [23]

    Cellvit: Vision transformers for precise cell segmentation and clas- sification

    Fabian H¨orst, Moritz Rempe, Lukas Heine, Constantin Sei- bold, Julius Keyl, Giulia Baldini, Selma Ugurel, Jens Siveke, Barbara Gr ¨unwald, Jan Egger, and Jens Kleesiek. Cellvit: Vision transformers for precise cell segmentation and clas- sification. Medical Image Analysis, 94...

    2024

  16. [24]

    Histopathological image classification with cell morphology aware deep neural networks

    Andrey Ignatov, Josephine Yates, and Valentina Boeva. Histopathological image classification with cell morphology aware deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6913–6925, 2024. 6, 7, 8

    2024

  17. [25]

    Quilt-1m: One million image-text pairs for histopathology

    Wisdom Oluchi Ikezogwo, Mehmet Saygin Seyfioglu, Fate- meh Ghezloo, Dylan Stefan Chan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-1m: One million image-text pairs for histopathology. arXiv preprint arXiv:2306.11207, 2023. 3

    2023 arXiv

  18. [26]

    Tsfd-net: Tissue spe- cific feature distillation network for nuclei segmentation and classification

    Talha Ilyas, Zubaer Ibna Mannan, Abbas Khan, Sami Azam, Hyongsuk Kim, and Friso De Boer. Tsfd-net: Tissue spe- cific feature distillation network for nuclei segmentation and classification. Neural Networks, 151:1–15, 2022. 5

    2022

  19. [27]

    Mazurowski

    Nicholas Konz, Yuwen Chen, Haoyu Dong, and Maciej A. Mazurowski. Anatomically-controllable medical image gener- ation with segmentation-guided diffusion models. In Proceed- ings of the Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, page pending. Spri...

    2024

  20. [28]

    Levine, Jason Peng, David Farnell, Mitchell Nursey, Yiping Wang, Julia R

    Adrian B. Levine, Jason Peng, David Farnell, Mitchell Nursey, Yiping Wang, Julia R. Naso, Hezhen Ren, Hossein Farahani, Colin Chen, Derek Chiu, Aline Talhouk, Brandon Sheffield, Maziar Riazy, Philip P. Ip, Carlos Parra-Herran, Anne Mills, Naveena Singh, Basile Tessier-Cloutier...

    2020

  21. [29]

    A visual-language foun- dation model for computational pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foun- dation model for computational pathology. Nature Medicine, 30:863–874, 2024. 13

    2024

  22. [30]

    Lucena, R

    O. Lucena, R. Souza, L. Rittner, R. Frayne, and R. Lotufo. Convolutional neural networks for skull-stripping in brain mr imaging using silver standard masks. Artificial Intelligence in Medicine, 98:48–58, 2019. 6

    2019

  23. [31]

    Co- synthesis of histopathology nuclei image-label pairs using a context-conditioned joint diffusion model

    Seonghui Min, Hyun-Jic Oh, and Won-Ki Jeong. Co- synthesis of histopathology nuclei image-label pairs using a context-conditioned joint diffusion model. In Computer Vision – ECCV 2024, pages 146–162, Cham, 2025. Springer Nature Switzerland. 3, 12

    2024

  24. [32]

    Mar- tin, Jochen Lennerz, Stephen Yip, Hossein Farahani, and Ali Bashashati

    Puria Azadi Moghadam, Sanne Van Dalen, Karina C. Mar- tin, Jochen Lennerz, Stephen Yip, Hossein Farahani, and Ali Bashashati. A morphology focused diffusion probabilis- tic model for synthesis of histopathology images. In 2023 IEEE/CVF Winter Conference on Applications of Comp...

    2023

  25. [33]

    M¨uller-Franzes, J.M

    G. M¨uller-Franzes, J.M. Niehues, F. Khader, et al. A multi- modal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical im- age synthesis. Scientific Reports, 13:12098, 2023. 2

    2023

  26. [34]

    Diffmix: Diffusion model- based data synthesis for nuclei segmentation and classification in imbalanced pathology image datasets

    Hyun-Jic Oh and Won-Ki Jeong. Diffmix: Diffusion model- based data synthesis for nuclei segmentation and classification in imbalanced pathology image datasets. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023: 26th International Conference, Vancouver...

    2023

  27. [35]

    Controllable and efficient multi-class pathology nuclei data augmentation using text- conditioned diffusion models

    Hyun-Jic Oh and Won-Ki Jeong. Controllable and efficient multi-class pathology nuclei data augmentation using text- conditioned diffusion models. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2024, pages 36–46, Cham, 2024. Springer Nature Switzerland. 1, 3, 4

    2024

  28. [36]

    Semantic image synthesis with spatially-adaptive nor- malization

    Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive nor- malization. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, 2019. 6

    2019

  29. [37]

    On aliased resizing and surprising subtleties in gan evaluation

    Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR,

  30. [38]

    Pathologygan: Learning deep representations of cancer tissue

    Adalberto Claudio Quiros, Roderick Murray-Smith, and Ke Yuan. Pathologygan: Learning deep representations of cancer tissue. In Proceedings of the Third Conference on Medical Imaging with Deep Learning, pages 669–695. PMLR, 2020. 2

    2020

  31. [39]

    Learning transferable vi- sual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable vi- sual models from natural language supervision. CoRR, abs/2103.000...

    2021 arXiv

  32. [40]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models, 2021. 2, 3, 4, 5, 17, 18

    2021

  33. [41]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020. 6

    2010 arXiv

  34. [42]

    Srinidhi, Ozan Ciga, and Anne L

    Chetan L. Srinidhi, Ozan Ciga, and Anne L. Martel. Deep neural network models for computational histopathology: A survey. Medical Image Analysis, 67:101813, 2021. 1

    2021

  35. [43]

    Pathasst: A generative foundation ai assistant towards artificial general intelligence of pathology

    Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yunlong Zhang, Honglin Li, and Lin Yang. Pathasst: A generative foundation ai assistant towards artificial general intelligence of pathology. Proceedings of the AAAI Conference on Artificial Intelligence ,...

  36. [44]

    Vale-Silva and Karl Rohr

    Lu´ıs A. Vale-Silva and Karl Rohr. Long-term cancer survival prediction using multimodal deep learning. Scientific Reports, 11(1):13505, 2021. Epub 2021 Jun 29. 1

    2021

  37. [45]

    Lennerz, Danny Ruta, Wen Ng, Selvam Thavaraj, Kalliopi P

    Gregory Verghese, Jochen K. Lennerz, Danny Ruta, Wen Ng, Selvam Thavaraj, Kalliopi P. Siziopikou, Threnesan Naidoo, Swapnil Rane, Roberto Salgado, Sarah E. Pinder, and Anita Grigoriadis. Computational pathology in cancer diagnosis, prognosis, and prediction - present day and p...

    2023

  38. [46]

    Multi-organ nuclei segmentation and classification challenge 2020

    Ruchika Verma, Neeraj Kumar, Abhijeet Patil, Nikhil Cherian Kurian, Swapnil Rane, and Amit Sethi. Multi-organ nuclei segmentation and classification challenge 2020. IEEE trans- actions on medical imaging, 39(1380-1391):8, 2020. 5, 6, 7, 12, 14, 17 10

    2020

  39. [47]

    Semantic image synthesis via diffusion models, 2022

    Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models, 2022. 3, 4, 5, 6, 12, 14

    2022

  40. [48]

    Pathldm: Text conditioned latent diffusion model for histopathology

    Srikar Yellapragada, Alexandros Graikos, Prateek Prasanna, Tahsin Kurc, Joel Saltz, and Dimitris Samaras. Pathldm: Text conditioned latent diffusion model for histopathology. In Pro- ceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 51...

    2024

  41. [49]

    Diffusion-based data augmentation for nuclei image segmentation

    Xinyi Yu, Guanbin Li, Wei Lou, Siqi Liu, Xiang Wan, Yan Chen, and Haofeng Li. Diffusion-based data augmentation for nuclei image segmentation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pages 592–602, Cham, 2023. Springer Nature Switzerland. 3

    2023

  42. [50]

    Wu, Weiyang Zeng, Yuxin Zhang, Hussain Salman, and Vladlen Koltun

    Lvmin Zhang, Maneesh K. Wu, Weiyang Zeng, Yuxin Zhang, Hussain Salman, and Vladlen Koltun. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023. 2, 3, 4, 5, 6, 7, 8, 12, 14, 16, 17 11 PathDiff: Histopathology Image Synthesis with ...

    2023 arXiv

  43. [54]

    These augmented datasets were generated from the training split, conditioned on mask, and evaluated on the real test split

    Scaling Augmentation in Downstream Tasks To evaluate the impact of synthetic data augmentation on downstream tasks, we designed three augmented sets for training the CellViT [23] model on the PanNuke [12] dataset. These augmented datasets were generated from the training split...

  44. [55]

    Qualitative Comparison of Synthetic Images In this section, we present a qualitative comparison of synthesized images generated by PathDiff, DiffMix [ 34], SDM [47], and ControlNet [50]. 7.1. Mask-to-Image examples Fig. 7 shows a comparison of synthetic images generated on the...

  45. [56]

    Therefore, we conduct expert evaluation to validate the efficiency of the generated samples

    Domain Expert Assessment We acknowledge that traditional fidelity metrics like FID [17] are only somewhat applicable to histological im- ages as large image datasets like ImageNet [ 5] would un- likely have images from this specific domain. Therefore, we conduct expert evaluat...

  46. [57]

    Each Quadruplet of images was generated using the same conditional mask

    generated from PathDiff, SDM [ 47], ControlNet [50], DiffMix [34]. Each Quadruplet of images was generated using the same conditional mask. Domain experts were asked to choose one of the four images that looked most real. As shown in Fig. 10, both domain experts preferred Path...

  47. [58]

    Algorithm 2 gives overview of the sampling

    Sampling Algorithm We use classifier-free guidance to sample from conditional and unconditional diffusion models to update the final score. Algorithm 2 gives overview of the sampling. We either randomly pair the conditions from non-overlapping M2I and T2I datasets or generate ...

  48. [59]

    We evaluate performance with three values of psplit: 0.2, 0.5, and 0.8

    Considerations for psplit When training jointly on two datasets—Text-to-Image and Mask-to-Image—psplit controls the proportion of data sam- pled from each of them. We evaluate performance with three values of psplit: 0.2, 0.5, and 0.8. Results using only text are shown in Tabl...

  49. [60]

    CONCH [29] is a foundation model trained on large pathology image-text pairs

    In-Domain FID Results For a more faithful assessment of pathology image quality, we compute an in-domain FID using the CONCH [29] en- coder rather than relying solely on CLIP or Inception-based features, which were trained on general natural images and may not capture the nuan...

  50. [61]

    We generate mask edges from instance mask using image processing technics

    Mask Depth Ablation We test the effect of using two types of conditioning mask, first cell type mask and other is instance mask. We generate mask edges from instance mask using image processing technics. Using both masks generates better quality images as seen in Tab. 12, subs...

  51. [62]

    15 Histological section of the adrenal metastatic disease showing tumor cells of varying sizes with oval shapes, eosinophilic cytoplasm, and prominent small nucleoli

    Choice of Pretrained Checkpoints We tried different pretrained checkpoints choices in three module components: V AE, Text-Encoder,and Unet. 15 Histological section of the adrenal metastatic disease showing tumor cells of varying sizes with oval shapes, eosinophilic cytoplasm, ...

  52. [63]

    We randomly swap method labels within each test example and compare mean F1/Dice across 1000 trials

    Significance Test on Downstream Task: We validate that PathDiff’s higher downstream scores aren’t due to chance by running a paired permutation test. We randomly swap method labels within each test example and compare mean F1/Dice across 1000 trials. The resulting p-values are...

  53. [64]

    pathological

    Generation Performance on Hard Pathology Cases To assess how well PathDiff handles challenging, clinically significant images, we split our test set into “pathological” (reports mentioning “carcinoma”) and “non-pathological” cases (reports describing benign findings). Table 15...

  54. [65]

    We use their official repositories to refer to their code

    Details on training previous works: Diffmix and SDM are trained on M2I datasets only. We use their official repositories to refer to their code. For both DiffMix and SDM we use same training settings for all M2I datasets that of PanNuke [12] in [ 34]. For ControlNet we pre-tra...

  55. [66]

    PathDiff trains 694 M parameters

    Computational Costs: Since PathDiff only trains Unet encoder and shallow mask embedder, training costs remain modest, even for joint train- ing. PathDiff trains 694 M parameters. Training time for the largest dataset combination (PathCap [ 43] + Pan- Nuke [12]) is 30 Hours on ...

  56. [67]

    Clear instructions were given to evaluate the images

    Survey Tool: We used an interactive web-based tool to conduct a domain experts survey. Clear instructions were given to evaluate the images. Fig. 12 and Fig. 13 show the web interface used for the domain expert image preference experiment and the Turing test respectively. 18 F...

  57. [2019]

    5, 6, 7, 12, 13, 14, 15, 17, 18

    Springer International Publishing. 5, 6, 7, 12, 13, 14, 15, 17, 18

  58. [2022]

    Springer Nature Switzerland. 2

  59. [2024]

    5, 6, 7, 8, 13, 15, 16, 17, 18

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.