Pith. sign in

REVIEW 4 major objections 7 minor 13 references

Position Paper: Building Trust in Synthetic Data for Clinical AI

T0 review · 4 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that trust in synthetic clinical data is earned by controlling its quality, diversity, and proportion, and that a balanced real/synthetic training mix—around 50/50—produces the most trustworthy brain-tumor segmentation…

desk verdict A clearly written position paper whose empirical illustration only varies mixing proportion, so the headline causal claim about quality, diversity, and trust is not supported; worth engaging as a position piece with heavy revision. read the letter →

arxiv 2502.02076 v1 pith:W6DJGAVE submitted 2025-02-04 cs.LG cs.CV

Krishan Agyakari Raja Babu , Supriti Mulay , Om Prabhu , Mohanasankar Sivaprakasam This is my paper · ORCID
classification cs.LGcs.CV
keywords syntheticmedicaldataclinicalAItrustbraintumorsegmentationDicesimilarityaugmentationdiffusionmodelsfactorsmodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that synthetic medical data can be trusted by clinicians, but only when its quality, diversity, and proportion are deliberately controlled. Using brain tumor segmentation as a testbed, it shows that a roughly 50/50 mix of real and synthetic training images makes a segmentation model's performance align most closely on real and synthetic test sets. It also finds that synthetic data is least trustworthy for small tumor regions, implying that generative models need targeted conditioning for small structures. These findings are offered as a framework for deciding how much synthetic data to use in clinical AI pipelines.

What carries the argument

The argument turns on an α-controlled training mixture: $|D_{\text{train}}| = \alpha |D_{R,T}| + (1-\alpha)|D_{S,T}|$, where $\alpha$ sweeps from all-synthetic to all-real training. The synthetic images are generated by Med-DDPM, a conditional diffusion model, using the exact same semantic masks as the real images, creating one-to-one paired real/synthetic samples. Performance is audited with five trust factors ($T_1$–$T_5$) derived from Dice similarity: overall performance, cross-region consistency, real-synthetic alignment via cosine similarity, absolute performance difference, and coefficient of variation. The $\alpha$-sweep plus these Dice-based factors is what lets the paper locate the trust optimum and identify small-region unreliability.

What would settle it

A randomized study in which radiologists rate the same model outputs under different $\alpha$ mixes, blind to the mix, would test whether the $\alpha \approx 0.5$ optimum corresponds to actual clinical trust rather than Dice alignment.

Watch

Extended reading notes

Core claim

The paper's central claim is that trust in synthetic data for clinical AI is not binary but a function of three controllable dataset properties: quality (how representative the generated images are), diversity (how well small and rare attributes are captured), and proportion (the fraction of synthetic images in the training set). From brain tumor segmentation experiments, it claims that the optimal proportion is a balanced mix, $\alpha \approx 0.5$, where the model's Dice scores on real and synthetic test sets are most similar. It further claims that synthetic data is more consistent (lower predictive variability) than real data, but that consistency is bought at the cost of failing to capture small structures like the necrotic tumor core, so synthetic data should not be trusted for small-region tasks without augmentation. The conclusion is operational: a curated blend of real and synthetic data, with attention to underrepresented regions, is what earns clinician trust.

Load-bearing premise

The paper equates clinician trust with consistency between Dice scores on real and synthetic test sets, so if real trust depends on calibration, explainability, data provenance, or integration into clinical workflows, the measured trust factors are only a proxy.

Editorial extensions

If this is right

  • Clinical AI pipelines should train on approximately half real and half synthetic data, since that mix yields the closest alignment between real and synthetic test performance.
  • Small anatomical regions (e.g., necrotic tumor core) will be the least trustworthy under synthetic data, so generative models need targeted conditioning or augmentation for small structures before deployment.
  • Models trained on real data may not generalize to synthetic test sets and vice versa, so evaluation should always include both real and synthetic test sets.
  • Synthetic data appears more consistent and less variable than real data, meaning a synthetic-only test set can understate real-world variability.
  • The five trust factors ($T_1$–$T_5$) provide a quantitative template for auditing synthetic data quality in other segmentation tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Dice-proxy framing suggests a testable extension: survey clinicians' subjective trust on the same model outputs across $\alpha$ values; the paper's five factors would be validated only if survey trust tracks $T_1$–$T_5$.
  • The small-region failure implies generative model evaluation should report per-structure fidelity, not just aggregate Dice; this extends naturally beyond brain tumors to any sparse anatomical target.
  • The $\alpha \approx 0.5$ optimum likely depends on the generator's realism gap, so the practical lesson is to tune $\alpha$ per model/data pair rather than adopt a universal 50/50 rule.
  • The paper does not address privacy-utility tradeoffs, but if balanced mixing becomes practice, future work should measure whether synthetic data still delivers its privacy benefit when mixed at 50/50 with real data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This position paper argues that clinician trust in synthetic medical data is essential for clinical adoption and that the quality, diversity, and proportion of synthetic data directly affect that trust. The empirical component trains a SwinUNETR on BraTS 2021 brain tumor MRI data, using a fixed pre-trained Med-DDPM to generate synthetic images conditioned on the same semantic maps as the real images. Only the mixing proportion α in Eq. (4) is varied, and the model is evaluated on paired real and synthetic test sets using Dice scores. Five 'trust factors' (T1–T5) are defined in Section IV-D from these Dice scores. The main finding is that a balanced mix (α ≈ 0.5) gives the closest Dice alignment between real and synthetic test sets, with synthetic data performing worse for small tumor regions such as NCR. The paper concludes that a balanced real-synthetic mix is crucial for enhancing trust and that synthetic data needs better quality and diversity for small structures.

Significance. If the empirical claim were supported, the paper would offer a practical, quantitative recommendation for mixing real and synthetic data in clinical AI training and an operational proxy for trust. It addresses an important question, uses a public benchmark (BraTS 2021), and includes a clear description of the method. However, the central causal claim is not supported by the presented evidence: the experiment varies only α, not quality or diversity, and 'trust' is an unvalidated Dice-consistency proxy. The balanced-mix observation was already reported in [8], so the novel empirical content is limited. The paper could still serve as a position piece stimulating discussion, but as an empirical demonstration it falls short of what the abstract and conclusion assert.

major comments (4)
  1. [Abstract and Section I] The abstract and Section I claim that the quality, diversity, and proportion of synthetic data directly impact trust in clinical AI models. In Section III, however, the synthetic data is generated by a single fixed pre-trained Med-DDPM and conditioned on exactly the same semantic maps as the real images, as stated in Eq. (2). Only the mixing proportion α in Eq. (4) is varied. Quality and diversity are held constant by construction, so the experiment provides no evidence at all about the first two named causal variables.
  2. [Section IV-D and Section V] Trust is never measured. The five trust factors T1–T5 are defined entirely from Dice similarity coefficients on paired real and synthetic test sets; there is no clinician survey, no calibration or explainability measure, and no validation that these Dice-based quantities are correlated with clinician trust. Because real and synthetic test images are generated from identical ground-truth masks (one-to-one correspondence, Eq. (2)), T3 and T4 may reflect conditioning similarity rather than trustworthiness. The conclusion in Section V-B that 'a balanced mix of real and synthetic datasets is crucial to enhancing trust' is an unsupported extrapolation from this proxy.
  3. [Section V-B and Figs. 4–8] The central balanced-mix claim is based on visual inspection of plots, with no confidence intervals, multiple training seeds, or statistical significance tests. The manuscript itself notes that the balanced-mix result was 'also emphasized in [8],' so the present experiment primarily reproduces a previous observation. Without uncertainty quantification, the assertion that a balanced proportion is 'crucial' for trust is overstated.
  4. [Section V-D and V-E] The paper infers that 'the reliability and trustworthiness of synthetic data are directly proportional to the size of the attributes' (Section V-D) and that synthetic data is less trustworthy for small regions like NCR (Section V-E). This conflates a mechanical property of the Dice score—small structures have lower Dice overlap by construction—with trustworthiness. The analysis does not control for region volume or disentangle size effects from synthetic-data quality effects, so this conclusion is not established.
minor comments (7)
  1. [Section IV-D, Eq. (6)] The notation for T1 is ambiguous: C_{β,k} is described as a 'dice vector,' but the formula sums over k without specifying how the vector is averaged over the M test-set samples. Please define the reduction explicitly.
  2. [Section IV-D, Eq. (7)] T2 uses 'std' inside a 1/K sum, but the exact operation on the samples is unclear. Please specify the standard deviation across regions versus across test-set cases.
  3. [Section V-B and V-F] The expressions 'α = 0.5 ± 0.1' and 'α ∈ [0.5 ± 0.2]' are informal. Use interval notation such as α ∈ [0.4, 0.6].
  4. [Section VI] The conclusion defines 'quality (representativeness)' and 'diversity (size of attributes),' but Section III never defines or measures these quantities. The terminology should be made consistent with the experimental setup.
  5. [Figure 1] Figure 1 is captioned 'Illustration of the Proposed Method' but is not described in the text; please add a description or remove it.
  6. [Section V-B] Since the balanced-mix result is attributed to [8], the manuscript should explicitly state what new evidence this paper adds beyond that reference.
  7. [General] There are typographical issues, including 'V olumetric' in the caption of Fig. 3 and 'α = 0to' in Section V-D. A copyediting pass is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the Dice-based trust analysis is self-contained and [8] is corroborative, not load-bearing.

full rationale

The paper's empirical derivation chain is self-contained. Section III fixes a pre-trained Med-DDPM and constructs Dsyn from the same semantic maps (Eq. 2), Section IV-B varies only alpha in Eq. 4, and Section IV-D defines T1-T5 as explicit functions of Dice scores (Eqs. 6-10). No parameter is fitted to a target quantity and then reported as a prediction of that quantity. The balanced-mix conclusion in Section V-B is read from the paper's own Fig. 4 and Fig. 7, with the citation to [8] serving as corroboration rather than as the load-bearing justification for the result. The self-citation shares the first author, but the central empirical observation does not reduce to that citation; it is supported by the figures generated from the present experiment. The main weaknesses are construct-validity and external-validity concerns: 'trust' is operationalized solely through Dice-based factors, and quality and diversity are not independently manipulated, so the abstract's causal claim about quality and diversity is not demonstrated by this design. These are substantive limitations but not circularity, because the trust factors are not defined in terms of the conclusions, nor are the conclusions forced by the definitions alone. Thus no circular step with a specific reduction can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its empirical claims rest on three assumptions: the paired synthetic test set is a faithful replica, Dice-based factors measure trust, and single-run estimates are stable. The first is standard for such studies, the second is adopted for this paper without validation, and the third is unexamined.

assumptions (3)
  • domain assumption The synthetic test set generated by Med-DDPM conditioned on the same semantic maps as the real test set is a faithful paired replica, so Dice differences isolate synthetic-data quality.
    Section III constructs D_syn with one-to-one correspondence to D_real; all T3/T4 comparisons assume the pairing is meaningful.
  • ad hoc to paper Dice-score consistency between real and synthetic test performance is a valid operationalization of clinician trust.
    Section IV-D defines the five trust factors solely from Dice; the paper provides no external validation linking these metrics to actual clinician trust.
  • domain assumption A single training run per alpha value, without repeated seeds or cross-validation, yields stable estimates of the trust factors.
    Section V reports point estimates for T1-T5 with no error bars; the alpha=0.5 conclusion depends on this stability assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position Paper: Building Trust in Synthetic Data for Clinical AI." pith.science (2026). https://pith.science/paper/W6DJGAVE

@misc{pith2026250202076,
  author       = {Pith},
  title        = {Pith review of: Position Paper: Building Trust in Synthetic Data for Clinical AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6DJGAVE}},
  note         = {Machine review of arXiv:2502.02076}
}
read the original abstract

Deep generative models and synthetic medical data have shown significant promise in addressing key challenges in healthcare, such as privacy concerns, data bias, and the scarcity of realistic datasets. While research in this area has grown rapidly and demonstrated substantial theoretical potential, its practical adoption in clinical settings remains limited. Despite the benefits synthetic data offers, questions surrounding its reliability and credibility persist, leading to a lack of trust among clinicians. This position paper argues that fostering trust in synthetic medical data is crucial for its clinical adoption. It aims to spark a discussion on the viability of synthetic medical data in clinical practice, particularly in the context of current advancements in AI. We present empirical evidence from brain tumor segmentation to demonstrate that the quality, diversity, and proportion of synthetic data directly impact trust in clinical AI models. Our findings provide insights to improve the deployment and acceptance of synthetic data-driven AI systems in real-world clinical workflows.

Figures

Figures reproduced from arXiv: 2502.02076 by the authors.

Figure 1
Figure 1. Illustration of the Proposed Method (a) Mask (b) Flair (R) (c) T1 (R) (d) T1ce (R) (e) T2 (R) (f) Mask (g) Flair (S) (h) T1 (S) (i) T1ce (S) (j) T2 (S) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Comparison of Real (R) and Synthetic (S) Brain Tumor Images. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Volumetric Distribution of Tumor Regions [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of Trust Factor T1 As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: Visualization of Trust Factor T3 T3 values from α = 0 to α = 1, indicating that bias is more prominent in the real data compared to the synthetic data, a finding consistent with T2. Notably, the decline in T3 is most pronounced for NCR, [PITH_FULL_IMAGE:figures/full_f…
Figure 7
Figure 7. Figure 7: Visualization of Trust Factor T4 The mean absolute difference in performance, as shown in [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Visualization of Trust Factor T5 trained entirely on real samples), a similar pattern is observed for both T5,R and T5,S. Specifically, the model trained solely on real data struggles to generalize to synthetic data and vice versa, despite the synthetic replicas being …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

  1. [8]

    Raja Babu, K.A., Sathish, R., Pattanaik, M., Venkataramani, R. (2025). Synthetic Simplicity: Unveiling Bias in Medical Data Augmentation. In: Bhattarai, B., et al. Data Engineering in Medical Imaging. DEMI

    work page 2025

  2. [1]

    Harnessing the power of synthetic data in healthcare: innovation, application, and privacy

    Giuffr `e, M., Shung, D.L. Harnessing the power of synthetic data in healthcare: innovation, application, and privacy. npj Digit. Med. 6, 186 (2023). https://doi.org/10.1038/s41746-023-00927-3

    doi:10.1038/s41746-023-00927-3 2023

  3. [2]

    PLOS Digit Health 2(1): e0000082

    Gonzales A, Guruswamy G, Smith SR (2023) Synthetic data in health care: A narrative review. PLOS Digit Health 2(1): e0000082. https://doi. org/10.1371/journal.pdig.0000082

    doi:10.1371/journal.pdig.0000082 2023

  4. [3]

    McDuff, D., Curran, T., Kadambi, A. (2023). Synthetic Data in Health- care. arXiv preprint arXiv:2304.03243. Available at: https://arxiv.org/ abs/2304.03243

    work page Pith review arXiv 2023

  5. [4]

    Hatamizadeh, A., Nath, V ., Tang, Y ., Yang, D., Roth, H.R., Xu, D. (2022). Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images. BrainLes 2021. Lecture Notes in Computer Science, vol 12962. Springer, Cham. https://doi.org/10.1007/ 978-3-031-08999-2 22

    work page 2022

  6. [5]

    B. H. Menze et al., ”The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS),” in IEEE Transactions on Medical Imaging, vol. 34, no. 10, pp. 1993-2024, Oct. 2015, https://doi.org/10.1109/TMI.2014. 2377694

    doi:10.1109/tmi.2014 1993

  7. [6]

    Dorjsembe, H

    Z. Dorjsembe, H. -K. Pao, S. Odonchimed and F. Xiao, ”Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis,” in IEEE Journal of Biomedical and Health Informatics, vol. 28, no. 7, pp. 4084- 4093, July 2024, https://doi.org/10.1109/JBHI.2024.3385504

    arXiv 2024

  8. [7]

    C. K. Leung, E. W. R. Madill, J. Souza and C. Y . Zhang, ”Towards Trustworthy Artificial Intelligence in Healthcare,” 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI), Rochester, MN, USA, 2022, pp. 626-632, https://doi.org/10.1109/ICHI54592.2022. 00127

    arXiv 2022

Show all 13 references
  1. [9]

    Usman Akbar, M., Larsson, M., Blystad, I. et al. Brain tumor seg- mentation using synthetic MR images - A comparison of GANs and diffusion models. Sci Data 11, 259 (2024). https://doi.org/10.1038/ s41597-024-03073-x

    2024

  2. [10]

    Trust and medical AI: the challenges we face and the expertise needed to overcome them

    Quinn TP, Senadeera M, Jacobs S, Coghlan S, Le V . Trust and medical AI: the challenges we face and the expertise needed to overcome them. J Am Med Inform Assoc. 2021 Mar 18;28(4):890-894. https://doi.org/ 10.1093/jamia/ocaa268. PMID: 33340404; PMCID: PMC7973477

    2021 doi

  3. [11]

    Kaabachi, B., Despraz, J., Meurers, T., Otte, K., Halilovic, M., Prasser, F., & Raisaro, J. L. (2023). Can We Trust Synthetic Data in Medicine? A Scoping Review of Privacy and Utility Metrics. medRxiv. https://doi. org/10.1101/2023.11.28.23299124

    2023 doi

  4. [12]

    Fernandez, V . et al. (2022). Can Segmentation Models Be Trained with Fully Synthetically Generated Data?. Simulation and Synthesis in Med- ical Imaging. SASHIMI 2022. Lecture Notes in Computer Science, vol 13570. Springer, Cham. https://doi.org/10.1007/978-3-031-16980-9 8

    2022 doi

  5. [2024]

    Springer, Cham

    Lecture Notes in Computer Science, vol 15265. Springer, Cham. https://doi.org/10.1007/978-3-031-73748-0 7

    doi

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.