Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities

Yichen Xu

REVIEW 2 major objections 4 minor 1 cited by

Generative tabular models can preserve predictive accuracy while badly distorting average treatment effect estimates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 15:28 UTC pith:MGGLWU5U

load-bearing objection The hybrid-data recipe and the empirical warning are worth engaging; the 'structural failure' theorem rests on an unproven loss-weight premise. the 2 major comments →

arxiv 2604.23904 v3 pith:MGGLWU5U submitted 2026-04-26 stat.ME cs.AIstat.ML

Generative Synthetic Data for Causal Inference: Pitfalls, Remedies, and Opportunities

Yichen Xu This is my paper

classification stat.ME cs.AIstat.ML

keywords synthetic datacausal inferenceaverage treatment effectgenerative modelsGANlarge language modelsoverlap weightprediction loss decomposition

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that fully generative tabular synthesizers, including GAN- and LLM-based models, can preserve predictive utility while distorting average treatment effect (ATE) estimates. The reason is structural: the outcome-prediction loss decomposes exactly into a prognostic-error term plus an overlap-weighted treatment-effect-error term, so when treatment groups are imbalanced or overlap is limited, a generator can reproduce dominant observed outcomes without learning the counterfactual contrast needed for causal inference. The paper proves this mismatch through a loss-decomposition theorem and an analogous result for next-token prediction, then proposes a hybrid approach that generates covariates separately from treatment and outcome mechanisms. In the paper's experiments, the hybrid approach restores causal fidelity, improves targeted augmentation for positivity problems, and serves as a pre-analysis simulation engine for comparing outcome regression, IPW, AIPW, and TMLE. A sympathetic reader would care because synthetic data is increasingly used for sharing, privacy, and augmentation, and this work identifies when it is safe for causal use.

Core claim

The central claim is that predictive accuracy and causal validity are different objectives, and the gap is not incidental but written into the loss function. For any fitted outcome model, the squared prediction loss can be written exactly as the sum of the squared error in the prognostic mean and the squared error in the treatment-effect contrast, weighted by the propensity-overlap product π(W)(1−π(W)). Because this weight is small in low-overlap or imbalanced regions, a model can minimize prediction loss while making large CATE or ATE errors. The paper formalizes this with a sensitivity bound separating covariate-law error from contrast error, and shows further that row-level joint reconstr

What carries the argument

The central object is the exact decomposition of the outcome-prediction loss, L_Q = E_W[Δm(W)^2] + E_W[π(W)(1−π(W)) Δτ(W)^2], which splits prediction error into a prognostic component and a treatment-effect-contrast component weighted by the overlap product. This identity carries the argument: it shows that prediction loss is small when the prognostic mean is well fit, regardless of how poorly the treatment-effect contrast is estimated in low-overlap regions. The paper pairs this with an L2 sensitivity bound that separates ATE error into covariate-law error and contrast error, and with a joint-reconstruction theorem showing that row-level models allocate only 1/(d+1) of the loss to the outco

Load-bearing premise

The 'structural' claim depends on the assumption that row-level generative training assigns the outcome exactly 1/(d+1) of the joint reconstruction loss, so that outcome error is diluted by the number of covariates; if a generator's objective does not have this equal-weight tokenwise form, the precise dilution mechanism does not apply.

What would settle it

Measure the effective gradient contribution of the outcome term versus covariate terms in a GAN-based tabular synthesizer's training objective (e.g., by comparing loss-gradient norms). If the outcome term receives substantially more than 1/(d+1) of the signal, the theorem's premise fails and the claimed structural dilution mechanism would not explain the ATE errors observed for GAN-generated data.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Fully generative synthetic datasets that pass distributional similarity, privacy-distance, and train-on-synthetic-test-on-real checks can still yield biased ATE estimates; those metrics should not certify causal validity.
Hybrid synthesis, where covariates are generated separately from treatment and outcome mechanisms, substantially improves ATE fidelity in the paper's experiments.
Targeted synthetic augmentation for positivity problems can reduce estimator error when improved conditional-effect learning outweighs the covariate shift it introduces.
Hybrid synthetic data can serve as a pre-analysis simulator: repeatedly sampling from a learned covariate generator with explicit outcome and treatment mechanisms lets analysts compare finite-sample bias, variance, and MSE of estimators before the real-data analysis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

We infer that causal-validity checks for synthetic data should report ATE error stratified by propensity or overlap, not just global TSTR or DCR; this would make the failure mode visible.
The analogous decomposition for next-token prediction implies that language-model training may systematically underlearn rare but compositionally important block dependencies; reweighting block frequencies or adding a distinguishability term could align language-model objectives with causal or structural downstream uses.
The paper's real-world HIV trial results show that the quality of the covariate generator is as important as the hybrid split: LLM-based generators were much more faithful than GAN-based ones, suggesting that hybrid synthesis with a weak covariate generator may not rescue causal fidelity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The hybrid-data recipe and the empirical warning are worth engaging; the 'structural failure' theorem rests on an unproven loss-weight premise.

read the letter

The practical message is solid: fully generative tabular synthesizers can look good on predictive and privacy diagnostics while badly distorting ATE estimates, and a hybrid setup—generate W, model A and Y separately, randomize A if you can—largely fixes it. The simulations support that, and the paper is honest about limits.

What is genuinely new: the framing of hybrid generation for causal inference, and the simulation-engine use case. The formal pieces are mostly correct but overclaimed. Theorem A.2 is a correct identity, but it is the standard overlap-weighted R-learner decomposition; not citing that prior work inflates the novelty. Theorem A.3 is a neat analogue for next-token prediction. Proposition 3.1 is fine.

The load-bearing soft spot is Theorem A.1. The 1/(d+1) outcome-loss share is assumed, not derived. GANs don't optimize that loss at all, and GReaT-style LLMs randomize feature order, so the outcome token's contribution isn't fixed at 1/(d+1). The stress-test note lands: the 'structural' diagnosis is not formally supported for either model class. The empirical demonstration doesn't need that theorem, though. The exact identity of A.2 plus the experiments already make the point that predictive fidelity doesn't guarantee causal fidelity.

Minor issues: Tables 1 and 2 are single-seed point estimates with no Monte Carlo error; the code URL is missing; augmentation pairing is under-specified; ACTG has no ground truth, so those ATE numbers are illustrative only.

If I were the editor, I'd send this to a careful referee. The practical contribution is real, the failure mode is important, and the formal flaw is repairable—either prove the loss-weight claim under actual GAN/LLM training, or downgrade it to a motivating heuristic. Worth engaging.

Referee Report

2 major / 4 minor

Summary. The paper argues that fully generative tabular synthesizers (GAN- and LLM-based) can achieve strong predictive or distributional fidelity while badly distorting average treatment effect (ATE) estimates. It formalizes this through: a sensitivity bound separating covariate-law error from treatment-effect-contrast error (Prop. 3.1); an exact decomposition of outcome-prediction loss showing treatment-effect error enters only through the overlap weight (Thm. A.2); a joint-reconstruction tradeoff claiming that equal-weight row-level objectives give the outcome only a 1/(d+1) share (Thm. A.1); and an analogous decomposition for block-level next-token prediction (Thm. A.3). The proposed remedy is hybrid synthesis: generate covariates with a GAN/LLM, then model treatment and outcome mechanisms separately, optionally with randomized synthetic treatment assignment. The paper evaluates this recipe in simulations and on ACTG, reporting improved ATE fidelity, targeted augmentation for positivity problems, and a synthetic simulation engine for estimator benchmarking.

Significance. If the results hold, the paper addresses an important and underexplored gap: standard synthetic-data evaluation metrics do not guarantee causal validity. The exact loss decomposition (Theorem A.2) is a clean and correct statement that prediction loss can be small while causal-contrast error is large, and the hybrid recipe is a practical, actionable remedy. The simulation studies use an external ground truth, and the reproducibility section gives substantial implementation detail. The weakest link is the theoretical claim that the failure is 'structural' for GAN- and LLM-based generators specifically: that claim rests on an assumed equal-weight joint reconstruction loss (Theorem A.1) that is not derived from either model class. This does not undermine the empirical demonstrations or the general decomposition, but it requires a revision of the paper's theoretical scope.

major comments (2)

[§3, Theorem A.1] The 'structural failure' claim for GAN- and LLM-based synthesizers depends on the assumed loss L_joint = L_W + (1/(d+1)) L_Y. This equal-weight premise is not established for either model class. GReaT/GPT-2 serializes rows as text with randomized feature order; the expected contribution of the Y token to the autoregressive log-loss is an order-averaged conditional probability, not 1/(d+1) times a balanced outcome loss, and A is also a generated token, so the denominator should involve d+2 tokens. GAN training does not optimize any tokenwise reconstruction loss, so Theorem A.1 does not apply. Theorem A.2 is correct for any fitted outcome model, but it only shows that low prediction loss can coexist with large causal-contrast error; it does not show that GAN/LLM generative training realizes that regime. Since the abstract and Section 3 describe the failure as structural for full generators
[§4, Proposition A.1] Proposition A.1 is an exact identity, not a result that by itself establishes when augmentation helps; it only restates the tradeoff between improved conditional-effect estimation and covariate shift. The empirical results in Table 1 are promising, but the implementation is underspecified: the paper does not state how many extreme-propensity samples were selected, how the synthetic covariate neighbourhood for pairing was chosen, or how the covariate shift μ_aug - μ_0 was monitored. Without these details, the interpretation that pairing 'reduces error when improved conditional-effect estimation outweighs the covariate shift' is post hoc. Please provide the pairing protocol and, if possible, report the induced covariate shift and the estimated improvement in τ in the extreme-propensity regions.

minor comments (4)

[§5, Table 2] The column 'Real bias' should explicitly state that the real-data benchmark is computed from the known data-generating process in the simulation setting, to avoid confusion with the ACTG application where no ground truth exists.
[§4, threshold definition] The extreme-propensity threshold p_i < 1/sqrt(n log n) is stated without derivation or citation. Please define the regime in which this threshold is appropriate and note that it depends on the sample size n of the seed data.
[§6, Table 3] In the ACTG application there is no ground-truth ATE, yet the text interprets the shift toward 'more consistently negative' hybrid estimates as evidence of reduced distortion. The TSTR and DCR improvements are credible, but the causal-fidelity interpretation is not directly verifiable. Please soften the wording or provide a validation with a known-effect overlay.
[Appendix, Reproducibility] The text promises a supplementary repository but gives no URL or artifact identifier. Add a link or DOI so that the described code and checkpoints can be located.

Circularity Check

1 steps flagged

Partial circularity: Theorem A.1's 1/(d+1) outcome-weight premise already contains the claimed dilution; the core decomposition and empirical benchmarks are independent.

specific steps

self definitional [Theorem A.1 (Appendix); Section 3 (paragraph after Prop. 3.1); Appendix Reproducibility (GReaT details)]
"Suppose the row-level reconstruction loss gives equal weight to the d covariate coordinates and one outcome component, so that L_joint(f) = L_W(f) + 1/(d+1) L_Y(f)."

The theorem's conclusion—that outcome loss is diluted by a factor d+1 and the contrast bound becomes 2 sqrt((d+1)ε)—is just the premise rearranged: assigning Y weight 1/(d+1) in L_joint directly forces L_Y ≤ (d+1)L_joint. The main text then uses this conditional statement to assert that 'joint reconstruction or next-token objectives can downweight the causal contrast' and applies it to GAN- and LLM-based synthesizers. But the paper never establishes that GAN training optimizes L_W + 1/(d+1)L_Y, and for GReaT the appendix says features are randomized in serialization, so the autoregressive token loss is not this fixed weighted-sum loss. The 'structural' failure mechanism for full generators is thus read out of an assumed loss structure, not derived from the generators' actual objectives.

full rationale

Most of the paper's formal apparatus is self-contained and non-circular. Proposition 3.1 is a direct triangle-inequality bound; Theorem A.2 is an exact algebraic decomposition of squared outcome-prediction loss, proved from definitions; Theorem A.3 is an exact chain-rule/KL identity. The simulation and ACTG evaluations are benchmarked against external ground truth (a known DGP and real ACTG data), so the empirical full-versus-hybrid comparison is not a fitted-value tautology. The one genuinely tautological step is Theorem A.1: its premise literally defines L_joint with the outcome weighted 1/(d+1), and its conclusion—outcome-loss dilution and the sqrt(d+1) contrast bound—is that premise rearranged. The paper then invokes this as evidence that GAN/LLM full generators 'structurally' downweight the causal contrast, without showing that either model class optimizes such a loss; the paper's own GReaT description uses randomized-order autoregressive tokens. Because the central decomposition (A.2) and the empirical demonstrations stand independently, the circularity is partial, not total. Self-citations (Xu et al. 2026a,b; Nakada et al. 2025) are contextual and not load-bearing. Score 4.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

No new physical or statistical entities are invented. The ledger's load is methodological: one hand-placed loss-weight assumption (the 1/(d+1) share carrying the 'structural failure' narrative), one hand-chosen cutoff, an underspecified augmentation budget, and two domain assumptions (Lebesgue densities; transportability of Q_hat).

free parameters (3)

Outcome-loss share 1/(d+1) = 1/(d+1) (assumed, not estimated)
Theorem A.1 posits L_joint = L_W + (1/(d+1))L_Y. This weight drives the claimed sqrt(d+1) inflation of the ATE-contrast certificate and is asserted rather than derived from actual GAN/LLM training losses.
Extreme-propensity threshold = 1/sqrt(n log n)
Section 4 defines extreme-propensity samples by p_i < 1/sqrt(n log n); Table 1 outcomes depend on this hand-chosen cutoff.
Augmentation pairing budget = unspecified
Section 4 pairs each extreme-propensity sample with 'nearby' synthetic covariates by Euclidean distance; the number of pairs and distance radius are never given, making the augmentation dose a hidden tuning choice.

axioms (4)

domain assumption Consistency, ignorability, positivity (standard causal identification)
Section 3 paragraph 3: 'Under standard causal assumptions, including consistency, ignorability, and positivity, the ATE is identified as...' The entire ATE framing inherits these.
domain assumption W admits a Lebesgue density on [0,1]^d
Proposition 3.1 states 'suppose P_W and P*_W admit densities p_W and p*_W with respect to Lebesgue measure,' but the experimental DGP includes Bernoulli W1, W2, so the L2 bound does not literally apply to the demonstrated settings.
ad hoc to paper Equal-weight joint reconstruction loss (outcome receives 1/(d+1) share)
Theorem A.1 premise: 'Suppose the row-level reconstruction loss gives equal weight to the d covariate coordinates and one outcome component.' This is the load-bearing structural assumption; not derived for GANs and only approximate for randomized-feature-order autoregressive LLMs.
domain assumption Transportability of fitted outcome model Q_hat to synthetic covariate support
Algorithm 1 and the augmentation experiments sample outcomes from Q_hat evaluated on generated covariates, including rare low-overlap regions where Q_hat was poorly identified; Section 4's flip experiments and Proposition A.1 acknowledge the tradeoff, but the core improvement claims assume this transport holds.

pith-pipeline@v1.3.0-alltime-deepseek · 18293 in / 23043 out tokens · 230314 ms · 2026-08-02T15:28:18.460754+00:00 · methodology

0 comments

read the original abstract

Synthetic tabular data are often evaluated by distributional similarity, privacy distance, or train-on-synthetic-test-on-real predictive performance, but these criteria do not ensure validity for causal inference. We show that fully generative tabular synthesizers, including GAN- and LLM-based models, can preserve predictive utility while distorting average treatment effect (ATE) estimates. The failure is structural: ATE preservation requires both a realistic covariate law and an accurate treatment-effect contrast, whereas prediction loss penalizes treatment-effect error only through an overlap-weighted term. Thus, under imbalance or limited overlap, a generator may reproduce dominant observed outcomes while underlearning intervention-relevant contrasts. We formalize this mismatch through sensitivity and loss-decomposition results. Motivated by this causal analysis and intuition, we propose a hybrid synthetic-data framework for causal inference that generates covariates while modeling treatment and outcome mechanisms separately. We evaluate the framework in three settings: ATE preservation under fully generative versus hybrid synthesis, augmentation for practical positivity problems, and diagnostic simulation engines for comparing OR, IPW, AIPW, and TMLE before real-data analysis. We also stress-test the hybrid construction across settings that vary overlap, covariate dimension, seed sample size, and treatment-effect complexity, including a logistic outcome-model misspecification check. Across controlled simulation experiments, hybrid synthesis improves causal fidelity relative to fully generative baselines; the ACTG application shows improved predictive fidelity and potential for finite-sample estimator benchmarking. LLM-based hybrid synthesis is often more faithful than CTGAN in settings where causal fidelity can be assessed.

Figures

Figures reproduced from arXiv: 2604.23904 by Yichen Xu.

**Figure 1.** Figure 1: Privacy and causal-fidelity diagnostics for synthetic data. Left: predictive utility (TSTR AUC) and view at source ↗

**Figure 2.** Figure 2: Finite-sample benchmarking results under the LLM-based hybrid ACTG simulator. Panels report view at source ↗

Review history (3 revisions) →

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

When Does Synthetic Data Augmentation Improve Score-Based Imbalanced Classification?
stat.ML 2026-06 unverdicted novelty 7.0

Synthetic minority augmentation improves threshold-integrated and optimized classification metrics only under model misspecification by correcting ranking errors, while providing no fundamental benefit beyond possible...

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[2007]

URLhttps://doi.org/10.1214/07-STS227

doi: 10.1214/07-STS227. URLhttps://doi.org/10.1214/07-STS227. Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large language models for text classification: Potential and limitations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi...

Pith/arXiv arXiv 2023
[2022]

doi: 10.1093/aje/kwac087

ISSN 0002-9262. doi: 10.1093/aje/kwac087. URLhttps://doi.org/10.1093/aje/kwac087. 9 D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663–685, 1952. doi: 10.1080/01621459.1952. 10483446. URLhttps://doi.org/10.1080/01621459.1952.10483446. Josep...

arXiv 1952

[1] [2007]

URLhttps://doi.org/10.1214/07-STS227

doi: 10.1214/07-STS227. URLhttps://doi.org/10.1214/07-STS227. Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large language models for text classification: Potential and limitations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi...

Pith/arXiv arXiv 2023

[2] [2022]

doi: 10.1093/aje/kwac087

ISSN 0002-9262. doi: 10.1093/aje/kwac087. URLhttps://doi.org/10.1093/aje/kwac087. 9 D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663–685, 1952. doi: 10.1080/01621459.1952. 10483446. URLhttps://doi.org/10.1080/01621459.1952.10483446. Josep...

arXiv 1952