{"id":"c3f4e206-0630-4fee-8c6c-780a36bcd074","arxiv_id":"2412.16335","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LLM-generated synthetic data improved AUROC for minority groups in 13 of 17 tested settings, but group-specific prompting added no consistent benefit over generic prompts.","lead":"This paper tests whether GPT4-Turbo can generate fake medical records to balance skewed training data and improve predictions for underrepresented patient groups. Across two health datasets, the synthetic-data approach often helped a little, but asking for a specific group rarely outperformed generic prompts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is underpowered: Framingham Asian hold-out sets have ~11 patients, yet the 13/17 win count relies on sub-0.01 AUROC differences with no confidence intervals.","rationale":"The paper is an honest, clearly written empirical comparison with useful exploratory analysis, but the quantitative claim of 'generally superior' performance depends on AUROC differences that are within the noise floor of the evaluation. The reader identified the same load-bearing concern: tiny Framingham test sets and missing uncertainty quantification. My stress test confirms that the 13/17 count is fragile, since many wins are sub-0.01 and the test sets for Asian participants contain about 11 patients. The paper's own exclusion of Asian CHF due to sparse positives underscores the problem. The exploratory analyses (correlation structure, L1 distances, probability density comparisons) are informative and partially independent, and the discussion appropriately hedges the results, which supports a conditional rather than reject verdict. The missing code/data and lack of confidence intervals are addressable. Therefore, the verdict stays CONDITIONAL, with the burden on the authors to demonstrate that the reported differences are not sampling artifacts.","tokens_in":13641,"tokens_out":7814,"duration_ms":63635,"concrete_test":"For each of the 17 settings, bootstrap the hold-out test set 10,000 times and compute the 95% confidence interval for the AUROC difference between the best GPT4T method and the best standard approach, using paired resampling of patient indices. If any of the 13 claimed wins (especially Framingham Asian CVD and the sub-0.01 MIMIC differences) yield intervals crossing zero, the headline claim is not statistically supported. Additionally, report the per-run AUROC spread across the 25 seeds to demonstrate the noise level.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that GPT4T augmentation 'generally outperforms' baselines (13/17 for AUROC) rests on point estimates from hold-out sets far too small to resolve the reported differences. For Framingham Asian participants, the hold-out set is 11 patients (Table S1: 111 total minus 100 used for training, Section 2.3). With CHD prevalence 11.7%, the test set contains roughly one positive case, so a single patient can swing the AUROC by ~0.1. Table 2 reports four decimals with no confidence intervals, and the 13/17 count includes wins with differences as small as 0.0007 (e.g., MIMIC-IV Asian hospitalization: 0.8139 vs 0.8132). Averaging over 25 training samples does not reduce test-set sampling variance because the hold-out set is fixed. Consequently, the superiority count could easily flip under resampling, and the 'little benefit from group-specific prompting' null result is equally underpowered. The paper's own exclusion of Asian CHF due to only 3 positive cases (Section 2.5) acknowledges the severity of sparsity but does not extend the same caution to the remaining Asian outcomes with equally tiny test sets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using GPT4-Turbo (GPT4T) to generate synthetic tabular data for underrepresented racial/ethnic groups, with prompts that either do or do not name the target group, and evaluates whether augmenting the minority training data with these synthetic samples improves downstream logistic-regression AUROC/AUPRC for that group. Experiments are run on MIMIC-IV and Framingham data for Asian, Black, and Hispanic subgroups, with 25 repeated training samples per setting, and are compared against baseline, upweighting, separate models, and SMOTE. The authors report that GPT4T augmentation outperforms the standard baselines in 13 of 17 AUROC comparisons (9 of 17 AUPRC), that group-specific prompting gives little consistent benefit over a generic prompt, and that the method should be viewed as one tool among several for improving subgroup performance.","tokens_in":13908,"tokens_out":4442,"duration_ms":40933,"significance":"If the empirical claims were statistically supported, the paper would be a useful practical contribution: it evaluates an out-of-the-box LLM augmentation strategy on two widely used health datasets, compares against several standard baselines, reports exploratory quality checks, and provides detailed appendices with the prompt template, temperature sensitivity, and minority-size sensitivity analyses. The comparative study design is appropriate for the research question, and the cautious discussion is more measured than the abstract. However, the headline result that GPT4T augmentation is 'generally superior' rests on point estimates from very small held-out test sets without confidence intervals or significance tests, so the central claim is not yet established at the level of precision the paper claims.","major_comments":[{"comment":"The evaluation is underpowered for the Framingham minority groups, and the test-set construction is ambiguous. Table S1 reports only 111 Asian participants in Framingham; Section 2.3 removes 100 minority individuals for training and Section 2.2 supplies 20 minority examples to the prompt, while Section 2.5 says the hold-out set is all remaining samples after removing both the prompt examples and the training samples. Under a natural reading, the Asian test set is at most 11 patients, and if the prompt examples are disjoint from the training sample it could be even smaller. With an 11.7% CHD prevalence, this yields roughly one positive test case, so a single patient can move the AUROC by a large amount. Nevertheless, Table 2 reports four-decimal AUROC differences (e.g., 0.8139 vs 0.8130 for Asian hospitalization). Since the test set is fixed, averaging over 25 training samples reduces only sampling variation in the training data, not the test-set sampling variance. The paper itself excludes Asian CHF because of only 3 positive cases, but the same caution is not applied to Asian CHD/CVD, where the test sets are similarly tiny. Please report bootstrap confidence intervals or significance tests for every cell, or explicitly relabel the table as point estimates not suitable for ranking methods.","section":"Section 2.5 and Table S1"},{"comment":"The central '13 out of 17' superiority count is not robust because many wins are numerically tiny. Examples include MIMIC-IV Asian hospitalization (0.8139 vs 0.8130, a difference of 0.0009) and multiple generic-vs-tailored comparisons reported as 'no additional benefit' with differences below 0.01. Without confidence intervals or paired significance tests, these point estimates are indistinguishable from noise, and the win count could easily flip under resampling. The conclusion that GPT4T augmentation 'generally outperforms' baselines should be softened unless the authors can show that the relevant differences are statistically reliable, including the null result that group-specific prompting provides little benefit, which currently reads as accepting the null hypothesis without equivalence bounds.","section":"Section 3.2 and Table 2"},{"comment":"The temperature hyperparameter is tuned on MIMIC-IV only, and the choice of 0.9 is then transferred to Framingham with the statement 'we expect similar results on the Framingham dataset.' This transfer is load-bearing because the main Table 2 includes all Framingham outcomes. The reported MIMIC-IV insensitivity (AUROC within 1% across temperatures) reduces the risk, but it does not establish that the same holds for Framingham, especially given different feature distributions and outcome prevalences. Please either report a Framingham temperature sensitivity analysis or state this transfer as an explicit limitation.","section":"Appendix D"},{"comment":"The comparison across methods treats each of the 17 settings as an independent test, but the settings share data, preprocessing, and the LLM generation pipeline, so the 13/17 and 9/17 counts are not independent Bernoulli trials. A formal analysis accounting for clustering by dataset/outcome, or at least a per-setting significance test, is needed before drawing aggregate conclusions such as 'in the majority of experiments our method outperforms standard modeling baselines.'","section":"Section 2.4 and Table 2"}],"minor_comments":[{"comment":"The abbreviation footnote says 'CHF: coronary heart failure,' but the standard term is 'congestive heart failure'; please correct this.","section":"Table 2"},{"comment":"The paper alternates between 'GPT4-Turbo' and 'GPT4T'; please define the abbreviation once at first use and use it consistently thereafter.","section":"Throughout"},{"comment":"Please clarify explicitly whether the 20 prompt examples in Section 2.2 are a subset of the 100 training patients or are drawn separately; the current description of the hold-out set does not make this clear.","section":"Section 2.5"},{"comment":"The density plots are difficult to distinguish in grayscale; using different line styles or hatching would improve readability.","section":"Figure 4"},{"comment":"The temperature table header formats 'T emp' instead of 'Temperature'; please fix the typo.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The central barrier is statistical: the headline '13/17' claim and the null result on group-specific prompting are both based on underpowered, fixed test sets without confidence intervals. This is fixable within the manuscript's scope by adding bootstrap CIs and significance tests, clarifying the hold-out construction, and tempering the abstract. The paper's own cautious conclusion is more defensible than the abstract's 'generally superior' framing, so I would encourage a reanalysis rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a careful, honest empirical comparison, and the research question is worth asking: can out-of-the-box GPT-4 Turbo generate synthetic tabular data that improves downstream model performance for underrepresented groups in health datasets? The authors deserve credit for comparing several baselines (upweighting, separate models, SMOTE) and for doing exploratory quality checks on the generated data (correlation structure, L1 distance to real data, racial similarity via a random forest). The null result—group-specific prompting adds little over generic prompts—is a useful datapoint, and the conclusion is appropriately hedged: they call it one tool in the toolbox, not a general solution.\n\nThe main soft spot is exactly what the stress-test note flags, and it lands. The Framingham Asian hold-out set has 11 patients, with roughly one or two positive cases for CHD/CVD. The paper reports four-decimal AUROC differences with no confidence intervals, and several of the 13/17 wins are under 0.01 AUROC. Averaging over 25 training samples does not reduce the test-set variance because the hold-out set is fixed. So the central “generally superior” claim is not as strong as the point estimates make it look. The group-prompting null is equally underpowered. The paper’s own exclusion of Asian CHF (3 positives) shows the authors know sparsity is a problem, but they do not apply the same caution to other Asian outcomes with similarly tiny test sets.\n\nOther concerns are minor. Temperature tuning was done on MIMIC-IV and assumed to transfer to Framingham, but the appendix shows scores within 1% across temperatures, so this is unlikely to change the conclusions. The minority-size sensitivity analysis (50 vs 100 vs 200) shows stable patterns. No code or data is shipped, which matters for reproducibility of prompt-based generation.\n\nWho is this for? Researchers working on LLM-based tabular data generation or fairness in health ML. It is a modest but legitimate contribution, and the authors are honest about its limits. With uncertainty quantification (bootstrap or resampling the hold-out set) and a toned-down conclusion, it would be solid. As it stands, I would send it to peer review because the question is important and the empirical work, while underpowered in places, is genuine and clearly reported. I would ask the authors to add confidence intervals, report test-set sizes alongside the results, and make code/data available. My own verdict would be conditional revision.","headline":"Honest, useful empirical comparison of LLM synthetic data augmentation for health equity, but the headline win rate rests on tiny test sets and missing confidence intervals.","tokens_in":14414,"tokens_out":2575,"would_cite":false,"duration_ms":23641,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper finds that augmenting minority-group training data with GPT-4-Turbo–generated synthetic tabular records improves downstream model performance in 13 of 17 AUROC comparisons, but the benefit is inconsistent and group-specific…","keywords":["large language models","synthetic data","health equity","tabular data","data augmentation","underrepresented groups","MIMIC-IV","Framingham Heart Study"],"falsifier":"Compute bootstrap confidence intervals for the AUROC differences between GPT-4-Turbo augmentation and the baseline for the Framingham minority groups; if most intervals include zero, the claim of superior augmentation is not supported. Alternatively, a replication using a larger held-out test set or an external cohort that finds no consistent improvement would falsify the central claim.","tokens_in":13472,"feed_emoji":"🩺","tokens_out":4599,"duration_ms":40647,"temperature":0.7,"pith_summary":"The paper asks whether synthetic tabular data generated by a large language model can improve predictive performance for underrepresented racial groups in health datasets, where models typically do worse because fewer samples exist. It builds a pipeline that prompts GPT-4-Turbo with 20 real examples from a minority group and uses the returned synthetic records to augment the training set for a downstream logistic regression. Across two datasets (MIMIC-IV and Framingham) and three minority groups, the augmented models beat standard baselines in 13 of 17 AUROC comparisons, with gains up to 0.11 AUROC for Hispanic participants in Framingham. However, the improvement is inconsistent, and explicitly naming the group in the prompt adds little over a generic prompt. The authors conclude the approach is a complementary tool, not a guaranteed fix, for subgroup performance gaps.","feed_headline":"GPT-4 synthetic data helps minority groups in 13 of 17 tests","feed_subtitle":"But naming the group in the prompt adds little, and gains are inconsistent across datasets.","key_machinery":"The central mechanism is a prompt-based synthetic data generation pipeline that follows the structure of Seedat et al. [26]: the model receives a role description, dataset context, 20 randomly selected real examples from the target minority group (resampled to include at least one positive outcome), and instructions to produce diverse new samples. A variant adds an explicit group label (e.g., 'generate data for 10 Asian patients') to test whether group specification matters. The generated records are concatenated with the real training data, and a logistic regression with a group indicator is trained and evaluated per group, compared against upweighting, separate models, and SMOTE as baselines.","core_discovery":"The central claim is that LLM-generated synthetic data can serve as a useful augmentation tool for improving prediction performance on underrepresented demographic groups in tabular health data, but it is not uniformly superior. In a majority of experiments, models trained with GPT-4-Turbo–augmented data achieved higher AUROC and AUPRC than the baseline and than standard approaches such as upweighting, separate models, and SMOTE; the largest AUROC gain was 0.1121 (17%) for Hispanic participants in Framingham on the CHD outcome. Yet the paper finds no consistent benefit from making the prompt group-specific: a generic prompt that does not name the racial group often performs as well or better. The authors also report that GPT-4-Turbo's synthetic data captures some but not all of the real data's correlation structure, and that it tends to fall between minority and majority groups in racial-classifier probability space. They frame the method as 'another tool in the toolbox' for health-equity modeling rather than a standalone solution.","pith_inferences":["The negligible difference between generic and race-tailored prompts suggests that GPT-4-Turbo's pretraining already encodes demographic distributions in the feature space, making explicit labels redundant; a testable extension is to compare prompts that also provide group-specific medical relationships from external literature.","The tiny test sets for Framingham minority groups (e.g., 111 Asian participants) imply that reported AUROC differences of a few hundredths may fall within sampling noise; larger validation cohorts or bootstrap confidence intervals are needed before relying on specific effect sizes.","The method's success appears tied to how separable the minority group's feature-outcome distribution is from the majority's; extending the analysis to other protected attributes, such as age or socioeconomic status, could reveal where augmentation helps most.","The correlation between synthetic and real data being weaker for some pairwise relationships (e.g., between visit types in MIMIC-IV) hints that improving prompt context with explicit causal or clinical knowledge could make generation more faithful and, potentially, more beneficial."],"forward_implications":["Healthcare modelers can use out-of-the-box LLM generation as a low-effort augmentation option for small subgroups without fine-tuning or training a custom generator.","If the mixed results hold, practitioners should evaluate augmentation effects per dataset, group, and outcome rather than assuming universal benefit.","The small difference between generic and group-specific prompting suggests that prompt engineering for demographic labels may not be a productive lever for tabular health data.","The method can be combined with existing fairness interventions, such as upweighting or separate models, for cases where LLM augmentation alone underperforms.","The observed improvements, though often modest, can be clinically meaningful in low-prevalence settings where AUROC gains of 0.1 are substantial."],"supporting_citations":[{"why":"Supplies the prompt structure for generating tabular data from few examples with an LLM.","marker":"[26]"},{"why":"Defines SMOTE, the interpolation baseline that the LLM augmentation must beat.","marker":"[5]"},{"why":"Provides the MIMIC-IV preprocessing and modeling pipeline and documents racial performance gaps.","marker":"[22]"},{"why":"Defines the feature set and Framingham prediction tasks used in the evaluation.","marker":"[3]"},{"why":"Is the MIMIC-IV dataset source.","marker":"[16]"},{"why":"Demonstrates prior LLM-based tabular data generation for class imbalance, motivating the subgroup focus.","marker":"[18]"},{"why":"Shows that group identifiers in prompts diversify image generation, motivating the group-specific prompt comparison.","marker":"[11]"}],"fun_headline_variants":["GPT-4 synthetic data helps minority groups in most, not all, tests","GPT-4 augmentation improves minority predictions, but not always","Group-specific prompts add little to GPT-4 synthetic data gains","LLM synthetic data: a tool for health equity, but inconsistent","GPT-4 synthetic data boosts underrepresented groups in most tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the held-out test sets for minority groups—especially the roughly 111 Asian participants in Framingham—are large enough for the reported AUROC differences to be reliable rather than dominated by sampling noise.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 synthetic data helps minority groups in most, not all, tests","GPT-4 augmentation improves minority predictions, but not always","Group-specific prompts add little to GPT-4 synthetic data gains","LLM synthetic data: a tool for health equity, but inconsistent","GPT-4 synthetic data boosts underrepresented groups in most tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1719,"prompt_tokens":1044,"completion_tokens":675,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":587}},"tokens_in":660,"tokens_out":675,"duration_ms":5869,"temperature":1.0,"reasoning_tokens":587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:41:05.341199+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute bootstrap confidence intervals for the AUROC differences between GPT-4-Turbo augmentation and the baseline for the Framingham minority groups; if most intervals include zero, the claim of superior augmentation is not supported. Alternatively, a replication using a larger held-out test set or an external cohort that finds no consistent improvement would falsify the central claim.","supporting_citations":[{"cited_title":"Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in ultra low-data regimes, February 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the prompt structure for generating tabular data from few examples with an LLM."},{"cited_title":"Smote: synthetic minority over-sampling technique","cited_arxiv_id":null,"evidence_quote":"Defines SMOTE, the interpolation baseline that the LLM augmentation must beat."},{"cited_title":"Coarse race data conceals dispar- ities in clinical risk score performance, August 2023","cited_arxiv_id":null,"evidence_quote":"Provides the MIMIC-IV preprocessing and modeling pipeline and documents racial performance gaps."},{"cited_title":"Comparison of machine learning algorithms for clinical event prediction (risk of coronary heart disease)","cited_arxiv_id":null,"evidence_quote":"Defines the feature set and Framingham prediction tasks used in the evaluation."},{"cited_title":"Mimic-iv","cited_arxiv_id":null,"evidence_quote":"Is the MIMIC-IV dataset source."},{"cited_title":"PreciseDebias: An Automatic Prompt Engineering Approach for Generative AI to Mitigate Image Demographic Biases","cited_arxiv_id":null,"evidence_quote":"Shows that group identifiers in prompts diversify image generation, motivating the group-specific prompt comparison."}],"review_version":1}