Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Embedding-Driven Diversity Sampling to Improve Few-Shot Synthetic Data Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Diversity sampling of few-shot examples makes synthetic clinical notes almost as useful as real data for fine-tuning classifiers.

desk verdict Useful idea, but the central diversity-vs-random comparison is confounded by different baseline sets, so the headline claims don't hold as stated. read the letter →

arxiv 2501.11199 v2 pith:M5NJK4IC submitted 2025-01-20 cs.CL

Ivan Lopez , Fateme Nateghi Haredasht , Kaitlin Caoili , Jonathan H Chen , Akshay Chaudhari This is my paper · ORCID
classification cs.CL
keywords diversitysamplingsyntheticclinicaltextfew-shotpromptingCheXpertlargelanguagemodelsdataaugmentationembedding-drivenselectionNLP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the way you pick the few real examples shown to a large language model matters as much as the generation itself. Selecting the 50 examples by embedding-driven diversity sampling—spreading them across the semantic space of clinical notes—produces synthetic chest radiology reports that are statistically indistinguishable from real ones and that train a classifier nearly as well as real data. The authors show that models fine-tuned with this synthetic data reach a 0.85 AUC with roughly 40% less data for AUROC and 30% less for AUPRC, and that augmenting with synthetic notes improves AUROC by 57% and AUPRC by 68% over random or zero-shot baselines. If correct, this gives a path to building clinical text classifiers when expert-annotated data is scarce.

What carries the argument

The central object is the 'diverse data' set: 50 real clinical notes chosen as k-means centroids in a two-dimensional UMAP projection of SFR-Embedding-Mistral contextualized sentence embeddings. These notes serve as the five-shot examples in prompts to LLaMA-3.1 8B Instruct, steering the generator to reproduce the syntactic range of real radiology reports. The mechanism that carries the argument is the selection step: covering the embedding space with centroids rather than picking examples at random, so the generated notes span the same semantic variety as the real corpus.

What would settle it

Re-run the same pipeline with the 50 diverse examples labeled by manual clinician annotation instead of CLEAR's automatic labels; if the reported AUROC/AUPRC gains and the real-to-synthetic gap change materially, the results depend on CLEAR's label accuracy rather than on diversity sampling itself.

Watch

Extended reading notes

Core claim

Diversity sampling produces synthetic clinical text that is closer to real notes in embedding space and in clinician judgment, and that delivers better downstream fine-tuning efficiency and final performance than random or zero-shot sampling. On CheXpert chest radiology reports, the approach generates notes that a clinical expert cannot reliably distinguish from real clinical text (52% correct synthetic identification, p=0.37), whereas zero-shot notes are easily spotted (96% correct, p<0.001). Fine-tuning Bio+Clinical BERT with these notes reaches the 0.85 AUROC threshold in 4.6 average steps versus 8.0 for random and 7.2 for zero-shot, and the final real-to-synthetic performance gap is 4.1% compared with 10.6% and 8.5% for the baselines. One diversity-sampled synthetic note is worth about 0.9 times a real note, a 60% improvement in value over random and zero-shot sampling.

Load-bearing premise

The pipeline assumes the CLEAR automatic labeling pipeline produces accurate presence/absence labels for the 50 diverse notes and the 650 real-world augmentation notes; if those labels are systematically wrong, the synthetic notes and the augmented training data inherit the errors and the performance-gap measurements are biased.

Editorial extensions

If this is right

  • Reaching the 0.85 AUROC threshold requires 112 augmentation points with diversity sampling versus 177 for random and 176 for zero-shot, so less training data is needed.
  • The final performance gap to real-data training is 4.1% for diversity sampling versus 10.6% (AUROC) and 13.2% (AUPRC) for random and 8.5% and 12.9% for zero-shot, meaning synthetic data comes close to matching real data.
  • One diversity-sampled note is roughly 0.9 times as effective as a real note, versus about 0.56 for random and 0.57 for zero-shot, a large gain in data value.
  • A clinician cannot reliably tell diversity- or random-sampled synthetic notes from real clinical text, while zero-shot notes are clearly artificial, supporting the authenticity of the generated data.
  • Fine-tuning with diversity-sampled augmentation reaches the 0.85 AUROC cutoff in 4.6 average steps, close to the 4.0 steps of real-world data and well below the 8.0 and 7.2 steps of the baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benefit likely comes from coverage of the embedding space rather than the specific embedding model, so the approach may transfer to other note types, languages, or clinical domains with any reasonable semantic encoder.
  • A practical workflow implied but not tested is to use automated labeling on the diverse 50 notes, generate synthetic notes for downstream tasks, and reserve expert annotation for a small one-time cost, reducing the annotation bottleneck.
  • The Turing test result suggests synthetic clinical text may be usable for privacy-restricted sharing or model development without exposing real patient notes, though the paper does not address de-identification risk.
  • The 0.9x effectiveness ratio is measured at a single AUC threshold; whether diversity sampling retains its advantage at higher performance ceilings or on multi-label tasks remains an open, testable question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an embedding-driven diversity sampling approach for few-shot synthetic clinical text generation. The method embeds real clinical notes with SFR-Embedding-Mistral, reduces dimensionality with UMAP, clusters with k-means, and selects 50 centroid-based diverse notes. These notes are used as few-shot prompts for LLaMA-3.1 8B to generate synthetic notes, which are then used to augment fine-tuning data for Bio+Clinical BERT on five CheXpert classification tasks. The authors compare against random few-shot sampling, zero-shot generation, and real-world sampling, reporting that diversity sampling improves synthetic data authenticity (cosine similarity and Turing test), reduces the number of augmentation steps needed to reach a 0.85 AUROC/AUPRC threshold, narrows the real-to-synthetic performance gap, and yields synthetic data that is approximately 0.9 times as effective as real data.

Significance. If the central claim is sound, the method offers a practical, low-compute way to bootstrap synthetic clinical text for fine-tuning when expert annotations are scarce. The downstream evaluation uses a manually annotated held-out test set, which is independent of the proposed method, and the comparison against multiple control conditions is a reasonable design. The Turing test with an expert clinician is a meaningful authenticity check. However, the paper's headline comparison between diversity and random sampling is confounded by the baseline selection procedure (Section 2.9), and the authenticity claim is not consistently supported by the reported numbers (Table 1). These issues are load-bearing for the paper's three stated contributions, so the present version does not yet establish the causal benefit of diversity sampling for synthetic data generation.

major comments (4)
  1. [Section 2.9] The diversity-vs-random comparison conflates baseline composition with synthetic-data quality. The text states that in the few-shot prompting experiments (both diversity and random), the 50 baseline real notes used for fine-tuning are the same notes used for prompting; for diversity these are k-means centroid-selected, while for random they are randomly sampled. Consequently, the diversity condition starts from a deliberately diverse and potentially more informative baseline, so the faster learning curves (Table 3, Figure 2), better final AUROC/AUPRC (Table 4), and the real-to-synthetic ratio (Table 5) may reflect baseline informativeness rather than the quality of the synthetic notes generated from diverse prompts. To isolate the effect of the synthetic data, the baseline should be held fixed across conditions (e.g., use the same 50 diverse notes as the fine-tuning baseline for both diversity and random prompting, or vary only the prompt-example selection).
  2. [Section 3.1 / Table 1] The claim that diversity sampling produces synthetic notes 'most similar to real-world data' is not supported by the reported numbers. Table 1 reports an average cosine similarity distance of 0.82 for both diversity sampling and zero-shot sampling, with random sampling at 0.73. The text says the diversity method was 'most similar' compared to zero-shot, but the table shows no difference on average. Please clarify the direction of the metric (whether lower or higher is better) and explain why zero-shot and diversity are equal; as written, the authenticity comparison does not differentiate diversity from zero-shot.
  3. [Sections 2.4 and 2.8] The cosine-similarity authenticity evaluation uses the same embedding model (SFR-Embedding-Mistral) that is used to perform diversity sampling. Because the diversity selection explicitly optimizes for spread in this embedding space, measuring distance to real data in that same space risks favoring the diversity condition by construction. An independent embedding model (for example, Bio+Clinical BERT or a different sentence encoder) should be used for evaluation, or the authors should explain why this is not circular.
  4. [Sections 2.5, 2.7, and 2.9] Label provenance is not specified consistently across conditions. The diverse baseline notes are labeled with the CLEAR pipeline (Section 2.5), and the real-world augmentation set is also labeled with CLEAR (Section 2.7), but the random sampling baseline and zero-shot baseline appear to use the original CheXpert agent labels from the working dataset without relabeling. If different label sources are used for different conditions, the comparison is confounded by label noise. Please specify the labeling procedure for every training subset and, ideally, report a validation of CLEAR's label accuracy on a sample of the working dataset.
minor comments (5)
  1. [Section 2.5] The prompt-generation procedure for the random sampling control is not stated as explicitly as for diversity sampling; specify whether the same number of prompts (325 per class) and the same five-shot example sampling procedure were used.
  2. [Table 1] The term 'cosine similarity distance' is ambiguous. Define the formula (e.g., 1 - cosine similarity) and clarify whether larger or smaller values indicate greater similarity to real-world data.
  3. [Figure 2 and Table 4] No information is given about the number of random seeds or repeated fine-tuning runs used to compute the reported means and 95% confidence intervals. Please state the number of runs and how the intervals were constructed.
  4. [Table 5] The method for computing 'additional data to reach AUROC threshold of 0.85' is not described; clarify whether this is interpolation between the 25-point increments or from a fitted curve.
  5. [Section 3.3] The phrase 'one diversity sampled data point is approximately 0.9 times as effective as a real-world data point' conflates the number of augmentation points needed from different baselines with per-point effectiveness; please rephrase or provide a direct per-point marginal-effect analysis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central utility claim is evaluated on a manually annotated held-out test set, and the reported efficiency ratios are arithmetic summaries of measured learning curves.

full rationale

The paper's central utility claim—that embedding-driven diversity sampling yields synthetic data that improves fine-tuning efficiency and final AUROC/AUPRC—is evaluated against a manually annotated held-out test set (Section 2.3), which is independent of all model inputs and fitted values. The learning-rate thresholds and real-to-synthetic ratios (Tables 3–5) are computed directly from measured fine-tuning curves, not derived from the selection procedure, so no prediction reduces to the method's own inputs by construction. The only self-citation, the CLEAR labeling pipeline (reference 32), is used as a data-labeling tool and is not a uniqueness theorem or an ansatz smuggled in to force the paper's conclusions; the central diversity-versus-random comparison would still be meaningful, though possibly confounded, even if CLEAR were replaced by another labeler. The authenticity metric's use of the same SFR-Embedding-Mistral model for both example selection and cosine similarity is a measurement-bias concern rather than a by-construction equivalence, and the differing 50-note baselines between conditions are a confound rather than a circular derivation. The Turing test provides partially independent support for the authenticity claim. The paper's stated limitation (Section 4) concerns generalizability, not circularity. Therefore no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method is empirical and does not introduce new theoretical entities. The hand-chosen hyperparameters (cluster count, prompt count, threshold) are methodological choices that shape the reported effect sizes; the key assumptions are automatic label quality and the representativeness of the embedding model.

free parameters (6)
  • Number of clusters (k) = 50
    Chosen by hand for k-means clustering of UMAP-reduced embeddings; determines the diversity pool and affects which examples are selected for few-shot prompting (Section 2.4).
  • Few-shot examples per prompt = 5
    Randomly sampled from the 50 diverse points; affects prompt content and synthetic output (Section 2.5).
  • Number of prompts per condition = 650 per entity (325 positive, 325 negative)
    Determines synthetic dataset size; chosen arbitrarily (Section 2.7).
  • AUC threshold for learning rate analysis = 0.85
    Used to define data/step efficiency metrics (Tables 3 and 5); choice affects reported improvements.
  • Number of baseline data points = 50
    Fine-tuning starting point; affects all conditions equally but the sampling of baseline differs between conditions (Section 2.9).
  • UMAP hyperparameters = not specified
    Parameters such as n_neighbors and min_dist are not reported, yet they control the 2D projection used for clustering (Section 2.4).
assumptions (4)
  • domain assumption CLEAR pipeline provides accurate labels for presence and negation of the five clinical concepts in radiology notes.
    Used to label the 50 diverse points and the 650 real-world augmentation points; if labels are noisy, synthetic data and augmentation inherit the noise (Sections 2.5, 2.7).
  • domain assumption SFR-Embedding-Mistral embeddings capture clinically relevant syntactic and semantic similarity.
    Both the diversity sampling and the cosine similarity evaluation rely on this model; selection and evaluation are therefore not independent (Sections 2.4, 2.8).
  • domain assumption LLaMA-3.1 8B Instruct can generate realistic clinical notes from few-shot prompts that include real examples.
    The entire synthetic data pipeline assumes the base model is capable of producing text that matches clinical style when given examples (Section 2.5).
  • domain assumption The manually annotated held-out test set of 200 notes per entity is a stable benchmark.
    All evaluations use a single clinician's annotations; no inter-annotator agreement is reported (Section 2.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Embedding-Driven Diversity Sampling to Improve Few-Shot Synthetic Data Generation." pith.science (2026). https://pith.science/paper/M5NJK4IC

@misc{pith2026250111199,
  author       = {Pith},
  title        = {Pith review of: Embedding-Driven Diversity Sampling to Improve Few-Shot Synthetic Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5NJK4IC}},
  note         = {Machine review of arXiv:2501.11199}
}
read the original abstract

Accurate classification of clinical text often requires fine-tuning pre-trained language models, a process that is costly and time-consuming due to the need for high-quality data and expert annotators. Synthetic data generation offers an alternative, though pre-trained models may not capture the syntactic diversity of clinical notes. We propose an embedding-driven approach that uses diversity sampling from a small set of real clinical notes to guide large language models in few-shot prompting, generating synthetic text that better reflects clinical syntax. We evaluated this method using the CheXpert dataset on a classification task, comparing it to random few-shot and zero-shot approaches. Using cosine similarity and a Turing test, our approach produced synthetic notes that more closely align with real clinical text. Our pipeline reduced the data needed to reach the 0.85 AUC cutoff by 40% for AUROC and 30% for AUPRC, while augmenting models with synthetic data improved AUROC by 57% and AUPRC by 68%. Additionally, our synthetic data was 0.9 times as effective as real data, a 60% improvement in value.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 26 canonical work pages

  1. [2]

    M., Savova, G

    Meystre, S. M., Savova, G. K., Kipper-Schuler, K. C. & Hurdle, J. F. Extracting information from textual documents in the electronic health record: a review of recent research. Yearb. Med. Inform. 128–144 (2008)

    work page 2008

  2. [3]

    Wornow, M. et al. Zero-Shot Clinical Trial Patient Matching with LLMs. Preprint at https://doi.org/10.48550/arXiv.2402.05125 (2024)

    arXiv doi:10.48550/arxiv.2402.05125 2024

  3. [4]

    Blankemeier, L. et al. Efficient Diagnosis Assignment Using Unstructured Clinical Notes. in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (eds. Rogers, A., Boyd-Graber, J. & Okazaki, N.) 485–494 (Association for Computational Linguistics, Toronto, Canada, 2023). doi:10.18653/v1/2023.acl-short.42

    work page doi:10.18653/v1/2023.acl-short.42 2023

  4. [5]

    https://www.nature.com/articles/s41746-023-00957-x

    Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models | npj Digital Medicine. https://www.nature.com/articles/s41746-023-00957-x

    work page

  5. [6]

    Ma, S. P. et al. Electronic Phenotyping of Urinary Tract Infections as a Silver Standard Label for Machine Learning. AMIA Summits Transl. Sci. Proc. 2024 , 182–189 (2024)

    work page 2024

  6. [7]

    Lopez, I. et al. Predicting premature discontinuation of medication for opioid use disorder from electronic medical records. AMIA. Annu. Symp. Proc. 2023 , 1067–1076 (2024)

    work page 2024

  7. [8]

    Nateghi Haredasht, F. et al. Predictability of buprenorphine-naloxone treatment retention: A multi-site analysis combining electronic health records and machine learning. Addiction 119 , 1792–1802 (2024)

    work page 2024

  8. [9]

    Swaminathan, A. et al. Selective prediction for extracting unstructured clinical data. J. Am. Med. Inform. Assoc. 31 , 188–197 (2024)

    work page 2024

Show all 33 references
  1. [10]

    Swaminathan, A. et al. Extraction of Unstructured Electronic Health Records to Evaluate Glioblastoma Treatment Patterns. JCO Clin. Cancer Inform. e2300091 (2024) doi:10.1200/CCI.23.00091

    2024 doi

  2. [11]

    Ghanzouri, I. et al. Performance and usability testing of an automated tool for detection of peripheral artery disease using electronic health records. Sci. Rep. 12 , 13364 (2022)

    2022

  3. [12]

    Omiye, J. A. et al. Clinical use of polygenic risk scores for detection of peripheral artery disease and cardiovascular events. PLOS ONE 19 , e0303610 (2024)

    2024

  4. [13]

    & Cohen, K

    Zweigenbaum, P., Demner-Fushman, D., Yu, H. & Cohen, K. B. Frontiers of biomedical text mining: current progress. Brief. Bioinform. 8 , 358–375 (2007)

    2007

  5. [14]

    Wang, Y. et al. Clinical information extraction applications: A literature review. J. Biomed. Inform. 77 , 34–49 (2018)

    2018

  6. [15]

    Wang, S. et al. Annotation-efficient deep learning for automatic medical image segmentation. Nat. Commun. 12 , 5915 (2021)

    2021

  7. [16]

    & Mahmoud, Q

    Goyal, M. & Mahmoud, Q. H. A Systematic Review of Synthetic Data Generation Techniques Using Generative AI. Electronics 13 , 3509 (2024)

    2024

  8. [17]

    & Shung, D

    Giuffrè, M. & Shung, D. L. Harnessing the power of synthetic data in healthcare: innovation, application, and privacy. Npj Digit. Med. 6 , 186 (2023)

    2023

  9. [18]

    Choi, E. et al. Generating Multi-label Discrete Patient Records using Generative Adversarial Networks. Preprint at https://doi.org/10.48550/arXiv.1703.06490 (2018)

    2018 doi arXiv

  10. [19]

    & Veeramachaneni, K

    Xu, L., Skoularidou, M., Cuesta-Infante, A. & Veeramachaneni, K. Modeling Tabular data using Conditional GAN. in Advances in Neural Information Processing Systems vol. 32 (Curran Associates, Inc., 2019)

    2019

  11. [20]

    & Talukdar, W

    Biswas, A. & Talukdar, W. Enhancing Clinical Documentation with Synthetic Data: Leveraging Generative Models for Improved Accuracy. Int. J. Innov. Sci. Res. Technol. IJISRT 1553–1566 (2024) doi:10.38124/ijisrt/IJISRT24MAY2085

    2024 doi

  12. [21]

    Sushil, M. et al. CORAL: Expert-Curated Oncology Reports to Advance Language Model Inference. NEJM AI 1 , (2024)

    2024

  13. [22]

    Van Veen, D. et al. Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. 30 , 1134–1142 (2024)

    2024

  14. [23]

    Tu, T. et al. Towards Conversational Diagnostic AI. Preprint at https://doi.org/10.48550/arXiv.2401.05654 (2024)

    2024 doi arXiv

  15. [24]

    Peng, C. et al. A study of generative large language model for medical research and healthcare. Npj Digit. Med. 6 , 1–10 (2023)

    2023

  16. [25]

    Curth, A. et al. Transferring Clinical Prediction Models Across Hospitals and Electronic Health Record Systems. in Machine Learning and Knowledge Discovery in Databases (eds. Cellier, P. & Driessens, K.) 605–621 (Springer International Publishing, Cham, 2020). doi:10.1007/978-...

    2020 doi

  17. [26]

    Li, R., Wang, X. & Yu, H. Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data. Proc. Conf. Empir. Methods Nat. Lang. Process. Conf. Empir. Methods Nat. Lang. Process. 2023 , 7129–7143 (2023)

    2023

  18. [27]

    Ive, J. et al. Generation and evaluation of artificial mental health records for Natural Language Processing. Npj Digit. Med. 3 , 1–9 (2020)

    2020

  19. [28]

    Brown, T. B. et al. Language Models are Few-Shot Learners. Preprint at https://doi.org/10.48550/arXiv.2005.14165 (2020)

    2020 doi arXiv

  20. [29]

    & Melville, J

    McInnes, L., Healy, J. & Melville, J. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. Preprint at https://doi.org/10.48550/arXiv.1802.03426 (2020)

    2020 doi arXiv

  21. [30]

    Irvin, J. et al. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison. Proc. AAAI Conf. Artif. Intell. 33 , 590–597 (2019)

    2019

  22. [31]

    & Reimers, N

    Muennighoff, N., Tazi, N., Magne, L. & Reimers, N. MTEB: Massive Text Embedding Benchmark. Preprint at https://doi.org/10.48550/arXiv.2210.07316 (2023)

    2023 doi arXiv

  23. [32]

    Lopez, I. et al. Clinical entity augmented retrieval for clinical information extraction. Npj Digit. Med. 8 , 1–11 (2025)

    2025

  24. [33]

    Meta AI https://ai.meta.com/blog/meta-llama-3/

    Introducing Meta Llama 3: The most capable openly available LLM to date. Meta AI https://ai.meta.com/blog/meta-llama-3/

  25. [34]

    Alsentzer, E. et al. Publicly Available Clinical BERT Embeddings. Preprint at https://doi.org/10.48550/arXiv.1904.03323 (2019)

    2019 doi arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.