Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that when LLM-generated data stays close to the real distribution, moderate diversity improves fine-tuning in low-label settings while excessive diversity hurts, and that distribution shift, not just diversity, is what…

desk verdict A useful first attempt at controlling diversity in synthetic SFT data, but the headline claim is undercut by a scale confound and missing error bars. read the letter →

arxiv 2506.19262 v2 pith:QQ2EMJDU submitted 2025-06-24 cs.CL cs.LG

Yuchang Zhu , Huazhen Zhong , Qunshu Lin , Haotong Wei , Xiaolong Sun , Zixuan Yu , Minghao Liu , Zibin Zheng
show 1 more author
Liang Chen
This is my paper
classification cs.CLcs.LG
keywords LLM-generateddatadiversitymodelcollapsesupervisedfine-tuningsyntheticdistributionshiftaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the diversity of LLM-generated training data is itself a lever on downstream model quality, separate from fluency and distributional fidelity. Using two controllable data-generation pipelines, it creates seven diversity levels for question-answering and story-completion datasets and fine-tunes models of three sizes on them. The central finding is non-monotonic: when the generated data stays close to the real distribution, moderately diverse generated data improves fine-tuning in low-label settings, while highly diverse generated data hurts. In the story-completion task, where the generated data is far from the real distribution, higher diversity does not translate into better performance. The paper concludes that diversity matters, but only after distribution shift is controlled.

What carries the argument

The argument is carried by two diversity-controlling generation pipelines plus a diversity metric. Paraphrasing-based augmentation replaces a controlled number of generated QA pairs with paraphrased near-duplicates, so the fraction of similar samples sets the diversity level; topic-guided generation varies the number of seed topics (from 20 to 10,000) used to prompt story generation. Diversity is measured by Distinct-5, the proportion of unique 5-grams, normalized within each comparison group. Together these tools let the paper hold fluency and distribution approximately fixed while moving diversity, and they expose the confound: diversity gains are cancelled by distribution shift.

What would settle it

Compute a two-sample classifier (or MMD) on sentence embeddings between Gen-D0 through Gen-D6 and real QA data; if the classifier separates generated from real data with high AUC despite the near-zero answer-length KL, then the 'minimal distribution shift' premise fails and the diversity explanation for QA gains is unsupported. Conversely, a strong separation only in the story-completion task would support the paper's attribution.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that data diversity and distribution shift are separable factors in LLM-generated data, and that their effects on fine-tuning go in opposite directions. In the replace-style setting, increasing diversity lowers test loss on both tasks, yet models trained on generated data never beat models trained on real data; for story completion the generated data is more diverse than the real data yet performs worse, which the paper attributes to a large distribution shift measured by KL divergence. In the accumulate-style setting with fixed dataset size, every mixing ratio above zero raises test loss relative to real data alone, matching prior collapse results even at 10% synthetic. When the dataset size is allowed to grow and real labels are scarce, adding moderately diverse generated data (not the most diverse) improves test loss, while the most diverse expansions degrade it, especially for the 1B model. The upshot is that moderate diversity acts as a beneficial data-quality axis only under minimal distribution shift.

Load-bearing premise

The load-bearing premise is that the QA task's generated data is distributionally nearly identical to real data except for diversity; the paper relies on answer-length KL divergence and a t-SNE plot for this, so a subtler distributional mismatch that those two checks miss would undo the central claim.

Editorial extensions

If this is right

  • At a fixed training-set size, adding even 10% LLM-generated data to real data raises test loss, so data augmentation via generated samples is not free.
  • In label-scarce regimes, generation pipelines should be tuned for moderate rather than maximal diversity, since high-diversity expansions degrade performance.
  • Distribution matching should be a standard precondition before diversity is used to explain or improve synthetic-data training.
  • The non-monotonic diversity effect reproduces across model scales (124M, 1B, 8B), architectures, and dataset sizes, so it is not a quirk of one model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stronger distributional test than answer-length KL—for example, a trained classifier or MMD on embeddings—could reveal whether the QA task truly has minimal shift; if it does not, the diversity explanation may be an artifact of hidden shift.
  • Moderate diversity may act as an implicit regularizer, and the optimal level should scale with model capacity; the 124M-versus-8B divergence in mixing-ratio results hints at this.
  • The two pipelines tie diversity to different mechanisms (near-duplicate replacement vs. topic breadth), so a single underlying 'diversity' quantity may not transfer across tasks; comparing the two control mechanisms directly would test generality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates how the diversity of LLM-generated training data affects the fine-tuning performance of downstream models. The authors construct seven diversity levels (Gen-D0 to Gen-D6) using two controlled generation pipelines: paraphrasing-based augmentation for the NarrativeQA question-answering task and topic-guided generation for the ROCStories story-completion task. They fine-tune GPT-2-124M, Llama 3.2-1B, and Llama 3.1-8B on fully generated data, on fixed-size synthetic mixtures, and on variable-size synthetic mixtures with a small real-data seed. The paper's central interpretive claim is that, with minimal distribution shift, moderately diverse LLM-generated data can improve performance under label scarcity, whereas highly diverse generated data can hurt; the authors also argue that distribution shift can override the benefits of diversity.

Significance. If the central claim held, the paper would provide actionable guidance for LLM-based data augmentation under label scarcity and would sharpen the model-collapse literature by separating diversity from other data-quality dimensions. The paper has real strengths: the two generation protocols make diversity an explicit control parameter (mr and mt), the evaluation spans multiple model sizes and architectures, and the headline comparisons use held-out real test loss. The fixed-size mixing-ratio experiments in Figure 5 are a useful contribution. However, the positive low-label result rests on a missing iso-size control, the 'minimal distribution shift' premise is supported only by a narrow proxy, and the experiments are single-run. These issues are load-bearing rather than cosmetic.

major comments (4)
  1. [§4.3, Eq. (7), Figure 6] The claim that 'without a distribution shift, integrating moderately diverse data into fields with limited labeled data can boost model performance' is not established by the variable-size experiment because total training-set size grows monotonically with the mixing ratio. The QA curves start from 2,204 real samples and add generated samples up to an 80% mixing ratio, so the observed loss decrease from ratio 0 to 80% can be explained by a pure scale effect. The paper compares Gen-D1, Gen-D3, and Gen-D5 at each ratio, which holds size fixed across diversity levels, but it never trains on an equal number of additional real QA pairs or on an iso-size synthetic set that keeps diversity fixed. A real-data expansion curve and an iso-diversity or iso-size control are needed to separate the diversity contribution from the dataset-size contribution, and the positive claim should be scoped to the QA task, since the story-completion variable-size experiment in Figure 7 shows degradation.
  2. [Table 2, Figure 4, Section 4.2] The 'minimal distribution shift' premise for the QA task is not adequately supported. The KL divergence values in Table 2 appear to be computed on answer-length distributions (the analysis in Section 4.2 that precedes Table 2 is about answer lengths, Figure 3), and this is a narrow proxy for the full (context, question, answer) distribution. Figure 4(a) itself shows that even Gen-D6, the most diverse generated set, occupies only a small portion of the real-data embedding cluster, which is evidence of a nontrivial semantic distribution shift. The conclusion in Section 5 acknowledges that distribution consistency is crucial, but this premise is not established for the QA task. The authors should report a distributional distance on the full text (for example, embedding-based MMD or a fine-tuned classifier accuracy) and show the real/generated overlap at each diversity level before relying on this premise.
  3. [§4.2 vs. §4.4, Figures 2 and 8] The paper contains a direct inconsistency about model-size dependence. Section 4.2 states that increasing diversity improves performance for both tasks, but Section 4.4 reports that for the QA task the 124M model performs worse as diversity increases and attributes this to small models' difficulty with long-context understanding. This contradicts the contribution that scaling behaviors are consistent across model sizes and weakens the claimed generality of the diversity effect. The authors should either provide a single explanation that reconciles Figure 2(a) with Figure 8, or explicitly scope the diversity-benefit claim to models above a certain capacity and justify that scope with the data.
  4. [§4.1.2, Figures 2, 6, 8] No random seeds, repetitions, or error bars are reported, and several cross-condition differences are small relative to the vertical range of the plots (for example, the 124M and 1B QA curves in Figure 6). Without variance estimates or significance tests, the monotonic trends asserted across Gen-D levels could be run-to-run noise. The authors should report at least three training seeds per condition with means and ranges, or an equivalent statistical analysis, for the headline comparisons in Figures 2, 6, and 8.
minor comments (5)
  1. [Section 1, contributions bullet 3] The bullet lists 'model size, dataset size, model architecture, and model size'; the second 'model size' should be a distinct factor such as 'training budget' or should be removed.
  2. [Eq. (5) and §4.1.1] Equation (5) is written with Distinct-n, while the experiments use Distinct-5; the notation should be aligned to avoid confusion.
  3. [Appendix C] The setting 'mr is set as {30,25,20,15,10,5,0} for Gen-D0/1/2/3/4/5/6' with mp equal to 5 is not fully specified because the resulting replacement ratios r depend on the number of questions per context, which varies across experiments; please state the actual r values or define the replacement count more carefully.
  4. [Section 2, last paragraph] The sentence 'Our study uniquely focuses on the theoretical and empirical analysis of model collapse from a diversity perspective' overstates the contribution: the paper contains no theoretical analysis, so 'theoretical' should be removed.
  5. [Appendix A.1, Figure 10 prompt] The prompt says 'These twenty question-answer pairs' but later asks to 'Create Thirty question and answer pairs'; both the text and the JSON format in Figure 10 should be made consistent regarding the number of pairs.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the central claim is an empirical benchmark, not a derived prediction.

full rationale

The paper does not derive a prediction from a fitted parameter. Its contribution is an empirical comparison: seven diversity levels Gen-D0 through Gen-D6 are produced by two explicit mechanisms (paraphrase replacement ratios mr and seed-topic counts mt), and diversity is then independently quantified with Distinct-5. Model performance is evaluated as test loss on held-out real-world test sets. The central claim that moderately diverse generated data can help under label scarcity comes from the variable-size mixing experiment, where a fixed small real subset is augmented with generated data. This experimental design has a genuine confound: total training set size grows with the mixing ratio, and there is no iso-size real-data control, so the loss improvement may partly reflect scale rather than diversity. That is a validity concern, not circularity, because the conclusion is not used to define the diversity labels or the distribution-shift condition. The 'minimal distribution shift' premise is supported by answer-length KL divergence and t-SNE inspection, which are external measurements rather than restatements of the performance result; even if those measurements are thin, that is an empirical-evidence issue. The paper cites prior work on model collapse, but none of the load-bearing cited results is authored by the present authors, and no uniqueness theorem or ansatz is imported from a same-author citation to force the interpretation. The fixed-size mixing experiments (Figure 5) and the architecture/size scaling checks in the appendix provide independent empirical grounding. In short, no equation or constructed variable makes the reported finding true by definition, so the circularity burden is not met.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper relies on standard empirical assumptions rather than fitted parameters. The hand-chosen diversity control lists are experimental conditions, not fitted constants. The load-bearing assumptions are the diversity proxy, the distribution shift proxy, the performance metric, and the fixed training setup. These are domain assumptions common in the field, but they should be stated explicitly because the central claim is conditional on them.

free parameters (2)
  • mr, number of replaced questions per context in QA paraphrasing = {30,25,20,15,10,5,0}
    Hand-chosen to span seven diversity levels for the QA task. Not fitted to optimize performance; the paper measures the resulting diversity after the fact.
  • mt, number of seed topics for story completion = {20,100,300,1000,3000,6000,10000}
    Hand-chosen to span seven diversity levels for story completion. Not fitted to optimize performance; serves as an experimental control knob.
assumptions (4)
  • domain assumption Distinct-n n-gram overlap is a valid proxy for the diversity of generated text.
    Section 3.2.2 uses Distinct-5 as the diversity metric. If this surface-level metric does not capture semantic diversity, the ordering of Gen-D0 to Gen-D6 may not reflect meaningful diversity in the training signal.
  • domain assumption KL divergence computed on answer lengths and t-SNE overlap adequately capture distribution shift between real and generated data.
    Section 4.2 and Table 2 use these to conclude minimal shift for QA and large shift for story completion. The exact feature space for the KL computation is not specified, so the conclusion is only as strong as this measurement.
  • domain assumption Test loss on the real-world test set is a faithful measure of downstream model performance.
    Section 4.1.1 adopts test loss following prior work. Other metrics such as accuracy, answer correctness, or human evaluation are not reported, so the performance ranking may differ under alternative metrics.
  • domain assumption LoRA fine-tuning with fixed hyperparameters is an adequate and fair training setup for all data conditions.
    Section 4.1.2 fixes rank 8, alpha 16, learning rate 1e-4, and 3 epochs. Results may change with full fine-tuning, different ranks, or more epochs, which limits the generality of the conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning." pith.science (2026). https://pith.science/paper/QQ2EMJDU

@misc{pith2026250619262,
  author       = {Pith},
  title        = {Pith review of: What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQ2EMJDU}},
  note         = {Machine review of arXiv:2506.19262}
}
read the original abstract

With the remarkable generative capabilities of large language models (LLMs), using LLM-generated data to train downstream models has emerged as a promising approach to mitigate data scarcity in specific domains and reduce time-consuming annotations. However, recent studies have highlighted a critical issue: iterative training on self-generated data results in model collapse, where model performance degrades over time. Despite extensive research on the implications of LLM-generated data, these works often neglect the importance of data diversity, a key factor in data quality. In this work, we aim to understand the implications of the diversity of LLM-generated data on downstream model performance. Specifically, we explore how varying levels of diversity in LLM-generated data affect downstream model performance. Additionally, we investigate the performance of models trained on data that mixes different proportions of LLM-generated data, which we refer to as synthetic data. Our experimental results show that, with minimal distribution shift, moderately diverse LLM-generated data can enhance model performance in scenarios with insufficient labeled data, whereas highly diverse generated data has a negative impact. We hope our empirical findings will offer valuable guidance for future studies on LLMs as data generators.

Figures

Figures reproduced from arXiv: 2506.19262 by the authors.

Figure 1
Figure 1. Diversity-controlling data generation pipelines in this study. (a) Paraphrasing-based [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Correlation between model performance and the diversity of LLM-generated data. (a), (b) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparison of answer length distribution [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: Model performance and dataset diversity on synthetic datasets with fixing size. (a), (b) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Model performance and dataset diversity on variable-size synthetic datasets for the QA [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Model performance and dataset diversity on variable-size synthetic datasets for the story [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Model performance on QA datasets (real-world or LLM-generated datasets) of varying [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Model performance on story completion datasets (real-world or LLM-generated datasets) [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Prompt for new QA pairs generation. Given the following question and answer pair, generate a similar question that mean the same thing but are worded differently. The answer should remain the same. # Example: Quesion: Why did James Norrington resign from the Navy? Ans…
Figure 11
Figure 11. Figure 11: Prompt for paraphrased question generation [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: The diversity of LLM-generated data and model performance under various model [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Model performance and dataset diversity under different mixing ratios across various [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Loss curves corresponding to Gen-D5-sourced models in Figure 6. [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Coverage of sparse-autoencoder-identified task features predicts post-training performance and can guide synthesis of small, high-impact datasets (2,000 vs. 300,000 samples).

  2. One Joke to Rule them All? On the (Im)possibility of Generalizing Humor

    cs.CL 2025-08 conditional novelty 6.0 of 10

    LLMs fine-tuned on one to three humor datasets transfer partially to unseen humor types (up to 75% accuracy); diverse training helps modestly, and dad jokes enable transfer best but resist it as a target.

  3. Epistemic diversity across language models mitigates knowledge collapse

    cs.LG 2025-12 reject novelty 5.0 of 10

    In repeated self-training loops on Wikitext2, ecosystems of four small language models show lower average perplexity than one, two, or sixteen models, but the paper's broader claims about monotonic optima, robustness,...

Reference graph

Works this paper leans on

41 extracted references · 9 canonical work pages · cited by 3 Pith papers

  1. [1]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling.arXiv preprint arXiv:2408.16737, 2024

    Hritik Bansal, Arian Hosseini, Rishabh Agarwal, Vinh Q Tran, and Mehran Kazemi. Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling.arXiv preprint arXiv:2408.16737, 2024

    arXiv 2024

  4. [4]

    Large language models suffer from their own output: An analysis of the self-consuming training loop.arXiv preprint arXiv:2311.16822, 2023

    Martin Briesch, Dominik Sobania, and Franz Rothlauf. Large language models suffer from their own output: An analysis of the self-consuming training loop.arXiv preprint arXiv:2311.16822, 2023

    arXiv 2023

  5. [5]

    Language gans falling short.arXiv preprint arXiv:1811.02549, 2018

    Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Charlin. Language gans falling short.arXiv preprint arXiv:1811.02549, 2018

    arXiv 2018

  6. [6]

    On the diversity of synthetic data and its impact on training large language models

    Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I Abdin. On the diversity of synthetic data and its impact on training large language models. arXiv preprint arXiv:2410.15226, 2024

    arXiv 2024

  7. [7]

    Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models.arXiv preprint arXiv:2406.12397, 2024

    Jie Chen, Yupeng Zhang, Bingning Wang, Wayne Xin Zhao, Ji-Rong Wen, and Weipeng Chen. Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models.arXiv preprint arXiv:2406.12397, 2024

    arXiv 2024

  8. [8]

    Auggpt: Leveraging chatgpt for text data augmentation.arXiv preprint arXiv:2302.13007, 2023

    Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, et al. Auggpt: Leveraging chatgpt for text data augmentation.arXiv preprint arXiv:2302.13007, 2023

    arXiv 2023

Show all 41 references
  1. [9]

    Universality of the π2/6 pathway in avoiding model collapse, 2024

    Apratim Dey and David Donoho. Universality of the π2/6 pathway in avoiding model collapse, 2024. 10

    2024

  2. [10]

    Is gpt-3 a good data annotator?arXiv preprint arXiv:2212.10450, 2022

    Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Shafiq Joty, Boyang Li, and Lidong Bing. Is gpt-3 a good data annotator?arXiv preprint arXiv:2212.10450, 2022

    2022 arXiv

  3. [11]

    Data augmentation using llms: Data perspectives, learning paradigms and challenges

    Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Luu Anh Tuan, and Shafiq Joty. Data augmentation using llms: Data perspectives, learning paradigms and challenges. InFindings of the Association for Computational Linguistics A...

    2024

  4. [12]

    Model collapse demystified: The case of regression.arXiv preprint arXiv:2402.07712, 2024

    Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression.arXiv preprint arXiv:2402.07712, 2024

    2024 arXiv

  5. [13]

    Strong model collapse

    Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. Strong model collapse. arXiv preprint arXiv:2410.04840, 2024

    2024 arXiv

  6. [14]

    A tale of tails: Model collapse as a change of scaling laws.arXiv preprint arXiv:2402.07043, 2024

    Elvis Dohmatob, Yunzhen Feng, Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse as a change of scaling laws.arXiv preprint arXiv:2402.07043, 2024

    2024 arXiv

  7. [15]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  8. [16]

    Beyond model collapse: Scaling up with syn-thesized data requires verification.arXiv preprint arXiv:2406.07515, 2024

    Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, Julia Kempe, and FAIR Meta. Beyond model collapse: Scaling up with syn-thesized data requires verification.arXiv preprint arXiv:2406.07515, 2024

    2024 arXiv

  9. [17]

    Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. arXiv pre...

    2024 arXiv

  10. [18]

    Chatgpt outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences, 120(30):e2305016120, 2023

    Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks.Proceedings of the National Academy of Sciences, 120(30):e2305016120, 2023

    2023

  11. [19]

    The curious decline of lin- guistic diversity: Training language models on synthetic text.arXiv preprint arXiv:2311.09807, 2023

    Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chloé Clavel. The curious decline of lin- guistic diversity: Training language models on synthetic text.arXiv preprint arXiv:2311.09807, 2023

    2023 arXiv

  12. [20]

    Targen: Targeted data genera- tion with large language models.arXiv preprint arXiv:2310.17876, 2023

    Himanshu Gupta, Kevin Scaria, Ujjwala Anantheswaran, Shreyas Verma, Mihir Parmar, Saurabh Arjun Sawant, Chitta Baral, and Swaroop Mishra. Targen: Targeted data genera- tion with large language models.arXiv preprint arXiv:2310.17876, 2023

    arXiv

  13. [21]

    Collapse or thrive? perils and promises of synthetic data in a self-generating world.arXiv preprint arXiv:2410.16713, 2024

    Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L Donoho, and Sanmi Koyejo. Collapse or thrive? perils and promises of synthetic data in a self-generating world.arXiv preprint arXiv:2410.16713, 2024

    arXiv

  14. [22]

    The narrativeqa reading comprehension challenge, 2017

    Tomáš Koˇciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017

    2017

  15. [23]

    Not all llm-generated data are equal: Rethinking data weighting in text classification.arXiv preprint arXiv:2410.21526, 2024

    Hsun-Yu Kuo, Yin-Hsiang Liao, Yu-Chieh Chao, Wei-Yun Ma, and Pu-Jen Cheng. Not all llm-generated data are equal: Rethinking data weighting in text classification.arXiv preprint arXiv:2410.21526, 2024

    2024 arXiv

  16. [24]

    A diversity- promoting objective function for neural conversation models

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. A diversity- promoting objective function for neural conversation models. InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang...

    2016

  17. [25]

    Synthetic data generation with large lan- guage models for text classification: Potential and limitations.arXiv preprint arXiv:2310.07849, 2023

    Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic data generation with large lan- guage models for text classification: Potential and limitations.arXiv preprint arXiv:2310.07849, 2023

    2023 arXiv

  18. [26]

    Api-guided dataset synthesis to finetune large code models.arXiv preprint arXiv:2408.08343, 2024

    Zongjie Li, Daoyuan Wu, Shuai Wang, and Zhendong Su. Api-guided dataset synthesis to finetune large code models.arXiv preprint arXiv:2408.08343, 2024. 11

    2024 arXiv

  19. [27]

    Generating training data with language models: Towards zero-shot language understanding.Advances in Neural Information Processing Systems, 35:462–477, 2022

    Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding.Advances in Neural Information Processing Systems, 35:462–477, 2022

    2022

  20. [28]

    A corpus and cloze evaluation for deeper understanding of commonsense stories

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. InProceedings of the 2016 Conference of the North American Chapter ...

    2016

  21. [29]

    I learn better if you speak my language: Understanding the superior performance of fine-tuning large language models with llm-generated responses

    Xuan Ren, Biao Wu, and Lingqiao Liu. I learn better if you speak my language: Understanding the superior performance of fine-tuning large language models with llm-generated responses. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pa...

    2024

  22. [30]

    How bad is training on synthetic data? a statistical analysis of language model collapse

    Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How bad is training on synthetic data? a statistical analysis of language model collapse. arXiv preprint arXiv:2404.05090, 2024

    2024 arXiv

  23. [31]

    The curse of recursion: Training on generated data makes models forget.arXiv preprint arXiv:2305.17493, 2023

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget.arXiv preprint arXiv:2305.17493, 2023

    2023 arXiv

  24. [32]

    Ai models collapse when trained on recursively generated data.Nature, 631(8022):755– 759, 2024

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained on recursively generated data.Nature, 631(8022):755– 759, 2024

    2024

  25. [33]

    Large language models for data annotation: A survey.arXiv preprint arXiv:2402.13446, 2024

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey.arXiv preprint arXiv:2402.13446, 2024

    2024 arXiv

  26. [34]

    Evaluating the evaluation of diversity in natural language generation.arXiv preprint arXiv:2004.02990, 2020

    Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation.arXiv preprint arXiv:2004.02990, 2020

    2004 arXiv

  27. [35]

    Self-instruct: Aligning language models with self-generated instruc- tions.arXiv preprint arXiv:2212.10560, 2022

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions.arXiv preprint arXiv:2212.10560, 2022

    2022 arXiv

  28. [36]

    Codeclm: Aligning language models with tailored synthetic data.arXiv preprint arXiv:2404.05875, 2024

    Zifeng Wang, Chun-Liang Li, Vincent Perot, Long T Le, Jin Miao, Zizhao Zhang, Chen-Yu Lee, and Tomas Pfister. Codeclm: Aligning language models with tailored synthetic data.arXiv preprint arXiv:2404.05875, 2024

    2024 arXiv

  29. [37]

    Progen: Pro- gressive zero-shot dataset generation via in-context feedback.arXiv preprint arXiv:2210.12329, 2022

    Jiacheng Ye, Jiahui Gao, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Progen: Pro- gressive zero-shot dataset generation via in-context feedback.arXiv preprint arXiv:2210.12329, 2022

    2022 arXiv

  30. [38]

    Zerogen: Efficient zero-shot learning via dataset generation.arXiv preprint arXiv:2202.07922, 2022

    Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. Zerogen: Efficient zero-shot learning via dataset generation.arXiv preprint arXiv:2202.07922, 2022

    2022 arXiv

  31. [39]

    Gpt3mix: Leveraging large-scale language models for text augmentation.arXiv preprint arXiv:2104.08826, 2021

    Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. Gpt3mix: Leveraging large-scale language models for text augmentation.arXiv preprint arXiv:2104.08826, 2021

    2021 arXiv

  32. [40]

    Large language model as attributed training data generator: A tale of diversity and bias.Advances in Neural Information Processing Systems, 36, 2024

    Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias.Advances in Neural Information Processing Systems, 36, 2024

    2024

  33. [41]

    question

    Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, Ermo Hua, Xingtai Lv, Ning Ding, Zhouhan Lin, Zilong Zheng, and Bowen Zhou. How to synthesize text data without model collapse?arXiv preprint arXiv:2412.14689, 2024. 12 A Details of LLM-generated Data A.1 Prompt for Paraphras...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.