Pith. sign in

REVIEW 3 major objections 6 minor 30 references

Does Training on Synthetic Data Make Models Less Robust?

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning on synthetic data does not amplify the lexical-overlap heuristic in NLI models.

desk verdict A useful but statistically thin null-result case study: the biased-synthetic control is the best part, and the abstract oversells the claim. read the letter →

arxiv 2502.07164 v2 pith:2JFPMTRP submitted 2025-02-11 cs.CL cs.AIcs.LG

Lingze Zhang , Ellie Pavlick This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords syntheticdatanaturallanguageinferenceHANSlexicaloverlapheuristicmodelrobustnessfine-tuningLlama-2blindspots
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many teams train language models on data generated by other language models, raising the worry that synthetic data reinforces the model's existing shortcuts rather than teaching it to generalize. This paper tests that worry on natural language inference, where the known shortcut is the lexical-overlap heuristic: relying on word overlap between premise and hypothesis instead of genuine reasoning. The authors fine-tune Llama-2-7B models on MultiNLI, then further fine-tune them on synthetic MultiNLI-like data produced by a Llama-2 generator, and probe the 'blindspot' with the HANS dataset. The central finding is that unfiltered synthetic data does not make the heuristic worse, though it also does not reliably reduce it; only a deliberately biased synthetic dataset drives the blindspot accuracy down sharply. A sympathetic reader would take the paper as evidence that the feared amplification effect is not automatic and depends on how synthetic data is sampled.

What carries the argument

The paper's central machinery is a controlled two-model simulation: a task model $T$ and a generator model $G$, both Llama-2-7B-hf fine-tuned on MultiNLI, with $T$ further fine-tuned on synthetic dataset $D_G$ produced by $G$. The probe for the blindspot is HANS, an adversarial NLI set built around three syntactic heuristics (lexical overlap, subsequence, constituent); the key comparison is accuracy on the MultiNLI matched test set versus accuracy on HANS and on the lexical-overlap/non-entailment subset. The design varies the starting checkpoint of $T$ (0 to 391,722 MultiNLI examples) and the training set (original, uniformly sampled synthetic, or biased synthetic), which lets the authors separate the effect of synthetic data from the effect of more fine-tuning.

What would settle it

Run the same fine-tuning recipe with the 20K and 392K starting checkpoints over at least five seeds each, using both original and synthetic data, and compute confidence intervals for HANS accuracy; if the synthetic-fine-tuned models consistently score lower on HANS than the original-fine-tuned models across seeds, the paper's conclusion that synthetic data does not make the heuristic worse would be overturned.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a negative result: fine-tuning an NLI model on unfiltered synthetic data generated by a similar Llama-2 model improves MultiNLI accuracy about as much as fine-tuning on original data, yet does not consistently lower accuracy on HANS, the targeted test of syntactic heuristics. The hypothesized trade-off—general accuracy rising while blindspot accuracy falls—appears only for the 20K starting checkpoint and partially for the 392K checkpoint, and the authors judge the overall pattern inconclusive. A deliberately biased synthetic set, in which every entailment example satisfies lexical overlap and no non-entailment example does, does drive HANS lexical-overlap accuracy toward zero, confirming that the probe works. The authors therefore conclude that standard unfiltered synthetic data does not overrepresent the heuristic strongly enough to exacerbate the blindspot.

Load-bearing premise

The central negative claim rests on treating single-run accuracy differences as meaningful; with no multiple seeds or error bars, the absence of a consistent HANS drop could be statistical noise rather than a true null effect.

Editorial extensions

If this is right

  • If the result holds, fine-tuning on unfiltered synthetic data from a similar model is not inherently more dangerous than fine-tuning on original data for this kind of heuristic.
  • The general-task benefit of synthetic data is roughly on par with original data when the synthetic set is large enough.
  • The feared trade-off between general accuracy and blindspot accuracy is not a consistent pattern; it depends on the starting checkpoint and dataset size.
  • Deliberately biased synthetic data can severely degrade blindspot performance, which shows the mechanism is real when the heuristic is actually overrepresented.
  • A practical corollary is that curation or filtering of synthetic data matters more than the fact that the data is synthetic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's null result is about one heuristic (lexical overlap) in one task with one model family; a reasonable extension would test subsequence and constituent heuristics and stronger or more diverse generators, where shortcut overrepresentation may differ.
  • The single-run, no-seed design means the negative claim should be treated as provisional; multi-seed runs with confidence intervals would tell whether the absence of a consistent HANS drop is real.
  • The finding implicitly shifts attention from 'is synthetic data harmful' to 'which biases does the generator overrepresent,' since the biased-synthetic control shows the probe can detect amplification when it exists.
  • If the result generalizes, the practical use of unfiltered synthetic data for fine-tuning is safer than model-collapse narratives suggest, but only for tasks where the generator's heuristic distribution is close to the original data's.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper asks whether fine-tuning an NLI model on synthetic data produced by a similar LLM (Llama-2-7B-hf) amplifies known heuristics, specifically the lexical-overlap heuristic measured by HANS. The authors fine-tune six starting models on varying amounts of MultiNLI, generate a large synthetic MultiNLI-like dataset with a fine-tuned Llama-2 generator, and further fine-tune the starting models on original, randomly sampled synthetic, and deliberately biased synthetic data at three sizes. They report MultiNLI Matched and HANS accuracy for each configuration. The main finding, as stated in the abstract, is that synthetic fine-tuning does not make HANS performance worse relative to the starting model, contrary to the authors' initial hypothesis, although a small and non-monotonic gap favoring original data exists under most settings. A control condition with a strongly biased synthetic dataset reproduces the expected blindspot deterioration, serving as a sanity check that the HANS lexical-overlap subset can detect heuristic amplification.

Significance. If the central claim is correct, the paper provides a useful empirical counterpoint to the intuition that unfiltered synthetic data from a similar model necessarily reinforces already-encoded heuristics. The experimental design has real strengths: the evaluation uses held-out benchmarks (MultiNLI Matched, HANS) not used in training; the synthetic and original datasets are matched in size and label balance; the biased-synthetic control (§3) demonstrates that the HANS instrument is capable of detecting an extreme heuristic; and the authors release their code. The paper is also honest in its Limitations section about the case-study nature of the result and the arbitrariness of several design choices. However, the central negative claim currently rests on single-run point estimates with no error bars, confidence intervals, or multiple seeds, which is a serious load-bearing weakness for a null result. The paper's significance is therefore conditional on adding uncertainty quantification or otherwise demonstrating that the observed flat HANS curves are not noise.

major comments (3)
  1. [§3, Figure 1] The central conclusion that synthetic fine-tuning 'does not make it worse' on HANS is a null claim, but all results in Figure 1 appear to be point estimates from single fine-tuning runs, with no error bars, confidence intervals, or multiple seeds reported anywhere in the paper. HANS accuracies are small differences on a possibly noisy evaluation surface, and the curves are explicitly described as non-monotonic; without variance estimates, the absence of a consistent drop cannot be distinguished from low statistical power. I request seed variance or bootstrapped confidence intervals across at least the key configurations (Synthetic vs. Original at 18K/36K/73K for the 20K and 392K starting models), and a statement of how many runs each reported point represents.
  2. [Abstract and §3] There is a tension between the abstract's claim that synthetic data 'does not make it worse' and the paper's own observation in §3 that 'under most settings, further fine-tuning with original MultiNLI data would always benefit more or harm less on HANS performance than synthetic data.' The latter sentence describes a systematic relative disadvantage of synthetic data on HANS; the two statements can be reconciled only if 'worse' means relative to the starting model rather than relative to original-data fine-tuning. The paper should make this distinction explicit in the abstract and results, and should quantify the size of the synthetic-vs-original HANS gap; if the gap is real and consistent, the appropriate conclusion may be 'synthetic data helps less and can harm more than original data, but not catastrophically,' which is materially different from the current abstract wording.
  3. [§3, Table 1] The biased-synthetic control shows that an extreme heuristic distribution can drive HANS lexical-overlap accuracy toward zero, so the null result for the random synthetic condition is not obviously a dead instrument. However, this does not by itself establish that the random synthetic condition has enough statistical power to detect a meaningful effect. Table 1 shows that the random synthetic data increases lexical-overlap prevalence and its correlation with the entailment label only modestly relative to original MultiNLI, so the expected effect size in the random condition is small. The paper should report the HANS accuracy difference between Original and Synthetic conditions with a confidence interval, and ideally a power analysis or an equivalence test, to support the negative claim.
minor comments (6)
  1. [§1] Typo: 'It is well know that LLMs' should read 'It is well known that LLMs.'
  2. [§2.2.3] Typo: 'no example with neural or contradiction label' should read 'neutral or contradiction label.'
  3. [§3] Typo: 'only happens in for the 20K starting point' should read 'only happens for the 20K starting point.'
  4. [References] The two Kotek et al. (2023a, 2023b) entries appear to cite the same paper; if so, they should be merged, and the in-text citation in §1 should point to a single reference.
  5. [Figure 1] The subplot figure is difficult to read as referenced in the text; consider enlarging individual panels, adding axis labels with accuracy values, and marking the starting-model performance explicitly on each panel so the 'benefit more or harm less' comparison is visually direct.
  6. [§2.3.3] The notation 'Synthetic' for the randomly sampled dataset and 'Biased Synthetic' for the biased dataset is clear enough, but the paper should state explicitly whether the three dataset sizes are fixed across all starting models and whether the same generated synthetic pool is reused for every starting model; this affects independence assumptions if any inferential statistics are added.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical evaluation with held-out benchmarks and no fitted-parameter derivation chain.

full rationale

This paper does not contain a derivation chain in which outputs are equivalent to inputs by construction. The experimental protocol is: fine-tune a task model and a generator on MultiNLI, generate synthetic data from the generator, further fine-tune the task model on synthetic or original data, and evaluate on held-out test sets (MultiNLI Matched and HANS). No parameter is fitted to the evaluation data, and no 'prediction' is derived from a fitted quantity: the reported accuracies are direct measurements on benchmarks that are not used in training. HANS is cited from McCoy, Pavlick, and Linzen (2019), which includes a co-author of this paper, but HANS is an external, widely used diagnostic benchmark and is used only as a held-out evaluation instrument; the paper does not rely on an unverified self-citation to justify its central claim. The authors' own limitations discussion—acknowledging arbitrary design choices and the need for further validation—concerns generalizability and statistical uncertainty, not circularity. The reader's concern about single-run point estimates and the strength of the negative claim is a statistical-power issue, not a circularity issue. Therefore, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. Its empirical claims rest on standard domain assumptions about the validity of its evaluation metrics and the representativeness of its pipeline. The most consequential unstated premise is that point estimates from single fine-tuning runs are stable enough to support a null conclusion.

assumptions (4)
  • domain assumption HANS accuracy is a valid probe of heuristic use in NLI.
    The paper equates lower HANS accuracy with greater reliance on the lexical overlap, subsequence, and constituent heuristics (Section 2.2.3, Section 3).
  • domain assumption MultiNLI Matched test set represents general NLI capability.
    Used as the general-task metric (Section 2.2.3); assumes no relevant distribution shift between development and matched test.
  • domain assumption The Llama-2-7B-hf model and the two-step fine-tuning pipeline are representative of synthetic-data training.
    The authors state this is a case study and that results may not generalize (Limitations).
  • domain assumption Point estimates from single training runs are stable enough for the null result.
    No multiple seeds, error bars, or significance tests are reported (Section 3, Figure 1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Does Training on Synthetic Data Make Models Less Robust?." pith.science (2026). https://pith.science/paper/2JFPMTRP

@misc{pith2026250207164,
  author       = {Pith},
  title        = {Pith review of: Does Training on Synthetic Data Make Models Less Robust?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2JFPMTRP}},
  note         = {Machine review of arXiv:2502.07164}
}
read the original abstract

An increasingly common practice is to train large language models (LLMs) using synthetic data. Often this synthetic data is produced by the same or similar LLMs as those it is being used to train. This raises the question of whether the synthetic data might in fact exacerbate certain "blindspots" by reinforcing heuristics that the LLM already encodes. In this paper, we conduct simulated experiments on the natural language inference (NLI) task with Llama-2-7B-hf models. We use MultiNLI as the general task and HANS, a targeted evaluation set designed to measure the presence of specific heuristic strategies for NLI, as our "blindspot" task. Our goal is to determine whether performance disparities between the general and blind spot tasks emerge. Our results indicate that synthetic data does not reinforce blindspots in the way we expected. Specifically, we see that, while fine-tuning with synthetic data doesn't necessarily reduce the use of the heuristic, it also does not make it worse as we hypothesized.

Figures

Figures reproduced from arXiv: 2502.07164 by the authors.

Figure 1
Figure 1. Augmented model performance under different settings. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 14 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard Baraniuk. 2024. https://openreview.net/forum?id=ShjMHfmPs0 Self-consuming generative models go MAD . In The Twelfth International Conference on Learning Representations

    work page 2024

  4. [4]

    Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.696 Improving question answering model robustness with synthetic adversarial data generation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8830--8848, Online and...

    work page doi:10.18653/v1/2021.emnlp-main.696 2021

  5. [5]

    Victor Besnier, Himalaya Jain, Andrei Bursuc, Matthieu Cord, and Patrick Pérez. 2020. https://doi.org/10.1109/ICASSP40776.2020.9053146 This dataset does not exist: Training models from generated images . In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5

    arXiv 2020

  6. [6]

    James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. 2024. Synthetic replacements for human survey data? the perils of large language models. Political Analysis, pages 401--416

    work page 2024

  7. [7]

    Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gul Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, et al. 2023. Going beyond nouns with vision & language models using synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20155--20165

    2023

  8. [8]

    Jie Chen, Yupeng Zhang, Bingning Wang, Xin Zhao, Ji-Rong Wen, and Weipeng Chen. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.873 Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 14855--14865, Miami, F...

    work page doi:10.18653/v1/2024.findings-emnlp.873 2024

Show all 30 references
  1. [9]

    Virginia Fernandez, Walter Hugo Lopez Pinaya, Pedro Borges, Petru-Daniel Tudosiu, Mark S Graham, Tom Vercauteren, and M Jorge Cardoso. 2022. Can segmentation models be trained with fully synthetically generated data? In International Workshop on Simulation and Synthesis in Med...

    2022

  2. [10]

    Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. 2021. Improving robustness using generated data. Advances in Neural Information Processing Systems, 34:4218--4233

    2021

  3. [11]

    Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. Advances in neural ...

    2021

  4. [12]

    Hadas Kotek, Rikker Dockum, and David Sun. 2023 a . Gender bias and stereotypes in large language models. In Proceedings of the ACM collective intelligence conference, pages 12--24

    2023

  5. [13]

    Hadas Kotek, Rikker Dockum, and David Sun. 2023 b . https://doi.org/10.1145/3582269.3615599 Gender bias and stereotypes in large language models . In Proceedings of The ACM Collective Intelligence Conference, CI '23, page 12–24, New York, NY, USA. Association for Computing Machinery

    2023

  6. [14]

    Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024 a . https://arxiv.org/abs/2402.10946 Culturellm: Incorporating cultural differences into large language models . Preprint, arXiv:2402.10946

    2024 arXiv

  7. [15]

    Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024 b . https://arxiv.org/abs/2405.15145 Culturepark: Boosting cross-cultural understanding in large language models . Preprint, arXiv:2405.15145

    2024 arXiv

  8. [16]

    Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.647 Synthetic data generation with large language models for text classification: Potential and limitations . In Proceedings of the 2023 Conference on Empirical Methods in Nat...

    2023 doi

  9. [17]

    Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. https://doi.org/10.18653/v1/P19-1334 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3...

    2019 doi

  10. [18]

    Bhargavi Paranjape, Matthew Lamm, and Ian Tenney. 2022. https://doi.org/10.18653/v1/2022.acl-long.117 Retrieval-guided counterfactual generation for QA . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1...

    2022 doi

  11. [19]

    Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan

    Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, D...

    2023

  12. [20]

    Sara Rajaee, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.725 Looking at the overlooked: An analysis on the word-overlap bias in natural language inference . In Proceedings of the 2022 Conference on Empirical Methods in ...

    2022 doi

  13. [21]

    Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Abdelkader DEBBAH. 2024. https://openreview.net/forum?id=t3z6UlV09o How bad is training on synthetic data? a statistical analysis of language model collapse . In First Conference on Language Modeling

    2024

  14. [22]

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. Ai models collapse when trained on recursively generated data. Nature, 631(8022):755--759

    2024

  15. [23]

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2024. https://openreview.net/forum?id=G8LaO1P0xv A long way to go: Investigating length correlations in RLHF . In First Conference on Language Modeling

    2024

  16. [24]

    Daniel Susser and Jeremy Seeman. 2024. Critical provocations for synthetic data. Surveillance and Society, 22(4):453--459

    2024

  17. [25]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  18. [26]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual Mee...

    2023 doi

  19. [27]

    Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2024. https://arxiv.org/abs/2308.03958 Simple synthetic data reduces sycophancy in large language models . Preprint, arXiv:2308.03958

    2024 arXiv

  20. [28]

    Cedric Deslandes Whitney and Justin Norman. 2024. Real risks of fake data: Synthetic data, diversity-washing and consent circumvention. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1733--1744

    2024

  21. [29]

    Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/N18-1101 A broad-coverage challenge corpus for sentence understanding through inference . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computation...

    2018 doi

  22. [30]

    Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.90 Generative data augmentation for commonsense reasoning . In Findings of ...

    2020 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.