Pith. sign in

REVIEW 3 major objections 6 minor 4 cited by

BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Base models beat instruct models at few-shot data generation

desk verdict A genuinely new few-shot synthetic data recipe with the right control experiment, but single-run numbers and one internal inconsistency make the headline gains look shakier than the prose suggests. read the letter →

arxiv 2502.01697 v3 pith:QE3DRJJR submitted 2025-02-03 cs.CL cs.AIcs.LG

Alan Zhu , Parth Asawa , Jared Quincy Davis , Lingjiao Chen , Boris Hanin , Ion Stoica , Joseph E. Gonzalez , Matei Zaharia This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords syntheticdatagenerationfew-shotlearningbaselanguagemodelsinstruction-tuneddiversityqualityBase-Refinefine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the standard practice of generating synthetic training data with instruction-tuned models fails in the few-shot regime because these models produce too little diversity, and that untuned base models, which have been largely ignored for this purpose, are a better source of variety. On this basis it introduces Base-Refine (BARE), a two-stage method that first samples a large, diverse set of candidate examples from a base model using just three seed examples, then has an instruction-tuned model refine each candidate for quality. The paper reports that fine-tuning a Llama 3.1 8B model on 1,000 BARE-generated coding examples reaches accuracy comparable to state-of-the-art similarly sized models on LiveCodeBench, and that BARE-generated math data improves a fine-tuned Llama 3.2 1B on GSM8K by 101% over instruction-model-only data. A sympathetic reader would care because this points to a data-efficient way to create training sets in domains where curated examples are scarce or expensive.

What carries the argument

The load-bearing object is the two-stage Base-Refine (BARE) pipeline: a base language model, such as Llama-3.1-70B-Base, independently samples candidate examples conditioned on three seed examples at temperature 0.7, and then an instruction-tuned model, such as Llama-3.1-70B-Instruct or GPT-4o, edits each candidate to improve realism and correctness while keeping its theme. The mechanism is the division of labour: diversity is inherited from the base model's untuned output distribution, and quality is imposed per entry by the refiner, so the final dataset keeps the base model's spread while gaining instruction-tuned fidelity. The paper measures spread with average pairwise embedding cosine similarity and quality with indistinguishability rate (IR), the rate at which a strong LLM discriminator fails to tell a synthetic entry apart from real examples.

What would settle it

Run the GSM8K comparison, BARE with Llama-3.1-70B-Base plus GPT-4o refinement versus instruct-only generation, for ten independent fine-tuning seeds and check whether the 35.8% versus 17.8% gap persists with non-overlapping confidence intervals; if the gap collapses, the central claim that BARE improves downstream performance is not established.

Watch

Extended reading notes

Core claim

The central discovery is a diversity-quality trade-off between model checkpoints: base models, which have not undergone post-training, generate far more diverse outputs from a few seed examples, while instruction-tuned models generate more realistic, in-distribution examples but collapse toward a narrow mode. BARE exploits this trade-off by separating generation from refinement: the base model supplies spread, and the instruction-tuned model repairs correctness, style, and ambiguity without removing the original diversity. The paper demonstrates that this separation is what matters by ablating the first stage: swapping the base generator for an instruction-tuned generator while keeping the same refine step lowers GSM8K accuracy from 29.8% to 25.4% with a Llama refiner and from 35.8% to 30.8% with GPT-4o, while dataset diversity stays similar. The paper concludes that base-model priors are an effective and largely untapped source of seed-efficient diversity for synthetic data.

Load-bearing premise

The central claim rests on the assumption that the reported downstream accuracy differences are real and not run-to-run noise: every fine-tuning result is a single run with no error bars, and HotpotQA and PubMedQA are evaluated on static test sets of only 100 items.

Editorial extensions

If this is right

  • With only 3 seed examples, BARE-generated data outperforms data from either base-only or instruct-only generation on downstream fine-tuning across all four generative domains tested: HotpotQA, PubMedQA, GSM8K, and LiveCodeBench.
  • On LiveCodeBench Test Output Prediction, fine-tuning Llama-3.1-8B-Instruct on 1,000 BARE examples reaches 28.1% accuracy, close to top similar-size models that were trained with seed sets of tens of thousands of examples.
  • BARE improves the RAFT pipeline for RAG data generation by up to 18.4% on HotpotQA with the 8B model family, setting a new state of the art in that pipeline.
  • On GSM8K, BARE is the only tested generation method whose data improves a fine-tuned Llama-3.2-1B model over its untrained baseline, and with GPT-4o refinement it reaches 35.8%, above the 26.6% obtained from the official human training set.
  • The advantage of BARE over instruct-only generation persists as the seed set grows, with the two methods converging around 100 seed examples, about 10% of the generated dataset size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the diversity-quality trade-off is general, then model families that have not been post-trained, including older or smaller checkpoints, could be recycled as cheap generators in low-resource domains without requiring a large curated seed pool.
  • The indistinguishability rate metric could be turned into a filtering criterion inside BARE: instead of refining all base samples, a discriminator could first select the most realistic candidates, potentially improving data efficiency further.
  • The convergence of BARE and instruct-only methods near 100 seed examples suggests the base model's diversity advantage matters most when seed data is scarcest; in high-resource settings the extra refinement cost may not pay off.
  • A natural stress test is to apply BARE to specialized domains such as clinical or legal text, where large seed sets are hardest to obtain and instruction-tuned-only methods would be expected to collapse to narrow modes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces BARE (Base-Refine), a two-stage method for few-shot synthetic data generation: a base language model generates a diverse set of candidate examples from three seed examples, and an instruction-tuned model refines each candidate for quality. The authors report that base-model generations are more diverse, that instruction-tuned generations are higher quality, and that fine-tuning small models on BARE-generated data improves downstream accuracy relative to instruction-tuned-only generation across six domains, including LiveCodeBench TOP, GSM8K, and RAFT-based RAG tasks.

Significance. The proposed setting is well motivated, and the central insight—that base models can supply the diversity that is normally bought with very large seed sets—is cleanly articulated. The Instruct-Instruct ablation in Appendix B.4 is an important control: it shows that replacing the base generator with an instruction-tuned generator lowers downstream accuracy, so the reported effect is not simply an artifact of the two-stage pipeline. If the results are stable under repeated trials, BARE would be a practical contribution to few-shot synthetic data generation. The main obstacles are experimental: the absence of uncertainty quantification and an internal numeric inconsistency in the PubMedQA results.

major comments (3)
  1. [Section 5 and Appendix B.2, Tables 5–8] All downstream fine-tuning results are single runs without error bars or multiple seeds, and the smallest evaluation sets are static n=100 test sets (HotpotQA and PubMedQA). The most decision-relevant comparison is not the headline 35.8% versus 17.8% on GSM8K (which confounds the choice of generator with the extra refinement pass) but the Instruct-Instruct control in Table 12: GPT-4o refining Llama-3.1-70B-Instruct data reaches 30.8%, while BARE with Llama-3.1-70B-Base plus GPT-4o reaches 35.8%; on n=500 that 5-point gap is roughly 1.7 binomial standard errors, before adding variance from data generation and LoRA fine-tuning. On LCB TOP, the 28.1% versus 26.0% comparison on n=442 is about 0.7 standard error. The paper should report repeated runs and confidence intervals, or otherwise demonstrate that the reported gaps are outside run-to-run noise.
  2. [Figure 9 versus Appendix Table 6] The same configuration (BARE with Llama-3.1-8B-Base generator and GPT-4o refiner) is reported as 0.79 accuracy on PubMedQA in Figure 9 and 0.72 in Table 6. If the correct value is 0.72, BARE does not outperform GPT-4o independent prompting (0.71) on PubMedQA, and the statements in Section 5 that BARE outperforms all GPT-4o prompting methods and also outperforms existing RAFT pipelines on PubMedQA are not supported. The discrepancy must be resolved before the aggregate claims can be accepted.
  3. [Section 3.3 and Table 1] The motivating claim that base models are more diverse is not universal: on LCB TOP, Llama-3.1-70B-Base has average pairwise cosine similarity 0.468, which is higher than the 0.389 of Llama-3.1-70B-Instruct, i.e., less diverse. The paper attributes the reversal to the base model repeating phrases from the examples, but no quantitative evidence is provided. Because diversity is the paper's proposed mechanism for downstream gains, this exception should be analyzed explicitly (for example, by recomputing similarity after removing verbatim repeated spans or by showing representative samples), rather than only acknowledged in a short remark.
minor comments (6)
  1. [Appendix B.3, Table 10] The temperature sweep on PubMedQA shows Llama-3.1-8B-Instruct at temperature 0.5 reaching 75% accuracy, which is above BARE's 73% in the same table; this contradicts the claim in Appendix B.3 that temperature gains are 'small relative to gains by using BARE.'
  2. [Figure 9 caption] The phrase 'all over methods' should presumably be 'all other methods'; more generally, the paper mixes fractional and percentage reporting (e.g., 0.79 in Figure 9 versus 28.1% in Figure 8), which makes numeric cross-checks harder.
  3. [Section 5] The statement that 'across all domains training with BARE-generated data leads to the highest model accuracy' is too strong, since Table 3 (Enron, BARE with Llama-3.1-70B: 0.771 F1 versus 0.848 for 70B Instruct) and Table 5 (HotpotQA, BARE with Llama-3.1-70B: 0.54 versus 0.55 for 70B Instruct) show BARE not being best.
  4. [Appendix A] There is a typo, 'temperauter' should be 'temperature'.
  5. [Appendix D, IR prompt] The indistinguishability-rate prompt asks the discriminator to identify the 'low quality' example rather than the 'synthetic' example; this conflation should be stated explicitly as a design choice, since it may affect the interpretation of the reported IR values.
  6. [General] No code or data-generation scripts are provided; for a method whose value is empirical, releasing generation and fine-tuning scripts would materially aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BARE's downstream gains are measured on external benchmarks, and the causal role of the base model is directly tested by an Instruct-Instruct ablation.

full rationale

The paper's central claims are supported by external downstream evaluations: fine-tuned models are measured on fixed test sets from GSM8K (n=500), LiveCodeBench TOP (n=442), HotpotQA and PubMedQA (n=100 each), plus BERT classifiers on Enron and Newsgroups. None of these accuracies is derived from the seed examples or the generation pipeline by construction; they are independent empirical outcomes. The diversity metric uses an external embedding model, and the indistinguishability-rate (IR) quality metric uses GPT-4o as a discriminator; even though GPT-4o also serves as a refiner in some BARE variants, IR is an auxiliary motivation metric and the downstream accuracy results do not depend on it. The paper also includes an Instruct-Instruct ablation (Appendix B.4, Table 12) that replaces the base generator with an instruction-tuned generator while keeping the refinement step, showing a drop in external accuracy; this directly addresses the alternative explanation that the multi-step refinement alone drives the gains. The RAFT baseline is prior work by overlapping authors, but it is used as an empirical comparison method rather than as a load-bearing theoretical premise or a uniqueness theorem. The absence of error bars on fine-tuning runs is a statistical robustness concern, not a circularity concern. No equation or fitted parameter is defined in terms of the target result, so there is no reduction of a prediction to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claims are empirical and rest on hand-chosen sampling temperatures, metric choices, and the representativeness of the fine-tuning evaluation. No new physical or mathematical entities are postulated. The main free parameters are sampling temperatures, which are not fitted to the target results.

free parameters (3)
  • Base model sampling temperature = 0.7
    Chosen by hand as a default; not tuned to downstream accuracy. The temperature ablations in Appendix B.3 vary instruct-model temperature only.
  • Instruction-tuned sampling temperature = 1.0 (Llama), 1.2 (GPT-4o)
    Chosen as the highest temperature at which generation remains coherent, per Appendix A; affects the diversity baseline of instruct-only methods.
  • Refinement temperature = 0.7
    Used for the instruction-tuned refiner in all BARE runs (Appendix A); not ablated in the paper.
assumptions (4)
  • domain assumption Base models without post-training generate more diverse outputs than instruction-tuned models under few-shot prompting.
    Demonstrated in Section 3.3 using embedding cosine similarity, but treated as a premise for designing BARE.
  • domain assumption OpenAI text-embedding-3-small cosine similarity is a valid measure of semantic diversity.
    Used in Section 3.1; no validation that this metric correlates with downstream utility beyond the paper's own experiments.
  • domain assumption GPT-4o's indistinguishability rate (IR) is a valid proxy for entry-wise quality.
    IR is introduced in Section 3.1 and used to motivate refinement; GPT-4o acts as a judge.
  • domain assumption LoRA fine-tuning of Llama-3.1-8B-Instruct for 4 epochs on 1,000 generated examples is a valid downstream utility measure.
    Used in Section 5; fixed hyperparameters from Appendix B.1.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation." pith.science (2026). https://pith.science/paper/QE3DRJJR

@misc{pith2026250201697,
  author       = {Pith},
  title        = {Pith review of: BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QE3DRJJR}},
  note         = {Machine review of arXiv:2502.01697}
}
read the original abstract

As the demand for high-quality data in model training grows, researchers and developers are increasingly generating synthetic data to tune and train LLMs. However, current data generation methods rely on seed sets containing tens of thousands of examples to prompt instruction-tuned models. This reliance can be especially problematic when the curation of high-quality examples is expensive or difficult. In this paper we explore the novel few-shot synthetic data generation setting -- generating a high-quality dataset from a few examples. We show that when working with only a few seed examples, instruction-tuned models used in current synthetic data methods produce insufficient diversity for downstream tasks. In contrast, we show that base models without post-training, largely untapped for synthetic data generation, offer substantially greater output diversity, albeit with lower instruction following abilities. Leveraging this insight, we propose Base-Refine (BARE), a novel two-stage method that combines the diversity of base models with the quality assurance of instruction-tuned models. BARE excels in few-shot synthetic data generation: using only 3 seed examples it generates diverse, high-quality datasets that significantly improve downstream task performance. We show that fine-tuning Llama 3.1 8B with 1,000 BARE-generated samples achieves performance comparable to state-of-the-art similarly sized models on LiveCodeBench tasks. Furthermore, data generated with BARE enables a 101% improvement for a fine-tuned Llama 3.2 1B on GSM8K over data generated by only instruction-models, and an 18.4% improvement for a fine-tuned Llama 3.1 8B over the state-of-the-art RAFT method for RAG data generation.

Figures

Figures reproduced from arXiv: 2502.01697 by the authors.

Figure 2
Figure 2. Accuracy of a Llama-3.2-1B-Instruct model fine-tuned on real-world data and 5 different sets of math problems synthetically generated using 3 seed examples, evaluated on a randomly selected n = 500 subset of GSM8K. Training with BARE￾generated data outperforms all other data sources. Synthetic data is most valuable when it is both high quality and diverse [Chen et al., 2024, Raventós et al., 2023]. However, while in… view at source ↗
Figure 3
Figure 3. Distribution of pairwise embedding co￾sine similarity scores for Llama-3.1-70B-Instruct and Llama-3.1-70B-Base generations for Enron spam. Base model distribution have more density in the low-similarity region and less density in the high-similarity region, indicating greater diversity. Diversity. From the pairwise cosine similarity distributions of the embeddings in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Indistinguishability Rate for Llama￾3.1-70B Base and Instruct data generation methods across various datasets. Llama-3.1-70B￾Instruct is almost uniformly better at generating examples that appear in-domain when compared alongside real-world data. Entry-wise quality [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: BARE combines base models with instruction tuned models. Instruction-tuned models provide high-quality but low-diversity data, while base models provide low-quality but high-diversity data. With minimal seed examples, BARE independently generates a diverse initial set …
Figure 6
Figure 6. Figure 6: Distribution of pairwise embedding cosine similarity scores for Llama-3.1-70B-Base and Llama 3.1 70B BARE generations for GSM8K (top) and LCB TOP (bottom). The distributions are extremely similar for both tasks, indicating that refinement retains the diversity of base …
Figure 8
Figure 8. Figure 8: Downstream accuracy of BARE compared to Llama-3.1 Base and Instruct models. Accuracy is measured on Llama-3.1-8B-Instruct Model (HotpotQA, PubMedQA, LCB TOP) and Llama-3.2-1B-Instruct Model (GSM8K) finetuned on synthetic data generated using Base, Instruct, and BARE me…
Figure 9
Figure 9. Figure 9: Downstream accuracy for BARE versus other diversification methods. Fine-tuned model accuracy on HotpotQA, PubMedQA, GSM8K, and LCB TOP for data generated from prompting methods using GPT-4o. Prompting methods have mixed effects on downstream performance when compared a…
Figure 10
Figure 10. Figure 10: Accuracy at varying seed (few-shot) sizes of a Llama-3.2-1B-Instruct Model finetuned on synthetic data generated using Instruct and BARE methods on GSM8K. BARE performance is consistent throughout while Instruct improves with seed set size before converging with BARE.…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Intent Factored Generation: Unleashing the Diversity in Your Language Model

    cs.AI 2025-06 conditional novelty 6.0 of 10

    Intent Factored Generation samples a high-temperature intent, such as keywords or a summary, and then samples the final response at lower temperature conditioned on that intent, increasing semantic diversity while kee...

  2. Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading

    cs.LG 2025-08 conditional novelty 5.0 of 10

    Fine-tuning GPT-4o-mini on about 150 examples raised short-answer grading F1 from 0.68 to 0.73; QLoRA fine-tuning of Llama 3.1 8B only reached 0.65 after adding synthetic data.

  3. The Prompt is Mightier than the Example

    cs.LG 2025-05 conditional novelty 5.0 of 10

    Injecting domain knowledge into prompts can substitute for many in-context examples in LLM-based synthetic tabular data generation, cutting required example counts by 40-90%.

  4. LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

    cs.CL 2025-05 conditional novelty 5.0 of 10

    By prompting an aligned LLM with a document and the special token that precedes a user query, LongMagpie synthesizes long-context instruction data that outperforms prior datasets when used to fine-tune Llama-3-8B.

Reference graph

Works this paper leans on

47 extracted references · 9 canonical work pages · cited by 4 Pith papers

  1. [1]

    Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I. Abdin. On the diversity of synthetic data and its impact on training large language models, 2024. URL https://arxiv.org/abs/2410.15226

    arXiv 2024

  2. [2]

    Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions

    John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprint arXiv:2306.04140, 2023

    arXiv 2023

  3. [3]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv 2021

  4. [4]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL http://arxiv.org/abs/1810.04805

    arXiv 2018

  5. [5]

    The Llama 3 herd of models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models, 2024

    work page 2024

  6. [6]

    Personas with attitudes: Controlling LLMs for diverse data annotation

    Leon Fr \"o hling, Gianluca Demartini, and Dennis Assenmacher. Personas with attitudes: Controlling LLMs for diverse data annotation. arXiv preprint arXiv:2410.11745, 2024

    arXiv 2024

  7. [7]

    Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, Hyung Won Chung, Sam Toyer, Johannes Heidecke, Alex Beutel, and Amelia Glaese. Deliberative alignment: Reasoning enables safer language models, 2025. URL https://arxiv.org/abs/2412.16339

    arXiv 2025

  8. [8]

    Deepseek-coder: When the large language model meets programming--the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming--the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024

    arXiv 2024

Show all 47 references
  1. [9]

    The curious decline of linguistic diversity: Training language models on synthetic text

    Yanzhu Guo, Guokan Shang, Michalis Vazirgiannis, and Chlo \'e Clavel. The curious decline of linguistic diversity: Training language models on synthetic text. arXiv preprint arXiv:2311.09807, 2023

    2023 arXiv

  2. [10]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020

    2020

  3. [11]

    LoRA : Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA : Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  4. [12]

    LiveCodeBench : Holistic and contamination free evaluation of large language models for code, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench : Holistic and contamination free evaluation of large language models for code, 2024. URL https://arxiv.org/abs/2403.07974

    2024 arXiv

  5. [13]

    Cohen, and Xinghua Lu

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA : A dataset for biomedical research question answering, 2019. URL https://arxiv.org/abs/1909.06146

    2019 arXiv

  6. [14]

    The Enron corpus: A new dataset for email classification research

    Bryan Klimt and Yiming Yang. The Enron corpus: A new dataset for email classification research. In Jean-François Boulicaut, Floriana Esposito, Fosca Giannotti, and Dino Pedreschi, editors, Machine Learning: ECML 2004, volume 3201 of Lecture Notes in Computer Science, pages 217...

    2004 doi

  7. [15]

    Self-directed synthetic dialogues and revisions technical report

    Nathan Lambert, Hailey Schoelkopf, Aaron Gokaslan, Luca Soldaini, Valentina Pyatkin, and Louis Castricato. Self-directed synthetic dialogues and revisions technical report. arXiv preprint arXiv:2407.18421, 2024

    2024 arXiv

  8. [16]

    Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024 a

    Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (a...

    2024 arXiv

  9. [17]

    Predicting vs

    Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, and Ari Holtzman. Predicting vs. acting: A trade-off between world modeling & agent modeling, 2024 b . URL https://arxiv.org/abs/2407.02446

    2024 arXiv

  10. [18]

    Making large language models better reasoners with step-aware verifier

    Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making large language models better reasoners with step-aware verifier. arXiv preprint arXiv:2206.02336, 2022

    2022 arXiv

  11. [19]

    Textbooks are all you need II : phi-1.5 technical report, 2023

    Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II : phi-1.5 technical report, 2023. URL https://arxiv.org/abs/2309.05463

    2023 arXiv

  12. [20]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on synthetic data, 2024. URL https://arxiv.org/abs/2404.07503

    2024 arXiv

  13. [21]

    Diversity of thought improves reasoning abilities of large language models

    Ranjita Naik, Varun Chandrasekaran, Mert Yuksekgonul, Hamid Palangi, and Besmira Nushi. Diversity of thought improves reasoning abilities of large language models. arXiv preprint arXiv:2310.07088, 2023

    2023 arXiv

  14. [22]

    Sky-T1 : Train your own o1 preview model within \ 450

    NovaSky Team . Sky-T1 : Train your own o1 preview model within \ 450. https://novasky-ai.github.io/posts/sky-t1, 2025. Accessed: 2025-01-09

    2025

  15. [23]

    Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Aleksander F...

    2024 arXiv

  16. [24]

    New embedding models and API updates

    OpenAI . New embedding models and API updates. https://openai.com/index/new-embedding-models-and-api-updates/, 2024 a

    2024

  17. [25]

    GPT-4 technical report, 2024 b

    OpenAI . GPT-4 technical report, 2024 b . URL https://arxiv.org/abs/2303.08774

    2024 arXiv

  18. [26]

    Hello GPT -4o

    OpenAI . Hello GPT -4o. https://openai.com/index/hello-gpt-4o/, 2024 c

    2024

  19. [27]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs , 2023. URL https://arxiv.org/abs/2305.15334

    2023 arXiv

  20. [28]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in P ython. In Journal of Machine Lear...

    2011

  21. [29]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  22. [30]

    Pretraining task diversity and the emergence of non-bayesian in-context learning for regression, 2023

    Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression, 2023. URL https://arxiv.org/abs/2306.15063

    2023 arXiv

  23. [31]

    Sentence- BERT : Sentence embeddings using S iamese BERT -networks

    Nils Reimers and Iryna Gurevych. Sentence- BERT : Sentence embeddings using S iamese BERT -networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084

    2019 arXiv

  24. [32]

    Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu

    Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse adversarial prompts, 20...

    2024 arXiv

  25. [33]

    LLM see, LLM do: Guiding data generation to target non-differentiable objectives, 2024

    Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. LLM see, LLM do: Guiding data generation to target non-differentiable objectives, 2024. URL https://arxiv.org/abs/2407.01490

    2024 arXiv

  26. [34]

    Ilia Shumailov, Zakhar Shumaylov, and Yue et al. Zhao. AI models collapse when trained on recursively generated data. Nature, 631: 0 755--759, 2024. doi:10.1038/s41586-024-07566-y

    2024 doi

  27. [35]

    Evaluating the evaluation of diversity in natural language generation

    Guy Tevet and Jonathan Berant. Evaluating the evaluation of diversity in natural language generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main V...

    2021 doi

  28. [36]

    Will we run out of data? limits of llm scaling based on human-generated data, 2024

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of llm scaling based on human-generated data, 2024. URL https://arxiv.org/abs/2211.04325

    2024 arXiv

  29. [37]

    Planning in natural language improves LLM search for code generation

    Evan Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, Will Song, Vaskar Nath, Ziwen Han, Sean Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves LLM search for code generation. arXiv preprint arXiv:2409.03733, 2024

    2024 arXiv

  30. [38]

    Magicoder: Empowering code generation with OSS -instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with OSS -instruct. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 52632--52657. PM...

    2024

  31. [39]

    Seshia, and Joseph E

    Justin Wong, Yury Orlovskiy, Michael Luo, Sanjit A. Seshia, and Joseph E. Gonzalez. SimpleStrat : Diversifying language model generation with stratification, 2024. URL https://arxiv.org/abs/2410.09038

    2024 arXiv

  32. [40]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA : A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600

    2018 arXiv

  33. [41]

    Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024. URL https://arxiv.org/abs/2309.12284

    2024 arXiv

  34. [42]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv.org/abs/2203.14465

    2022 arXiv

  35. [43]

    Accessing GPT -4 level mathematical olympiad solutions via Monte Carlo tree self-refine with LLaMa-3 8B

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing GPT -4 level mathematical olympiad solutions via Monte Carlo tree self-refine with LLaMa-3 8B . arXiv preprint arXiv:2406.07394, 2024 a

    2024 arXiv

  36. [44]

    Improving diversity of commonsense generation by large language models via in-context learning

    Tianhui Zhang, Bei Peng, and Danushka Bollegala. Improving diversity of commonsense generation by large language models via in-context learning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024...

    2024 doi

  37. [45]

    Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E

    Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. RAFT : Adapting language model to domain specific RAG , 2024 c . URL https://arxiv.org/abs/2403.10131

    2024 arXiv

  38. [46]

    BERT score: Evaluating text generation with BERT

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. BERT score: Evaluating text generation with BERT . arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  39. [47]

    Texygen: A benchmarking platform for text generation models

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1097--1100, 2018

    2018

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.