Evaluating Language Models as Synthetic Data Generators

Carolin Lawrence; Graham Neubig; Juyoung Suk; Kiril Gashteovski; Sean Welleck; Seongyun Lee; Seungone Kim; Vijay Viswanathan; Xiang Yue; Yizhong Wang

REVIEW 3 major objections 5 minor 2 cited by

Evaluating Language Models as Synthetic Data Generators

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AgoraBench, a controlled benchmark that trains 99 student models on synthetic data from six language models, finds that data-generation ability does not track problem-solving ability and that intrinsic data-quality signals predict it…

desk verdict A genuinely useful benchmark with a misstated PCA claim; the headline finding about intrinsic indicators needs rework before it can be trusted at full strength. read the letter →

arxiv 2412.03679 v2 pith:4BK5RD6T submitted 2024-12-04 cs.CL

Seungone Kim , Juyoung Suk , Xiang Yue , Vijay Viswanathan , Seongyun Lee , Yizhong Wang , Kiril Gashteovski , Carolin Lawrence

show 2 more authors

Sean Welleck Graham Neubig

This is my paper · ORCID

classification cs.CL

keywords syntheticdatagenerationlanguagemodelevaluationabilityPerformanceGapRecoveredinstructiondifficultyresponsequalitydiversityselection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds a standardized arena for ranking language models (LMs) as generators of synthetic post-training data. It creates nine settings (three domains—math, code, instruction-following—times three generation methods: instance generation, response generation, and quality enhancement), holds every variable except the generator LM fixed, and trains Llama-3.1-8B on 10,000 instances produced by each of six generator LMs. It then measures each generator by Performance Gap Recovered (PGR), the fraction of the gap between the base model and Llama-3.1-8B-Instruct that the generated data closes. The central claim is that a model's problem-solving score does not predict its data-generation score, and that a combination of intrinsic data features—instruction difficulty, response quality, and response perplexity—explains far more of the variation in student-model improvement. The paper argues this matters because practitioners choosing an LM for synthetic data currently lack a standardized way to compare data-generation ability.

What carries the argument

The central machinery is the PGR metric together with the controlled benchmark design. PGR is defined as $(\text{score}_B(S_{D_G}) - \text{score}_B(S_\emptyset)) / (\text{score}_B(S_{ref}) - \text{score}_B(S_\emptyset)) \times 100$, where $S_\emptyset$ is the base Llama-3.1-8B model, $S_{D_G}$ is that model supervised-fine-tuned on the generated data, and $S_{ref}$ is Llama-3.1-8B-Instruct; it reports how much of the reference post-training improvement the synthetic data recovers. The benchmark holds the meta-prompt, seed data, data volume, and student model fixed across generators, isolating the generator as the only variable. A second piece of machinery is the intrinsic-metric analysis: each generated dataset is scored on instruction difficulty, response quality (via two LLM-as-judge models and a reward model), response perplexity under the student model, and diversity, and principal component analysis is used to find low-dimensional combinations of these features that track PGR.

What would settle it

Take the same six generators and the same nine settings, but train the student model on 50,000 instances per setting and with a second base model (e.g., Llama-3.1-70B-Base or Qwen2.5-7B), then recompute PGR and the rank correlation with problem-solving scores. If the generator ordering changes substantially, or if the top-five principal components of intrinsic metrics no longer explain most PGR variance, the paper's central claims are specific to its 10K-instance, Llama-3.1-8B regime rather than general properties of data-generation ability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that data generation ability is a distinct capability from problem solving. Across 1.26 million synthesized training instances and 99 trained student models, AgoraBench shows that no single LM dominates every data-generation setting: GPT-4o is strongest at generating new instances, Claude-3.5-Sonnet at enhancing existing ones, and cheaper or weaker models sometimes beat their stronger counterparts (e.g., Llama-3.1-8B-Instruct outperforms GPT-4o on code instance generation). Regression of benchmark problem-solving scores against PGR gives weak or non-significant correlations, while the top five principal components of intrinsic metrics (instruction difficulty, response quality from two judges and a reward model, response perplexity, and diversity) account for 93.4% of the variance in PGR. A linear regression on those components reaches statistical significance with $R^2 = 0.325$, against $R^2 < 0.1$ for problem-solving scores, leading the authors to conclude that intrinsic data-quality features collectively are the better indicator of an LM's data-generation ability.

Load-bearing premise

The whole ranking rests on treating the PGR score computed with a single student model (Llama-3.1-8B), a single random seed, and 10,000 generated instances per setting as a stable measure of data-generation ability; if a different student model, more training data, or another seed reorders the generators, the headline conclusions about problem-solving vs. data-generation ability would not generalize.

Editorial extensions

If this is right

Practitioners can choose data generators by matching method to model: GPT-4o for generating novel instances, Claude-3.5-Sonnet for refining existing data, and cheap models when scaling volume, since 50K instances from GPT-4o-mini can beat 10K from GPT-4o at lower cost.
Problem-solving leaderboards should not be used as a proxy for data-generation ability, because the regression shows the two are nearly uncorrelated.
Intrinsic data features could serve as a low-cost screening signal: measuring instruction difficulty, response quality, and perplexity of a candidate generator's output predicts a meaningful share of downstream student improvement without training a student model.
Output format matters: free-form meta-prompts outperform JSON-format prompts by about 4.5% in PGR, so structuring the generation format can silently change rankings.
Because no single model wins all nine settings, benchmarks like this one create a case for specialized data-generation models tuned for particular domains and methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because PGR is measured with a single student base model (Llama-3.1-8B), the ordering of generators might change if the student were, say, a 70B or multilingual model; the paper's own limitations note this. A natural next experiment is to re-run the benchmark with a second student model and see whether the top-5 principal-component predictor transfers.
The 93.4% variance explained is computed over 54 setting-level measurements, not over individual instances, so it reflects how well intrinsic features separate settings rather than how accurately any single dataset's downstream gain can be forecast; a practitioner should expect larger error at the individual-dataset level.
The finding that weaker generators can outperform stronger ones hints that data-generation skill is trainable independently of problem-solving skill; one could test this by fine-tuning a small model specifically on high-PGR generated data and checking whether its PGR rises without changing its benchmark scores.
The meta-prompt comparison (unoptimized vs. optimized vs. JSON) implies that prompt engineering can shift PGR by about 4% on average, so future benchmarks should report the exact meta-prompt alongside any ranking.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things you should know. AgoraBench is a real contribution: it standardizes nine settings (three domains times three generation methods), controls meta-prompts, seed data, instance counts, and the student model, and the PGR metric sensibly measures how much of a reference model's post-training gap a synthetic dataset recovers. The paper also includes careful scaling and meta-prompt robustness checks, which is more than most benchmark papers do. But the central analytical claim leans on a misread number: the abstract says top-5 principal components from intrinsic measurements explain 93.4% of variance in PGR, when that 93.4% is actually the cumulative variance of the intrinsic feature matrix. The regression of PGR on those same five components gives R^2 = 0.325, in-sample, on 54 points with five predictors. That is a meaningful difference, and it is the basis for saying intrinsic features are better indicators than problem-solving ability.

What is new and good: this is the first systematic comparison of LMs as generators across instance generation, response generation, and quality enhancement in one controlled framework. The observation that weaker solvers can beat stronger generators in some settings is interesting and worth following up, though with only six generators it is a pattern, not a law. The cost analysis is practical and honest. Contemporaneous work by Xu et al. (2024c) only covered response generation, so the benchmark occupies a distinct slot. Code, data, and checkpoints are public, which makes the whole thing more credible.

Soft spots, in proportion: beyond the PCA wording, the experiments use a single training seed, a single student model (Llama-3.1-8B), and no confidence intervals. Many PGR differences are small (20.6 vs. 16.1 in math instance generation), so fine-grained rankings should not be over-read. The authors explicitly acknowledge the base-model and 10K-instance limitations, and they did run partial scaling experiments to 50K, which deserves credit. Still, the paper's strongest conceptual claim about prediction requires out-of-sample validation and ideally multiple seeds before it is presented as a finding.

Bottom line: practitioners choosing data generators will get real value from AgoraBench, and the benchmark itself is citable. The paper deserves a serious referee, but the referee should push for corrected PCA reporting and multi-seed evidence. I would bring it to reading group, mostly to argue about the analysis section.

Referee Report

3 major / 5 minor

Summary. The paper introduces AgoraBench, a benchmark for comparing LMs as synthetic data generators under controlled conditions. It fixes meta-prompts, seed datasets, data volume, and student model (Llama-3.1-8B), and measures teacher quality by Performance Gap Recovered (PGR), the relative improvement of a student SFT model over the base model toward Llama-3.1-8B-Instruct. Across three domains (math, code, instruction following) and three generation methods (instance generation, response generation, quality enhancement), six generators produce 10K instances each, yielding 99 student models. The paper reports (i) generator-specific strengths (e.g., GPT-4o in instance generation, Claude-3.5-Sonnet in quality enhancement), (ii) weak correlation between problem-solving ability and PGR, (iii) an intrinsic-feature PCA analysis claiming top-5 components explain 93.4% of variance in PGR, with a linear regression R²=0.325, and (iv) experiments on cost, scaling, and meta-prompt format.

Significance. AgoraBench addresses a real gap: previous synthetic-data work varies too many factors to compare generators. The fully specified protocol, public code/checkpoints/data, 1.26M generated instances, and 99 trained student models are valuable resources. The PGR metric is a reasonable extrinsic measure, and the robustness checks on meta-prompts and data volume are useful. The headline non-correlation claim, if supported, would matter for model selection. However, the current quantitative evidence for the intrinsic-feature conclusion is weaker than presented, and the ranking claims lack variance estimates; the benchmark itself remains a useful contribution.

major comments (3)

[Section 5.2 / Figure 5 / Abstract] The claim that 'top-5 principal components ... can explain 93.4% of the variance in the PGR values' conflates two different quantities. The percentages in Figure 5 (39.2, 30.4, 11.9, 7.0, and 4.9) sum to 93.4% of the variance of the nine standardized intrinsic features, not of PGR. The actual regression of PGR on those components, reported in Figure 6, gives R²=0.325 on the same 54 observations. The abstract and Section 5.2 should be corrected to say that the PCs explain 93.4% of the intrinsic-feature variance; the explained variance of PGR is roughly one-third. Because the paper's conclusion that intrinsic features are better indicators than problem-solving ability rests on this evidence, the conflation is load-bearing.
[Section 5.2 / Figure 6 / Table 4] The regression in Figure 6 is an in-sample fit: the regression coefficients are estimated on the same PGR values that Figure 6 then 'predicts,' with five predictors on n=54 observations and no out-of-sample or cross-validated assessment. The word 'predict' in the text therefore overstates what is shown; the adjusted R², cross-validated R², or a permutation test is needed before claiming intrinsic metrics are better predictors. Table 4 is also presented as contribution 'to the PGR results,' but the numbers are normalized average loading magnitudes across the principal components, which do not measure each feature's contribution to PGR; this should be relabeled or replaced with a regression-based contribution analysis.
[Section 4 / Table 2 / Appendix E] The central ranking comparisons are point estimates from a single training seed (seed 42), a single student model (Llama-3.1-8B), and one data volume (10K instances). Several differences that drive the narrative are small relative to the likely noise (e.g., math instance generation 20.6 vs 16.1; quality enhancement averages 17.9 vs 5.5). Without multiple seeds, confidence intervals, or a bootstrap over benchmarks, Table 2 does not establish that the observed ordering is stable. The limitations section acknowledges the single base model and 10K scale but not the single seed. Either add variance estimates or soften the ranking claims to 'point estimates under one configuration.' This also affects the non-correlation analysis in Figure 4, whose R² values cannot be separated from seed noise.

minor comments (5)

[Figure 4] The caption mixes R and R² and the reported p-values appear inconsistent with the sample sizes; please report regression statistics (slope, SE, F, adjusted R²) and ensure axes match the text.
[Section 4, paragraph after Table 3] The sentence 'and average performance on AGORA BENCH are listed in Table 3. and average performance on AGORA BENCH for all six LMs' is garbled and should be rewritten.
[Table 8] The row labeled 'Instruction Difficulty (Perplexity)' is misleading because the text defines the feature as response perplexity conditioned on the instruction; correct the row label to avoid confusing it with instruction difficulty.
[Figure 7] The horizontal axis is unlabeled and has no tick values, making the scaling comparison difficult to verify; adding the actual data volumes and cost ratios would strengthen the presentation.
[Section 2, Equation (2)] PGR is undefined if the reference model and base model have equal scores on a benchmark; a sentence noting degenerate cases would be helpful for practitioners applying the metric.

Circularity Check

2 steps flagged · score 6.0 of 10

PCA's 93.4% 'variance explained' is the variance of the intrinsic-feature inputs, not of PGR, so the headline indicator claim substitutes this self-variance for the real regression fit (R2=0.325).

fitted input called prediction [Abstract; Section 5.2 (Figure 5)]
"Furthermore, we demonstrate that the top-5 principal components extracted from intrinsic measurements can explain 93.4% of the variance in the PGR values. [Section 5.2:] The results, shown in Figure 5, reveal that the top five principal components explain approximately 93.4% of the variance in AGORA BENCH results (39.2%, 30.4%, 11.9%, 7.0%, and 4.9% respectively)."

The PCA is computed on the intrinsic features listed in Figure 5 (judge/reward scores, perplexity, diversities) plus problem-solving ability; PGR is not one of the PCA input variables. The 'explained variance ratio' percentages are, by construction, the share of variance of those input features captured by each principal component, and they sum to 93.4%. Reporting this as 'variance in the PGR values' or 'variance in AGORA BENCH results' substitutes the variance of the fitted input for the variance of the target variable. The paper's own Figure 6 regression of PGR on the same PCs yields R2=0.325, so 0.934 is not PGR-explained variance. The abstract's headline indicator claim therefore leans on a number that is self-referential to the PCA input by construction.
fitted input called prediction [Section 5.2 (Figure 6)]
"Lastly, we predict data generation capabilities by performing linear regression on the top-5 principal components, weighting each component by its corresponding regression coefficient, as shown in Figure 6. Compared to using problem-solving scores alone (Figure 4), this approach yields a statistically significant relationship (p <0.001) with improved explanatory power (R2 = 0.325)."

The coefficients in the weighted linear combination are fitted to the same n=54 PGR values that are then called 'predicted' in Figure 6; no held-out set, cross-validation, or adjusted R2 is reported. The resulting R2=0.325 is therefore an in-sample goodness-of-fit, not an out-of-sample prediction. Insofar as the paper presents this as evidence that intrinsic features 'predict' or 'serve as better indicators' of data generation ability, the prediction statistic is the fit itself. This is a milder, statistical form of the fitted-input-called-prediction pattern: the weights are chosen to maximize fit to the target, so the reported explanatory power is a property of the fitted model rather than an independent forecast.

full rationale

AgoraBench itself is a controlled, externally grounded benchmark: PGR is computed from student models trained on generated data and evaluated on standard benchmarks, and the per-generator rankings in Table 2 are independent measurements. The problem-solving-versus-generation correlation analysis is honest in reporting weak R2 and p-values. The circularity is concentrated in Section 5.2's intrinsic-indicator analysis. The 93.4% figure quoted in the abstract and Figure 5 is the cumulative explained variance of the PCA performed on intrinsic features; since PGR is not an input to that PCA, the number cannot be the explained variance of PGR by construction. The actual relationship between the PCA scores and PGR is the in-sample regression R2=0.325 in Figure 6, which is a fit rather than a validated prediction. No load-bearing self-citation chain or uniqueness argument appears. The paper's own limitations (single base model, 10K instances) are acknowledged weaknesses, not circular steps. Because the headline 'intrinsic features are better indicators' claim borrows a number that is self-referential to the fitted input, but the benchmark and the R2=0.325 comparison retain independent empirical content, the overall circularity is partial.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central comparison of LMs via PGR is directly measured and does not rely on fitted parameters. The PCA/regression analysis introduces fitted components and coefficients, but these are used only as an explanatory model, not as a derivation of PGR. Main assumptions are the validity of SFT as a probe and the representativeness of Llama-3.1-8B as the student model.

free parameters (2)

Top-5 principal component loadings = not reported
Used to construct intrinsic quality scores; the number of components was chosen to explain ~93.4% of variance in intrinsic metrics.
Linear regression coefficients mapping PCs to PGR = not reported
Fitted on the 54 AgoraBench observations to produce the R^2=0.325 'prediction' result.

assumptions (5)

domain assumption Supervised fine-tuning on generated data, with loss only on response tokens, is a valid probe of data generation ability.
The paper relies on this to define PGR and compare generators; if SFT is not a clean probe, the rankings may reflect training dynamics rather than data quality.
domain assumption Llama-3.1-8B is a representative base model for evaluating data generators.
All PGR values are measured with this student; the authors acknowledge that findings may not generalize to other base models.
domain assumption The chosen benchmarks (GSM8K, MATH, MBPP, HumanEval, AlpacaEval 2.0, Arena-Hard) measure the intended capabilities of the student model.
PGR is calculated on these benchmarks, so any benchmark limitation propagates to the comparison.
domain assumption The seed datasets are of sufficient quality and appropriateness for each domain.
Instance generation expands seed data, quality enhancement refines it, and response generation uses fixed instructions; biased seed data would affect all generators equally but still shape absolute PGR.
domain assumption The outputs of proprietary LMs are stable and representative, and the released data faithfully captures them.
The original generation used GPT-4o and Claude APIs; if those outputs are not reproducible from the released data, replication is impossible.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Language Models as Synthetic Data Generators." pith.science (2026). https://pith.science/paper/4BK5RD6T

@misc{pith2026241203679,
  author       = {Pith},
  title        = {Pith review of: Evaluating Language Models as Synthetic Data Generators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4BK5RD6T}},
  note         = {Machine review of arXiv:2412.03679}
}

read the original abstract

Given the increasing use of synthetic data in language model (LM) post-training, an LM's ability to generate high-quality data has become nearly as crucial as its ability to solve problems directly. While prior works have focused on developing effective data generation methods, they lack systematic comparison of different LMs as data generators in a unified setting. To address this gap, we propose AgoraBench, a benchmark that provides standardized settings and metrics to evaluate LMs' data generation abilities. Through synthesizing 1.26 million training instances using 6 LMs and training 99 student models, we uncover key insights about LMs' data generation capabilities. First, we observe that LMs exhibit distinct strengths. For instance, GPT-4o excels at generating new problems, while Claude-3.5-Sonnet performs better at enhancing existing ones. Furthermore, our analysis reveals that an LM's data generation ability doesn't necessarily correlate with its problem-solving ability. Instead, multiple intrinsic features of data quality-including response quality, perplexity, and instruction difficulty-collectively serve as better indicators. Finally, we demonstrate that strategic choices in output format and cost-conscious model selection significantly impact data generation effectiveness.

Figures

Figures reproduced from arXiv: 2412.03679 by the authors.

**Figure 1.** Illustration of the motivation of AGORABENCH: Prior works focused on developing new methods to generate synthetic data. In contrast, our work focuses on systematically comparing different LMs as data generators based on existing data generation methods. Further explanation of data generation methods are covered in Section 2. an identical number of training instances. Llama3.1-8B is trained on each synthetic dataset… view at source ↗

**Figure 2.** AGORABENCH tests three data generation methods: generating new instruction and response pairs (left), generating responses (middle), and enhancing the quality of the instruction and/or the response (right). Data Generation Methods. As shown in Figure 2, among the various methods for generating data, most can be grouped into three categories: instance generation, response generation, and quality enhancement. These … view at source ↗

**Figure 3.** Illustration of Performance Gap Recovered metric: The performance gap recovered metric captures the relative improvement of SDG with respect to Sref where SDG and Sref is both trained from SØ. where SØ denotes a pre-trained LM, SDG denotes SØ trained on DG, Sref denotes a reference model that shares the same pre-trained model SØ as a base model, and scoreB(·) denotes the score on benchmark B. In our experiments, we … view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Problem-solving and data generation capabilities do not strongly correlate: Linear regression between problem-solving ability and data generation ability scores at multiple granularity levels yields either low R2 values (R2 < 0.1) or non-significant relationships (p > …

**Figure 5.** Figure 5: Through a PCA analysis on multiple intrinsic evaluation metrics, we find that there exists [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: Principal Components from Intrinsic Metrics Show Stronger Correlation with Data Generation ability: Linear regression using the weighted top-5 principal components yields a higher explained variance (R2 = 0.325) and statistical significance (p < 0.001) compared to us…

**Figure 7.** Figure 7: With a fixed budget, generating large amounts of data with weaker LMs could sometimes be more effective and cheaper than generating a few instances with stronger LMs: Since GPT-4o-mini is 17 times cheaper than GPT-4o, generating 50K instances is 3.4 times cheaper than …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation
cs.CL 2025-09 reject novelty 3.0 of 10

A synthetic long-context data generation framework is described, but with no empirical evaluation or comparison to existing methods.
Generative Models for Synthetic Data: Transforming Data Mining in the GenAI Era
cs.LG 2025-08 unverdicted novelty 1.0 of 10

A tutorial proposal outlining how generative models can synthesize data across modalities for data mining, with no new research results.

Reference graph

Works this paper leans on

51 extracted references · 22 canonical work pages · cited by 2 Pith papers

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

Claude 3.5 sonnet model card addendum

Anthropic, A. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 2024

work page 2024
[3]

Program synthesis with large language models

Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

arXiv 2021
[4]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 2020

work page 2020
[5]

A survey on evaluation of large language models

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 0 (3): 0 1--45, 2024

2024
[6]

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

arXiv 2021
[7]

Training verifiers to solve math word problems

Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

arXiv 2021
[8]

How abilities in large language models are affected by supervised fine-tuning data composition

Dong, G., Yuan, H., Lu, K., Li, C., Xue, M., Liu, D., Wang, W., Yuan, Z., Zhou, C., and Zhou, J. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492, 2023

arXiv

Show all 51 references

[9]

The llama 3 herd of models

Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[10]

Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024

2024 arXiv
[11]

Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023

2023 arXiv
[12]

Evaluating large language models: A comprehensive survey

Guo, Z., Jin, R., Liu, C., Huang, Y., Shi, D., Yu, L., Liu, Y., Li, J., Xiong, B., Xiong, D., et al. Evaluating large language models: A comprehensive survey. arXiv preprint arXiv:2310.19736, 2023

2023 arXiv
[13]

Measuring mathematical problem solving with the math dataset

Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021

2021
[14]

R., and Levy, O

Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10782, 2022

2022 arXiv
[15]

Unnatural instructions: Tuning language models with (almost) no human labor

Honovich, O., Scialom, T., Levy, O., and Schick, T. Unnatural instructions: Tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 14409--14428, 2023

2023
[16]

P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

2024 arXiv
[17]

The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning

Kim, S., Joo, S., Kim, D., Jang, J., Ye, S., Shin, J., and Seo, M. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\...

2023
[18]

J., Jang, Y., Chae, H., and Yeo, J

Kim, S., Joo, S. J., Jang, Y., Chae, H., and Yeo, J. Cotever: Chain of thought prompting annotation toolkit for explanation verification. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pp.\...

2023
[19]

Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M

Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

2024
[20]

Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al

Lambert, N., Pyatkin, V., Morrison, J., Miranda, L., Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787, 2024

2024 arXiv
[21]

E., and Stoica, I

Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024

2024 arXiv
[22]

Universal and context-independent triggers for precise control of llm outputs

Liang, J., Li, G., and Yu, Y. Universal and context-independent triggers for precise control of llm outputs. arXiv preprint arXiv:2411.14738, 2024

2024 arXiv
[23]

Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y

Liu, C. Y., Zeng, L., Liu, J., Yan, R., He, J., Wang, C., Yan, S., Liu, Y., and Zhou, Y. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451, 2024 a

2024 arXiv
[24]

S., Wang, Y., and Zhang, L

Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chat GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023 a . URL https://openreview.net/forum?id...

2023
[25]

Best practices and lessons learned on synthetic data

Liu, R., Wei, J., Liu, F., Si, C., Zhang, Y., Rao, J., Zheng, S., Peng, D., Yang, D., Zhou, D., et al. Best practices and lessons learned on synthetic data. In First Conference on Language Modeling, 2024 b

2024
[26]

What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

Liu, W., Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, 2023 b

2023
[27]

W., Tay, Y., Zhou, D., Le, Q

Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pp.\ 22631--22648. PMLR, 2023

2023
[28]

Introducing meta llama 3: The most capable openly available llm to date

MetaAI. Introducing meta llama 3: The most capable openly available llm to date. 2024. URL https://ai.meta.com/blog/meta-llama-3/

2024
[29]

Cross-task generalization via natural language crowdsourcing instructions

Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational ...

2022
[30]

Mteb: Massive text embedding benchmark

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp.\ 2014--2037, 2023 a

2014
[31]

S., Shen, S., Yong, Z

Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational ...

2023
[32]

Orca: Progressive learning from complex explanation traces of gpt-4

Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707, 2023

2023 arXiv
[33]

Mixeval: Deriving wisdom of the crowd from LLM benchmark mixtures

Ni, J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neubig, G., and You, Y. Mixeval: Deriving wisdom of the crowd from LLM benchmark mixtures. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=6A29LUZhfv

2024
[34]

Leverage the Latest Open Models for Synthetic Data Generation with NVIDIA Nemotron-4-340B

Nvidia . Leverage the Latest Open Models for Synthetic Data Generation with NVIDIA Nemotron-4-340B . https://developer.nvidia.com/blog/leverage-our-latest-open-models-for-synthetic-data-generation-with-nvidia-nemotron-4-340b/ https://developer.nvidia.com/blog/leverage-our-late...

2024
[35]

J., and Hashimoto, T

Ruan, Y., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of langauge model performance. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=On5WIN7xyD

2024
[36]

B., Cardenas, E., Sharma, A., Trengrove, J., and van Luijt, B

Shorten, C., Pierse, C., Smith, T. B., Cardenas, E., Sharma, A., Trengrove, J., and van Luijt, B. Structuredrag: Json response formatting with large language models. arXiv preprint arXiv:2408.11061, 2024

2024 arXiv
[37]

R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-y., and Chen, Y.-N

Tam, Z. R., Wu, C.-K., Tsai, Y.-L., Lin, C.-Y., Lee, H.-y., and Chen, Y.-N. Let me speak freely? a study on the impact of format restrictions on performance of large language models. arXiv preprint arXiv:2408.02442, 2024

2024 arXiv
[38]

Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023

2023
[39]

Qwen2.5: A party of foundation models, September 2024

Team, Q. Qwen2.5: A party of foundation models, September 2024. URL https://qwenlm.github.io/blog/qwen2.5/

2024
[40]

Prompt2model: Generating deployable models from natural language instructions

Viswanathan, V., Zhao, C., Bertsch, A., Wu, T., and Neubig, G. Prompt2model: Generating deployable models from natural language instructions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp.\ 413--421, 2023

2023
[41]

S., Arunkumar, A., Stap, D., et al

Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Emp...

2022
[42]

A., Khashabi, D., and Hajishirzi, H

Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)...

2023
[43]

W., Lester, B., Du, N., Dai, A

Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021

2021
[44]

Wizard LM : Empowering large pre-trained language models to follow complex instructions

Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. Wizard LM : Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openrevi...

2024
[45]

Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024 b

2024 arXiv
[46]

Y., and Poovendran, R

Xu, Z., Jiang, F., Niu, L., Lin, B. Y., and Poovendran, R. Stronger models are not stronger teachers for instruction tuning. arXiv preprint arXiv:2411.07133, 2024 c

2024 arXiv
[47]

Learning to mine aligned code and natural language pairs from stack overflow

Yin, P., Deng, B., Chen, E., Vasilescu, B., and Neubig, G. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Mining Software Repositories, MSR, pp.\ 476--486. ACM, 2018. doi:https://doi.org/10.1145/3196398.3196408

2018
[48]

Mammoth: Building math generalist models through hybrid instruction tuning

Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2023

2023
[49]

MA mmo TH 2: Scaling instructions from the web

Yue, X., Zheng, T., Zhang, G., and Chen, W. MA mmo TH 2: Scaling instructions from the web. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=yVu5dnPlqA

2024
[50]

Z., Ye, X., Yang, X., Chen, L., Wang, W

Zhang, X., Chen, Z. Z., Ye, X., Yang, X., Chen, L., Wang, W. Y., and Petzold, L. R. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. arXiv preprint arXiv:2405.20535, 2024

2024 arXiv
[51]

Lima: Less is more for alignment

Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

Claude 3.5 sonnet model card addendum

Anthropic, A. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 2024

work page 2024

[3] [3]

Program synthesis with large language models

Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

arXiv 2021

[4] [4]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 2020

work page 2020

[5] [5]

A survey on evaluation of large language models

Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 0 (3): 0 1--45, 2024

2024

[6] [6]

Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

arXiv 2021

[7] [7]

Training verifiers to solve math word problems

Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

arXiv 2021

[8] [8]

How abilities in large language models are affected by supervised fine-tuning data composition

Dong, G., Yuan, H., Lu, K., Li, C., Xue, M., Liu, D., Wang, W., Yuan, Z., Zhou, C., and Zhou, J. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arXiv:2310.05492, 2023

arXiv