Pith. sign in

REVIEW 4 major objections 6 minor 45 references

LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read BenchMaker, a purely LLM-driven pipeline, generates a custom multiple-choice benchmark from a written assessment demand, and the paper reports that its model rankings reproduce MMLU-Pro's ordering of 12 LLMs with 0.967 Pearson correlation…

desk verdict Well-engineered benchmark generator with a strong external-validity claim, but missing overlap checks against the human benchmarks it validates on—fix that before trusting the 0.967 correlation. read the letter →

arxiv 2502.01683 v1 pith:ZV6PMYKE submitted 2025-02-02 cs.CL cs.AI

Peiwen Yuan , Shaoxiong Feng , Yiwei Li , Xinglin Wang , Yueqi Zhang , Jiayi Shi , Chuyi Tan , Boyuan Pan
show 2 more authors
Yao Hu Kan Li
This is my paper · ORCID
classification cs.CLcs.AI
keywords benchmarkgenerationsyntheticdataLLM-as-a-judgemultiple-choicequestionsevaluationframeworkdifficultycontrolcontaminationBenchMaker
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single LLM pipeline, BenchMaker, can generate a custom multiple-choice benchmark from nothing more than a written assessment demand, and that the resulting benchmark matches or beats human-annotated benchmarks on the authors' ten evaluation criteria. The headline result is that model rankings produced by BenchMaker's benchmark correlate with MMLU-Pro at 0.967 Pearson across 12 LLMs, at a cost of about $0.005 and 0.38 minutes per sample. To make this credible, the paper builds a four-dimension, ten-criteria evaluation framework and argues that LLM-judge biases in this scenario are mediated entirely by judgment length, so regression on judge length yields unbiased faithfulness and alignment scores. If true, this would let practitioners produce tailored, contamination-resistant, difficulty-controllable benchmarks on demand.

What carries the argument

The load-bearing mechanism is the automated evaluation framework itself, a four-dimension, ten-criteria scoring system: credibility (faithfulness and alignment), diversity (lexical, semantic, and knowledge), difficulty (controllability and boundary), and benchmark-level properties (effectiveness, robustness, and efficiency). Inside it, the debiasing step fits a multiple-regression model $f(i)=\beta_i+\beta_{\text{len}}\cdot \text{judge\_length}+\epsilon$, treating judge length as a covariate so that the coefficient $\beta_i$ becomes the debiased faithfulness or alignment score. On the generation side, the difficulty diffusion mechanism carries the argument: the hardest generated samples are fed back as references so that sample difficulty rises continuously, while conflict-guided contrastive discrimination treats the generator as a test-taker and uses self-consistency prediction conflicts to correct questionable rationales and labels.

What would settle it

Run the Figure 2 partial-correlation analysis on a dataset with a known judge bias unrelated to length, such as a preference for a particular answer position or phrasing style, and check whether faithfulness and alignment scores still agree with human labels after controlling for judge length; if any such factor remains significant, the mediation-by-length assumption fails.

Watch

Extended reading notes

Core claim

The central claim is that a generic benchmark generator, not a task-specific augmenter, can be made reliable enough to substitute for human-annotated benchmarks. With only assessment demands as input, BenchMaker generates multiple-choice samples containing a question, rationale, options, and label, and it patches the weaknesses of directly prompting an LLM: stepwise self-correction and conflict-guided contrastive discrimination raise faithfulness, difficulty strategy guidance and a difficulty diffusion mechanism extend difficulty controllability and boundary, and AttrPrompt plus in-batch redundancy filtering raise diversity. Under the proposed framework, BenchMaker is reported to achieve faithfulness comparable to human benchmarks, higher alignment, knowledge diversity, difficulty controllability and boundary, and an average of 0.953 Pearson and 0.966 Spearman correlation with the model rankings of human benchmarks, including 0.967 Pearson against MMLU-Pro across 12 LLMs. The paper also derives a formula intended to show that unbiased label noise does not change the statistical significance of a model ranking.

Load-bearing premise

The whole evaluation framework is only as trustworthy as the claim that every LLM-judge bias in this scenario is mediated by judgment length, so any bias that does not run through response length survives the debiasing regression.

Editorial extensions

If this is right

  • If BenchMaker is right, producing a tailored benchmark for a new assessment demand requires only a written demand, not an existing seed benchmark.
  • The reported 0.967 Pearson correlation with MMLU-Pro across 12 LLMs implies that benchmark effectiveness can be bought for roughly $0.005 per sample, making large custom evaluation suites affordable.
  • The improved difficulty controllability and boundary imply that generated benchmarks can include hardest subsets capable of differentiating advanced models, addressing benchmark saturation.
  • The reliability formula implies that a known fraction of incorrect labels does not by itself weaken the statistical significance of a model ranking, provided the errors are unbiased.
  • Because samples are generated on demand rather than drawn from a static pool, the approach could serve as a dynamic benchmark generator to mitigate data contamination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 0.967 correlation is measured with MMLU-Pro as the ground truth, so the strongest version of the claim is about reproducing an existing human benchmark rather than about discovering a correct ranking for a brand-new domain.
  • Editorial inference: a natural testable extension is to apply BenchMaker to an assessment demand outside MATH, MMLU-Pro, and HellaSwag with an independent gold standard, such as a professional licensing exam, and check whether the effectiveness correlation stays above 0.9.
  • Editorial inference: the debiasing assumption implies that judge biases not channeled through response length, such as option-position preference or topic aversion, would escape the regression correction, so the unbiasedness claim is only as broad as that assumption.
  • Editorial inference: the paper's own format-conversion experiment shows that open-ended versions score lower on knowledge diversity and effectiveness than the MCQ versions, suggesting the headline numbers may not transfer directly to free-form generative benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. BenchMaker is a pipeline that generates multiple-choice benchmarks solely from written assessment demands. The paper first defines a ten-criteria evaluation framework for benchmark generators, with LLM-judge scores debiased via regression on judge length. It then analyzes a direct-prompt baseline, adds modules for faithfulness, difficulty, and diversity, and evaluates the resulting BenchMaker on math (MATH), language understanding (MMLU-Pro), and commonsense (HellaSwag). The headline result is a 0.967 Pearson correlation between model accuracies on the generated MMLU-Pro-style benchmark and on MMLU-Pro across 12 LLMs, at a cost of about $0.005 and 0.4 minutes per sample.

Significance. The claimed result is potentially important: if a cheap, generic generator can produce benchmarks whose model rankings match human-curated benchmarks, it would lower the cost of customized evaluation and help mitigate contamination. The paper's use of external human benchmarks for validation is the right methodology, and the per-sample cost estimate is compelling. The framework also usefully separates credibility, diversity, difficulty, and benchmark-level criteria. However, the headline claim is only as strong as the exclusion of memorization and the validity of the debiasing adjustment; both need additional evidence before the 'reliable, generic' label is warranted. The paper ships code and includes a manual check and a reliability formula, which are positive features.

major comments (4)
  1. [§5.3, Table 2] The effectiveness claim (0.967 Pearson against MMLU-Pro) is vulnerable to memorization contamination. Section 5.3 argues that because the input demands do not mention MMLU-Pro, the model cannot achieve high alignment by 'memorizing and replicating samples from MMLU-Pro,' but this does not rule out GPT-4o mini reproducing or lightly paraphrasing benchmark items seen during pretraining. The t-SNE and word-cloud analyses show global distributional differences but cannot detect a substantial subset of near-duplicates. Please report quantitative overlap statistics between the generated benchmarks and MMLU-Pro, MATH, and HellaSwag (e.g., n-gram overlap, embedding-similarity percentiles, or membership tests). Without such analysis, the 'generic generation' interpretation of the 0.967 correlation is not established.
  2. [§3.1] The framework's 'unbiased' metrics rely on the claim that all LLM-judge biases are mediated by judgment length. This is supported by partial correlations on a single dataset (MATH) with two judges (Qwen-Plus and GPT-4o mini). Unmeasured confounders (e.g., sample topic, rationale style, or position bias) could invalidate the regression adjustment in Eq. (1), and the faithfulness/alignment scores in Table 2 depend on it. Please validate the mediation assumption on at least one additional dataset and provide a sensitivity analysis with alternative covariate sets.
  3. [§3.3–§3.4, Appendix E] Difficulty controllability and knowledge diversity are computed using the error rates and correctness patterns of the same 12-model set M1:|M| that is later used for the effectiveness correlation. This creates a risk of self-reference: the difficulty labels are defined by the models' behavior, and the benchmark is then evaluated on those same models. Please report whether the difficulty and effectiveness results are stable when M1:|M| is split into separate sets for label construction and evaluation, or otherwise justify why this circularity does not affect the claims.
  4. [Abstract, §5.1, Table 2] The abstract states that BenchMaker 'achieves superior or comparable performance to human-annotated benchmarks on all metrics,' but Table 2 shows faithfulness of 0.930 (MATH) and 0.918 (BenchMaker4o) against a human benchmark score of 1.000, and the manual check in §5.3 reports a 7.5% error rate. These point estimates are below the human benchmark. Please replace the blanket claim with a metric-by-metric comparison that includes confidence intervals or a statement of statistical significance.
minor comments (6)
  1. [Throughout] The method name 'BENCH MAKER' appears with and without a space (e.g., 'BENCHMAKER', 'BenchMaker', 'BENCH MAKER'); please unify the notation.
  2. [Appendix A, Eq. (8)] The variance formula for ¯a − ¯b ignores the covariance between the two models' accuracies because both are scored on the same benchmark items; please state this approximation explicitly.
  3. [Table 2] The 'Human Benchmark' rows show '-' for effectiveness and robustness; adding a one-sentence explanation would help readers understand why these cells are empty.
  4. [Figure 2 caption] The phrase 'The most relevant path' is not defined; please specify the selection rule (e.g., highest absolute correlation along a directed path).
  5. [§5.3] The manual check of 80 samples should describe the sampling procedure and whether the multiple authors annotated independently, as this affects the reported 7.5% error rate.
  6. [Appendix I] References to 'Appendix I' are ambiguous because the appendix contains several prompt templates; please cite the specific prompt names (e.g., 'LLM as Faithfulness Judge').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 0.967 effectiveness correlation is an external, empirical benchmark comparison, not a fitted identity.

full rationale

The paper's central effectiveness claim (0.967 Pearson correlation against MMLU-Pro across 12 LLMs) is an empirical correlation between model accuracies on generated samples and on a human-annotated benchmark; neither quantity is fitted to the other, and no equation in Sections 3-5 defines one in terms of the other. BenchMaker's components (stepwise self-correction, contrastive discrimination, difficulty diffusion, AttrPrompt) are validated by ablations on the same framework that motivated them, which creates a mild design-evaluation coupling, but the headline correlation is not optimized by any component and is reported against external benchmarks. Difficulty labels use the generator's self-consistency error (Eq. 3), while the difficulty ground truth is the average error rate of a distinct model set (Appendix E, which excludes GPT-4o mini), so the reported Spearman values are empirical rather than identities. The debiased faithfulness/alignment metrics rest on an explicit assumption (Section 3.1) that judge biases are mediated solely by judgment length; this is an unverified modeling assumption and a correctness/validity risk, not a definitional reduction. There are no load-bearing self-citations or imported uniqueness theorems. The memorization/overlap concern raised by a skeptical reading (Section 5.3) is a data-contamination threat to generalization, not a circularity of the derivation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The main free parameters are pipeline hyperparameters set by preliminary experiments without sensitivity analysis. The axioms are the debiasing assumption, the representativeness of the 12-model set, and the equal-p assumption in the reliability formula. No new entities are postulated.

free parameters (6)
  • decay_factor_0_9 = 0.9
    Decay factor in Calibrate_Difficulty formula (Eq. 10, Appendix F), set without sensitivity analysis.
  • reference_number = 8
    Number of reference samples in difficulty diffusion, tuned by preliminary experiments (Appendix F).
  • in_batch_candidates_L = 5
    Number of candidates generated per sample for in-batch diversity boosting, set as default (Section 4.5).
  • self_consistency_T = not reported
    Number of generations for self-consistency in conflict-guided discrimination and difficulty label estimation (Eq. 3); value not stated in the main text.
  • difficulty_level_descriptions = 10 levels
    Hand-crafted difficulty level descriptions (Appendix D) used in prompts; arbitrary scaling.
  • hardest_subset_quantile = 20%
    Top 20% difficulty samples used to measure difficulty boundary (Section 4.2); no sensitivity analysis.
assumptions (5)
  • domain assumption Judge length mediates all LLM-judge biases in this setup
    Section 3.1: partial correlation controlling for judge length removes significance of other factors; assumes no other confounders.
  • domain assumption The 12-model set provides ground-truth difficulty and knowledge embeddings
    Sections 3.2, 3.3: error rates and correctness patterns of the chosen 12 models are treated as ground truth for difficulty and knowledge diversity.
  • domain assumption Models A and B have equal accuracy p on incorrectly labeled samples
    Appendix A, Step 1: the reliability estimation assumes identical p for both models on noisy samples; if violated, the de-noised difference is biased.
  • standard math Observed accuracy difference is normally distributed
    Appendix A, Step 4: CLT approximation for hypothesis testing.
  • ad hoc to paper Judge scores on human benchmarks rescale to 1 and generated benchmarks can exceed 1
    Section 4.2 footnote: alignment scores above 1.0 are reported after rescaling; this is a modeling choice that makes 'superior alignment' possible by construction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient." pith.science (2026). https://pith.science/paper/ZV6PMYKE

@misc{pith2026250201683,
  author       = {Pith},
  title        = {Pith review of: LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZV6PMYKE}},
  note         = {Machine review of arXiv:2502.01683}
}
abstract

The rapid advancement of large language models (LLMs) has led to a surge in both model supply and application demands. To facilitate effective matching between them, reliable, generic and efficient benchmark generators are widely needed. However, human annotators are constrained by inefficiency, and current LLM benchmark generators not only lack generalizability but also struggle with limited reliability, as they lack a comprehensive evaluation framework for validation and optimization. To fill this gap, we first propose an automated and unbiased evaluation framework, structured around four dimensions and ten criteria. Under this framework, we carefully analyze the advantages and weaknesses of directly prompting LLMs as generic benchmark generators. To enhance the reliability, we introduce a series of methods to address the identified weaknesses and integrate them as BenchMaker. Experiments across multiple LLMs and tasks confirm that BenchMaker achieves superior or comparable performance to human-annotated benchmarks on all metrics, highlighting its generalizability and reliability. More importantly, it delivers highly consistent evaluation results across 12 LLMs (0.967 Pearson correlation against MMLU-Pro), while taking only $0.005 and 0.38 minutes per sample.

Figures

Figures reproduced from arXiv: 2502.01683 by the authors.

Figure 1
Figure 1. The trends of LLMs released and open-source [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pearson correlations among key factors of benchmark evaluation and LLM (Qwen-Plus) judge scores [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of BENCHMAKER. we get the self-consistency (Wang et al., 2023a) re￾sult a¯˙ i through majority voting. If a¯˙ i ̸= ai , the conflict suggests differing ri and r¯˙ i . As Zheng et al. (2023) finds that comparison-based judges are more accurate than item-wise judges, we have the LLM conduct a contrastive discrimination be￾tween ri and r¯˙ i to determine the final rationale and label for si . 4.4 Difficulty Op… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Trends of real and labeled difficulty over the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Word cloud of MMLU-Pro and the benchmark [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: T-SNE results on the text embeddings of benchmarks. 5.3 A Closer Look at the Generated Benchmark After metric analysis, we perform a more thorough examination of BENCHMAKER. Some of the gen- [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Pearson correlations among key factors of benchmark evaluation and LLM (GPT-4o mini) judge scores [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 13 canonical work pages

  1. [1]

    Claude 3.5

    Anthropic. Claude 3.5. https://www.anthropic.com/news/3-5-models-and-computer-use

    work page

  2. [2]

    Simone Balloccu, Patr \' cia Schmidtov \' a , Mateusz Lango, and Ondrej Dusek. 2024. https://aclanthology.org/2024.eacl-long.5 Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms . In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2024 - Volume 1: Long...

    work page 2024

  3. [3]

    Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. https://doi.org/10.48550/ARXIV.2406.20094 Scaling synthetic data creation with 1,000,000,000 personas . CoRR, abs/2406.20094

    arXiv doi:10.48550/arxiv.2406.20094 2024

  4. [4]

    Yu, Qiang Yang, and Xing Xie

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. https://doi.org/10.1145/3641289 A survey on evaluation of large language models . ACM Trans. Intell. Syst. Technol. , 15(3):39:1--39:45

    doi:10.1145/3641289 2024

  5. [5]

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean - Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli J \" a rviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Gre...

    arXiv doi:10.48550/arxiv.2411.04872 2024

  6. [6]

    Xiaobo Guo and Soroush Vosoughi. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.984 Length does matter: Summary length can bias summarization metrics . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 , pages 15869--15879. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-main.984 2023

  7. [7]

    Richard W Hamming. 1950. Error detecting and error correcting codes. The Bell system technical journal, 29(2):147--160

    work page 1950

  8. [8]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net

    2021

Show all 45 references
  1. [9]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html Measuring mathematical problem...

    2021

  2. [10]

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. https://openreview.net/forum?id=IkmD3fKBPQ Large language models cannot self-correct reasoning yet . In The Twelfth International Conference on Learning Representati...

    2024

  3. [11]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  4. [12]

    Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.123 Towards mitigating LLM hallucination via self reflection . In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, ...

    2023 doi

  5. [13]

    Kusner, and Ricardo Silva

    Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva. 2022. https://doi.org/10.48550/ARXIV.2206.15475 Causal machine learning: A survey and open problems . CoRR, abs/2206.15475

    2022 doi arXiv

  6. [14]

    Fangyu Lei, Qian Liu, Yiming Huang, Shizhu He, Jun Zhao, and Kang Liu. 2023. https://doi.org/10.48550/ARXIV.2310.15147 S3eval: A synthetic, scalable, systematic evaluation suite for large language models . CoRR, abs/2310.15147

    2023 doi arXiv

  7. [15]

    u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \" u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e...

    2020

  8. [16]

    Jiatong Li, Renjun Hu, Kunzhe Huang, Yan Zhuang, Qi Liu, Mengxiao Zhu, Xing Shi, and Wei Lin. 2024 a . https://doi.org/10.48550/ARXIV.2405.19740 Perteval: Unveiling real knowledge capacity of llms with knowledge-invariant perturbations . CoRR, abs/2405.19740

    2024 doi arXiv

  9. [17]

    Sihang Li, Jin Huang, Jiaxi Zhuang, Yaorui Shi, Xiaochen Cai, Mingjun Xu, Xiang Wang, Linfeng Zhang, Guolin Ke, and Hengxing Cai. 2024 b . https://doi.org/10.48550/ARXIV.2408.15545 Scilitllm: How to adapt llms for scientific literature understanding . CoRR, abs/2408.15545

    2024 doi arXiv

  10. [18]

    Zenan Li, Zhi Zhou, Yuan Yao, Yu - Feng Li, Chun Cao, Fan Yang, Xian Zhang, and Xiaoxing Ma. 2024 c . https://doi.org/10.48550/ARXIV.2412.04857 Neuro-symbolic data generation for math reasoning . CoRR, abs/2412.04857

    2024 doi arXiv

  11. [19]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://aclanthology.org/2023.emnlp-main.153 G-eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proc...

    2023

  12. [20]

    Lin Long, Rui Wang, Ruixuan Xiao, Junbo Zhao, Xiao Ding, Gang Chen, and Haobo Wang. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.658 On llms-driven synthetic data generation, curation, and evaluation: A survey . In Findings of the Association for Computational Linguisti...

    2024 doi

  13. [21]

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Qingwei Lin, Jianguang Lou, Shifeng Chen, Yansong Tang, and Weizhu Chen. 2024 a . https://doi.org/10.48550/ARXIV.2407.10627 Arena learning: Build data flywheel for llms post-training via simulated chatbot arena . CoRR, abs/2407.10627

    2024 doi arXiv

  14. [22]

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2024 b . https://openreview.net/forum?id=UnUwSIgK5W Wizardcoder: Empowering code large language models with evol-instruct . In The Twelfth International Co...

    2024

  15. [23]

    Gaurav Maheshwari, Dmitry Ivanov, and Kevin El Haddad. 2024. https://doi.org/10.48550/ARXIV.2409.11968 Efficacy of synthetic data as a benchmark . CoRR, abs/2409.11968

    2024 doi arXiv

  16. [24]

    Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. 2019. http://arxiv.org/abs/1904.03971 Jointly measuring diversity and quality in text generation models . CoRR, abs/1904.03971

    2019 arXiv

  17. [25]

    text-embedding-ada-002

    OpenAI. text-embedding-ada-002. https://platform.openai.com/docs/guides/embeddings

  18. [26]

    Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan

    Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, D...

    2023

  19. [27]

    Zhi Rui Tam, Cheng - Kuang Wu, Yi - Lin Tsai, Chieh - Yen Lin, Hung - yi Lee, and Yun - Nung Chen. 2024. https://aclanthology.org/2024.emnlp-industry.91 Let me speak freely? A study on the impact of format restrictions on large language model performance . In Proceedings of th...

    2024

  20. [28]

    Zhengwei Tao, Ting - En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou. 2024. https://doi.org/10.48550/ARXIV.2404.14387 A survey on self-evolution of large language models . CoRR, abs/2404.14387

    2024 doi arXiv

  21. [29]

    Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. https://doi.org/10.48550/ARXIV.2406.12624 Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges . CoRR, abs/2406.12624

    2024 doi arXiv

  22. [30]

    Raphael Vallat. 2018. https://doi.org/10.21105/JOSS.01026 Pingouin: statistics in python . J. Open Source Softw., 3(31):1026

    2018 doi

  23. [31]

    Ke Wang, Jiahui Zhu, Minjie Ren, Zeming Liu, Shiwei Li, Zongye Zhang, Chenkai Zhang, Xiaoyu Wu, Qiqi Zhan, Qingjie Liu, and Yunhong Wang. 2024 a . https://doi.org/10.48550/ARXIV.2410.12896 A survey on data synthesis and augmentation for large language models . CoRR, abs/2410.12896

    2024 doi arXiv

  24. [32]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023 a . https://openreview.net/forum?id=1PL1NIMMrw Self-consistency improves chain of thought reasoning in language models . In The Eleventh International Confe...

    2023

  25. [33]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023 b . https://doi.org/10.18653/V1/2023.ACL-LONG.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual ...

    2023 doi

  26. [34]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024 b . https://doi.org/10.48550/ARXIV.2406.01574 Mmlu-pro: A mor...

    2024 doi arXiv

  27. [35]

    Siyuan Wu, Yue Huang, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xiangliang Zhang, Jianfeng Gao, Chaowei Xiao, and Lichao Sun. 2024. https://doi.org/10.48550/ARXIV.2406.18966 Unigen: A unified framework for textual dataset generation using large language mod...

    2024 doi

  28. [36]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 doi arXiv

  29. [37]

    Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2024. https://openreview.net/forum?id=N8N0hgNDRt Metamath: Bootstrap your own mathematical questions for large language models . In The Twelfth I...

    2024

  30. [38]

    Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang

    Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J. Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/ae9500c4f5607caf2eff033c67daa9d7-Abstract-Datasets\_and\_Benchmarks.html Large language model as attributed ...

    2023

  31. [39]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://doi.org/10.18653/V1/P19-1472 Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, ...

    2019 doi

  32. [40]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstr...

    2023

  33. [41]

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. 2024 a . https://openreview.net/forum?id=gjfOL9z5Xr Dyval: Dynamic evaluation of large language models for reasoning tasks . In The Twelfth International Conference on Learning Representations,...

    2024

  34. [42]

    Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. 2024 b . https://doi.org/10.48550/ARXIV.2402.14865 Dyval 2: Dynamic evaluation of large language models by meta probing agents . CoRR, abs/2402.14865

    2024 doi arXiv

  35. [43]

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. https://doi.org/10.1145/3209978.3210080 Texygen: A benchmarking platform for text generation models . In The 41st International ACM SIGIR Conference on Research & Development in Informatio...

    2018

  36. [44]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  37. [45]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.