Pith. sign in

REVIEW 5 major objections 5 minor 43 references

InfoSynth: Information-Guided Benchmark Synthesis for LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read InfoSynth sets out to show that benchmark novelty and diversity can be computed cheaply from embedding geometry and that optimizing those scores yields automatically generated Python coding benchmarks whose solutions and tests are correct 9

desk verdict InfoSynth is a real, mostly working synthesis pipeline, but its central novelty/diversity claim is undercut by using the same embedding geometry to select and to measure; referee it, expect major revision. read the letter →

arxiv 2601.00575 v2 pith:VHS654F5 submitted 2026-01-02 cs.CL

Ishir Garg , Neel Kolhe , Xuandong Zhao , Dawn Song This is my paper
classification cs.CL
keywords benchmarksynthesisLLMevaluationcodegenerationnoveltymetricsdiversitygeneticalgorithmsKLdivergencedifferentialentropy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the quality of a coding benchmark can be decomposed into two measurable quantities—novelty, defined as KL divergence from a seed distribution over problem embeddings, and diversity, defined as the differential entropy of that distribution—and that these quantities can guide an automatic generation pipeline. The pipeline mutates and crosses over seed problems, keeps candidates that are farthest from what has already been generated, and verifies each problem by running an LLM-generated solution against LLM-generated tests in an isolated Python environment, feeding execution output back for up to five rounds of repair. The authors report that the resulting benchmarks are more novel and diverse than their seed datasets, that difficulty can be dialed up or down through mutation prompts, and that human verification found the generated solutions and tests correct 97% of the time. If true, this provides a cheap way to assess and produce evaluation benchmarks instead of relying on expensive test-taker statistics or manual construction.

What carries the argument

The load-bearing machinery is the pair of information-theoretic estimators—a k-NN KL-divergence estimator for novelty and a k-NN differential-entropy estimator for diversity—applied to sentence embeddings of problem statements after a joint nonlinear projection to 8–12 dimensions. These scores serve both as the paper's quality metric and as the selection signal for keeping or discarding generated problems. Around that core sits a genetic generation loop: LLM prompts that mutate a seed problem in three difficulty directions or cross over several problems, a k-farthest-neighbor filter that retains problems least similar to the already-selected set, an iterative code-feedback loop that runs can

What would settle it

Take a set of paraphrased copies of the seed problems (same ideas, different wording) and compute the paper's novelty score against the seeds; if the paraphrased copies score as high as genuinely new problems, the embedding-based metric is too weak to detect memorization. Alternatively, run the pipeline twice with and without k-farthest-neighbor filtering and check whether the higher-scoring dataset produces systematically lower pass rates on a held-out model that was not used in generation; if it does not, the claimed link between the scores and evaluation value fails.

Watch

Extended reading notes

Core claim

The central discovery is that benchmark-level properties can be treated as an optimization target: novelty is estimated by a k-nearest-neighbor KL-divergence estimator comparing a new dataset's embedding distribution to a seed dataset's, and diversity is estimated by a k-nearest-neighbor differential-entropy estimator on the new dataset. The paper validates these estimators by showing that known dataset relationships (topic subsets versus full sets, easy versus hard problem collections) rank as intuition predicts, then applies the same scores inside a genetic algorithm. Each generation round an LLM produces easy, medium, and hard mutations or crossover combinations of seed problems; problems

Load-bearing premise

The whole approach depends on the premise that KL divergence and differential entropy over low-dimensional projections of problem-statement embeddings measure the novelty and diversity that actually matter for evaluating LLMs—and in particular that high scores imply resistance to contamination, which the paper explicitly says it does not directly address.

Editorial extensions

If this is right

  • Synthesized MBPP- and Leetcode-seeded datasets are reported to have higher novelty and diversity than their seeds on the proposed metrics.
  • Hard-mutation variants make benchmarks harder: model pass rates drop by roughly 8–15 percentage points compared with the original MBPP dataset, providing a difficulty-control mechanism.
  • k-farthest-neighbor filtering reliably increases novelty and diversity, but produces easier problems, exposing a novelty–diversity–difficulty tradeoff that the pipeline can steer.
  • Iterative code feedback raises the fraction of passing solution–test pairs by about 20% over five iterations, with most of the gain achieved after three iterations.
  • Post-processing problem statements to specify edge-case behavior improves downstream model pass rates by 5–15 percentage points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not claimed by the paper, but a natural next step is to turn the novelty/diversity scores into a differentiable reward inside the generator rather than a post-hoc filter; the k-farthest-neighbor selection is a hard version of that idea.
  • The metrics operate on problem-statement embeddings only, so the paper's scores cannot certify that a benchmark is contamination-free; a testable extension is to correlate these scores with probe models' accuracy drops on the synthesized problems.
  • The pipeline's correctness is bounded by the generator's ability to write numerically accurate tests; the paper itself notes that problems with heavy numerical computation are filtered out, suggesting a ceiling on the problem types this approach can synthesize.
  • The reported 97% correctness comes from manual verification of 100 sampled problems per dataset; an extension would be automated verification over the entire generated set, since the execution environment already exists.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. InfoSynth proposes an automatic pipeline for synthesizing Python coding benchmarks from seed datasets, guided by information-theoretic measures of novelty and diversity. The novelty metric is the KL divergence between embedding distributions, and the diversity metric is differential entropy, both estimated with k-NN estimators on UMAP projections of problem-statement embeddings. The pipeline applies mutation and crossover, k-farthest-neighbor filtering, deduplication, iterative code-execution feedback, and postprocessing. The authors report that roughly 97% of human-checked sampled problems have correct solutions/tests, and that the generated benchmarks are often more novel and diverse than their seeds, with controllable difficulty. They validate the metrics on known dataset categories and compare with GeneticInstruct and KodCode.

Significance. If the central claims are supported, InfoSynth would be a useful contribution: it offers a relatively cheap, model-free way to estimate benchmark novelty and diversity, and a self-verifying generator that produces executable Python problems with adjustable difficulty. The use of code execution for automated correctness feedback, the human verification of sampled problems, the extensive hyperparameter ablations in Appendix A, and the embedding-model invariance analysis are strengths. However, the current evidence for the headline novelty/diversity gains is weakened by the circular relationship between selection and evaluation, uncontrolled comparisons to prior methods, and limitations in the metric validation. The contribution is therefore conditional on additional evidence.

major comments (5)
  1. [Abstract; §5.1, Fig. 4; Table 2] The abstract claims synthesized benchmarks 'consistently exhibit higher novelty and diversity compared to their seed datasets' and 'higher difficulty compared to prior works.' This is not supported as stated. Figure 4a's own caption says MBPP-Hard has lower novelty than MBPP-Original, and Figures 4c/4d show guided variants with reduced novelty. Table 2 shows MBPP-Guided often has higher pass rates than MBPP-Original (e.g., GPT-4.1-Mini: 71.93 vs 66.04; Gemini-2.0-Flash: 71.93 vs 68.72), so difficulty is not consistently higher, and no difficulty comparison to GeneticInstruct or KodCode is provided. The claims need to be restricted to the datasets and configurations where they hold, or the wording revised to be non-absolute.
  2. [§4.1, §5.1, Algorithm 1, Eqs. (1)–(4)] The k-farthest-neighbor selection criterion (Algorithm 1, EvolveColony) and the novelty/diversity evaluation (Eqs. (1)–(4)) use the same underlying embedding geometry: the pipeline retains problems with lowest cosine similarity to the seed+generated set, while the reported novelty/diversity is measured as KL/entropy over UMAP projections of those same embeddings. The claimed gains are therefore at least partly by construction. The §3.2 validation only shows that the metric orders hand-picked categories (e.g., String vs. Hash Table); it does not establish that optimizing the metric returns genuinely novel tasks. An independent held-out measure (e.g., n-gram overlap with seeds, LLM-judged topic novelty, or human blind ratings) and a random-selection control with identical prompts and generation budget are needed before accepting the novelty/diversity claims.
  3. [§3.2.2, §5.1, Fig. 4] Differential entropy estimates are strongly affected by sample size, as the paper itself notes in §3.2.2 when it equalizes N for Figure 2. Figure 4 compares datasets of different sizes (e.g., MBPP-New filtered vs. original; Leetcode-New variants) without stating whether the same sample-size equalization was applied. If it was not, higher diversity of the generated sets may simply reflect smaller sample sizes. The authors should apply the equalization procedure to all comparisons or provide control experiments demonstrating that the reported rankings are invariant to sample size.
  4. [§5.7, Fig. 6] The comparison with GeneticInstruct and KodCode is not controlled. Those methods are not run under the same seed subset, generation budget, or reference distribution, and novelty is measured against Leetcode Original, which is InfoSynth's own seed, rather than the seeds of the other pipelines. This setup cannot support the conclusion that 'InfoSynth stands out as the most novel dataset.' A fair comparison should use identical seed data and equal generation budgets, or the conclusion should be limited to InfoSynth's behavior relative to its own seed.
  5. [§3.2.1, Eq. (3)] The KL estimator in Eq. (3) produces negative values in the subset-superset comparison in §3.2.1, which is impossible for the true KL divergence. This indicates a substantial estimator bias in exactly the type of comparison the paper later uses to evaluate generated vs. seed datasets. The statement that 'we only care about relative differences' is not sufficient, because a biased estimator can distort relative rankings when dataset densities differ substantially. The authors should validate the estimator's monotonic behavior on synthetic perturbations or use a nonnegative estimator.
minor comments (5)
  1. [References] References [11] and [13] are duplicates of the same paper; please deduplicate and renumber.
  2. [§3.1, Eq. (3)] The phrase 'we take the KL-divergence of p with q as the null hypothesis' is confusing. Equation (1) defines D_KL(q||p); the wording should be clarified to reflect the actual direction.
  3. [§5.1, Fig. 4] Figure 4 captions do not state the number of UMAP runs or whether confidence intervals are shown, unlike Figure 2. Please state the sampling/equalization procedure used (or that it matches Figure 2).
  4. [Abstract, Table 1] The 97% figure in the abstract is derived from manual verification of 100 randomly sampled problems per dataset, not from all generated problems. This should be stated explicitly to avoid overgeneralization.
  5. [§5.7] The text says all three pipelines use Leetcode-based seeds, but earlier sections suggest GeneticInstruct and KodCode use multiple seed datasets. Clarify the exact seed configuration used for each method in the comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: correctness and difficulty are externally verified; the novelty/diversity metric is partially aligned with the selection rule but not a definitional reduction.

full rationale

InfoSynth's main claims are: (i) generated solutions/tests are accurate ~97%, (ii) generated benchmarks are harder, and (iii) generated benchmarks are more novel/diverse. Claim (i) is supported by execution and manual verification of 100 problems per dataset (Table 1), an external check outside the embedding metrics. Claim (ii) is supported by test-taker accuracy drops (Table 2), which are independent of any embedding-distance measure. Claim (iii) is the only part entangled with the proposed metric: Section 4.1's k-farthest-neighbor filter selects generated problems by low cosine similarity to the seed+generated set, while Section 5.1 evaluates novelty/diversity with kNN-distance-based KL/entropy estimators (Eqs. 3-4) over UMAP projections of the same problem embeddings, with distances renormalized to correspond to cosine similarity. This creates a same-ruler concern: some of the reported novelty/diversity gain of guided over unguided datasets is an expected consequence of the selection heuristic, not an independent discovery. However, this is not a formal circular reduction: the selection and the estimators are not the same function (selection uses cosine similarity; metrics use KL/entropy of UMAP-projected kNN distances), unguided generation still shows gains over seeds (Figures 4a/4b), and Section 5.6 provides a separate LLM-topic-label check of diversity. The paper also explicitly disclaims directly addressing contamination-free benchmarks, and the only self-citation (AgentSynth, ref. [25]) appears in related work and is not load-bearing. Overall, no significant circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The load-bearing extras are the semantic-embedding assumption and the self-verification assumption; both are domain assumptions the paper does not independently validate.

free parameters (6)
  • kNN neighbor count k = k=4 for KL; k/N∈[0.02,0.04] for entropy
    Hyperparameter in Wang et al. and Kozachenko-Leonenko estimators; authors select via bias-variance tradeoff (§3.3). Rankings stable but absolute values vary.
  • UMAP projection dimension d = 8–12
    High-dimensional entropy estimation is unreliable, so authors project; recommendation is post hoc from ablations (§3.2–3.3).
  • UMAP n_neighbors / min_dist = 80 / 0.1 (30/0.1 in Fig. 4d)
    UMAP hyperparameters affect the embedding coordinates used for all metrics; ablations in Appendix A show relative rankings stable.
  • Dedup similarity threshold = 0.75 similarity, 250 permutations
    MinHash+LSH threshold controls which generated problems survive; affects dataset composition (§4.1).
  • Code feedback iterations Nit = 3–5 depending on dataset
    Authors find >3 iterations marginal; selected to balance pass rate and cost (§5.3, Appendix B).
  • Evolution parameters N,Nc,Bs,C,Bc = e.g., N=1000, Nc=10, Bs=30, C=2, Bc=5
    Per-dataset generation capacity and crossover batch affect difficulty/diversity (Appendix B).
assumptions (5)
  • standard math Wang et al. and Kozachenko-Leonenko kNN estimators are consistent for the embedding distributions used here
    Eqs. (3)-(4) are invoked without checking the support/smoothness assumptions on text-embedding distributions (§3.1).
  • domain assumption Cosine similarity in the projected embedding space captures semantic/reasoning similarity of coding problems
    K-farthest selection and metric validation both presuppose this; no downstream contamination test is run (§3.2, §4.1).
  • domain assumption Code-execution self-verification of LLM-generated solution/test pairs is sufficient evidence of benchmark correctness
    Only 100 problems per dataset are human-verified; the rest rely on generated tests passing against generated solutions (§4.1, Table 1).
  • domain assumption High KL novelty and high entropy diversity approximate contamination resistance
    Paper's motivation is contamination, but it explicitly disclaims directly addressing contamination-free benchmarks (§1).
  • domain assumption GPT-4o is a reliable generator/verifier for mutation, crossover, and test synthesis
    Generator choice is fixed; failures such as long-horizon crossover problems are noted as model limitations (§5.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of InfoSynth: Information-Guided Benchmark Synthesis for LLMs." pith.science (2026). https://pith.science/paper/VHS654F5

@misc{pith2026260100575,
  author       = {Pith},
  title        = {Pith review of: InfoSynth: Information-Guided Benchmark Synthesis for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHS654F5}},
  note         = {Machine review of arXiv:2601.00575}
}
read the original abstract

Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation, but efficiently creating new benchmarks to evaluate these capabilities remains a challenge. Traditional benchmark creation relies on manual human effort, which is expensive and time-consuming. Furthermore, existing benchmarks often contaminate LLM training data, necessitating novel and diverse benchmarks to accurately assess their genuine capabilities. This work introduces InfoSynth, a novel framework for automatically generating and evaluating reasoning benchmarks guided by information-theoretic principles. We propose metrics based on KL-divergence and entropy to quantify benchmark novelty and diversity without relying on costly model evaluations. Building on this framework, we develop an end-to-end pipeline that synthesizes robust Python coding problems from seed datasets using genetic algorithms and iterative code feedback. Our method generates accurate test cases and solutions to new problems 97% of the time, and the synthesized benchmarks consistently exhibit higher difficulty compared to prior works. Moreover, our algorithm provides a method for controlling the novelty/diversity and difficulty of generated problems. InfoSynth offers a scalable, self-verifying pipeline for constructing high-quality, challenging coding benchmarks for LLMs. Project Page: https://ishirgarg.github.io/infosynth_web/

Figures

Figures reproduced from arXiv: 2601.00575 by the authors.

Figure 1
Figure 1. Left: The full Leetcode dataset has higher novelty than its Hash Table and String subsets [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Left: Leetcode vs. MBPP entropy. MBPP shows lower entropy due to simpler, more [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Generation Pipeline. Each colony receives a subset of the seed problems and applies [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Novelty and diversity analysis across MBPP and Leetcode variants. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Fraction of problems relating to each topic for the 10 most common topics [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Novelty and diversity of various problem generation pipelines [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Embedding model invariance analysis for novelty and diversity metrics [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity analysis across all hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Self-verification for MBPP-New 60 80 100 10 20 % Passed % Failed % Erred % Unparsable 0 1 2 3 4 5 Number of Code Feedback Iterations 0 2 4 6 Percentage of Generated Dataset (%) [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages

  1. [2]

    BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions.ArXiv preprint, abs/2406.15877, 2024

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Davi...

    arXiv 2024

  2. [3]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. Program Synthesis with Large Language Models.ArXiv preprint, abs/2108.07732, 2021. URLhttps: //arxiv.org/abs/2108.07732

    arXiv 2021

  3. [4]

    Evaluat- ing Large Language Models Trained on Code.ArXiv preprint, abs/2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluat- ing Large Language Models Trained on Code.ArXiv preprint, abs/2107.03374, 2021. URL https://arxiv.org/abs/2107.03374

    arXiv 2021

  4. [6]

    Learning Code Preference via Synthetic Evolution.ArXiv preprint, abs/2410.03837, 2024

    Jiawei Liu, Thanh Nguyen, Mingyue Shang, Hantian Ding, Xiaopeng Li, Yu Yu, Varun Ku- mar, and Zijian Wang. Learning Code Preference via Synthetic Evolution.ArXiv preprint, abs/2410.03837, 2024. URLhttps://arxiv.org/abs/2410.03837

    arXiv 2024

  5. [7]

    Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models.ArXiv preprint, abs/2407.21077, 2024

    Somshubra Majumdar, Vahid Noroozi, Sean Narenthiran, Aleksander Ficek, Jagadeesh Balam, and Boris Ginsburg. Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models.ArXiv preprint, abs/2407.21077, 2024. URLhttps://arxiv. org/abs/2407.21077

    arXiv 2024

  6. [8]

    CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom.ArXiv preprint, abs/2503.01836, 2025

    Yisen Li, Lingfeng Yang, Wenxuan Shen, Pan Zhou, Yao Wan, Weiwei Lin, and Dongping Chen. CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom.ArXiv preprint, abs/2503.01836, 2025. URLhttps://arxiv.org/abs/2503.01836

    arXiv 2025

  7. [9]

    Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis from Scratch.ArXiv preprint, abs/2410.18693, 2024

    Yuyang Ding, Xinyu Shi, Xiaobo Liang, Juntao Li, Qiaoming Zhu, and Min Zhang. Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis from Scratch.ArXiv preprint, abs/2410.18693, 2024. URLhttps://arxiv.org/abs/2410.18693. 11

    arXiv 2024

  8. [10]

    MATH-Perturb: Benchmarking LLMs’ Math Reasoning Abilities against Hard Perturbations.ArXiv preprint, abs/2502.06453, 2025

    Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, Yue Wu, Ming Yin, Shange Tang, Yangsibo Huang, Chi Jin, Xinyun Chen, Chiyuan Zhang, and Mengdi Wang. MATH-Perturb: Benchmarking LLMs’ Math Reasoning Abilities against Hard Perturbations.ArXiv preprint, abs/2502.06453, 2025. URLhttps://ar...

    arXiv 2025

Show all 43 references
  1. [12]

    Data Contamination Quiz: A Tool to Detect and Esti- mate Contamination in Large Language Models.ArXiv preprint, abs/2311.06233, 2023

    Shahriar Golchin and Mihai Surdeanu. Data Contamination Quiz: A Tool to Detect and Esti- mate Contamination in Large Language Models.ArXiv preprint, abs/2311.06233, 2023. URL https://arxiv.org/abs/2311.06233

    2023 arXiv

  2. [13]

    Investigating data contamination in modern benchmarks for large language models

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors,Proceedings of the 2024 Conference of the North Amer- ican Chapter ...

    2024

  3. [14]

    A careful examination of large language model performance on grade school arithmetic

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Ka- plan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade sch...

    2024

  4. [15]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st ...

    2023 doi

  5. [16]

    PromptCoT: Synthesizing Olympiad- level Problems for Mathematical Reasoning in Large Language Models.ArXiv preprint, abs/2503.02324, 2025

    Xueliang Zhao, Wei Wu, Jian Guan, and Lingpeng Kong. PromptCoT: Synthesizing Olympiad- level Problems for Mathematical Reasoning in Large Language Models.ArXiv preprint, abs/2503.02324, 2025. URLhttps://arxiv.org/abs/2503.02324

    2025

  6. [17]

    Wizardlm: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. InThe Twelfth International Conference on Learning Rep- resentations, ICLR 20...

    2024

  7. [18]

    Kod- Code: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding.ArXiv preprint, abs/2503.02951, 2025

    Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kod- Code: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding.ArXiv preprint, abs/2503.02951, 2025. URLhttps://arxiv.org/abs/2503.02951

    2025 arXiv

  8. [19]

    DSTC: Direct Preference Learning with Only Self-generated Tests and Code to Improve Code LMs.ArXiv preprint, abs/2411.13611, 2024

    Zhihan Liu, Shenao Zhang, and Zhaoran Wang. DSTC: Direct Preference Learning with Only Self-generated Tests and Code to Improve Code LMs.ArXiv preprint, abs/2411.13611, 2024. URLhttps://arxiv.org/abs/2411.13611. 12

    2024 arXiv

  9. [20]

    RLTF: Reinforcement Learning from Unit Test Feedback.ArXiv preprint, abs/2307.04349, 2023

    Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. RLTF: Reinforcement Learning from Unit Test Feedback.ArXiv preprint, abs/2307.04349, 2023. URLhttps://arxiv.org/abs/2307.04349

    2023 arXiv

  10. [21]

    Codet: Code generation with generated tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net,

    2023

  11. [22]

    ACECODER: Acing Coder RL via Automated Test-case Synthesis.ArXiv preprint, abs/2502.01718, 2025

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. ACECODER: Acing Coder RL via Automated Test-case Synthesis.ArXiv preprint, abs/2502.01718, 2025. URLhttps://arxiv.org/abs/2502.01718

    2025 arXiv

  12. [23]

    AutoBencher: Towards Declarative Benchmark Construction

    Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. AutoBencher: Towards Declarative Benchmark Construction. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28,

    2025

  13. [24]

    Truong, Yuheng Tu, Percy Liang, Bo Li, and Sanmi Koyejo

    Sang T. Truong, Yuheng Tu, Percy Liang, Bo Li, and Sanmi Koyejo. Reliable and Efficient Amortized Model-based Evaluation.ArXiv preprint, abs/2503.13335, 2025. URLhttps: //arxiv.org/abs/2503.13335

    2025 arXiv

  14. [25]

    Agentsynth: Scalable task generation for generalist computer-use agents.arXiv preprint arXiv:2506.14205, 2025

    Jingxu Xie, Dylan Xu, Xuandong Zhao, and Dawn Song. Agentsynth: Scalable task generation for generalist computer-use agents.arXiv preprint arXiv:2506.14205, 2025

    2025

  15. [26]

    TaskE- val: Assessing Difficulty of Code Generation Tasks for Large Language Models.ArXiv preprint, abs/2407.21227, 2024

    Florian Tambon, Amin Nikanjam, Cyrine Zid, Foutse Khomh, and Giuliano Antoniol. TaskE- val: Assessing Difficulty of Code Generation Tasks for Large Language Models.ArXiv preprint, abs/2407.21227, 2024. URLhttps://arxiv.org/abs/2407.21227

    2024

  16. [27]

    Proximal Policy Optimization Algorithms.ArXiv preprint, abs/1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms.ArXiv preprint, abs/1707.06347, 2017. URLhttps:// arxiv.org/abs/1707.06347

    2017 arXiv

  17. [28]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors,2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http://arxiv.org/abs...

    2014 arXiv

  18. [29]

    Kulkarni, and Sergio Verd ´u

    Qing Wang, Sanjeev R. Kulkarni, and Sergio Verd ´u. Divergence estimation for multidimen- sional densities via k-nearest-neighbor distances.IEEE Trans. Inf. Theory, 55(5):2392–2405,

  19. [30]

    Mpnet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors,Ad- vances in Neural Information Processing...

    2020

  20. [31]

    UMAP: Uniform Manifold Approximation and Projection.J

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. UMAP: Uniform Manifold Approximation and Projection.J. Open Source Softw., 3(29):861, 2018. doi: 10.21105/JOSS.00861. URLhttps://doi.org/10.21105/joss.00861

    2018 doi

  21. [32]

    leetcode-problem-set.https://huggingface.co/datasets/kaysss/ leetcode-problem-set, 2025

    kaysss. leetcode-problem-set.https://huggingface.co/datasets/kaysss/ leetcode-problem-set, 2025. Accessed: 2025-05-11

    2025

  22. [33]

    Measuring coding challenge competence with apps.ArXiv preprint, abs/2105.09938, 2021

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with apps.ArXiv preprint, abs/2105.09938, 2021. URLhttps:// arxiv.org/abs/2105.09938. 13

    2021 arXiv

  23. [34]

    Codeforces Problems Dataset.https://huggingface.co/datasets/open-r1/ codeforces, 2024

    open-r1. Codeforces Problems Dataset.https://huggingface.co/datasets/open-r1/ codeforces, 2024. Accessed: 2025-05-11

    2024

  24. [35]

    Estimating mutual information

    Alexander Kraskov, Harald St ¨ogbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, 69(6), 2004. ISSN 1550-2376. doi: 10.1103/physreve.69.066138. URL http://dx.doi.org/10.1103/PhysRevE.69.066138

    2004 doi

  25. [36]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors,...

    2022

  26. [37]

    Gpt-4o system card, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, 2024. URL https://arxiv.org/abs/2410.21276

    2024 arXiv

  27. [38]

    LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs.ArXiv preprint, abs/2504.14655, 2025

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs.ArXiv preprint, abs/2504.14655, 2025. URLhttps://arxiv.org/ abs/2504.14655

    2025 arXiv

  28. [39]

    mutation

    Hirokazu Kiyomaru, Issa Sugiura, Daisuke Kawahara, and Sadao Kurohashi. A Compre- hensive Analysis of Memorization in Large Language Models. In Saad Mahamood, Minh Le Nguyen, and Daphne Ippolito, editors,Proceedings of the 17th International Natural Language Generation Confere...

    2024 doi

  29. [45]

    The sum of the fourth power of the first 'k' odd numbers from the array

  30. [46]

    "" Returns the second shortest string containing all vowels, sorted alphabetically

    The sum of the bitwise XOR of all pairs of numbers formed by selecting one element from the first 'k' even numbers and one element from the last 'k' elements in the array. The function should take two arguments - the array of integers and an integer 'k'. Ensure that 'k' is les...

  31. [2009]

    URLhttps://doi.org/10.1109/TIT.2009

    doi: 10.1109/TIT.2009.2016060. URLhttps://doi.org/10.1109/TIT.2009. 2016060

    2009

  32. [2021]

    URLhttps://arxiv.org/abs/2110.14168

    arXiv

  33. [2023]

    URLhttps://openreview.net/pdf?id=ktrw68Cmu9c

  34. [2024]

    URLhttps://arxiv.org/abs/2403.07974

    arXiv

  35. [2025]

    URLhttps://openreview.net/forum?id=ymt4crbbXh

    OpenReview.net, 2025. URLhttps://openreview.net/forum?id=ymt4crbbXh

    2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.