Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

HardTests: Synthesizing High-Quality Test Cases for LLM Coding

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HardTestGen's synthesized test suites classify LLM-generated code more accurately than existing synthetic tests, improving precision by 11.3 points and recall by 17.5 points on average.

desk verdict A practically valuable test-synthesis pipeline with a clean AtCoder evaluation, but the Codeforces evaluation protocol biases the headline numbers and needs fixing before the paper is fully convincing. read the letter →

arxiv 2505.24098 v1 pith:456F6KSN submitted 2025-05-30 cs.CL

Zhongmou He , Yee Man Choi , Kexun Zhang , Jiabao Ji , Junting Zhou , Dejia Xu , Ivan Bercovich , Aidan Zhang
show 1 more author
Lei Li
This is my paper
classification cs.CL
keywords testcasesynthesisLLMcodeverifierscompetitiveprogrammingreinforcementlearningself-distillationgenerationqualityoracleprograms
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that high-quality verifier test cases for competitive programming can be synthesized with LLMs, without access to proprietary human-written tests. It claims its pipeline, HardTestGen, produces test suites that agree with human oracle labels substantially better than prior synthetic suites, with precision 11.3 percentage points higher and recall 17.5 percentage points higher when classifying LLM-generated code. The gap grows on harder problems, up to 40 precision points, and the better tests measurably improve reinforcement learning and self-distillation for code LLMs. If correct, this means training verifiers for coding reasoning no longer depends on scarce human-written test cases.

What carries the argument

The pipeline runs an LLM to produce an input validator, then three kinds of test-input generators: Type 1 directly generated small inputs, Type 2 random regular inputs from an LLM-written generator, and Type 3 hacking inputs designed to trigger worst-case time complexity or edge-case bugs. Outputs come from human-written oracle programs, with outputs accepted when two or more oracles agree on at least 90% of cases, and special judge functions handle problems where string comparison is insufficient. The validator filters out invalid inputs before they enter the test suite.

What would settle it

Deliberately give the pipeline two oracle programs that share the same subtle bug on a set of competition problems, then check whether the synthesized suite marks buggy candidate programs as correct; if a shared bug slips through in a substantial fraction of cases, the 90% agreement rule is insufficient.

Watch

Extended reading notes

Core claim

The central claim is that converting LLM test synthesis from directly writing inputs to writing and filtering test-generator programs produces verifier suites that are much closer to human oracle tests. Across AtCoder and Codeforces problems, HardTestGen's tests beat TACO and CodeContests on both precision and recall in nearly every setting, with the widest margin on the hardest problems and on programs generated by weaker LLMs. The paper further claims that using these tests during RL training and self-distillation improves downstream LiveCodeBench pass@k, while teacher distillation is less sensitive to test quality and benefits more from question scaling.

Load-bearing premise

The pipeline assumes that LLM-written input validators and oracle programs are reliable enough to define the tests, even though validators may skip complex constraints and oracle outputs are trusted when two or more programs agree on more than 90% of cases.

Editorial extensions

If this is right

  • RL training for code LLMs can use these synthesized tests to assign 0/1 rewards with fewer false positives, so models are less likely to be rewarded for inefficient or subtly wrong programs.
  • Self-distillation can be steered by verifier quality: filtering a 5k training subset with HardTestGen improved pass@5 and pass@10 over random or bad subsets.
  • Teacher distillation with a strong reasoning model benefits more from scaling the number of questions than from filtering trajectories by test correctness.
  • The approach transfers across 13 online judges and 47k problems, so a single pipeline can produce verifiers for a large, diverse competition dataset.
  • The 90%-agreement rule among oracle programs and the explicit allowance for skipping hard-to-verify constraints mean the pipeline's reliability depends on the quality of its LLM-generated validators and oracle consistency.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pipeline generalizes beyond standard-I/O competition problems, it could reduce the need for human-written tests in other execution-based coding domains, such as repository-level bug fixing, wherever a trustworthy oracle executable exists.
  • The multi-category output generator idea could be adapted to imbalanced test generation in general, ensuring rare output classes are represented rather than relying on random sampling.
  • A direct testable extension would be measuring how precision/recall of the synthesized suite changes as the number of agreed oracle programs increases from two to eight, since the 90% threshold is a heuristic.
  • The oracle-free variant described in the appendix suggests that even without accepted solutions, brute-force oracle programs can seed tests, though its false positive rate remains much higher than the full pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HardTestGen, an LLM-based pipeline that synthesizes test cases for competitive programming problems. Tests are produced in three types (directly generated, regular/random, and hacking inputs), using LLM-written input validators and human-written oracle programs to generate outputs. The authors release HardTests, a dataset of 47,136 problems (32.5k with generated tests) from 13 online judges, and evaluate test quality by measuring precision and recall of the synthesized suites as binary classifiers of LLM-generated and human-written candidate programs, with AtCoder official tests and Codeforces judge verdicts as ground truth. They report 11.3 percentage-point higher precision and 17.5 percentage-point higher recall than TACO and CodeContests, larger gains on harder problems, and downstream improvements in self-distillation and reinforcement learning on LiveCodeBench.

Significance. The evaluation design is largely sound: AtCoder official tests and Codeforces judge verdicts are external human-written oracles, and the post-training evaluation uses LiveCodeBench, which is not used to fit constants. The three-type input generation strategy is plausible, and the ablations in Table 1 support the value of Type 2 and Type 3 tests. The downstream experiments address an important and underexplored question: how verifier quality affects RL and distillation for code LLMs. If the reported gains survive a corrected Codeforces evaluation, the dataset and pipeline would be a useful resource for verifier-based code post-training. The main weaknesses are the biased Codeforces evaluation protocol, the reliance on unverified LLM-generated validators and oracle-agreement heuristics, and the absence of uncertainty quantification.

major comments (3)
  1. [4.3 / Appendix A.4] The Codeforces evaluation protocol is not a valid estimate of population precision/recall. The authors submit only 5% of candidate-program/problem pairs on which TACO, CodeContests, and HardTests agree, and 50% of pairs on which the suites conflict, and then pool the outcomes without inverse-probability weighting. Because agreement/conflict status is correlated with HardTests' verdict (HardTests is designed to be stricter), the pooled sample overrepresents conflicts, which disproportionately contain HardTests negatives and baseline false positives. This can inflate HardTests' measured precision relative to the baselines and distort recall for all suites. Table 2, and the 11.3/17.5 percentage-point headline that averages over Tables 1 and 2, therefore do not currently support the claimed effect size. The authors should either reweight each stratum by the inverse of the sampling probability or report precision and recall separately for the agreement and conflict strata.
  2. [3.2 / 3.3] The correctness of the released HardTests dataset depends on two unverified heuristics. Section 3.2 and the prompt in Appendix A.2.1 explicitly allow the LLM-written input validator to skip constraints that are too complex to verify, and Section 3.3 accepts outputs when two or more oracle programs agree on more than 90% of cases, discarding the remaining cases. If a validator misses a structural constraint or two oracle programs share a common bug, the synthesized tests will contain invalid inputs or wrong expected outputs, which would propagate into downstream RL training and into the precision/recall measurements. The aggregate numbers against official tests suggest the problem is not catastrophic, but the paper should report how often validators skipped constraints, the distribution of oracle agreement rates, and an audit of a random sample of synthesized inputs against the official test constraints for the 32.5k generated problems.
  3. [Tables 1, 2, 5; Figure 3] All quality comparisons are single point estimates with no uncertainty quantification. Tables 1 and 2 report one precision/recall number per cell despite candidate programs being sampled with temperature 0.7 and human submissions being randomly selected, and the RL results in Table 5 and Figure 3 come from one training run per condition. The paper uses the word 'significantly' in Sections 4.4 and 5.2 without statistical tests or repeated seeds. At minimum, bootstrap confidence intervals for the AtCoder and Codeforces tables and multiple independent seeds for the RL and self-distillation comparisons are needed to support the headline claims.
minor comments (5)
  1. [Abstract / 3.4] The abstract says '47k problems and synthetic high-quality tests,' but Section 3.4 states that HardTestGen was applied to only 32.5k of the 47,136 problems; please clarify the counts in the abstract and contribution list.
  2. [Table 1 caption] The caption contains a typo: 'TH-TYPE 1+2' should be 'HT-TYPE 1+2'.
  3. [Figure 1] Please define what 'AtCoder Hard' and 'Codeforces Hard' mean (e.g., difficulty threshold) and add sample sizes or error bars to the bars.
  4. [4.4] The sentence describing the 'intelligence' ordering is confusing: human-written programs are not less intelligent than 7B LLM-generated programs, yet the text says 'as the source of programs becomes less intelligent (ranging from human-written to 7B LLM-generated)'; please reword to match the observed ordering of the results.
  5. [A.7 / Table 8] The oracle-free evaluation in Appendix A.7 is reported only on 165 AtCoder problems with 50 sample solutions per problem, and Table 8 gives FPR/FNR without sample sizes or confidence intervals; please state these limitations in the main text if this result is advertised.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: quality evaluation is anchored to external human-written oracles and a decontaminated held-out benchmark; the Codeforces sampling concern is a measurement-bias issue, not circularity.

full rationale

The paper's central claim is that HardTestGen produces test suites whose precision and recall exceed those of TACO and CodeContests when judged against oracle outcomes, and that training with these suites improves downstream code generation. No circular step is present in this chain. The evaluation is externally grounded: for AtCoder the paper states "we run candidate programs on official tests that have been previously made available," and for Codeforces it states "we submit candidate programs to the website to obtain ground-truth verdicts." These ground-truth labels come from human-written judge systems, not from the synthesized suites or from the pipeline's internal oracle-agreement rule, so the measured precision and recall are not forced by construction. The downstream experiments are also decontaminated: Section 3.4 says "We conduct decontamination by removing the problems that are in LiveCodeBench from our dataset," and Section 5.1 evaluates on LiveCodeBench-105, a held-out benchmark not used to fit any parameter. The only self-citation with author overlap is the appendix's oracle-free method "based on ALGO (Zhang et al., 2023)"; this is not load-bearing because the appendix describes its own generation procedure and reports an independent comparison against AceCoder on 165 AtCoder problems. The reviewer concern about the Codeforces evaluation subsampling agreements at 5% and conflicts at 50% without inverse-probability weighting is a legitimate statistical-validity issue that could bias the headline effect size, but it is not circularity: it does not make any prediction equivalent to its input by definition, and it does not rely on a self-citation chain. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on the trustworthiness of LLM-generated validators, oracle-program agreement, and the completeness of official test labels. None of these is formalized or independently verified in the preprint, so the ledger contains no standard-math axioms and several domain assumptions plus hand-chosen pipeline constants.

free parameters (3)
  • n_D, n_R, n_H test counts = 10, 20, 10 per generator
    Hand-chosen sample sizes in Section 3.2; larger or smaller counts would change how many false positives the suites catch and therefore the measured precision and recall.
  • Oracle agreement threshold = 90%
    Section 3.3 accepts outputs when two oracle programs agree on more than 90% of cases; the threshold determines which test cases are discarded and affects test suite quality.
  • m_R and m_H category and hacking function counts = determined by LLM per problem
    The number of output-category generators and hacking generators is not fixed by a principled rule; the LLM decides, and this affects coverage of hard cases.
assumptions (5)
  • domain assumption Oracle programs y* exist and are correct for almost all coding problems in online competitions.
    Section 3.1 assumes availability of accepted human-written programs; the limitation section acknowledges this may fail in other domains.
  • ad hoc to paper LLM-generated input validators correctly enforce problem constraints, including structural constraints they were allowed to skip.
    Section 3.2 prompt tells the model it can skip constraints too complex to verify; no guarantee that skipped constraints do not affect the validity of generated tests.
  • ad hoc to paper Two oracle programs agreeing on more than 90% of outputs is sufficient evidence that the outputs are correct and the remaining cases can be discarded.
    Section 3.3 sets this threshold without validation against a gold standard; shared bugs or hard cases could be mislabeled.
  • domain assumption Official AtCoder test archive and Codeforces submissions provide complete ground-truth labels for the sampled candidate programs.
    Section 4.3 and Appendix A.4 use a partial AtCoder archive and sampled Codeforces submissions; incomplete or biased ground truth would distort precision and recall.
  • domain assumption The candidate programs sampled from three LLMs and human submissions are representative of programs a verifier will face.
    Section 4.3 uses 10 samples per model per problem; the reported precision and recall may not generalize to other model families or prompt distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HardTests: Synthesizing High-Quality Test Cases for LLM Coding." pith.science (2026). https://pith.science/paper/456F6KSN

@misc{pith2026250524098,
  author       = {Pith},
  title        = {Pith review of: HardTests: Synthesizing High-Quality Test Cases for LLM Coding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/456F6KSN}},
  note         = {Machine review of arXiv:2505.24098}
}
read the original abstract

Verifiers play a crucial role in large language model (LLM) reasoning, needed by post-training techniques such as reinforcement learning. However, reliable verifiers are hard to get for difficult coding problems, because a well-disguised wrong solution may only be detected by carefully human-written edge cases that are difficult to synthesize. To address this issue, we propose HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this pipeline, we curate a comprehensive competitive programming dataset HARDTESTS with 47k problems and synthetic high-quality tests. Compared with existing tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points higher and recall that is 17.5 percentage points higher when evaluating LLM-generated code. For harder problems, the improvement in precision can be as large as 40 points. HARDTESTS also proves to be more effective for model training, measured by downstream code generation performance. We will open-source our dataset and synthesis pipeline at https://leililab.github.io/HardTests/.

Figures

Figures reproduced from arXiv: 2505.24098 by the authors.

Figure 1
Figure 1. HARDTESTS test cases are significantly better than the baselines. The large improvement in precision indicates that our tests greatly reduce false positives and are indeed harder. The low quality of synthetic tests is due to the challenging nature of coding problems. Coding competitions often require efficient solutions with advanced data structures and algorithms. A bad choice of algorithm can lead to a well-disgui… view at source ↗
Figure 2
Figure 2. Comparison of the input generation process between previous test synthesizers (left) and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. RL Validation Rewards Over Time. Reward from H [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The distribution of the number of Type1, Type2, and Type3 test cases, as well as the total [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: The result status distribution of our test case generation pipeline H [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Number of problems from each OJs [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A target-blind agent-generated test suite with a consensus-and-validator certification chain found 906 verified accepted-but-buggy submissions on AtCoder and led all baselines on post-cutoff Codeforces problems.

  2. Efficiency of turbulence

    physics.flu-dyn 2025-08 unverdicted novelty 6.0 of 10

    The efficiency of turbulence, the fraction of input energy stored in the flow, appears bounded and may saturate in a power-law manner across several turbulent flows.

Reference graph

Works this paper leans on

35 extracted references · 6 canonical work pages · cited by 2 Pith papers

  1. [1]

    1 1 0\n1000000000\n1000000000

    - 1 ≤ ai, bi ≤ 109 The first 3 edge test input generators created by ALGO are shown below, corresponding to the following test inputs. Note that the values are at the boundaries of the input bounds and follow clearly defined structures. 1 ["1 1 0\n1000000000\n1000000000", 2 "10 10 1\n1000 2000 3000 4000 5000 6000 7000 8000 9000 10000\n1000 2000 3000 4000 ...

    work page 2000

  2. [3]

    123 * The Python code block under each field should be independent

    122 * Do not wrap the Python code in ```python```, just provide it plainly. 123 * The Python code block under each field should be independent. In other words, they should not call or reference each other. If one block imports a library, other blocks must re-import it as needed. ,→ ,→ 124 * In a Python block, you should first import the necessary librarie...

    work page 2000

  3. [4]

    Caia Costello, Simon Guo, Anna Goldie, and Azalia Mirhoseini

    URL https://arxiv.org/abs/2107.03374. Caia Costello, Simon Guo, Anna Goldie, and Azalia Mirhoseini. Think, prune, train, improve: Scaling reasoning without scaling models. arXiv preprint arXiv: 2504.18116,

    arXiv

  4. [5]

    {n} { m}

    5 m = random.randint(4 * 10**8, 5 * 10**8) 6 return f"{n} { m}" 7 8 def gen_hacking_input_edge_case() -> str: 9 n = random.randint(1, 5 * 10**8) 10 return f"{n} { n}" For this problem, the LLM believes that a string comparison function would be enough for output judging. A.3 Details of the Collection of Problem Specifications and Oracle Programs in HARD T...

    work page 2024

  5. [6]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938,

    arXiv

  6. [7]

    Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning

    Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of-thought of llms via reinforcement learning. arXiv preprint arXiv: 2504.01296,

    arXiv

  7. [8]

    Testgeneval: A real world unit test generation and test completion benchmark, 2025a

    Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. Testgeneval: A real world unit test generation and test completion benchmark, 2025a. URL https://arxiv.org/abs/2410.00752. Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. R2E: Turning any github repository into a programming agent environment. In Ruslan Salakhutdinov, Zico...

    arXiv

  8. [9]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi

    URL https://arxiv.org/abs/2501.12599. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314–21328,

    arXiv

Show all 35 references
  1. [12]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

    arXiv

  2. [13]

    11 Jiawei Liu and Lingming Zhang

    URL https://arxiv.org/abs/2411.05010. 11 Jiawei Liu and Lingming Zhang. Code-r1: Reproducing r1 for code with reliable rewards

    arXiv

  3. [14]

    Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, and Mao Yang

    URL https://arxiv.org/abs/2305.01210. Yifei Liu, Li Lyna Zhang, Yi Zhu, Bingcheng Dong, Xudong Zhou, Ning Shang, Fan Yang, and Mao Yang. rstar-coder: Scaling competitive code reasoning with a large-scale verified dataset,

    arXiv

  4. [15]

    Stephan Lukasczyk and Gordon Fraser

    URL https://arxiv.org/abs/2505.21297. Stephan Lukasczyk and Gordon Fraser. Pynguin: automated unit test generation for python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Com- panion Proceedings , ICSE ’22. ACM, May

    arXiv

  5. [16]

    URL http://dx.doi.org/10.1145/3510454.3516829

    doi: 10.1145/3510454.3516829. URL http://dx.doi.org/10.1145/3510454.3516829. Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A ful...

  6. [17]

    URL https://arxiv.org/abs/2406. 12952. OpenAI. Openai o1 system card. arXiv preprint arXiv: 2412.16720,

    arXiv

  7. [19]

    URL https://arxiv.org/ abs/2502.06807. Guilherme Penedo, Anton Lozhkov, Hynek Kydlí ˇcek, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces. https://huggingface.co/datasets/open-r1/c...

    arXiv

  8. [20]

    URL https://arxiv.org/abs/2502.02827. Z. Z. Ren, Zhihong Shao, Junxiao Song, Huajian Xin, Haocheng Wang, Wanjia Zhao, Liyue Zhang, Zhe Fu, Qihao Zhu, Dejian Yang, Z. F. Wu, Zhibin Gou, Shirong Ma, Hongxuan Tang, Yuxuan Liu, Wenjun Gao, Daya Guo, and Chong Ruan. Deepseek-prover...

    arXiv

  9. [21]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu

    URL https://arxiv.org/abs/2402.03300. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256,

    arXiv

  10. [22]

    Beyond human data: Scaling self-training for problem-solving with language models

    Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585,

    arXiv

  11. [23]

    Testeval: Benchmarking large language models for test case generation, 2025a

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. Testeval: Benchmarking large language models for test case generation, 2025a. URL https://arxiv.org/abs/2406.04531. 12 Yiping Wang, Qing Yang, Zhiyuan Zeng, L...

    arXiv

  12. [24]

    URL https://arxiv.org/abs/ 2503.02951. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Kemi...

    arXiv

  13. [25]

    Limo: Less is more for reasoning

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv: 2502.03387,

    arXiv

  14. [26]

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman

    URL https://arxiv.org/abs/2305.04207. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488,

    arXiv

  15. [27]

    Acecoder: Acing coder rl via automated test-case synthesis, 2025a

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis, 2025a. URL https://arxiv.org/abs/2502. 01718. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl...

    arXiv

  16. [28]

    Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen

    URL https://arxiv.org/abs/ 2305.14591. Quanjun Zhang, Ye Shang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. Testbench: Evaluating class-level test case generation capability of large language models,

    arXiv

  17. [29]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma

    URL https://arxiv.org/abs/2409.17561. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Comp...

    arXiv

  18. [30]

    URL http://arxiv.org/abs/2403.13372

    Association for Computational Linguistics. URL http://arxiv.org/abs/2403.13372. 13 A Appendix A.1 More Related Work on Synthetic Test Quality and its Implications Although existing LLM test synthesis methods prove to be useful in many scenarios, such as improving the quality o...

    arXiv

  19. [31]

    and software engineering(Mündler et al., 2025; Jain et al., 2024), their quality is far from perfect (Yuan et al.,

    2025

  20. [32]

    core logic

    and are bounded in complexity, because direct generations of complicated data structures often result in inconsistency (Zhang et al., 2023). Weak verifiers can harm downstream code generation and search performance (Light et al., 2025). The quality of those synthetic tests and...

    2023

  21. [1000]

    3\n1 10\n2 8\n3 10

    sub-tasks. Each sub-task gives a pair of integers l, r(1 ≤ l ≤ r ≤ 998244353), and the goal is to find a pair of integers x, ysuch that l ≤ x, y≤ r, x ̸= y, and y is divisible by x. It is guaranteed that every sub-task has a valid solution. Note: It can be mathematically prove...

    2000

  22. [2011]

    ISBN 9781450304436

    Association for Computing Machinery. ISBN 9781450304436. doi: 10.1145/2025113.2025179. URL https://doi.org/10.1145/2025113.2025179. 10 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Inc...

  23. [2021]

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen

    URL https://arxiv.org/abs/2108.07732. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In ICLR,

    arXiv

  24. [2022]

    Taco: Topics in algorithmic code generation dataset

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852,

    arXiv

  25. [2023]

    Limr: Less is more for rl scaling

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv: 2502.11886,

    arXiv

  26. [2024]

    org/abs/2412.02883

    URL https://arxiv. org/abs/2412.02883. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models,

    arXiv

  27. [2025]

    Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha

    URL https://arxiv.org/abs/2504.01943. Toufique Ahmed, Martin Hirzel, Rangeet Pan, Avraham Shinnar, and Saurabh Sinha. Tdd-bench verified: Can llms generate tests for issues before they get resolved?,

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.