Pith. sign in

REVIEW 4 major objections 7 minor 6 cited by

rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Independent-solution consensus can verify 380K synthetic code problems, and training on them lifts small models past frontier baselines.

desk verdict rStar-Coder is a serious dataset-focused empirical paper with large, credible gains; the main risk is that the 'verified' label on 380K synthetic problems is a transfer assumption, not a measured fact. read the letter →

arxiv 2505.21297 v1 pith:SNY45MXK submitted 2025-05-27 cs.CL

Yifei Liu , Li Lyna Zhang , Yi Zhu , Bingcheng Dong , Xudong Zhou , Ning Shang , Fan Yang , Mao Yang This is my paper · ORCID
classification cs.CL
keywords codereasoningcompetitiveprogrammingsyntheticdatasettestcasegenerationmutualverificationlongchain-of-thoughtfine-tuningLiveBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that competition-level code reasoning can be scaled up without waiting for more human-written data: a pipeline that synthesizes problem statements from expert seeds, generates test inputs of controlled scale, and labels outputs by majority agreement among independent solutions can produce hundreds of thousands of verified training problems. It then argues that this dataset, not raw model size or sheer solution count, is what drives code reasoning gains: fine-tuning Qwen2.5-Coder models on the 418K problems lifts LiveCodeBench pass@1 from 17.4% to 57.3% at 7B and from 23.3% to 62.5% at 14B, with the 14B model surpassing o3-mini (low) by 3.1%. The practical claim is that strong reasoning LLMs can verify their own synthetic training data through consensus, and small models trained on that data rival much larger systems on hard algorithmic problems.

What carries the argument

The load-bearing object is the mutual-verification loop, powered by three-step input generation. For each problem, a frontier model writes two functions: one that generates random but valid inputs with exposed scale parameters, and one that validates inputs against the problem's stated constraints; scale parameters are swept from small values up to $10^5$, and only validated inputs are kept. Sixteen long-reasoning candidate solutions are executed on at least 50 such inputs, and a problem enters the dataset only if a majority of the candidates return identical outputs on all of them. The oracle solutions of the 37.7K seed problems serve as a second, high-confidence labeling channel for augmented tests and as a small calibration check of the consensus mechanism.

What would settle it

Take a random sample of the 380K synthetic problems, obtain ground-truth outputs from an independent trusted oracle such as a human-written solution or a brute-force verifier, and compare them with the mutual-verification labels; a large drop below the 96.8% seen on 64 seed problems, or one problem family where all 16 candidates converge on the same wrong output, would undercut the central claim.

Watch

Extended reading notes

Core claim

The central discovery is a reliable labeling recipe for synthetic competition problems that have no reference solution. Instead of asking a model to write input-output pairs directly, the method separates input generation from output labeling: it first builds valid, constraint-checked test inputs spanning sizes from roughly 10 up to $10^5$, then samples 16 long-reasoning candidate solutions from a frontier reasoning model and accepts both the solutions and the outputs only when a majority produce identical outputs on every test input. On 64 seed problems where oracle answers exist, this consensus check labels outputs with 96.8% accuracy, versus 12.7% for direct generation. The paper reports that scaling this verified data across 418K problems, of which 380K are synthetic, yields state-of-the-art code reasoning from models as small as 1.5B, including a 7B model that outscores QWQ-32B on USACO 2025.

Load-bearing premise

The pipeline treats majority agreement among independently sampled solutions as proof that a synthetic problem's outputs are correct, so if the candidates share a systematic misreading or make the same wrong assumption, the labels are wrong; the oracle check covers only 64 seed problems, not the 380K synthetic ones.

Editorial extensions

If this is right

  • Fine-tuning on the 580K verified question-solution pairs takes Qwen2.5-Coder-7B from 17.4% to 57.3% and the 14B model from 23.3% to 62.5% on LiveCodeBench, putting the 14B model above o3-mini (low).
  • The same 7B model reaches 16.15% average pass@1 on USACO 2025, beating QWQ-32B's 15.62%, even though QWQ-32B generated many of the training solutions.
  • Dataset breadth is a more efficient scaling axis than solution count: 580K data points built from 480K unique problems outperform 603K data points built from only 37.7K problems with 16 solutions each.
  • Mutual verification is what makes output labeling reliable: it agrees with oracle labels on 96.8% of checked seed-problem outputs, compared with 12.7% when a frontier LLM directly writes input-output pairs.
  • Scale-diverse test inputs improve downstream reasoning: replacing direct LLM-generated inputs with the three-step method raises LiveCodeBench accuracy from 42.9% to 44.6% overall and from 10.6% to 12.6% on hard problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the 96.8% label accuracy transfers from the 64 seed problems to the full 380K synthetic set, consensus labeling could be reused in any domain that has executable checkers and a strong enough candidate generator, without needing a reference solver.
  • Editorial inference: the relaxed agreement threshold for hard Codeforces problems (40% instead of 60%) is a likely source of label noise, so filtering those problems or repeating verification on them could improve dataset quality further.
  • Editorial inference: the fact that a 7B model outperforms the model that generated its training solutions suggests dataset breadth, not teacher strength alone, is doing much of the work, and a controlled comparison with a different teacher at equal breadth would test that.
  • Editorial inference: because the pipeline depends on constraints stated explicitly in problem statements, it probably under-serves problems whose constraints are implied by context, so adding constraint inference to input generation would extend coverage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces rStar-Coder, a pipeline for building a large-scale verified competitive-programming dataset from 37.7K expert-written seed problems with oracle solutions. It uses GPT-4o to synthesize 380K new problems, generates valid test inputs with a three-step method based on CYaRon utility functions, labels outputs through mutual verification of 16 QWQ-32B solutions, and augments seed problems with long reasoning chains. Fine-tuning Qwen2.5-Coder models (1.5B-14B) on the resulting 580K question-solution pairs yields large LiveCodeBench and USACO gains, with the 14B model surpassing o3-mini (low) on LiveCodeBench and the 7B model surpassing QWQ-32B on USACO 2025. The paper also ablates the contribution of each dataset component and the verification mechanism.

Significance. If the verification pipeline is reliable beyond the 64 seed problems on which it is audited, rStar-Coder is a significant contribution: it provides a scalable recipe for producing competition-level training data without oracle solutions, and it demonstrates that small models can reach frontier-level code reasoning performance through such data. The paper's strengths are its well-designed ablations, the large scale of the resulting dataset, the inclusion of decontamination, and the impressive 96.8% mutual-verification accuracy on the oracle-audited subset. The main open risks are whether the verification accuracy transfers to the 380K synthetic problems and whether the evaluation protocol supports the headline claims with statistical confidence.

major comments (4)
  1. [Section 3.2.2, Table 5 Left] The 96.8% mutual-verification accuracy is measured on only 64 expert-written seed problems with oracle solutions, but the dataset's scale comes from 380K synthetic problems generated by GPT-4o, which have no reference solutions. The paper offers no evidence that the mutual-verification accuracy transfers to this distribution; synthetic statements may be ambiguous or subtly different from seeds, and a majority of QWQ-32B solutions could adopt the same misparse or the same wrong algorithmic assumption. The 40% agreement threshold for Codeforces-derived hard problems (Section 3.3) further lowers the bar to 7/16 solutions. I request a direct audit on a sample of synthetic problems, for example by having expert annotators write reference solutions for a stratified sample, or by using an independent verifier (different model family or brute-force oracle for tractable classes) to quantify label accuracy on the synthetic distribution. Without such an audit, the 'verified' label for the bulk of the dataset rests on an unverified distribution-transfer assumption.
  2. [Section 4.1, Tables 2-3] The evaluation protocol for pass@1 is under-specified. The text states that 16 solutions are sampled per problem and the average pass@1 is reported, but it does not define the estimator (e.g., whether a problem counts as solved if at least one of 16 samples passes, or a majority, or an unbiased pass@1 estimator) nor does it report per-problem variance or confidence intervals. This matters for the headline comparisons: LiveCodeBench +3.1% over o3-mini (low) and USACO +0.53% over QWQ-32B are small margins, and USACO 2025 contains only 12 problems (3 per tier), so the difference is less than one problem. I ask the authors to specify the exact pass@1 estimator and to report confidence intervals (e.g., bootstrap CIs over problems) or at least the number of problems solved per tier. This is needed to support the claim that rStar-Coder 'outperforms' frontier models.
  3. [Section 3.3] The paper labels the dataset as 'verified', but the seed-problem augmentation step explicitly retains all 16 generated QWQ-32B solutions for challenging problems where no solution passes all generated tests ('we follow prior work and retain all generated solutions to include more diverse and potentially correct intermediate reasoning steps'). This means a subset of the 580K question-solution pairs is not test-verified. The authors should quantify how many seed problems fall into this category, and ideally report training results with and without these unverified pairs, so that the reader can judge the impact on the claimed verification quality and on the final benchmark numbers.
  4. [Section 3.3] The 16-gram decontamination procedure is weak for detecting semantically similar or lightly paraphrased problems. Because synthetic problems are generated from seed problems taken from the same platforms (Codeforces, AtCoder, LeetCode, etc.) that also appear in LiveCodeBench and other benchmarks, a 16-gram match may be rare even when a problem tests the same algorithm in a similar context. The paper should report the number of problems removed by decontamination, and perform a more robust similarity check (e.g., embedding-based or using problem-solution pairs) to rule out contamination from the seed and synthetic sets into the evaluation benchmarks. This is directly relevant to interpreting the large LiveCodeBench gains as evidence of reasoning improvement rather than memorization or near-duplicate retrieval.
minor comments (7)
  1. [Section 3.1] 'CodeFroces' should be 'Codeforces'.
  2. [Section 4.2] 'OCR-Owen' should be 'OCR-Qwen' to match the model names in Table 2.
  3. [Section 4.3] 'An key component' should be 'A key component'.
  4. [Table 5 Left] 'Dual verification' should be 'Mutual verification' for consistency with the method name used throughout the paper.
  5. [Figure 4] In the test-output grid, 'Test Outputₙ₁' appears to be a mislabel; it should likely be 'Test Outputₘₙ' to correspond to the m-th solution on the n-th input.
  6. [Section 3.1] 'In totoal' should be 'In total'.
  7. [Appendix A.1] The Limitations appendix acknowledges that many generated problems are discarded but does not report the filter statistics; adding per-stage success rates would help readers calibrate the pipeline's cost and yield.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central benchmark results are measured on external, decontaminated tests, and the mutual-verification accuracy is audited against oracle solutions on seed problems.

full rationale

The paper's derivation chain is: curate seed problems with oracle solutions, synthesize new problems from those seeds, generate test inputs with GPT-4o utility functions, label outputs by majority voting among QWQ-32B solutions, fine-tune Qwen2.5-Coder models, and evaluate on LiveCodeBench, USACO 2025, HumanEval, MBPP, and their plus variants. No step in this chain defines a benchmark result in terms of a fitted parameter or a quantity that is itself the target of prediction. The 60%/40% mutual-verification thresholds are hand-set constants, not tuned on the evaluation benchmarks, and the paper does not claim to predict benchmark performance from a fitted model. The mutual-verification mechanism is not definitionally circular: it is independently audited on 64 expert-written seed problems with oracle solutions, where it achieves 96.8% labeling accuracy against ground truth, and this audit is reported as an ablation rather than assumed. The fact that QWQ-32B both proposes and votes on labels for the 380K synthetic problems is a legitimate external-validity concern, but it is a generalization and data-quality risk, not a circular reduction: the paper never equates 'majority agreement' with 'correct by definition' when reporting benchmark gains, and the evaluation benchmarks are external and decontaminated via 16-gram overlap removal. The Limitations appendix explicitly acknowledges reliance on frontier-LLM interpretation of problem statements and the discarding of many generated problems, which further indicates that the authors do not treat majority agreement as a logical guarantee. Self-citations such as rStar-Math are not load-bearing for the central benchmarking claim. Therefore no specific circular step can be exhibited, and the correct finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's empirical pipeline relies on four domain assumptions: majority agreement of LLM-written solutions indicates correctness, LLM-generated validators correctly enforce problem constraints, seed oracle solutions are correct, and 16-gram decontamination is sufficient. None of these is proven, but the first two are partially validated on 64 seed problems. There are no new physical or mathematical entities.

free parameters (4)
  • Mutual verification agreement threshold = 60% (40% for Codeforces rating > 1600)
    Synthetic problems are discarded when fewer than the threshold fraction of 16 QWQ-32B solutions agree on all test outputs. The 60/40 thresholds are chosen by hand in Section 3.3, with no sensitivity analysis.
  • Candidate solution count = 16
    The paper samples 16 long-reasoning solutions per problem for mutual verification and for seed augmentation; the count is fixed without an ablation in Section 3.2.2.
  • Minimum test input count = at least 50
    Each problem is evaluated on a shared set of at least 50 inputs; the exact number and difficulty mix are not specified beyond the scale set in Algorithm 1.
  • Input scale set = {1,...,9} union {10^i for 0 <= i <= e}
    Algorithm 1 fixes the scale set used to instantiate each scale-controlling parameter; the choice of e, typically 5, controls the hardest generated tests.
assumptions (4)
  • domain assumption Majority agreement among independently generated solutions implies the outputs are correct
    Invoked in Section 3.2.2 to label synthetic problem outputs without oracle solutions. The assumption fails if candidate solutions share a systematic bias or if all misinterpret an ambiguous problem statement.
  • domain assumption LLM-generated utility functions correctly encode the problem's input semantics and constraints
    Section 3.2.1 relies on GPT-4o-written generate_test_input and validate_test_input functions. If the validator is incomplete or wrong, invalid inputs can enter the test set and corrupt the verification.
  • domain assumption Seed oracle solutions are correct and authoritative
    Section 3.1 assumes reference solutions from competition platforms are correct when used to label augmented seed test inputs and when used as prompts for synthesis. A wrong oracle would mislabel that problem and bias synthesis.
  • domain assumption 16-gram decontamination is sufficient to prevent evaluation benchmark leakage
    Section 3.3 removes problems with 16-gram overlap with evaluation benchmarks, but semantic or template-level duplicates are not checked, so some style overlap with LiveCodeBench and USACO may remain.

how reviews work

0 comments
Cite this review

Pith. "Pith review of rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset." pith.science (2026). https://pith.science/paper/SNY45MXK

@misc{pith2026250521297,
  author       = {Pith},
  title        = {Pith review of: rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SNY45MXK}},
  note         = {Machine review of arXiv:2505.21297}
}
read the original abstract

Advancing code reasoning in large language models (LLMs) is fundamentally limited by the scarcity of high-difficulty datasets, especially those with verifiable input-output test cases necessary for rigorous solution validation at scale. We introduce rStar-Coder, which significantly improves LLM code reasoning capabilities by constructing a large-scale, verified dataset of 418K competition-level code problems, 580K long-reasoning solutions along with rich test cases of varying difficulty. This is achieved through three core contributions: (1) we curate competitive programming code problems and oracle solutions to synthesize new, solvable problems; (2) we introduce a reliable input-output test case synthesis pipeline that decouples the generation into a three-step input generation method and a mutual verification mechanism for effective output labeling; (3) we augment problems with high-quality, test-case-verified long-reasoning solutions. Extensive experiments on Qwen models (1.5B-14B) across various code reasoning benchmarks demonstrate the superiority of rStar-Coder dataset, achieving leading performance comparable to frontier reasoning LLMs with much smaller model sizes. On LiveCodeBench, rStar-Coder improves Qwen2.5-7B from 17.4% to an impressive 57.3%, and Qwen2.5-14B from 23.3% to 62.5%, surpassing o3-mini (low) by3.1%. On the more challenging USA Computing Olympiad, our 7B model achieves an average pass@1 accuracy of 16.15%, outperforming the frontier-level QWQ-32B. Code and the dataset will be released at https://github.com/microsoft/rStar.

Figures

Figures reproduced from arXiv: 2505.21297 by the authors.

Figure 1
Figure 1. Pass@1 accuracy on code reasoning benchmarks. rStar-Coder consistently delivers [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Examples of standard input-output test case pairs from competitive programming datasets. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example of LLM-generated utility functions for test input generation and validation. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Mutual verification between candidate solutions and test outputs. Given a diverse set of [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: rStar-Coder generates more diverse and larger-scale test inputs, while directly LLM [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Standard input-ouput based problem example [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 8
Figure 8. Figure 8: New code problem synthesis prompt. New Code Problem Synthesis Prompt in rStar-Coder I will provide you with a programming problem along with its solution. Your task is to create a new, transformed programming problem based on the original one. You need to complete the …
Figure 9
Figure 9. Figure 9: Test Input Generation Prompt for Standard Input/Output based Problems [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Ablation study: Prompt for directly generating test input-output pairs with GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Ablation study: Prompt for directly generating test input with GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Embarrassingly Simple Self-Distillation Improves Code Generation

    cs.CL 2026-04 conditional novelty 7.0 of 10

    Simple self-distillation—fine-tuning a code model on its own temperature-sampled, truncated outputs—raises LiveCodeBench pass@1 substantially without verifiers, teachers, or RL.

  2. Toward Training Superintelligent Software Agents through Self-Play SWE-RL

    cs.SE 2025-12 unverdicted novelty 6.0 of 10

    Self-play RL on bug injection and repair in sandboxed repositories yields +10.4 and +7.8 point gains on SWE-bench Verified and Pro while outperforming human-data baselines.

  3. Efficiency of turbulence

    physics.flu-dyn 2025-08 unverdicted novelty 6.0 of 10

    The efficiency of turbulence, the fraction of input energy stored in the flow, appears bounded and may saturate in a power-law manner across several turbulent flows.

  4. HardTests: Synthesizing High-Quality Test Cases for LLM Coding

    cs.CL 2025-05 conditional novelty 6.0 of 10

    HardTestGen generates higher-precision, higher-recall test suites for 47,136 competitive programming problems, improving test precision by 11.3 points and recall by 17.5 points over TACO and CodeContests when judging ...

  5. Hermes 4 Technical Report

    cs.AI 2025-08 conditional novelty 5.0 of 10

    Hermes 4 releases three open-weight reasoning models (14B, 70B, 405B) trained with synthetic data and a length-control SFT stage, evaluated on mathematics, code, knowledge, and alignment benchmarks.

  6. Xolver: Multi-Agent Reasoning with Holistic Experience Learning Just Like an Olympiad Team

    cs.CL 2025-06 conditional novelty 5.0 of 10

    A training-free multi-agent framework with episodic and shared memory reports new best results on GSM8K, AIME 2024/2025, Math-500, and LiveCodeBench.

Reference graph

Works this paper leans on

39 extracted references · 15 canonical work pages · cited by 6 Pith papers

  1. [1]

    Phi-4- reasoning technical report.arXiv preprint arXiv:2504.21318, 2025

    Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4- reasoning technical report.arXiv preprint arXiv:2504.21318, 2025

    arXiv 2025

  2. [2]

    Opencodereasoning: Advancing data distillation for competitive coding.arXiv preprint arXiv:2504.01943, 2025

    Wasi Uddin Ahmad, Sean Narenthiran, Somshubra Majumdar, Aleksander Ficek, Siddhartha Jain, Jocelyn Huang, Vahid Noroozi, and Boris Ginsburg. Opencodereasoning: Advancing data distillation for competitive coding.arXiv preprint arXiv:2504.01943, 2025

    arXiv 2025

  3. [3]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021

    2021

  4. [4]

    Code alpaca: An instruction-following llama model for code generation

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. GitHub repository, 2023

    work page 2023

  5. [5]

    Codet: Code generation with generated tests.arXiv preprint arXiv:2207.10397, 2022

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests.arXiv preprint arXiv:2207.10397, 2022

    arXiv 2022

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    work page 2021

  7. [7]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. 2023

    2023

  8. [8]

    rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking.arXiv preprint arXiv:2501.04519, 2025

    arXiv 2025

Show all 39 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  2. [10]

    Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938, 2021

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938, 2021

    2021 arXiv

  3. [11]

    Enhancing large language models in coding through multi-perspective self-consistency.arXiv preprint arXiv:2309.17272, 2023

    Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. Enhancing large language models in coding through multi-perspective self-consistency.arXiv preprint arXiv:2309.17272, 2023

    2023 arXiv

  4. [12]

    Opencoder: The open cookbook for top-tier code large language models.arXiv preprint arXiv:2411.04905, 2024

    Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models.arXiv preprint arXiv:2411.04905, 2024

    2024 arXiv

  5. [13]

    Key-point-driven data synthesis with its enhancement on mathematical reasoning.arXiv preprint arXiv:2403.02333, 2024

    Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning.arXiv preprint arXiv:2403.02333, 2024

    2024 arXiv

  6. [14]

    Codeforces-python-submissions

    Huggingface. Codeforces-python-submissions

  7. [15]

    Huggingface. Open r1

  8. [16]

    Qwen2.5-coder technical report

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou...

    2024

  9. [17]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  10. [18]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  11. [19]

    Numina- math

    Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numina- math. [https://huggingface.co/AI-MO/NuminaMa...

    2024

  12. [21]

    Taco: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852, 2023

    Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset.arXiv preprint arXiv:2312.14852, 2023. 11

    2023 arXiv

  13. [22]

    Competition-level code generation with alphacode.Science, 378(6624):1092–1097, 2022

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode.Science, 378(6624):1092–1097, 2022

    2022

  14. [23]

    Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

    2023

  15. [24]

    Wizardcoder: Empowering code large language models with evol-instruct

    Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  16. [25]

    Deepcoder: A fully open-source 14b coder at o3-mini level

    Roy Huang Ameen Patel Alpay Ariyak Qingyang Wu Xiaoxiang Shi Rachel Xin Colin Cai Maurice Weber Ce Zhang Li Erran Li Raluca Ada Popa Ion Stoica Michael Luo, Sijun Tan. Deepcoder: A fully open-source 14b coder at o3-mini level. https://pretty-radio-b75.notion.site/ DeepCoder-A-...

  17. [26]

    Open r1: Update 3, 2025

    Guilherme Penedo, Lewis Tunstall, Anton Lozhkov, Hynek Kydlicek, Edward Beeching, Loubna Ben Allal, Quentin Gallouedec, Leandro von Werra, Agustín Piqueres Lajarín, and Nathan Habib. Open r1: Update 3, 2025

    2025

  18. [27]

    Can language models solve olympiad programming?arXiv preprint arXiv:2404.10952, 2024

    Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming?arXiv preprint arXiv:2404.10952, 2024

    2024 arXiv

  19. [28]

    Open Thoughts

    OpenThoughts Team. Open Thoughts. https://open-thoughts.ai, January 2025

    2025

  20. [29]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  21. [30]

    Magicoder: Empow- ering code generation with OSS-instruct

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with OSS-instruct. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 52632–52657. PMLR...

    2024

  22. [31]

    Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025

    Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding, 2025

    2025

  23. [32]

    Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. InProceedings of the 62nd Annual Meeting of the Association for Computat...

    2024

  24. [33]

    Acecoder: Acing coder rl via automated test-case synthesis.arXiv preprint arXiv:2502.01718, 2025

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis.arXiv preprint arXiv:2502.01718, 2025

    2025 arXiv

  25. [34]

    Algo: Synthe- sizing algorithmic programs with generated oracle verifiers.Advances in Neural Information Processing Systems, 36:54769–54784, 2023

    Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthe- sizing algorithmic programs with generated oracle verifiers.Advances in Neural Information Processing Systems, 36:54769–54784, 2023

    2023

  26. [35]

    846903"Output:304689 Example 2:Input:s =

    Xiaojiang Zhang, Jinghui Wang, Zifei Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang, Yinghan Cui, Chao Wang, Junyi Peng, et al. Srpo: A cross-domain imple- mentation of large-scale reinforcement learning on llm.arXiv preprint arXiv:2504.14286, 2025. 12 Table 6: A...

    2025 arXiv

  27. [36]

    Identify the reasoning steps (e.g., Step 1, Step 2, Step 3) and summarize the knowledge points tested in the original problem

    Analyze and understand the original problem and its solution. Identify the reasoning steps (e.g., Step 1, Step 2, Step 3) and summarize the knowledge points tested in the original problem

  28. [37]

    as in the original problem

    Design a new problem that is similar to the original one and can be solved using the same knowledge points. If you reference any conditions or descriptions from the original problem, rewrite them clearly and avoid phrases like "as in the original problem". • Provide two exampl...

  29. [38]

    Parse the constraintson the input from the problem description, such as the range of input data, specific input constraints, etc

  30. [39]

    The function should validate that the parameters fall within the specified constraints

    Write a function generate_test_input using the CYaRon library to randomly generate test inputs based on a specified problem size. The function should validate that the parameters fall within the specified constraints. If any parameter is out of range, the function should retur...

  31. [40]

    t e s t _ i n p u t s

    Write a function validate_test_input to verify whether the generated test input satisfies the requirements specified in the problem description. This includes checking the input data type and constraints parsed in step 1, such as range and other conditions. The function should...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.