Pith. sign in

REVIEW 2 major objections 6 minor 2 cited by

OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OIBench is a contamination-resistant benchmark of 250 original olympiad-level programming problems that separates reasoning models (21.4% average) from conventional instruction-tuned models (3.6%), while still showing an efficiency gap to…

desk verdict OIBench is a genuinely useful benchmark resource, but the paper's two headline claims—contamination resistance and beating 'most' human contestants—are both stronger than the evidence supports. read the letter →

arxiv 2506.10481 v1 pith:R5HAUX3D submitted 2025-06-12 cs.AI

Yaoming Zhu , Junxin Wang , Yiyang Li , Lin Qiu , ZongYu Wang , Jun Xu , Xuezhi Cao , Yuhuai Wei
show 3 more authors
Mingshi Wang Xunliang Cai Rong Ma
This is my paper
classification cs.AI
keywords OIBencholympiadininformaticscodegenerationbenchmarkcontaminationreasoningmodelstest-timescalingalgorithmicefficiencyhuman-modelcomparison
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OIBench is an attempt to build a coding benchmark that cannot be gamed by memorization: 250 original, bilingual, olympiad-level informatics problems written by experienced competition coaches and checked to be absent from the public web before release. The paper's central claim is that, on this private test, models that do extended reasoning before answering solve about 21.4% of problems on average, versus about 3.6% for conventional instruction-tuned models, with the strongest evaluated model reaching 36.35%. It also introduces Time/Space Completion Curves to show how close generated code comes to the canonical solution's time and memory use, and finds that even the best models remain substantially less efficient. On a 44-problem subset, frontier models outperformed the average human contestant while still trailing top human players. If the problems are as original as claimed, the benchmark offers a discrimination signal that saturated pass-rate benchmarks no longer provide.

What carries the argument

The central instrument is the Time/Space Completion Curve, a cumulative distribution that plots the fraction of test cases a model's submissions pass against the maximum time or memory used, expressed as a percentage of the canonical solution's usage on a log scale. This turns a binary pass/fail acceptance rate into a detailed view of algorithmic efficiency. The benchmark itself is the second mechanism: 250 original problems from experienced competition coaches, each with large test cases, a verified C++ canonical solution, bilingual statements, and difficulty labels, checked for absence from the web. The Risk-Score contamination metric, defined as the improvement from adding in-distribution problems to supervised fine-tuning divided by the remaining headroom, is the third mechanism, quantifying how much a leaked training set could inflate scores.

What would settle it

Check the released 250 problems against a large pre-training corpus and public code dumps before release; finding any near-verbatim match would falsify the originality claim. Independently, run the paper's contamination experiment with reinforcement learning instead of supervised fine-tuning, training a strong base model on 100 held-out problems and measuring the solve-rate jump on the remaining 150; a jump far above the reported Risk-Score below 0.01 would falsify the contamination-resistance claim.

Watch

Extended reading notes

Core claim

The paper establishes that the frontier in code generation is not on memorized, easy tasks but on unseen, hard problems where a model must discover the right algorithm and implement it efficiently. Its leaderboard shows a clear separation between reasoning models and conventional models, while the pseudocode variant reveals that giving a solution outline narrows the gap, placing much of the difficulty in problem discovery rather than code writing. The Time/Space Completion Curves show that all models' generated code uses more time and memory than the canonical C++ solutions, even at the same correctness level. The human comparison places current state-of-the-art models above most ACM-level contestants but below the strongest humans, and the paper's contamination experiments report a Risk-Score below 0.01, indicating that supervised fine-tuning on in-distribution problems does not inflate scores.

Load-bearing premise

The central claims rest on the assumption that all 250 problems were genuinely unseen by the evaluated models during training; the support is coach attestation, web-search checks, and a fine-tuning experiment that does not cover reinforcement-learning-phase contamination.

Editorial extensions

If this is right

  • Passing a coding benchmark's acceptance tests no longer guarantees competitive efficiency; reporting completion curves alongside pass rates should become standard.
  • Because pseudocode hints lift scores even for the strongest models, future progress in code reasoning will come as much from discovering the right algorithm as from writing code.
  • The leading model's superiority comes with better reasoning efficiency, solving more problems per inference token, making long-chain-of-thought efficiency a concrete optimization target.
  • Static, controlled human comparisons on a fixed problem set can serve as a reproducible alternative to months-long online Elo evaluation.
  • Open-source models trail closed-source ones, but the strong result of a reasoning-distilled model suggests that distilling chain-of-thought into conventional models transfers some reasoning ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the completion-curve method transfers to any generation task with resource budgets, such as tokens used per correct math answer or memory used per data-science pipeline, as a finer-grained alternative to accuracy-only reporting.
  • Beyond the paper: since the contamination experiment covers supervised fine-tuning but not reinforcement learning, the benchmark's long-term value depends on a refresh protocol; running the Risk-Score experiment with reinforcement learning on held-out problems would be the decisive follow-up.
  • Beyond the paper: the bimodal "peanut-shaped" human-rank distribution suggests current long-chain-of-thought models specialize on some problem types and fail on others; grouping the human-comparison problems by algorithm tag would test whether failures cluster.
  • Beyond the paper: the observation that conventional models do relatively better at very small time budgets hints that they fall back on memorized naive algorithms that are fast only on small test cases, which is testable by comparing their solutions' complexity classes across problem sizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents OIBench, a benchmark of 250 olympiad-level informatics problems collected from ACM-ICPC/OI coaches, with bilingual statements, test cases, canonical C++ solutions, difficulty labels, and a public release that includes model responses and compute costs. The evaluation covers 18 LLMs on four languages in zero-shot settings, reports AC rates on OIBench and a pseudocode-aided variant, introduces Time/Space Completion Curves for efficiency, runs an SFT contamination simulation with a Risk-Score metric, and compares model rankings with human ACM contestants on 44 problems. The main results are that reasoning models (O4-mini-high 36.35% overall) far outperform instruction-tuned models (average 3.6%), closed models outscore open models, all models remain below canonical-solution efficiency, and O4-mini-high ranks highest among evaluated models in the human comparison. The paper also proposes four derived code-understanding tasks (Bugfix, Complete, Translate, Interpret).

Significance. OIBench is a valuable and carefully constructed benchmark artifact. The authors verify canonical solutions by executing them on all test cases, perform search-engine originality checks, document prompts and environments, release model responses and reproduction costs, and propose a more informative efficiency visualization than a single time/space ratio. If the contamination-resistance claim survives, OIBench would fill a real gap between saturated code benchmarks and much harder competition data. The main claims in need of repair are the abstract/conclusion statement about beating 'most' human participants, which Figure 4 contradicts, and the §3.1 Risk-Score experiment, whose design as written cannot distinguish memorization from generalization.

major comments (2)
  1. [Abstract / §6 / Fig. 4] The abstract and conclusion claim that 'current SOTA models already outperform most human participants' and that 'frontier reasoning models surpass the majority of ACM-level competitors.' Figure 4 reports average relative rankings of 42.64% for O4-mini-high, 36.86% for DeepSeek-R1, and 31.47% for O3-mini-high, with all other models below 30%. None of these values exceed 50%, so the data do not support the 'most' or 'majority' wording. Please either soften the claim to 'a substantial fraction' or report a per-problem analysis that would justify a majority interpretation.
  2. [§3.1 / A.3 / Table 2] The Risk-Score experiment does not establish contamination resistance as written. A.3 states that the 100 problems mixed into the SFT data are '100 samples from OIBench,' but §3.1 does not state whether these 100 problems are part of the 250-problem evaluation set. If they are included, a model that memorizes even a fraction of them should raise the overall AC rate far above the 1–4% baselines; for Qwen2.5-72B, memorizing 40 of the 100 problems would add roughly 40/250 = 16 percentage points, implying a Risk-Score near 0.16 rather than the reported 0.01. If they are excluded, the experiment tests generalization from OIBench-like data rather than contamination of the benchmark itself. The paper must clarify the evaluation-set composition, report S_baseline and S_contaminated on the exact evaluation set, and either include the 100 SFT problems in the evaluation or explicitly exclude them and state the consequence. Without this, the central 'contamination-resistant' claim in the abstract and §1 is under-supported, as the authors themselves partly acknowledge in A.1 regarding RL-stage contamination.
minor comments (6)
  1. [§4] The sentence 'By default, we give all the information' is incomplete; finish it or delete it.
  2. [Table 4] The difficulty comparison mixes pass@1 results (HumanEval, OIBench) with pass@8 results (CodeElo) and results from different sources; state the metrics explicitly and consider reporting a single metric for comparability.
  3. [§4.2] The paper says R1 converts canonical solutions into pseudocode, but it does not describe how the pseudocode was validated to be language-agnostic and faithful to the intended algorithm; please add a verification step or at least a manual-check description.
  4. [§5.1 / Fig. 4] The violin plots do not specify how many human participants attempted each of the 44 problems or how ties in the IOI-style ranking were broken; adding per-problem participant counts and tie-breaking rules would improve reproducibility.
  5. [Table 1] The column 'avg. size of test per prob' does not state the unit; specify whether it is bytes, number of test cases, or something else, since OIBench's value of 37.5M is orders of magnitude larger than the others.
  6. [Throughout] There are numerous typos, including 'Learderboards' in the §4 heading, 'fine-grand' in §4.3, 'inprovment' in §4.3, 'serveral' and 'erros' in §5.2, 'elaborted' in the baseline-model description, 'Univerisity' in the affiliation, and 'Comparsion'/'resutles' in A.2; a light copyedit is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OIBench's claims are empirical measurements on a newly constructed dataset, and no derived quantity reduces to its inputs by construction.

full rationale

The paper's central claims are empirical: the benchmark's difficulty and discrimination power are measured by running frontier models on the problem set (Table 3), and the efficiency findings are read off the measured Time/Space Completion Curves, which the paper itself describes as 'essentially the cumulative distribution of solving time/space' (Section 4.3). No result is defined into existence: the pseudocode condition is explicitly an intentional hint ('the solution hints greatly reduce the difficulty that originally relied on complex reasoning'), and it is used as a separate evaluation mode, not as evidence for correctness. The Risk-Score anti-contamination experiment in Section 3.1 is a leakage-sensitivity check whose reported values (Table 2) are empirical outcomes, not quantities forced by the definition of the metric; a circular 'fit renamed as prediction' structure is absent. The paper does honestly disclose its evidentiary limits in A.1: 'the dataset may still carry potential risks of internet leakage,' 'we cannot completely ensure the absolute originality of the coaching questions,' and 'Our anti-contamination experiments excluded reinforcement learning (RL) training components.' The skeptic's concern that Section 3.1 does not clearly state whether the 100 SFT problems are part of the evaluation set is a validity gap in the contamination-resistance argument, not a circularity: the conclusion could be under-supported, but it is not identical to an input, a fitted parameter, or a self-citation chain. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via prior work. Accordingly, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper relies on trust in coaches for problem originality, on verified canonical solutions, on a limited SFT-based anti-contamination experiment, on voluntary ACM participants for human comparison, and on model-generated pseudocode; none of these are formal guarantees, so they are recorded as domain assumptions.

assumptions (5)
  • domain assumption Problem originality: each of the 250 problems is original and unpublished, attested by coaches and checked via search engines.
    Section 3 and A.1 state this; search-engine verification cannot prove absence from private repositories or model pre-training corpora.
  • domain assumption Canonical solutions and test cases are correct and comprehensive.
    Section 3 requires canonical solutions to pass all test cases; correctness of test cases is verified by execution, but there is no independent proof of solution optimality.
  • domain assumption Anti-contamination experiment is representative: SFT on 100 held-out problems approximates real contamination.
    Section 3.1; paper acknowledges in A.1 that RL-based contamination is not tested.
  • domain assumption Human participants are representative ACM-level contestants under controlled conditions.
    Section 5.1; participants are volunteers from CCPC student teams, and the last-submission rule may not reflect contest behavior.
  • domain assumption R1-generated pseudocode faithfully represents the logic of canonical solutions.
    Section 4.2; pseudocode is produced by a model, not manually verified for every problem.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics." pith.science (2026). https://pith.science/paper/R5HAUX3D

@misc{pith2026250610481,
  author       = {Pith},
  title        = {Pith review of: OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R5HAUX3D}},
  note         = {Machine review of arXiv:2506.10481}
}
read the original abstract

As models become increasingly sophisticated, conventional algorithm benchmarks are increasingly saturated, underscoring the need for more challenging benchmarks to guide future improvements in algorithmic reasoning. This paper introduces OIBench, a high-quality, private, and challenging olympiad-level informatics dataset comprising 250 carefully curated original problems. We detail the construction methodology of the benchmark, ensuring a comprehensive assessment across various programming paradigms and complexities, and we demonstrate its contamination-resistant properties via experiments. We propose Time/Space Completion Curves for finer-grained efficiency analysis and enable direct human-model comparisons through high-level participant evaluations. Our experiments reveal that while open-source models lag behind closed-source counterparts, current SOTA models already outperform most human participants in both correctness and efficiency, while still being suboptimal compared to the canonical solutions. By releasing OIBench as a fully open-source resource (https://huggingface.co/datasets/AGI-Eval/OIBench), we hope this benchmark will contribute to advancing code reasoning capabilities for future LLMs.

Figures

Figures reproduced from arXiv: 2506.10481 by the authors.

Figure 1
Figure 1. Example problem. 3.1 Anti-Contamination A challenge for benchmark design is the in-distribution contamination [34] problem, as modern LLMs typically scrape Internet data for pre-training and fine-tuning, potentially including contents with the same domain of the benchmark (e.g. the training set of GSM8k). The contamination not only renders leaked problems ineffective as test cases, but also artificially inflates per… view at source ↗
Figure 2
Figure 2. Time/Space Completion Curves. Note that in the figure, we employ the test case pass rate [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Inference budgets measured by the number of response tokens vs pass rate. The size of the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Violin plots of model’s relative ranking among human participants. The number indicates [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Time/Space Completion Curves A.8 Error Type Attribution The error type are shown in [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: The error type for each model. A.9 Participant Recruitment and Compensation For the six Informatics Olympiad participants, they are all undergraduate students from Chinese universities with a computer science background, and have achieved at least a silver medal in the…

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

    cs.CL 2025-08 conditional novelty 6.0 of 10

    AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.

  2. UniCode: Augmenting Evaluation for Code Reasoning

    cs.SE 2025-10 conditional novelty 5.0 of 10

    UniCode's LLM-generated coding benchmark drops top-model pass@1 to 70.3% and indicates current LLMs rely on memorized seed logic instead of generalizing to new algorithmic problems.

Reference graph

Works this paper leans on

61 extracted references · 23 canonical work pages · cited by 2 Pith papers

  1. [1]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  2. [2]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  3. [3]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Seyed-Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. CoRR, abs/2410.05229, 2024

    arXiv

  4. [4]

    On memorization of large language models in logical reasoning

    Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. CoRR, abs/2410.23123, 2024

    arXiv 2024

  5. [5]

    Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pu...

    arXiv 2022

  6. [6]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. CoRR, abs/2501.01257, 2025

    arXiv 2025

  7. [7]

    Can language models solve olympiad programming? CoRR, abs/2404.10952, 2024

    Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? CoRR, abs/2404.10952, 2024

    arXiv 2024

  8. [8]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. CoRR, abs/2403.07974, 2024

    arXiv 2024

Show all 61 references
  1. [9]

    Effibench: Bench- marking the efficiency of automatically generated code

    Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie Zhang. Effibench: Bench- marking the efficiency of automatically generated code. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, edi- tors, Advances in ...

    2024

  2. [10]

    A performance study of llm-generated code on leetcode

    Tristan Coignion, Clément Quinton, and Romain Rouvoy. A performance study of llm-generated code on leetcode. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024, Salerno, Italy, June 18-21, 2024, pages 79–89. ACM, 2024

    2024

  3. [12]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  4. [13]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  5. [14]

    Towards reasoning in large language models: A survey

    Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049– 1065, 2023

    2023

  6. [15]

    Towards large reasoning models: A survey of reinforced reasoning with large language models

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arXiv:2501.09686, 2025

    2025 arXiv

  7. [16]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021

    arXiv

  8. [17]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing...

    2021

  9. [18]

    Cohen, Ruslan Salakhut- dinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsu- jii, editor...

    2018

  10. [19]

    Logicbench: Towards systematic evaluation of logical reasoning ability of large language models

    Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar, edito...

    2024

  11. [20]

    Criticbench: Benchmarking llms for critique-correct reasoning

    Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang. Criticbench: Benchmarking llms for critique-correct reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, T...

    2024

  12. [21]

    Inference-time computations for LLM reasoning and planning: A benchmark and insights

    Shubham Parashar, Blake Olson, Sambhav Khurana, Eric Li, Hongyi Ling, James Caverlee, and Shuiwang Ji. Inference-time computations for LLM reasoning and planning: A benchmark and insights. CoRR, abs/2502.12521, 2025

    2025 arXiv

  13. [22]

    Logicvista: Multimodal LLM logical reasoning benchmark in visual contexts

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal LLM logical reasoning benchmark in visual contexts. CoRR, abs/2407.04973, 2024

    2024 arXiv

  14. [23]

    Test of time: A benchmark for evaluating llms on temporal reasoning

    Bahare Fatemi, Mehran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinyeong Yim, John Palowitch, Sungyong Seo, Jonathan Halcrow, and Bryan Perozzi. Test of time: A benchmark for evaluating llms on temporal reasoning. CoRR, abs/2406.09170, 2024

    2024 arXiv

  15. [24]

    Logicgame: Benchmarking rule-based reasoning abilities of large language models

    Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. Logicgame: Benchmarking rule-based reasoning abilities of large language models. CoRR, abs/2408.15778, 2024. 17

    2024 arXiv

  16. [25]

    Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data

    Xiao Liu, Zirui Wu, Xueqing Wu, Pan Lu, Kai-Wei Chang, and Yansong Feng. Are llms capable of data-based statistical and causal reasoning? benchmarking advanced quantitative reasoning with data. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Associat...

    2024

  17. [26]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models

    Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianxiang Wang, and Tao Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software ...

    2024

  18. [27]

    Beyond correctness: Benchmarking multi-dimensional code generation for large language models

    Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yaojie Lu, Xianpei Han, and Le Sun. Beyond correctness: Benchmarking multi-dimensional code generation for large language models. CoRR, abs/2407.11470, 2024

    2024 arXiv

  19. [28]

    Codescore: Evaluating code generation by learning code execution

    Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin. Codescore: Evaluating code generation by learning code execution. CoRR, abs/2301.09043, 2023

    2023 arXiv

  20. [29]

    Cruxeval: A benchmark for code reasoning, understanding and execution

    Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. Cruxeval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 202...

    2024

  21. [30]

    Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yaofeng Sun, Yuan Liu, Thiago S. F. X. Teixeira, Diyi Yang, Ke Wang, and Alex Aiken. Equibench: Benchmarking code reasoning capabilities of large language models via equivalence checking. CoRR, abs/2502.1...

    2025

  22. [31]

    Isolat- ing language-coding from problem-solving: Benchmarking llms with pseudoeval

    Jiarong Wu, Songqiang Chen, Jialun Cao, Hau Ching Lo, and Shing-Chi Cheung. Isolat- ing language-coding from problem-solving: Benchmarking llms with pseudoeval. CoRR, abs/2502.19149, 2025

    2025 arXiv

  23. [32]

    Is your benchmark (still) useful? dynamic benchmarking for code language models

    Batu Guan, Xiao Wu, Yuanyuan Yuan, and Shaohua Li. Is your benchmark (still) useful? dynamic benchmarking for code language models. CoRR, abs/2503.06643, 2025

    2025 arXiv

  24. [33]

    Dynamic benchmarking of reasoning ca- pabilities in code large language models under data contamination

    Simin Chen, Pranav Pusarla, and Baishakhi Ray. Dynamic benchmarking of reasoning ca- pabilities in code large language models under data contamination. CoRR, abs/2503.04149, 2025

    2025 arXiv

  25. [34]

    DICE: detecting in- distribution contamination in llm’s fine-tuning phase for math reasoning.CoRR, abs/2406.04197, 2024

    Shangqing Tu, Kejian Zhu, Yushi Bai, Zijun Yao, Lei Hou, and Juanzi Li. DICE: detecting in- distribution contamination in llm’s fine-tuning phase for math reasoning.CoRR, abs/2406.04197, 2024

    2024 arXiv

  26. [35]

    Skywork: A more open bilingual foundation model

    Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanq...

    arXiv

  27. [36]

    Benchmarking benchmark leakage in large language models

    Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. CoRR, abs/2404.18824, 2024

    2024 arXiv

  28. [37]

    Investigating data contamination in modern benchmarks for large language models

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, He- lena Gómez-Adorno, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American ...

    2024

  29. [38]

    Tahar Kechadi

    Cheng Xu, Shuhao Guan, Derek Greene, and M. Tahar Kechadi. Benchmark data contamination of large language models: A survey. CoRR, abs/2406.04244, 2024

    2024 arXiv

  30. [39]

    Docker: lightweight linux containers for consistent development and deployment

    Dirk Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux J., 2014(239), March 2014

    2014

  31. [40]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  32. [41]

    Adding error bars to evals: A statistical approach to language model evaluations

    Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint arXiv:2411.00640, 2024

    2024 arXiv

  33. [42]

    Nolazco-Flores, Lori Landay, Matthew Thomas Jackson, Paul Röttger, Philip H

    Francisco Eiras, Aleksandar Petrov, Bertie Vidgen, Christian Schröder de Witt, Fabio Pizzati, Katherine Elkins, Supratik Mukhopadhyay, Adel Bibi, Botos Csaba, Fabro Steibel, Fazl Barez, Genevieve Smith, Gianluca Guadagni, Jon Chun, Jordi Cabot, Joseph Marvin Imperial, Juan A. ...

    2024

  34. [43]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  35. [44]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  36. [45]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  37. [46]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  38. [47]

    doubao-pro-32k

    Doubao Team. doubao-pro-32k. https://www.volcengine.com/docs/82379/1554680,

  39. [48]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  40. [49]

    claude 3.5 sonnet

    claude Team. claude 3.5 sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet, 2025. Accessed: 2025-05-10

    2025

  41. [50]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. 19

    2025

  42. [51]

    OpenAI Team. O3-mini. https://openai.com/index/openai-o3-mini/ , 2025. Ac- cessed: 2025-05-10

    2025

  43. [52]

    OpenAI Team. O4-mini. https://openai.com/index/introducing-o3-and-o4-mini/ ,

  44. [53]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024

    2024 arXiv

  45. [54]

    Accessed: 2025-05-10

    2025

  46. [55]

    Chao Lei, Yanchuan Chang, Nir Lipovetzky, and Krista A. Ehinger. Planning-driven program- ming: A large language model programming workflow. CoRR, abs/2411.14503, 2024

    2024 arXiv

  47. [56]

    Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution

    Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I Wang. Swe-rl: Advancing llm reasoning via reinforcement learning on open software evolution. arXiv preprint arXiv:2502.18449, 2025

    2025 arXiv

  48. [57]

    Mdeval: Massively multilingual code debugging

    Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, Hualei Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yunlong Duan, Yu Hao, Liqun Yang, Guanglin Niu, Ge Zhang, and Zhoujun Li. Mdeval: Massively multilingual code debugging. CoRR, abs/2411.02310, 2024

    2024 arXiv

  49. [58]

    Megatron-lm: Training multi-billion parameter language models using model parallelism

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019

    1909 arXiv

  50. [59]

    Codetransocean: A comprehensive multilingual benchmark for code translation

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. Codetransocean: A comprehensive multilingual benchmark for code translation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, ...

    2023

  51. [60]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and S...

    2021

  52. [62]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems...

    2023

  53. [2025]

    Accessed: 2025-01-22

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.