Pith. sign in

REVIEW 5 major objections 6 minor 7 cited by

LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Frontier coding models solve none of the hardest live contest problems, and their top Elo scores come from tools and repeated tries, not superior reasoning, according to an Olympiad-medalist-judged benchmark.

desk verdict A genuinely useful benchmark with a plausible core message, but the 'contamination-free' and 'tool-driven performance' headline claims are stronger than the evidence. read the letter →

arxiv 2506.11928 v1 pith:C7KH3B52 submitted 2025-06-13 cs.SE cs.AIcs.CLcs.LG

Zihan Zheng , Zerui Cheng , Zeyu Shen , Shang Zhou , Kaiyuan Liu , Hansen He , Dongruixuan Li , Stanley Wei
show 11 more authors
Hangyi Hao Jianzhu Yao Peiyao Sheng Zixuan Wang Wenhao Chai Aleksandra Korolova Peter Henderson Sanjeev Arora Pramod Viswanath Jingbo Shang Saining Xie
This is my paper
classification cs.SEcs.AIcs.CLcs.LG
keywords LiveCodeBenchProcompetitiveprogrammingLLMevaluationdatacontaminationEloratingalgorithmicreasoningcognitive-focustaxonomyCodeforces
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that reports of large language models surpassing elite human competitors are premature, and supports that claim with a new benchmark, LiveCodeBench Pro, of 584 problems from Codeforces, ICPC, and IOI captured at the moment of contest release so that no solutions or editorials are yet online. Olympiad medalists annotate every problem by algorithmic topic and by cognitive focus—knowledge-heavy, logic-heavy, or observation-heavy—and perform line-by-line triage of failed submissions. Without external tools, the best model achieves 53% pass@1 on medium problems and 0% on hard problems, and model ratings collapse on observation-heavy tasks. The paper concludes that top reported Elo numbers are largely driven by implementation precision, multiple attempts, and tool augmentation rather than by reasoning at grandmaster level.

What carries the argument

The load-bearing object is the benchmark itself: a set of 584 problems harvested in real time from premier contests, before accepted solutions, editorials, or discussions exist online, which makes contamination much harder than in static archives. The argument is carried by two analytic instruments bolted onto that corpus: a cognitive-focus taxonomy (knowledge-heavy, logic-heavy, observation-heavy) in which every problem is annotated through a triple-blind process by Olympiad medalists, and a Bayesian maximum-a-posteriori Elo estimation that converts each model's accept/reject outcomes into a rating on the same scale as human contestants. The taxonomy does the explanatory work—it is what shows that performance collapses specifically when a problem demands a novel observation—and the failure triage of 125 o3-mini submissions against 125 human submissions, labeled by verdict and root cause, is what shows that model errors are conceptual rather than implementational.

What would settle it

Take the same benchmark's hard problems and check whether any evaluated model's training data actually contains them, for instance by asking the model to auto-complete a hard problem's statement from its first sentence or to reproduce an official solution nearly verbatim; if supposedly unseen hard problems get solved, or if pass rates drop sharply when hard problems are paraphrased, then the 0% result conflates memorization with reasoning. A cleaner temporal probe is to evaluate the same models on hard problems created after all training cutoffs and see whether any model scores above 0%.

Watch

Extended reading notes

Core claim

The central discovery is that frontier LLMs divide cleanly by problem type: they approach human expert levels on knowledge-heavy and logic-heavy problems, where success means applying known templates and derivations, and they fail hard on observation-heavy problems, where success requires a novel insight. The headline numbers are that the strongest evaluated model, o4-mini-high, solves 0% of problems rated above 3000 Elo and only 53% of the medium tier without tools or extra attempts, and that o3-mini's failed submissions show 34 more algorithmic-logic errors and 25 fewer implementation errors than matched human contestants in a 125-problem comparison. From this the authors conclude that aggregate leaderboard performance overstates native reasoning ability: the gap between their tool-free, single-attempt rating of about 2116 and the reported 2719 with tools and multiple attempts is attributed to tool augmentation and rerolling, not to stronger reasoning.

Load-bearing premise

The load-bearing premise is that grabbing problems the moment they appear in live contests keeps them out of every evaluated model's training data, so the 0% score on hard problems reflects reasoning limits rather than memorized answers.

Editorial extensions

If this is right

  • Reported leaderboard Elo numbers for frontier models should be decomposed into a tool-free pass@1 component and a tool-augmented pass@k component; the paper's numbers put that gap at roughly 400 to 600 rating points.
  • Evaluation and training should shift toward observation-heavy problems and corner-case reasoning, since those are where models fail even when given multiple attempts.
  • Test-time reasoning (chain-of-thought) produces its largest gains in combinatorics and knowledge-heavy categories and nearly none in game theory, greedy, constructive, and ad-hoc tasks.
  • The 0% hard-tier score means that claims of benchmark saturation are an artifact of test selection: implementation-heavy problems are nearly exhausted, while problems requiring novel insight remain unsolved.
  • Interactive problems need separate treatment in benchmarks because models can exhibit reward-hacking behavior, exploiting a judge's hack mode rather than solving the task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper leaves implicit is to use the cognitive-focus taxonomy prospectively: generate new hard problems in each category and track whether future models close the observation-heavy gap or remain stuck, with the paper's data predicting the latter.
  • The contamination-control assumption could be validated directly by running the same benchmark on hard problems authored after all model cutoffs; a non-zero hard pass rate there would suggest the 0% result is partly a vintage effect rather than a pure reasoning limit.
  • If tool access becomes the default deployment mode, the practical human-model gap may close even while the native-reasoning gap persists; separating the two, as this paper does, makes that split visible and worth tracking independently.
  • The large 'fails sample' surplus for tool-free models implies that a cheap, mechanical source of error is being removed in deployed settings, so tool-free API evaluations likely understate what the same model achieves with local compilation and sample checking.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. LiveCodeBench Pro introduces a continuously collected benchmark of 584 competitive-programming problems from Codeforces, ICPC, IOI, and related contests, captured at contest release, with expert medalist annotations of topic tags and cognitive focus (knowledge-heavy, logic-heavy, observation-heavy). The paper evaluates 21 models, reports pass@1 by difficulty tier and Codeforces-equivalent Elo ratings, analyzes tag-wise strengths and weaknesses, compares failure modes of o3-mini versus human contestants on 125 rejected submissions each, studies pass@k effects for o4-mini variants, and attributes part of the gap between its measured o4-mini-high rating and OpenAI's reported Codeforces rating to tool use. The headline claims are that, without tools, the best model achieves 53.5% pass@1 on medium-difficulty problems and 0% on hard problems, and that high performance is driven by implementation precision and tool augmentation rather than superior reasoning.

Significance. If the findings hold, the benchmark is a useful diagnostic instrument that challenges vendor reports claiming frontier models outperform elite humans in competitive programming. The paper deserves credit for its live problem-capture protocol, expert annotation of the full corpus, open leaderboard and evaluation code, and line-by-line failure triage of human versus model submissions. The direct measurements are plausible and internally consistent. However, the headline conclusions rest on an unverified contamination-freeness assumption, on an unreported hard-tier sample size, and on a tool-augmentation comparison against a vendor-reported number rather than a controlled experiment. These gaps are load-bearing for the abstract's strongest claims, so the paper needs revision before the central conclusions can be accepted.

major comments (5)
  1. [A3.1 and Section 4] Section A3.1 states that capturing problems at release 'significantly mitigates' contamination risk, but Section 4 calls the benchmark 'contamination-free.' The capture protocol ensures only that no accepted solutions, editorials, or discussion threads existed at the moment of capture; it does not prevent post-contest editorials and solutions from entering the training data of models whose knowledge cutoffs fall after a problem's release. Table A1 lists cutoffs such as January 2025 for Gemini 2.5 Pro and May 2024 for o4-mini-high, while the corpus spans problems released up to April 2025 and also includes older problems. The paper provides no canary or leakage probe, no per-model filtering by problem release date relative to cutoff, and no analysis of performance as a function of release-date distance. Because the hard-tier headline is a 0% rate over an unreported number of problems, even one or two leaked hard problems would change the headline. Please add per-model release-date filtering or a contamination probe and rerun the headline numbers on the filtered subset.
  2. [Table 1 and Section 3.3] Table 1 reports 0.0% hard pass@1 for every model, and Section 3.3 states that 'all variants show 0% pass rate on the hard tier,' but the number of hard-tier problems is never given. With 584 total problems and tier thresholds defined in Section 2, the hard subset is likely small, and the uncertainty of a 0/k result depends entirely on k. Please report the number of problems per difficulty tier and binomial confidence intervals for each model-tier cell, and avoid unqualified 0% claims without those counts.
  3. [A3.2] The Bayesian Elo estimation in A3.2 defines a Gaussian prior N(mu, sigma^2) and a MAP objective, but the numerical values of mu and sigma are not reported anywhere. Table 1 and Figures 2, 5, and 6 all depend on these ratings, and tag-wise ratings are particularly sensitive to prior strength when per-category sample sizes are small. Please state the prior parameters and provide a sensitivity analysis, or at least report the per-tag and per-tier sample sizes used in the rating computation.
  4. [Section 3.3 and Appendix A7] The claim that tool augmentation is a primary driver of high performance is not directly tested. Section 3.3 compares the measured o4-mini-high pass@1 rating of 2116 (without tools) against OpenAI's reported rating of 2719 obtained with terminal access and pass@k, subtracts the pass@k gains, and then conjectures that the remaining approximately 400 points are due to tool calls. Appendix A7 lists plausible mechanisms but provides no controlled experiment with tools enabled on LiveCodeBench Pro. Please run the same model with tools on this benchmark, or soften the abstract's causal claim to a hypothesis that remains to be tested.
  5. [Section 3.2 and Appendix A9] The line-by-line failure analysis covers only o3-mini and human participants, as acknowledged in Appendix A9. The abstract's claim that high performance is 'largely driven by implementation precision and tool augmentation, not superior reasoning' and the general statement that LLMs 'struggle with nuanced algorithmic reasoning' are generalized from one model's failure pattern plus 'preliminary manual checks' in Section 3.2. Please either extend the diagnosis to at least one additional frontier model or explicitly restrict the conclusion to o3-mini.
minor comments (6)
  1. [Appendix A8] The title and text of Appendix A8 repeatedly refer to 'o3-mini-high' when the model being discussed is o3-mini; please standardize the naming to match the rest of the paper.
  2. [Appendix A11] The Graph Theory example statement contains the typo 'such taht', and the Game Theory input line 'The second line contains n integers a1, a n, . . .an' mixes notation; these should be cleaned up.
  3. [Section 2 and Appendix A3.3] Section 2 says the triple-blind adjudication process is described in Section A3.2, but the actual description appears in Appendix A3.3; the cross-reference is incorrect.
  4. [Figure 1] The caption refers to a gray region that zooms in on the non-reasoning models, but no gray region is visible in the rendered figure; please clarify or adjust the figure.
  5. [Table A1] The row for GPT-4.5 lists 'Llama 3.1' under the license column, which appears to be a formatting error; please correct the entry.
  6. [Section 3.3] The footnote states that o4-mini-high evaluation was limited to pass@3, but the surrounding text refers to a 'converged rating' as though pass@10 results were available for that model; please clarify this point.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: headline pass@1 results are direct measurements, and the only overlapping-author citation (Bayesian Elo) is non-load-bearing.

full rationale

LiveCodeBench Pro is an empirical measurement paper, not a derivation, so most circularity modes do not apply. The headline claims (53% pass@1 on medium-difficulty problems, 0% on hard problems without tools) are direct pass/fail counts on a fixed problem set, not outputs of a fitted model or a self-cited theorem. The only overlapping-author citation is [55], used for the Bayesian MAP Elo rating in Appendix A3.2; the appendix fully states the Gaussian prior, logistic probability model, posterior, and curvature-based standard error, so the citation is an attribution for a standard method and is not load-bearing for the central results. The Elo ratings are estimates from the observed accept/reject data and are presented as such (Table 1, Section A3.2), not as independent predictions, so there is no fitted-input-called-prediction step. The cognitive-focus findings rest on expert annotations and measured pass rates; the taxonomy is defined independently of model outputs. Section A3.1's 'contamination-free' language is an empirical assumption about data leakage rather than a circular derivation; it is a validity risk (especially given model cutoffs in Table A1), but it does not make any stated result equivalent to its inputs by construction. No circular step was found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on three categories of support: the Bayesian Elo calibration, the contamination-free collection claim, and the expert annotation labels. The Elo calibration requires assuming Codeforces difficulty ratings behave like a logistic Elo model and the prior is well chosen; prior parameters are unreported. The contamination-free claim is an unverified assumption because model training cutoffs overlap the corpus. The expert taxonomy is asserted as ground truth without inter-annotator agreement metrics. No physical entities are introduced.

free parameters (2)
  • Gaussian prior mean and variance for Bayesian Elo = not reported (mu_0, sigma_0)
    A3.2 defines a Gaussian prior r ~ N(mu, sigma^2) and log-posterior with (r-mu0)^2/(2 sigma0^2) but never gives numeric values; these shift the MAP ratings and percentiles in Table 1.
  • Difficulty tier thresholds = Easy <= 2000; Medium (2000, 3000]; Hard > 3000
    The headline '0% on hard' depends on the authors' chosen Elo cutoffs in Section 2. With different thresholds the hard-tier pass rate would change.
assumptions (4)
  • domain assumption Codeforces official problem rating d satisfies P(solve | rating r) = 1/(1+10^((d-r)/400))
    Used in A3.2 to compute model Elo. Codeforces derives ratings from a more complex system; treating its difficulty as a direct Elo solve probability is an approximation that the paper does not test.
  • domain assumption Real-time collection before accepted solutions or editorials appear prevents any evaluated model from having seen benchmark problems
    Stated in Section 2 and A3.1. It is not verified per model; Table A1 lists cutoffs like January 2025 while the corpus includes problems from before that date, so training leakage is not ruled out.
  • domain assumption Expert annotations and triple-blind adjudication produce ground-truth cognitive-focus labels
    Section A3.3 describes manual re-tagging and cross-validation, but inter-annotator agreement is not quantified, and the labels drive the main diagnostic findings.
  • domain assumption The 125 failed o3-mini submissions are representative of all evaluated LLMs
    Section 3.2 only fully annotates o3-mini; generalization to other models is asserted from preliminary checks and acknowledged as a limitation in A9.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?." pith.science (2026). https://pith.science/paper/C7KH3B52

@misc{pith2026250611928,
  author       = {Pith},
  title        = {Pith review of: LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C7KH3B52}},
  note         = {Machine review of arXiv:2506.11928}
}
read the original abstract

Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algorithmic contests, we revisit this claim, examining how LLMs differ from human experts and where limitations still remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination. A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions. Using this new data and benchmark, we find that frontier models still have significant limitations: without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems, domains where expert humans still excel. We also find that LLMs succeed at implementation-heavy problems but struggle with nuanced algorithmic reasoning and complex case analysis, often generating confidently incorrect justifications. High performance appears largely driven by implementation precision and tool augmentation, not superior reasoning. LiveCodeBench Pro thus highlights the significant gap to human grandmaster levels, while offering fine-grained diagnostics to steer future improvements in code-centric LLM reasoning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Fine-tuning LLMs on an unseen language teaches syntax but fails to transfer semantic competence, leaving Python with up to a 19% performance advantage and no tested intervention closing the gap.

  2. GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks

    cs.AI 2026-08 conditional novelty 6.0 of 10

    GABench provides the first agentic graph-analysis benchmark with 10,400 executable tasks, and finds existing LLM agents succeed on under 40% of complex graph tasks.

  3. PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A profiler-guided, verifier-in-the-loop workflow more than doubles the rate at which an off-the-shelf LLM agent matches human-expert speedups on two repository-level code-optimization benchmarks.

  4. SWE-IF: Aligning Code Evaluation with Human Preference

    cs.CL 2025-10 conditional novelty 6.0 of 10

    A blend of functional correctness and instruction-following predicts human preference for code LLMs better than pass@k alone.

  5. AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

    cs.CL 2025-08 conditional novelty 6.0 of 10

    AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.

  6. When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions

    cs.SE 2025-07 conditional novelty 6.0 of 10

    Code-generating LLMs lose 20 to 40 percentage points in pass rate when task descriptions are ambiguous, incomplete, or contradictory.

  7. Evaluating and Improving Large Language Models for Competitive Program Generation

    cs.SI 2025-06 conditional novelty 4.0 of 10

    DeepSeek-R1 solves only 5 of 80 recent ICPC/CCPC competitive programming problems with a basic prompt, and 46 of 80 after a taxonomy-guided repair and regeneration pipeline.

Reference graph

Works this paper leans on

66 extracted references · 51 canonical work pages · cited by 7 Pith papers

  1. [1]

    URLhttps://github.com/openai/human-eval

    HumanEval: Hand-Written Evaluation Set. URLhttps://github.com/openai/human-eval

    work page

  2. [2]

    URLhttps://icpc.foundation/

    ICPC Foundation, . URLhttps://icpc.foundation/

    work page

  3. [3]

    URLhttps://icpc.global/

    The ICPC International Collegiate Programming Contest, . URLhttps://icpc.global/

    work page

  4. [4]

    URLhttps://ioinformatics.org/

    International Olympiad in Informatics. URLhttps://ioinformatics.org/

    work page

  5. [5]

    URLhttps://mitit.org/About

    MIT Informatics Tournament. URLhttps://mitit.org/About

    work page

  6. [6]

    URLhttps://noi.cn/

    Chinese National Olympiad in Informatics. URLhttps://noi.cn/

    work page

  7. [7]

    URLhttps://thusaac.com/public

    Tsinghua University Programming Contest. URLhttps://thusaac.com/public

    work page

  8. [8]

    URLhttps://usaco.org/

    USA Computing Olympiad. URLhttps://usaco.org/

    work page

Show all 66 references
  1. [9]

    URL https://icpc.global/worldfinals/fact-sheet/ ICPC-Fact-Sheet.pdf

    ICPC Fact Sheet, 2025. URL https://icpc.global/worldfinals/fact-sheet/ ICPC-Fact-Sheet.pdf

    2025

  2. [10]

    Model card addendum: Claude 3.5 sonnet

    Anthropic. Model card addendum: Claude 3.5 sonnet. https://www-cdn.anthropic.com/ fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf, 2024

    2024

  3. [11]

    Claude 3.5 Sonnet, 2024

    Anthropic. Claude 3.5 Sonnet, 2024. URL https://www.anthropic.com/news/ claude-3-5-sonnet

    2024

  4. [12]

    System card: Claude 3.7 sonnet (max reasoning)

    Anthropic. System card: Claude 3.7 sonnet (max reasoning). https://assets.anthropic.com/ m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf, 2025

    2025

  5. [13]

    System card: Claude 3.7 sonnet (no reasoning variant)

    Anthropic. System card: Claude 3.7 sonnet (no reasoning variant). https://assets.anthropic. com/m/785e231869ea8b3b/original/claude-3-7-sonnet-system-card.pdf, 2025

    2025

  6. [14]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    2021 arXiv

  7. [15]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  8. [16]

    Official website of the china collegiate programming contest (ccpc), 2025

    China Collegiate Programming Contest. Official website of the china collegiate programming contest (ccpc), 2025. URLhttps://ccpc.io/. Accessed: 2025-05-11

    2025

  9. [17]

    Model card: Qwen-max (qwen 2.5 max).https://huggingface.co/Qwen, 2025

    Alibaba Cloud. Model card: Qwen-max (qwen 2.5 max).https://huggingface.co/Qwen, 2025

    2025

  10. [18]

    Interactive Problems: Guide for Participants, 2015

    Codeforces. Interactive Problems: Guide for Participants, 2015. URL https://codeforces.com/ blog/entry/45307

    2015

  11. [19]

    Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation.arXiv preprint arXiv:2405.11430, 2024

    Jianbo Dai, Jianqiao Lu, Yunlong Feng, Dong Huang, Guangtao Zeng, Rongju Ruan, Ming Cheng, Haochen Tan, and Zhijiang Guo. Mhpp: Exploring the capabilities and limitations of language models beyond basic code generation.arXiv preprint arXiv:2405.11430, 2024. 13 LiveCodeBench Pro

    2024 arXiv

  12. [20]

    Model card: Gemini 2.0 flash reasoning.https://blog.google/technology/ google-deepmind/gemini-model-updates-february-2025/, 2025

    Google DeepMind. Model card: Gemini 2.0 flash reasoning.https://blog.google/technology/ google-deepmind/gemini-model-updates-february-2025/, 2025

    2025

  13. [21]

    Model card: Gemini 2.5 flash

    Google DeepMind. Model card: Gemini 2.5 flash. https://blog.google/products/gemini/ gemini-2-5-flash-preview/, 2025

    2025

  14. [22]

    Model card: Gemini 2.5 pro

    Google DeepMind. Model card: Gemini 2.5 pro. https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/, 2025

    2025

  15. [23]

    Model card: Gemma 3 27b

    Google DeepMind. Model card: Gemma 3 27b. https://blog.google/technology/ developers/gemma-3/, 2025

    2025

  16. [24]

    Model card: Deepseek v3

    DeepSeek AI. Model card: Deepseek v3. https://huggingface.co/deepseek-ai/DeepSeek-V3, 2024

    2024

  17. [25]

    Model card: Deepseek r1

    DeepSeek AI. Model card: Deepseek r1. https://huggingface.co/deepseek-ai/DeepSeek-R1, 2025

    2025

  18. [26]

    Model card: Deepseek -r1-distill-llama-70b

    DeepSeek AI. Model card: Deepseek -r1-distill-llama-70b. https://huggingface.co/ deepseek-ai/DeepSeek-R1-Distill-Llama-70B, 2025

    2025

  19. [27]

    Model card: Deepseek -v3-0324

    DeepSeek AI. Model card: Deepseek -v3-0324. https://huggingface.co/deepseek-ai/ DeepSeek-V3-0324, 2025

    2025

  20. [28]

    Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.Advances in Neural Information Processing Systems, 36:46701–46723, 2023

    Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.Advances in Neural Information Proces...

    2023

  21. [29]

    Evaluating the performance of large language models in competitive programming: A multi-year, multi-grade analysis

    Adrian Marius Dumitran, Adrian Cˇatˇalin Badea, and Stefan-Gabriel Muscalu. Evaluating the performance of large language models in competitive programming: A multi-year, multi-grade analysis. In2024 International Conference on INnovations in Intelligent SysTems and Application...

    2024

  22. [30]

    Mathematics and games.Eureka, 2:6–8, 1939

    Patrick M Grundy. Mathematics and games.Eureka, 2:6–8, 1939

    1939

  23. [31]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  24. [32]

    Llm-pros: Analyzing large language models’ performance in competitive problem solving.arXiv preprint arXiv:2502.04355, 2025

    Md Sifat Hossain, Anika Tabassum, Md Fahim Arefin, and Tarannum Shaila Zaman. Llm-pros: Analyzing large language models’ performance in competitive problem solving.arXiv preprint arXiv:2502.04355, 2025

    2025

  25. [33]

    Competition-level problems are effective llm evaluators.arXiv preprint arXiv:2312.02143, 2023

    Yiming Huang, Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, et al. Competition-level problems are effective llm evaluators.arXiv preprint arXiv:2312.02143, 2023

    2023 arXiv

  26. [34]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 14 LiveCodeBench Pro

    2024 arXiv

  27. [35]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  28. [36]

    Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, 2024

    2024

  29. [37]

    Competition-level code generation with alphacode.Science, 378(6624):1092–1097, 2022

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R´emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode.Science, 378(6624):1092–1097, 2022

    2022

  30. [38]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572, 2023

    2023

  31. [39]

    Repobench: Benchmarking repository-level code auto-completion systems.arXiv preprint arXiv:2306.03091, 2023

    Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-level code auto-completion systems.arXiv preprint arXiv:2306.03091, 2023

    2023 arXiv

  32. [40]

    Model card: Meta llama 3.1 405b instruct

    Meta AI. Model card: Meta llama 3.1 405b instruct. https://huggingface.co/meta-llama/ Llama-3.1-405B-Instruct, 2024

    2024

  33. [41]

    Codeforces, 2010

    Mike Mirzayanov. Codeforces, 2010. URLhttps://codeforces.com/

    2010

  34. [42]

    System card: Gpt-4o.https://openai.com/index/gpt-4o-system-card/, 2024

    OpenAI. System card: Gpt-4o.https://openai.com/index/gpt-4o-system-card/, 2024

    2024

  35. [43]

    Release note: Gpt -4.1

    OpenAI. Release note: Gpt -4.1. https://www.theverge.com/news/647896/ openai-chatgpt-gpt-4-1-mini-nano-launch-availability, 2025

    2025

  36. [44]

    Release note: Gpt -4.1 mini

    OpenAI. Release note: Gpt -4.1 mini. https://www.theverge.com/news/647896/ openai-chatgpt-gpt-4-1-mini-nano-launch-availability, 2025

    2025

  37. [45]

    System card: Gpt-4.5.https://openai.com/index/gpt-4-5-system-card/, 2025

    OpenAI. System card: Gpt-4.5.https://openai.com/index/gpt-4-5-system-card/, 2025

    2025

  38. [46]

    System card: Openai o3 -mini

    OpenAI. System card: Openai o3 -mini. https://openai.com/index/o3-mini-system-card/ , 2025

    2025

  39. [47]

    System card: Openai o4 -mini (including the o4-mini-high variant)

    OpenAI. System card: Openai o4 -mini (including the o4-mini-high variant). https://openai. com/index/o3-o4-mini-system-card/, 2025

    2025

  40. [48]

    Introducing openai o3 and o4-mini

    OpenAI. Introducing openai o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/, 2025

    2025

  41. [49]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings.arXiv preprint arXiv:2501.01257, 2025

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings.arXiv preprint arXiv:2501.01257, 2025

    2025 arXiv

  42. [50]

    Model card: Llama 4 maverick 17b instruct.https://huggingface

    Unsloth (community release). Model card: Llama 4 maverick 17b instruct.https://huggingface. co/unsloth/Llama-4-Maverick-17B-128E-Instruct-GGUF, 2025

    2025

  43. [51]

    Can language models solve olympiad programming?arXiv preprint arXiv:2404.10952, 2024

    Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming?arXiv preprint arXiv:2404.10952, 2024. 15 LiveCodeBench Pro

    2024 arXiv

  44. [52]

    New features: friends, tags and more, 2011

    Maxim Shipko. New features: friends, tags and more, 2011. URL https://codeforces.com/ blog/entry/1679

    2011

  45. [53]

    Learning task decomposition to assist humans in competitive programming

    Jiaxin Wen, Ruiqi Zhong, Pei Ke, Zhihong Shao, Hongning Wang, and Minlie Huang. Learning task decomposition to assist humans in competitive programming. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 117...

    2024

  46. [54]

    Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms

    Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint arXiv:2504.14655, 2025

    2025 arXiv

  47. [55]

    Evaluating the smooth control of attribute intensity in text generation with llms

    Shang Zhou, Feng Yao, Chengyu Dong, Zihan Wang, and Jingbo Shang. Evaluating the smooth control of attribute intensity in text generation with llms. InFindings of the Association for Computational Linguistics: ACL 2024, pages 4348–4362, 2024

    2024

  48. [56]

    ultimate

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions.arXiv preprint arXiv:2406.15877...

    2024 arXiv

  49. [57]

    Topic tagging.Each problem is assigned to a refined taxonomy (Figure2) that corrects the noise and omissions of the crowd-edited Codeforces tag system

  50. [58]

    Difficulty and pitfalls.Annotators make a fine-grained categorization scheme for all problems, classify each problem by their underlying algorithmic ideas, and record the official Codeforces difficulty rating x (50 % success rate at Elo x), and note the key observations, commo...

  51. [59]

    Fails Sample

    Submission triage.For every model or human solution, we log the verdict (Accepted, Wrong Answer, Time Limit Exceeded, . . . ), attach a root-cause label (idea-level vs. implementation-level error), and flag “Fails Sample” if the code cannot pass the problem’s own sample I/O pr...

    2024

  52. [60]

    Therefore every element that appears before that minimum in the original array must be moved to the back (and thus increased by 1)

    The first position must contain theglobal minimum. Therefore every element that appears before that minimum in the original array must be moved to the back (and thus increased by 1)

  53. [61]

    56 LiveCodeBench Pro

    For the second position we now choose the smallest possible value among • the suffix minimum that follows the global minimum, and • the set of elements that have already been moved to the back. 56 LiveCodeBench Pro

  54. [62]

    suffix minima sequence

    We repeat this procedure while we can still take the next value from thesuffix-minimum sequence. Eventually the next candidate issmallerin the pile of “moved” elements than in the remaining suffix minima. At that moment every element still to the right must be moved once (henc...

  55. [63]

    Precompute the minimum to the right of each element

  56. [64]

    For each element, decide whether to move it (if there’s a smaller element to the right)

  57. [65]

    Collect the non-moved elements in order and the moved elements in a list

  58. [66]

    op",i, "moved id

    Sort the moved elements and append them to the non-moved elements. This works because the moved elements are all added to the end of the array in sorted order, which is lex smallest. [But actually this doesn’t work.] Reasoning Process ofo4-mini-high The user asked me to solve ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.