REVIEW 12 cited by
Can Language Models Solve Olympiad Programming?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Computing olympiads contain some of the most challenging problems for humans, requiring complex algorithmic reasoning, puzzle solving, in addition to generating efficient code. However, it has been understudied as a domain to evaluate language models (LMs). In this paper, we introduce the USACO benchmark with 307 problems from the USA Computing Olympiad, along with high-quality unit tests, reference code, and official analyses for each problem. These resources enable us to construct and test a range of LM inference methods for competitive programming for the first time. We find GPT-4 only achieves a 8.7% pass@1 accuracy with zero-shot chain-of-thought prompting, and our best inference method improves it to 20.2% using a combination of self-reflection and retrieval over episodic knowledge. However, this is far from solving the benchmark. To better understand the remaining challenges, we design a novel human-in-the-loop study and surprisingly find that a small number of targeted hints enable GPT-4 to solve 13 out of 15 problems previously unsolvable by any model and method. Our benchmark, baseline methods, quantitative results, and qualitative analysis serve as an initial step toward LMs with grounded, creative, and algorithmic reasoning.
Forward citations
Cited by 12 Pith papers
-
CodeContests+: High-Quality Test Case Generation for Competitive Programming
A new dataset, CodeContests+, uses an LLM agent system to generate and validate competitive programming test cases, improving evaluation accuracy and reinforcement learning training.
-
When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration
Model benchmark performance only weakly predicts how well people learn from AI explanations, with notable outliers across code and math.
-
From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs
On 2,520 programming tasks, matched Qwen general and coder models reliably raise Bloom cognitive demand but fail to lower it, so execution skill does not imply educational control.
-
Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning
MetaEvolve trains LLMs with reinforcement learning on synthesized code-refinement trajectories, reporting large gains on coding and numerical-optimization benchmarks.
-
Efficiency of turbulence
The efficiency of turbulence, the fraction of input energy stored in the flow, appears bounded and may saturate in a power-law manner across several turbulent flows.
-
LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
A live, medalist-annotated competitive programming benchmark finds frontier LLMs solve 53% of medium and 0% of hard problems without tools, and succeed mainly on implementation-heavy tasks.
-
OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics
A private, contamination-resistant benchmark of 250 olympiad-level programming problems shows top reasoning models reaching about 36% solve rates, far above conventional models.
-
ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests
ICPC-Eval benchmarks LLMs on 118 recent ICPC contest problems with local test suites and a Refine@K metric that quantifies iterative refinement, showing even the best models solve only about 29% of problems.
-
rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
A large verified dataset of 418K competition code problems built by synthesis, test-input utility functions, and majority-vote output labeling improves small model code reasoning to frontier level.
-
Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report
An evaluation of 18 frontier AI models across seven catastrophic-risk categories finds all models in green or yellow zones, with none crossing the report's proposed red lines.
-
Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights
The paper claims a state-of-the-art, low-cost open-weight web agent, but its WebVoyager result is inflated by training on the evaluation benchmark.
-
Secure Code Generation at Scale with Reflexion
Three-round reflexion prompting raises LLM-generated code security from 70.74% to 79.43% as measured by the ICD static detector on the Instruct Prime benchmark.
Discussion (0). Continue with ORCID to comment.