REVIEW 15 cited by
Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent math benchmarks for large language models (LLMs) such as MathArena indicate that state-of-the-art reasoning models achieve impressive performance on mathematical competitions like AIME, with the leading model, Gemini-2.5-Pro, achieving scores comparable to top human competitors. However, these benchmarks evaluate models solely based on final numerical answers, neglecting rigorous reasoning and proof generation which are essential for real-world mathematical tasks. To address this, we introduce a comprehensive evaluation of full-solution reasoning for challenging mathematical problems. Using expert human annotators, we evaluated several state-of-the-art reasoning models on the six problems from the 2025 USAMO within hours of their release. Our results reveal that all tested models struggled significantly: only Gemini-2.5-Pro achieves a non-trivial score of 25%, while all other models achieve less than 5%. Through detailed analysis of reasoning traces, we identify the most common failure modes and find several unwanted artifacts arising from the optimization strategies employed during model training. Overall, our results suggest that current LLMs are inadequate for rigorous mathematical reasoning tasks, highlighting the need for substantial improvements in reasoning and proof generation capabilities.
Forward citations
Cited by 15 Pith papers
-
Cost-Effective Automated Judging of Natural-Language Mathematical Proofs
Cheap open-weight judges match frontier models on pass/fail grading of IMO proofs at up to 100x lower cost, with the best voting rule still pending replication.
-
QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs
On QEDBench, frontier LLM judges over-score university math proofs by up to +0.36 on average relative to human experts, while some solver models fail badly on discrete-combinatorial problems.
-
Large Lemma Miners: Can LLMs do Induction Proofs for Hardware?
LLMs, verified by a symbolic model checker, produced correct inductive strengthenings for 82 of 94 curated RTL safety properties.
-
Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models
A dynamic replay and reweighting scheduler (RECAP) preserves general capabilities during RLVR while keeping reasoning performance at least as good as reasoning-only finetuning.
-
Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
When a cosine schedule would halve the learning rate, Seesaw cuts it by √2 and doubles the batch, matching loss curves with ~36% fewer serial steps.
-
SSRL: Self-Search Reinforcement Learning
SSRL, a training pipeline that uses an LLM's own repeated sampling as a search environment for RL, improves question answering without external tools and transfers to real search engines.
-
Seed-Prover: Deep and Broad Reasoning for Automated Theorem Proving
Seed-Prover and Seed-Geometry prove 121 of 155 formalized past IMO problems, reach 99.6% on MiniF2F-test, and solve 5 of 6 IMO 2025 problems after the competition deadline.
-
Reviving DSP for Advanced Theorem Proving in the Era of Reasoning Models
An inference-only neuro-symbolic pipeline, DSP+, solves 80.7% of miniF2F and the previously unsolved imo_2019_p1, matching heavily RL-trained theorem provers without fine-tuning.
-
Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematics
A self-evolving curriculum that retrains a language model on variants of problems it can mostly get right lifts AIME pass@1 from 5.6% to 16.5%, beating static augmentation under the same data budget.
-
Too long; didn't solve
Prompt length and solution length both rise with LLM failure on expert-authored adversarial math problems, linking structural length to empirical difficulty.
-
No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
Internal feedback rewards (entropy and self-certainty) improve base LLM math reasoning only in early training and degrade later, with little benefit for instruct models.
-
Right Is Not Enough: The Pitfalls of Outcome Supervision in Training LLMs for Math Reasoning
On a new Olympiad-style benchmark, outcome-rewarded LLMs reach 80.1% answer accuracy but only 39.7% reasoning correctness, and a step-by-step LLM verifier outperforms holistic judges at spotting flawed reasoning.
-
Using Reasoning Models to Generate Search Heuristics that Solve Open Instances of Combinatorial Design Problems
LLM-generated search heuristics run through the CPro1 protocol with the reasoning model o3-mini-high produced verified constructions resolving open instances in 7 Handbook design families and newer problems.
-
The Mathematician's Assistant: Integrating AI into Research Practice
AI tools currently augment rather than replace mathematicians, and a five-principle, seven-area framework can help researchers use them responsibly.
-
On the Surprising Efficacy of LLMs for Penetration-Testing
A critical review arguing that LLMs are surprisingly effective for penetration testing because the task is largely pattern-matching, while noting serious reliability, safety, and cost barriers to autonomous use.
Discussion (0). Sign in to comment.