{"id":"41b44988-de9e-418f-a159-818152d48966","arxiv_id":"2603.03202","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Code agents with test-time exploration and dual verification can synthesize solvable math problems that are harder for LLM solvers than their seed problems.","lead":"This paper tests whether code-writing AI agents can turn easy math problems into harder, still-solvable ones by exploring numerical and symbolic structure. It reports that a three-agent pipeline with code execution produces problems that reduce solver accuracy while keeping human-judged validity high.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Origin-SR is computed on all 100 seeds while Evolution-SR is computed only on the accepted subset; if the subset is not representative, the reported difficulty drops may be an artifact.","rationale":"The reader correctly identified the solvability proxy as a load-bearing assumption: an LLM-judged 'flaw-less logical chain' may not guarantee true solvability for subtle Olympiad-level problems, and the human audit covers only 127 of roughly 477 accepted problems. However, the paper already provides external-judge agreement and a human-validity precision of 99.2%, which partially mitigates that concern. In contrast, the unmatched baseline between Origin-SR (full seed set) and Evolution-SR (accepted subset) is not discussed or mitigated anywhere in the manuscript. This is a more immediate threat to the central quantitative claim that evolved problems are 'more challenging.' It is also easily testable with the existing data, making it the most concrete next check. I therefore mark agreement as 'partial' because I share the reader's caution about solvability but believe the matched-baseline issue is the single most load-bearing concern to settle first. The verdict remains CONDITIONAL: the paper's framework is plausible and the human audit is promising, but the difficulty comparison needs rematching or explicit justification before the conclusion can be accepted.","tokens_in":31208,"tokens_out":6694,"duration_ms":63548,"concrete_test":"For each evolver, define S_e as the set of seed problems whose evolved outputs were externally certified (the denominator in each AR cell). Recompute Origin-SR for every solver using only S_e, and compare these matched Origin-SR values to the reported full-seed Origin-SR. If the drops (Evolution-SR minus matched Origin-SR) shrink substantially or become non-negative for several solver–evolver pairs, the difficulty claim is confounded by seed selection and should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that evolved problems are 'more challenging' rests on the comparison between Origin-SR and Evolution-SR in Table 1. Origin-SR is reported as a single value per solver across the full 100-seed dataset, while Evolution-SR is computed only on the problems that passed internal verification and external certification (e.g., 74–98 problems per evolver, with AR cells like 83/94, 94/98, 74/90). The paper does not state that Origin-SR is recomputed on the exact subset of seeds whose evolved outputs were evaluated. If the seeds that failed evolution are systematically easier or harder than the successful subset, the observed drops (e.g., GPT-5.2-High falling from 70% to 61–64%) could be overstated or even reversed. This is a direct confound for the difficulty-escalation conclusion, independent of the solvability-proxy issue. Even if every evolved problem is genuinely solvable, an unmatched baseline means the quantitative evidence for 'more challenging' is not yet established. The human audit does not address this because it focuses on validity and score agreement, not on rematching the seed subset.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Code2Math, a multi-agent framework for autonomously evolving mathematical problems. An Evolution Agent analyzes a seed problem and uses executable Python tools to explore structural variants; a Solvability Verification Agent screens generated problems by checking proposed solution steps for logical flaws; a Difficulty Verification Agent scores adaptations on a 1–5 scale according to 'Burden of Discovery'. Using 100 seed problems, five evolution backbones, and six solver models, the authors report that most internally accepted problems are certified solvable by an external judge (74–98 per evolver), that solve rates generally decrease from originals to evolved problems, that token consumption increases, and that a human audit supports the reliability of the GPT-5.2-High judge. A code ablation and a rollout-efficiency analysis are also presented. The central claim is that, given sufficient test-time exploration, code agents can synthesize solvable problems that are structurally distinct from and more challenging than the originals.","tokens_in":31502,"tokens_out":5332,"duration_ms":56843,"significance":"If the central claim holds, the paper makes a useful contribution to automated problem synthesis and benchmark self-evolution. The strengths are real: the evaluation crosses five evolvers and six solvers, the authors ship code and data, full prompts are provided in the appendix, and the human audit (126/127 validity precision, 92.5% answer-agreement, 95.9% within-one difficulty agreement) is a serious attempt to validate the LLM judge. The failure analysis and code ablation are also informative. However, the main quantitative evidence for 'more challenging' is currently an unmatched comparison between solve rates on different problem sets, and the solvability guarantee is an LLM-based proxy rather than a formal guarantee. Both issues are fixable, but they currently weaken the strength of the paper's headline conclusion.","major_comments":[{"comment":"Origin-SR is reported on the full 100-seed dataset, whereas Evolution-SR is computed only on the subset of problems that passed internal verification and external certification (CSC range 74–98 per evolver). The manuscript does not state that Origin-SR is recomputed on the matched subset of seeds whose evolved problems were evaluated. If successful evolution is correlated with seed difficulty, the reported drops (e.g., GPT-5.2-High: 70→64; Gemini-3-Flash-Thinking: 56→24–35) could be overstated or reversed. This is a direct confound for the 'more challenging' claim. Please recompute Origin-SR on the exact matched seed subset, or report paired per-seed deltas restricted to seeds with certified evolved problems.","section":"Table 1; §3.4; Appendix A.1"},{"comment":"The solvability guarantee rests on the assumption that a logically flaw-less proposed solution implies existence of a solution path. The external judge is itself an LLM, and the human audit covers only 127 GPT-valid problems, not the full certified sets (e.g., 74, 83, 94, 97, 98 per evolver). The audit establishes high precision on a sample of GPT-valid problems, but it cannot rule out false positives among the remaining certified problems or false negatives among rejected ones. Since 'solvable' is a central claim, either extend the human audit to all accepted problems or add independent mechanical/formal verification where feasible.","section":"§3.2; §4.3"},{"comment":"No confidence intervals or significance tests are reported for the solve-rate differences. Several cells are small or even positive (e.g., DeepSeek-Chat row: Origin-SR 16 → Evolution-SR 19 on DeepSeek-Chat evolutions; GPT-5.2-High: 70 → 70 on DeepSeek-Chat). With N≈74–98, differences of 3–9 points are often within sampling noise. Report bootstrap confidence intervals, McNemar tests, or paired tests by seed to support the 'most pairs lower' and 'systematically harder' conclusions.","section":"Table 1; §4.2"},{"comment":"The code ablation shows only a 2.6-point average difference (25.7→23.1) and the direction is inconsistent across pairs (e.g., DS-NT/DS-NT +2, Seed/DS-T +7). No uncertainty estimates are given. Because this is the only direct evidence that code execution, rather than general LLM reasoning, drives the difficulty effect, the 'code-driven' component of the central claim is not yet quantitatively established. Provide paired significance tests and, ideally, a more detailed qualitative coding of the generated problem types.","section":"§4.4; Table 5"}],"minor_comments":[{"comment":"Typo: 'evalaute' should be 'evaluate'.","section":"§3.4"},{"comment":"The cell '70 70±0' is unexplained. What does the ±0 denote — a standard error, a range, or a formatting artifact? Please clarify or remove.","section":"Table 1"},{"comment":"Typo: 'riangleABC' should be 'triangle ABC'.","section":"§A.6, Case 1"},{"comment":"Imputing the maximum token limit for all timeout samples may substantially shift the ATC distribution. At a minimum, report the fraction of timeouts separately and show the distribution without imputation as a sensitivity check.","section":"Figure 3; Appendix A.1"},{"comment":"The header abbreviates 'Gemini-3-Pro-Preview-Thinking' as 'Gemini-3-Pro-Thinking' while the text uses the full name. Use one consistent abbreviation.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the human audit is a strong point. The main obstacle is the unmatched Origin-SR baseline: the solve-rate comparison in Table 1 is not an apples-to-apples comparison unless Origin-SR is recomputed on the certified subset. I would support acceptance after a matched reanalysis and the addition of significance tests. The use of GPT-5.2-High as both judge and solver is a lesser concern; separate invocations help but do not fully eliminate potential contamination, so a brief discussion or sensitivity analysis would strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper has a real new idea and a real flaw in how it's measured. The idea is to have a code agent explore a seed math problem computationally, then run two LLM filters — one for solvability, one for 'burden of discovery' — and accept only problems that pass both. That's a sensible way to scale hard math problem generation, and it's distinct from the rule-based perturbations in MATH-Perturb and EvolMathEval, and from the coding-task focus in TRACE/AutoCode.\n\nThe flaw: the difficulty comparison in Table 1 is not apples-to-apples. Origin-SR is computed on all 100 seeds. Evolution-SR is computed only on the certified subset — 74–98 problems per evolver. If the seeds that failed evolution were systematically easier or harder than the successful ones, the reported solve-rate drops (e.g., GPT-5.2-High 70→61, Gemini-3-Flash 56→24) are confounded. The paper never says it recomputed Origin-SR on the exact success subset. This is fixable, but without the matched baseline the headline claim that evolved problems are 'more challenging' is not yet established.\n\nWhat the paper does well: the checks are substantial. It uses a held-out GPT-5.2-High as external judge for solvability, and the agreement with the internal verifier is high (74–98 of internally accepted). The human audit, though author-run and sampled, shows 126/127 validity and 92.5% correctness agreement. The code ablation is honest — 7 of 9 pairs show lower solve rates with code, but it's not a guaranteed effect. The case studies give a concrete look at why code helps (structural exploration, not just bigger numbers). And they release the code and data, which will make it easy to verify the subset issue we're worried about.\n\nOther soft spots, in proportion: no confidence intervals or significance tests on any SR differences, so we don't know which drops are noise. The solvability proxy — a flawless logical chain implies solvability — is reasonable but only as strong as the verifier; the external judge and human audit mitigate it, though not at full scale. And the paper itself admits the seed set is only 100 and there's no downstream training evaluation. The 'Aha moment' / 'Burden of Discovery' framing is a bit fuzzy, but the 1–5 rubric operationalizes it enough.\n\nBottom line: this deserves a serious referee, and I'd bring it to a reading group to discuss the methodology. But I'd ask the authors to fix the baseline mismatch and add error bars before I trust the quantitative conclusion. If the difficulty drop survives rematching, it's a useful contribution to math data synthesis.","headline":"Worth a serious look; the code-driven evolution framework is new and mostly sound, but the headline difficulty claim has a baseline mismatch that needs fixing.","tokens_in":32022,"tokens_out":4099,"would_cite":true,"duration_ms":39184,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Code-enabled agents can evolve existing math problems into new, solvable ones that are harder for current reasoning models—solve rates drop up to 32 points even for the strongest tested solver.","keywords":["code agents","math problem generation","test-time exploration","solvability verification","difficulty escalation","LLM reasoning","multi-agent system","problem evolution"],"falsifier":"Take a random sample of internally accepted evolved problems, remove the reference solutions, and ask strong independent solvers (or a proof-assistant formalization) to solve them from scratch. If a nontrivial fraction yield no correct solution or no verifiable proof, then 'solvable because the proposed solution had no detected flaw' is too weak. A simpler version: seed the pipeline with deliberately ill-posed problems that have plausible but subtly wrong reference solutions and count how often the solvability verifier passes them.","tokens_in":31116,"feed_emoji":"🧮","tokens_out":4604,"duration_ms":42967,"temperature":0.7,"pith_summary":"This paper asks whether large language models, given a Python sandbox and test-time exploration, can turn existing math problems into new problems that are both solvable and genuinely harder. The authors build a three-agent pipeline: an evolution agent that rewrites a seed problem while aiming to hide the key insight, a solvability verifier that audits proposed solutions for logical flaws, and a difficulty verifier that rejects merely tedious increases. Across five evolver models and six solver models, evolved problems lower solve rates relative to seeds for nearly every pair, with the strongest tested solver falling from 70% to 61% and another dropping 32 points, while an external judge certifies most internally accepted problems as solvable. The paper's point is that executable exploration lets a code agent discover structural variants—new constraints, extremal targets, hidden patterns—rather than only surface paraphrase. A sympathetic reader should care because automatic, scalable generation of hard, valid math problems would relieve a bottleneck for training and evaluating reasoning models.","feed_headline":"Agent-evolved math problems cut solve rates by up to 32 points","feed_subtitle":"Code agents with a Python sandbox turn seed problems into solvable variants that stump six reasoning models.","key_machinery":"The load-bearing mechanism is the three-agent validation loop. The Evolution Agent operates in two phases—bottleneck analysis then free exploration—and is instructed to create a 'Burden of Discovery' by hiding the entry point to the solution. The Solvability Verification Agent uses the proposed solution as a proxy: a logically consistent chain implies a solution path exists. The Difficulty Verification Agent scores adaptations from 1 to 5 and rejects scores 1–2 as artificial complexity. A code environment with symbolic computation, constraint solving, and enumeration tools supplies deterministic feedback during exploration and verification.","core_discovery":"The central claim is that a code-enabled autonomous agent can evolve a mathematical problem into a new problem that is (1) recognized as solvable by a held-out judge and (2) harder for current solvers than the original, where hardness is measured by lower solve rate and greater token usage. The mechanism combines executable exploration with dual verification: the evolution agent writes and runs Python to probe candidate structures, the solvability agent checks the proposed solution chain for logical flaws, and the difficulty agent scores whether the change demands a new insight rather than extra computation. The paper reports consistent Evolution-SR deficits across model pairs, a 49.1% failu","pith_inferences":["The paper does not show that training on evolved problems improves models; a natural follow-up would test whether the evolved set transfers as training data, which the authors explicitly defer.","The solvability proxy is the fragile joint; feeding accepted problem/solution pairs into a formal proof checker would give a stronger guarantee, and the pipeline's code-first format makes this test look feasible.","Difficulty as 'Burden of Discovery' invites a falsifiable check: whether human experts solving blind take longer and make more failed attempts on evolved problems than on seeds matched for topic and length.","A concrete extension would compare evolved problems against human-written olympiad problems of matched topic to see whether solve-rate drops are due to obscurity or ambiguity rather than mathematical depth."],"forward_implications":["Automatic evolution could produce large sets of novel olympiad-style problems without hand curation.","The observed capability asymmetry—evolvers producing problems that defeat stronger solvers—suggests models can generate training or evaluation data beyond their own current solving ceiling.","Because the pipeline filters for structural rather than computational difficulty, generated problems should remain useful for reasoning evaluation rather than merely longer arithmetic.","The efficiency numbers (1.56–6.55 failed rollouts per success, with solvability failures dominating) imply that making the solvability check cheaper or stronger would directly lower data-synthesis cost."],"fun_headline_variants":["Code agents evolve math problems into tougher, solvable variants","Tougher math from code: agents craft problems that stump LLMs","Python sandbox lets agents breed harder math problems","Evolved math problems reduce AI solve rates by 32 points","Agent-generated math: solvable but more challenging for LLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The claim that evolved problems are solvable rests on the paper's solvability verifier accepting a proposed solution whose logical chain it cannot find a flaw in; if that check is systematically over-permissive for subtle olympiad-level statements, the certified solvability counts and the difficulty comparisons built on them are unsupported, and the human audit covers only 127 validity cases, not the full generated set.","fun_headline_variants_meta":{"raw":{"variants":["Code agents evolve math problems into tougher, solvable variants","Tougher math from code: agents craft problems that stump LLMs","Python sandbox lets agents breed harder math problems","Evolved math problems reduce AI solve rates by 32 points","Agent-generated math: solvable but more challenging for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000361,"raw_usage":{"total_tokens":1764,"prompt_tokens":699,"completion_tokens":1065,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":981}},"tokens_in":443,"tokens_out":1065,"duration_ms":9301,"temperature":1.0,"reasoning_tokens":981,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:07:41.203101+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of internally accepted evolved problems, remove the reference solutions, and ask strong independent solvers (or a proof-assistant formalization) to solve them from scratch. If a nontrivial fraction yield no correct solution or no verifiable proof, then 'solvable because the proposed solution had no detected flaw' is too weak. A simpler version: seed the pipeline with deliberately ill-posed problems that have plausible but subtly wrong reference solutions and count how often the solvability verifier passes them.","supporting_citations":[],"review_version":1}