{"id":"5813fdc9-bc07-4ae9-b012-f7b5aa480b64","arxiv_id":"2504.17665","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Code-assisted LLMs often get correct math answers with brute-force or memorized programs, so execution accuracy overstates reasoning quality.","lead":"The paper inspects the Python programs five language models write to solve math problems and labels each as conceptually sound or not. It finds that brute-force and memorized programs often produce correct answers, so final-answer accuracy hides how models really reason.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Brute-force enumeration is classified as 'unsound' and then shown to reach comparable accuracy; under standard logical-soundness usage a terminating exhaustive search is valid, so the headline finding may reduce to an efficiency/style claim.","rationale":"The paper contributes a useful first taxonomy and an honest attempt at automated judging; the manual annotations, disagreement rates, and worked examples are real evidence. My concern is not that the data are fabricated or the analysis sloppy, but that the central interpretation overreaches the authors' own stipulative definition of 'logical soundness.' Standard usage distinguishes correctness and validity from efficiency and style; the paper's definition collapses them. The concrete re-annotation test would show whether the central finding is definitional. Until that is done, the paper should be accepted conditionally: the taxonomy and the 'No Logic can score well' observation are valuable, but the broader claim about unsound reasoning outperforming sound reasoning should be softened. I therefore keep the reader's CONDITIONAL verdict unchanged. The difficulty confound and the acknowledged low recall on the Disorganized class are secondary but reinforce the need for a revised presentation.","tokens_in":16833,"tokens_out":7032,"duration_ms":78754,"concrete_test":"Re-annotate a stratified sample of the generated programs with a revised taxonomy that treats exhaustive-but-terminating searches as logically sound (or at least as a separate 'efficient vs inefficient' axis), then recompute the per-class execution accuracies in Table 1 and the distributions in Figures 4-5. If the sound/unsound accuracy gap remains only when brute-force programs are excluded from the sound set, the central claim must be narrowed to no-processing/guessing programs; if the parity persists without brute-force in the unsound set, the claim is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central assertion in Section 5.2 ('Code-assisted LLMs can achieve comparable performance regardless of the type of logic they employ, hindering the trustworthiness of their generated programs') depends on the taxonomy's definition of logical soundness as 'grounding the implementation in a math concept' (Section 3.4; Section 3.5). The paper places terminating exhaustive searches into class 4 (Brute-Force) and treats them as unsound. The GPT-4 LCM example in Figure 1 illustrates the problem: a while-loop that checks divisibility of successive integers until it finds a common multiple is a correct, terminating, and verifiable algorithm for the given inputs. It is inefficient relative to math.gcd, but it is not logically unsound. Similarly, a program that returns a correct precomputed answer may reflect memorization, but the program itself is not a 'reasoning process' at all, which is a different failure than unsoundness. Because Table 1 shows that these 'unsound' classes achieve accuracy close to the 'sound' classes, the headline may be partially an artifact of labeling algorithmic style as logical invalidity. The weaker, more defensible claim—that models sometimes return answers without any programmatic reasoning (No Logic) and still get credit—is enough to motivate better evaluation, but the paper states the stronger conclusion. A secondary confound: the ASDiv and MATH500 comparisons in Section 5.2 use different prompts (PAL vs MATH prompts) and zero-shot settings for GPT4o-mini/Qwen on MATH500 (Section 4.2), so 'difficulty decreases soundness' is not cleanly identified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the Python programs that code-assisted large language models generate when solving math word problems, moving beyond execution accuracy to an analysis of the intermediate program logic. The authors propose a six-class taxonomy of program logic—Conceptual, Primitive, From-scratch, Brute-Force, Disorganized, and No Logic—and label the first three as logically sound and the last three as unsound. They manually annotate 300 programs, train a decision-tree classifier called Code-Structure Judge on AST-derived features, compare it with an o3-mini LLM judge, and apply the better judge to generations from five LLMs on ASDiv and MATH500. The main reported findings are that model capability affects the type of logic used, that the harder MATH500 problems reduce the share of sound programs, and that execution accuracy is comparable across sound and unsound classes, so accuracy-based evaluation hides reasoning-quality problems.","tokens_in":17120,"tokens_out":5006,"duration_ms":51204,"significance":"The paper makes a useful empirical contribution by drawing attention to program-level reasoning quality in code-assisted math solving, an area that is usually evaluated only by final-answer correctness. The manual annotation process is careful, with a per-line protocol and 93% line-level agreement, and the result that a small AST-feature decision tree outperforms an o3-mini LLM judge is an interesting and practical methodological finding. The taxonomy, while valuable as an analytical instrument, is built on a nonstandard definition of logical soundness, and several empirical claims are stated more strongly than the data support. If the terminology and claims are revised to target 'groundedness' or 'transparency' rather than logical soundness, the paper would be a solid contribution to more holistic evaluation of code-assisted LLMs.","major_comments":[{"comment":"The taxonomy labels Brute-Force programs as unsound because they do not ground the implementation in a math concept, but this is not logical unsoundness in the standard sense. A terminating exhaustive search is a valid algorithm: the GPT-4 while-loop in Figure 1 correctly computes the LCM for the given inputs, and it is verifiable and correct, though inefficient. Likewise, a No Logic program that directly returns a precomputed answer is not an unsound reasoning process; it is an absence of programmatic reasoning. Since Table 1 then shows that these 'unsound' classes often achieve accuracy comparable to the 'sound' classes, the central assertion in Section 5.2—that LLMs achieve comparable performance regardless of the logic they employ—partly conflates algorithmic style and efficiency with reasoning correctness.","section":"Section 3.5; Figure 1; Table 1"},{"comment":"The claim that execution accuracy is comparable across all logic classes is not supported by the reported numbers. On ASDiv, GPT4 achieves 86% for Conceptual but 68% for Brute-Force and 63% for No Logic; GPT4o-mini achieves 90% for Conceptual but 77% for Brute-Force and 85% for No Logic. These are differences of up to 23 percentage points. No sample sizes per class, confidence intervals, or significance tests are reported, so the reader cannot tell whether the observed differences are real or noise. The data support a weaker claim: unsound or ungrounded classes can still achieve nontrivial accuracy, not that accuracy is comparable regardless of logic.","section":"Section 5.2; Table 1"},{"comment":"The comparison between ASDiv and MATH500 is confounded by more than problem difficulty. ASDiv uses the PAL prompt with three demonstrations, while MATH500 uses a different prompt with demonstrations from the MATH training split; GPT4o-mini and Qwen are evaluated in zero-shot settings on MATH500, and the model subsets differ between the two datasets. In addition, Figure 5 shows only GPT4o-mini, Qwen2.5, and StarCoder2, so the statement that increasing difficulty decreases sound generations 'for all models' is not supported for GPT4 and Llama3.1. The difficulty-driven conclusion should either be supported by a controlled comparison or restricted to the models and settings actually evaluated.","section":"Section 4.2; Figure 5"},{"comment":"The automated judge is the instrument behind the large-scale per-class accuracy results, but its reliability is uneven. Code-Structure Judge has 81% overall accuracy, yet the Disorganized class has recall of only 0.16 and precision of 0.50, and the paper acknowledges that the notably high accuracy of Disorganized programs in Table 1 is mainly due to false positives from the judge. Because per-class execution accuracy is load-bearing for the central claim, the paper should report the confusion-matrix-corrected estimates or explicitly exclude the least reliable classes from the accuracy comparison.","section":"Section 5.1; Table 1"}],"minor_comments":[{"comment":"The heading contains the typo 'Evauation', and the same section later contains 'Primitve' instead of 'Primitive'.","section":"Section 5.2 heading and text"},{"comment":"The caption should state why GPT4 and Llama3.1 are absent from the MATH500 logic distribution, since Figure 4 reports all five models on ASDiv.","section":"Figure 5 caption"},{"comment":"The table header says 'Execution (macro) accuracy' but the paper does not define how the macro averaging is computed; please clarify whether accuracy is averaged per problem, per class, or per model.","section":"Table 1"},{"comment":"The paper says programs that fail to parse are discarded, but it does not report how many programs were discarded or whether the discard rate varies by model; this information is needed to assess potential bias in the logic distributions.","section":"Section 3.1"},{"comment":"Spelling of the taxonomy class is inconsistent: 'Disorganised' appears in Section 3.5 and some figures, while 'Disorganized' appears elsewhere; please standardize.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is the first paper I've seen that looks inside the generated programs of code-assisted LLMs on math and classifies the reasoning style, rather than scoring only the final answer. The taxonomy (library call, primitive, from-scratch, brute-force, disorganized, no logic) is useful and the manual annotation is carefully done, with documented guidelines and high agreement on a double-annotated subset. Second, the central empirical observation holds in the data: execution accuracy can be high for programs that got the answer via a brute-force loop or a memorized constant. Table 1 shows Qwen2.5 at 100% brute-force on ASDiv and StarCoder2's No Logic accuracy above its Conceptual accuracy. That is a genuine problem for evaluation that stops at execution.\n\nThe soft spot is the paper's definition of 'unsound.' The authors define sound as 'grounding the implementation in a math concept,' which is a normative style preference, not logical validity. The GPT-4 LCM example in Figure 1 is a terminating, correct search for a common multiple; it would be fair to call it inefficient, but calling it unsound overstates the finding. The No Logic and Disorganized cases are genuinely unverifiable, and that alone is enough to support the paper's main recommendation—evaluation should look at more than execution accuracy. But the strong version of the claim, stated in the abstract and Section 5.2, depends on a definition that will not hold up under scrutiny.\n\nThere are two secondary issues. The difficulty comparison between ASDiv and MATH500 is confounded: different prompts, different numbers of demonstrations, and different model subsets, so 'harder problems lead to more unsound reasoning' is not cleanly identified. And there is no released code or data and no error bars on the per-class accuracies, which makes it harder to judge how stable the small-cell results are.\n\nWho is this for? Anyone evaluating code-assisted LLMs on math or building evaluation metrics for reasoning. The taxonomy and the empirical picture are worth publishing, but the 'unsound' terminology should be softened and the difficulty analysis either re-run in a matched setting or reinterpreted. I'd send it to review—it deserves a serious referee—and the authors are honest about their limitations, which is a good sign.","headline":"A solid first look at the logic inside code-assisted math solutions, with a taxonomy worth using and a headline overstated by calling brute-force 'unsound.'","tokens_in":17677,"tokens_out":2863,"would_cite":true,"duration_ms":28004,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs can post high execution accuracy on math benchmarks even when their generated programs are logically unsound, so accuracy alone does not measure mathematical reasoning.","keywords":["code-assisted LLMs","mathematical reasoning","logical soundness","program taxonomy","execution accuracy","decision tree judge","abstract syntax tree","benchmark evaluation"],"falsifier":"Run one model on matched easy/hard variants of the same word problem, with the prompt and demonstrations held fixed and only the numbers made larger, and measure the share of concept-grounded programs; if the hard variant does not shift the distribution toward brute-force and no-logic programs, the paper's difficulty effect is refuted.","tokens_in":16605,"feed_emoji":"🤖","tokens_out":9016,"duration_ms":75299,"temperature":0.7,"pith_summary":"This paper argues that execution accuracy on math benchmarks is a misleading measure of how well code-assisted LLMs actually reason. The authors analyze the Python programs that five LLMs generate as intermediate reasoning steps and classify each program as logically sound (grounded in a mathematical concept) or unsound (relying on brute-force search, memorized numbers, or no real logic at all). They find that models can achieve comparable execution accuracy regardless of whether their programs are sound, and that harder problems push all models toward unsound reasoning. Because soundness is what makes a generated solution verifiable and trustworthy, the paper concludes that code-assisted LLM evaluation must look beyond final-answer correctness.","feed_headline":"LLMs can ace math benchmarks with unsound code","feed_subtitle":"A taxonomy and decision-tree judge show brute-force and memorized programs score as well as concept-based ones.","key_machinery":"The paper's central object is a six-class taxonomy of generated programs — Conceptual, Primitive, From-scratch Implementation, Brute-Force, Disorganized, and No Logic — where the first three are considered logically sound because they ground the solution in a mathematical concept, and the last three are considered unsound. The taxonomy is operationalized by the Code-Structure Judge, a decision tree trained on structural features of each program's abstract syntax tree: counts of function calls, imports, built-in operations, control-flow statements, and variables that are defined-but-unused or used-but-undefined. This judge lets the authors scale a manual line-level annotation to all generated programs and then relate each program's logic class to its execution accuracy.","core_discovery":"The central finding is that execution accuracy is decoupled from the logical soundness of the program an LLM generates. On the grade-school ASDiv set and the competition-level MATH500 set, the authors find that programs classified as Brute-Force or No Logic match the execution accuracy of concept-grounded programs, especially on the harder set. They also find that increasing problem difficulty decreases the share of sound generations for every evaluated model, including capable closed-source ones. To establish this, they build a six-class taxonomy of program logic and a decision-tree classifier, the Code-Structure Judge, that labels programs from abstract syntax tree features with 81% accuracy, outperforming an LLM-as-judge baseline at 73%.","pith_inferences":["Inference: If brute-force search were reclassified as logically sound — a finite exhaustive search is a legitimate algorithm — the headline finding would weaken to a statement about implementation style and efficiency rather than reasoning correctness.","Inference: The ASDiv-versus-MATH500 difficulty comparison is confounded by different prompts, demonstrations, and model subsets; a within-dataset matched-difficulty study could confirm or refute the difficulty effect.","Inference: The same taxonomy could be applied to natural-language chain-of-thought outputs, testing whether accuracy also hides unsound reasoning outside code.","Inference: Because the Code-Structure Judge has low recall on Disorganized programs (0.16), the reported soundness distributions contain non-trivial classification noise; a judge with semantic features would sharpen the quantitative claims."],"forward_implications":["Benchmark accuracy should no longer be read as evidence of mathematical reasoning quality, because unsound programs match sound ones in execution accuracy on hard problems.","As problem difficulty rises, even strong models shift from concept-grounded programs toward brute-force and no-logic solutions, meaning harder benchmarks can specifically reward reasoning hacks.","A correct execution outcome does not certify a trustworthy program; verifiability requires concept-grounded logic, not just the right answer.","Evaluations of code-assisted LLMs should report a soundness measure alongside accuracy, not accuracy alone.","Model capability shapes reasoning style: the more capable evaluated models ground more of their programs in math concepts, while the open-source models rely more on memorized and exhaustive-search approaches."],"supporting_citations":[{"why":"Supplies the program-aided prompting method and the ASDiv prompt that the paper reuses to elicit code solutions.","marker":"Gao et al., 2023"},{"why":"Provides the ASDiv dataset of grade-school math word problems used for the easy-difficulty half of the evaluation.","marker":"Miao et al., 2020"},{"why":"Provides the MATH dataset, whose difficulty characterization motivates the use of MATH500 as the hard benchmark.","marker":"Hendrycks et al., 2021"},{"why":"Source of the MATH500 subset of 500 MATH test problems used for the hard-difficulty comparison.","marker":"Lightman et al., 2023"},{"why":"Supplies the demonstration style adapted for the prompt used on MATH500.","marker":"Gou et al., 2023"},{"why":"Inspires the LLM-Judge baseline, which the paper compares against its Code-Structure Judge.","marker":"Tong and Zhang, 2024"},{"why":"Defines StarCoder2, one of the open-source models whose unsound reasoning patterns anchor the main finding.","marker":"Lozhkov et al., 2024"}],"fun_headline_variants":["Math LLMs ace benchmarks with unsound code","Accuracy alone hides faulty reasoning in LLM math code","Code-assisted LLMs pass math tests using brute force","High scores on math don't mean sound LLM logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central finding hinges on its normative classification of logical soundness as 'grounding the implementation in a math concept,' a standard that labels brute-force search and direct-answer returns as unsound.","fun_headline_variants_meta":{"raw":{"variants":["Math LLMs ace benchmarks with unsound code","Accuracy alone hides faulty reasoning in LLM math code","Code-assisted LLMs pass math tests using brute force","High scores on math don't mean sound LLM logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000546,"raw_usage":{"total_tokens":2582,"prompt_tokens":886,"completion_tokens":1696,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":1633}},"tokens_in":502,"tokens_out":1696,"duration_ms":12318,"temperature":1.0,"reasoning_tokens":1633,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:33:46.451560+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run one model on matched easy/hard variants of the same word problem, with the prompt and demonstrations held fixed and only the numbers made larger, and measure the share of concept-grounded programs; if the hard variant does not shift the distribution toward brute-force and no-logic programs, the paper's difficulty effect is refuted.","supporting_citations":[],"review_version":1}