{"id":"a20a04f5-bad8-4e15-b0a6-e2ddee61d3c1","arxiv_id":"2506.15455","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Language models score substantially lower on automatically generated variations of standard math, logic, and code benchmarks, indicating partial reliance on memorization.","lead":"A Microsoft Research team built an automated pipeline that rewrites existing AI reasoning problems into new versions, and found that language models solve the rewritten versions much worse than the originals. It gives evaluators a practical way to separate memorized benchmark answers from genuine reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The L1→L2→L3 accuracy drops are real, but they do not uniquely support 'statistical recall': the Level-3 format adds parsing and diff-application complexity beyond the code-operation count controlled in §6.1, so a surface-form matched control is missing.","rationale":"The reader identifies natural-language fidelity as the weakest assumption. My concern is related but broader: even when the natural-language translation is faithful, the Level-3 format systematically adds linguistic and task-execution complexity that is not controlled for, so the accuracy drops do not uniquely establish statistical recall. The code benchmarks (CRUXEval and Loop) are a genuine strength because they avoid NL translation entirely and still show declines, which supports the framework's empirical core. However, the Level-3 presentation in CRUXEval as 'original code plus a diff' still introduces an extra task component, and the §6.1 control in GSM8K matches only the number of arithmetic operations in the mutated code, not the added sentence parsing or the need to resolve a contradiction in the natural language. Appendix C itself concedes that the mapping to Pearl's ladder is partial, which further weakens the strong causal interpretation. The proposed surface-form control directly tests whether the format confound explains the observed drops. If the control fails, the central interpretive claim would need to be softened, but the benchmark synthesis framework and the reported accuracy reductions would remain valuable. The reader's CONDITIONAL verdict already makes release of code/data and stronger distribution-shift controls a condition of acceptance, so my concern does not change the verdict; it sharpens the specific control that should be required.","tokens_in":29934,"tokens_out":10269,"duration_ms":109068,"concrete_test":"Run a matched surface-form control: for every Level-3 mutation in GSM8K and CRUXEval, generate a control question with the same length, GPT-4o generation pipeline, and sentence or diff structure as the Level-3 mutation, but where the added sentence or diff leaves the computational logic and answer unchanged (e.g., a restatement or an irrelevant but plausible note, rather than a logic-altering assumption). Evaluate the same models on the original, control, and Level-3 sets. If the control set shows accuracy drops comparable to the Level-3 drops, the observed decline is a task-format or distribution-shift artifact rather than evidence for the reasoning hierarchy or statistical recall. This also addresses the possibility that the back-translated code check passed because GPT-4o found one favorable interpretation of an ambiguous sentence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the monotonic accuracy decline across Level-1, Level-2, and Level-3 reflects the reasoning hierarchy and, in particular, that the drops indicate reliance on statistical recall. The evidence does not yet isolate the reasoning level from the task format. In GSM8K, Level-3 mutations append a GPT-4o-generated 'Suppose/Assume ... instead' sentence and require the solver to override a previously stated fact. Compared with originals, these mutations add not only one code operation (the only variable controlled in §6.1) but also a new sentence to parse, a contradiction to detect, and a surface form systematically different from the original benchmark text. The back-translation check (Figure 16) verifies only that GPT-4o can find one code interpretation of the mutated natural language; it does not show that the natural language is unambiguous or that a typical model would parse it as intended. The reported human review of 50 examples per mutation type is too small to rule out systematic ambiguity, and the same confound appears in CRUXEval, where the Level-3 version is the original code plus a diff (Appendix G.2): applying a diff is an additional task component, so the comparison is not matched. A model could fail on Level-3 because instructions are harder to follow or because the input distribution shifted, without any reliance on memorized training answers. Thus 'a degree of reliance on statistical recall' is one possible explanation, but it is not the uniquely supported one.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RE-IMAGINE, a framework that classifies LLM reasoning evaluation into three levels (Observe, Mutate, Imagine) inspired by Pearl's ladder of causation, and an automated pipeline that converts benchmark questions into executable symbolic (Python) representations, applies code-level mutations, and translates the mutated code back into natural language. The authors apply the pipeline to GSM8K, CLadder, CRUXEval, and Loop, and report that model accuracy declines monotonically from Level-1 to Level-2 to Level-3 across these benchmarks. They interpret this decline as evidence that current LLM performance on reasoning benchmarks reflects, at least in part, statistical recall of training data rather than robust reasoning. The paper includes multiple validity checks: execution-based ground truth, back-translation verification for GSM8K, Frama-C verification for Loop, human review of 50 examples per GSM8K mutation, matched factual accuracy for CRUXEval, and a reasoning-step control in Section 6.1.","tokens_in":30264,"tokens_out":3444,"duration_ms":35125,"significance":"If the central claim is supported, the paper makes a useful contribution: it provides a scalable, cross-benchmark methodology for generating benchmark variations that are not answerable by memorizing the original training answers, and it quantifies a hierarchy of reasoning difficulty. The paper's strengths include an automated pipeline with executable ground truth, conservative upper-bound adjustments for invalid GSM8K mutations, matched factual baselines for CRUXEval, Frama-C verification for Loop, and statistical accuracy over 10 random test-set samples. The three-level hierarchy is a clean organizing principle that could influence future benchmark design. However, the strength of the central interpretation—that the observed accuracy drops indicate statistical recall—depends on isolating the reasoning level from task-format confounds, which the current experiments do not fully establish.","major_comments":[{"comment":"Section 6.1 controls for the number of code operations, but this does not isolate the reasoning level from surface-form difficulty. In GSM8K, Level-3 mutations append a 'Suppose/Assume ... instead' sentence that requires the model to detect a contradiction and override a previously stated fact; in CRUXEval, Level-3 mutations require the model to apply a diff to the original code. Both add parsing and instruction-following load beyond the single code operation controlled in §6.1. To support the claim that the L1→L2→L3 accuracy declines reflect the reasoning hierarchy and statistical recall, the authors should add a control that introduces the same added surface complexity (e.g., a 'Suppose' sentence or a diff-like format) without changing the underlying logic, and show that accuracy does not drop under that control.","section":"§6.1 and §4/§5.1"},{"comment":"The Loop benchmark contains only Level-1 and Level-2 mutations; Appendix H states, 'we limit ourselves to a category of Level-2 mutations that add useless information.' The abstract and conclusion, however, claim a consistent decline 'across all evaluated benchmarks' as mutations move from Level-1 to Level-2 to Level-3. This overstates the evidence: no Level-3 loop task is evaluated. The authors should either soften the cross-benchmark claim to reflect the absence of Level-3 Loop data or add Level-3 variants (e.g., counterfactual initializations for which a valid invariant still exists) to make the claim testable on all four benchmarks.","section":"§5.2, Appendix H, and Conclusion"},{"comment":"The back-translation check (Figure 16) verifies only that GPT-4o can find one code interpretation of the mutated natural language; it does not show that the mutated question is unambiguous or that the evaluated models parse it in the intended way. The human review of 50 examples per mutation type, with reported invalid rates of 0.00%–6.67%, is too small to rule out systematic ambiguity or awkward phrasing that could depress accuracy independently of reasoning ability. The authors should provide larger validity samples with confidence intervals, or filter the test set using stricter checks such as multiple independent back-translations or consistency across paraphrases.","section":"§4.1 and Figure 16"}],"minor_comments":[{"comment":"The row for 'Llama 3.3 (70B)' lists '#Parameters' as '8B', which appears to be a typo; it should be '70B'.","section":"Table 3"},{"comment":"The hashed blocks above the accuracy bars represent the percentage of invalid mutated examples; the text should explicitly state that this percentage is added to the accuracy to form the upper bound, and clarify whether the percentage is computed over the same test-set size used for accuracy.","section":"Figure 2"},{"comment":"The text says 50 questions are taken 'from the validation set of the benchmark,' while the rest of the paper refers to the GSM8K test set; please clarify which split is used and why.","section":"§4.3"},{"comment":"The aggregated accuracy in Figure 7 is averaged over all tested models; the authors should specify whether this is a macro-average or micro-average and report the number of models included.","section":"Figure 7 and Appendix E.5"},{"comment":"The distinction between SampleValues and CounterFactual is described only by how the change is presented in natural language; the authors should state explicitly that both mutations can affect the same nodes in the computational graph and that the difference lies in the presentation as an assumption statement.","section":"§3.2"},{"comment":"The mutation name 'Unrelate-dIrrelevantInfo' contains a hyphen artifact; it should be 'UnrelatedIrrelevantInfo' consistently.","section":"Appendix F.2.2"},{"comment":"The paper reports human-validated invalid rates for GSM8K mutations but does not report the coverage rate of the GSM8K transformation (i.e., the fraction of original problems for which all constants align with the code); a coverage statistic analogous to that in Appendix G.3 would help assess selection bias.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central interpretation leans on the authors' own prior frameworks (the Bi-Counterfactual metrics and the Loop benchmark), which is acceptable but should be made more explicit in the framing. The citation of 'Kambhampati et al. (2024)' for Phi-3 appears to be incorrect; the authors should verify this reference. Overall, the pipeline and evaluation are solid, but the main claim about statistical recall needs an additional control for surface-form confounds before it can be regarded as established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid engineering contribution to benchmark synthesis, not a decisive proof that LLMs rely on statistical recall. The three-level hierarchy is a useful organizing device, and the pipeline genuinely generalizes across math, causal QA, and code. I would send it to peer review, but the headline interpretation needs a matched control before it can carry the weight the paper puts on it.\n\nWhat is actually new: prior work had value replacement, irrelevant-info injection, counterfactual feedback, and graph-based dynamic evaluation. This paper unifies those under a single Observe-Mutate-Imagine ladder and, more importantly, shows one automated symbolic pipeline spanning four benchmarks, including code tasks where ground truth is executable. The execution-based verification, Frama-C checking for Loop, matched factual accuracy for CRUXEval, and the reasoning-step ablation in Section 6.1 are all real strengths. The invalid-example upper-bound treatment is conservative and correctly signed.\n\nWhere it is soft: the stress-test note is right. Level-3 mutations in GSM8K add not just one code operation but also a new sentence to parse, a contradiction to detect, and a surface form that is systematically different from the original benchmark text. Back-translation only proves GPT-4o can find one code interpretation; it does not show the mutated natural language is unambiguous or that typical models parse it as intended. Fifty human-reviewed examples per mutation is too small to rule out systematic awkwardness. The same confound appears in CRUXEval, where Level-3 is original code plus a diff, so applying the diff is an extra task component. A model can fail Level-3 because instructions are harder to follow or because the input distribution shifted, without relying on memorized training answers. So the monotonic drop is real, but 'statistical recall' is one explanation, not the uniquely supported one.\n\nMinor caveats: the GSM8K subset is filtered to code-answerable problems, which limits breadth but does not break the comparison; Loop only exercises Level-2, so the code-domain evidence for Level-3 rests entirely on CRUXEval; no artifact is released, which is a practical nuisance but not a scientific flaw.\n\nWho this is for: LLM evaluators and anyone building robustness benchmarks. The framework and adapters are worth building on even if the causal interpretation is dialed back. A serious referee should engage with it, mainly to push for a surface-form-matched control and a larger human validation sample.","headline":"Useful, reusable benchmark-synthesis pipeline with a real Level-3 confound: the accuracy drops are solid, but the 'statistical recall' interpretation outruns the evidence.","tokens_in":30820,"tokens_out":1470,"would_cite":true,"duration_ms":17505,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By mutating benchmark problems in an executable symbolic form, RE-IMAGINE shows that every tested LLM's accuracy falls as reasoning demands rise, indicating that high benchmark scores partly reflect memorization.","keywords":["RE-IMAGINE","reasoning evaluation","benchmark synthesis","statistical recall","memorization","ladder of causation","counterfactual reasoning","large language models"],"falsifier":"Re-run the evaluation on mutations whose natural-language translations have been independently certified by multiple human raters as equivalent to the code, and look for any tested model whose accuracy stays flat or rises from level 1 to level 3; the paper's claim predicts a monotone drop across all four benchmarks for every model, so a single family with flat accuracy on certified mutations would undercut the universal reliance-on-recall conclusion. The paper's own reported invalid rates (up to 6.7% of reviewed examples) make this the natural decisive check.","tokens_in":29742,"feed_emoji":"🧠","tokens_out":13153,"duration_ms":111261,"temperature":0.7,"pith_summary":"RE-IMAGINE is a framework for deciding whether large language models solve reasoning benchmarks by reasoning or by recalling training data. The paper's claim is that translating benchmark questions into executable Python code, mutating that code, and translating the mutation back into natural language yields arbitrarily many new problems whose answers can be computed exactly and cannot be memorized. Tested on four benchmarks — GSM8K (math), CLadder (causal reasoning), CRUXEval and Loop (code) — every model in the GPT, Llama, and Phi families loses accuracy on the mutated test sets as problems move from unchanged (level 1, observe) to superficially altered (level 2, mutate) to logically amended (level 3, imagine). If correct, this means high scores on standard benchmarks overstate genuine reasoning, and the gap can be measured at scale rather than with hand-crafted probe questions.","feed_headline":"LLMs lose accuracy on mutated benchmarks memorization can't solve","feed_subtitle":"Across math, logical, and code tasks, scores fall from original to mutated to imagined questions.","key_machinery":"The machinery is the executable symbolic representation: each natural-language problem is converted into a Python snippet whose computational graph has variables as nodes and dependencies as edges. Six graph mutations are defined — SampleValues and UselessInfo at level 2, AddDependence, InsertConditional, CounterFactual, and Bi-CounterFactual at level 3 — and because the mutated code still runs, the ground-truth answer to every new problem is obtained by executing it, which guarantees correctness without manual annotation. Mutations are admitted only after a back-translation pass in which a second LLM rewrites the mutated question into code that must reproduce the execution answer, and a human review of 50 examples per mutation type found 0–6.7% invalid. For benchmarks that are already programs (CRUXEval and Loop), the mutation is applied directly to the code, with outputs checked by execution and by an automated program verifier respectively; the Loop mutations are restricted to level 2 because changing program values can make loop invariants cease to exist. The computational graph also plays the role of a structural causal model, which lets the Bi-CounterFactual mutation evaluate models with counterfactual metrics (probability of necessity and sufficiency, summarized as N-IR and S-IR).","core_discovery":"The paper's central claim is that part of LLMs' success on reasoning benchmarks is statistical recall of training data, and that this can be demonstrated and quantified with a three-level hierarchy modeled on Pearl's ladder of causation. Level 1 ('observe') is the original benchmark. Level 2 ('mutate') keeps the problem's logical core intact while changing surface content: new numbers, added irrelevant information, renamed variables, or small code-level edits. Level 3 ('imagine') rewrites the logic: an added dependency, a conditional branch, or a counterfactual assumption appended to the question, so that solving requires revising a previously stated fact. The framework renders each question as an executable Python snippet, applies one of six mutations to the snippet's computational graph, and translates the mutated code back to natural language with a second LLM, so every generated problem has an exact execution-computed answer. Across GSM8K, CLadder, CRUXEval, and Loop, overall performance degrades as ladder level increases, the decline persists when the number of reasoning steps is held constant, and composing mutations widens it — evidence, the paper argues, that the mutations themselves, not added complexity, reveal the reliance on recall.","pith_inferences":["My inference: the same pipeline that produces evaluation variants also produces execution-verified training pairs, so level-2 and level-3 mutations could be used to fine-tune models on exactly the skills the hierarchy measures — a use the paper does not pursue.","My inference: the hierarchy implies a testable prediction — models trained with program-execution or causal supervision should show a smaller level-1-to-level-3 gap, and comparing such models with standard pretraining would separate training-data effects from architectural limits.","My inference: if the drops are genuine, leaderboard comparisons are partly confounded by test-set leakage, and a normalized 'reasoning gap' (level-1 accuracy minus level-3 accuracy) could become a standard complement to raw accuracy in future evaluations."],"forward_implications":["Raw benchmark scores should be reported together with scores on level-2 and level-3 variants, since the gap between them estimates how much of the original score came from recall rather than reasoning.","Any claim that a new model 'reasons' becomes testable at scale: the pipeline can generate an arbitrary number of unseen variants at each level, making memorization an insufficient route to a high score.","Level-3 questions — new logic, revisions, and counterfactuals — remain largely unsolved even by the strongest tested models, which identifies the next target for training and evaluation.","Composed mutations (for example, new values plus a counterfactual) enlarge the accuracy gap, so combining mutation types is an immediate way to build harder evaluation sets.","The pattern is cross-domain: the same hierarchy and pipeline apply to math, causal reasoning, and code, so the observed reliance on statistical recall is not specific to one benchmark format."],"supporting_citations":[{"why":"Supplies the ladder of causation (association, intervention, counterfactual) that the paper's three-level observe–mutate–imagine hierarchy is modeled on.","marker":"(Pearl, 2009)"},{"why":"OpenMathInstruct's validated GSM8K question–Python-solution pairs supply the initial symbolic representations for the math pipeline.","marker":"(Toshniwal et al., 2024)"},{"why":"GSM8K is the primary math benchmark whose questions are converted to code and then mutated.","marker":"(Cobbe et al., 2021)"},{"why":"Earlier value and template mutations that RE-IMAGINE generalizes; the Level-2 SampleValues mutation is its direct continuation.","marker":"(Mirzadeh et al., 2024)"},{"why":"CLadder supplies the causal QA benchmark whose parser and causal engine produce executable ground truth for the mutations.","marker":"(Jin et al., 2023a)"},{"why":"CRUXEval supplies the code-understanding benchmark (800 Python functions) that is mutated directly at the code level.","marker":"(Gu et al., 2024)"},{"why":"Loop supplies the loop-invariant tasks, the 250-item evaluation subset, and the prompting and verification setup reused for the code mutations.","marker":"(Kamath et al., 2024)"},{"why":"Frames counterfactual LLM evaluation with probability of necessity and sufficiency, the basis for the Bi-CounterFactual mutation.","marker":"(González & Nori, 2024)"},{"why":"Supplies the N-IR and S-IR inconsistency rates used to score models on bi-counterfactual questions.","marker":"(Hüyük et al., 2024)"},{"why":"Supplies the reasoning-step complexity measure used in the ablation that separates mutation difficulty from added reasoning steps.","marker":"(Ye et al., 2025)"}],"fun_headline_variants":["LLMs' reasoning scores drop when benchmark logic is mutated","Three-level 'reasoning ladder' exposes LLMs' recall crutch","Symbolic mutations reveal LLMs lean on memorized answers","Benchmark mutations slash LLM scores, exposing recall reliance","RE-IMAGINE test: LLMs rely on memorized patterns, not logic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole measurement depends on each mutated natural-language question being a faithful, unambiguous rendering of the mutated code — a correspondence validated only by execution-matching, back-translation, and a 50-example human review per mutation type, so if the translations are systematically awkward or subtly wrong, the accuracy drops would be artifacts rather than evidence about reasoning.","fun_headline_variants_meta":{"raw":{"variants":["LLMs' reasoning scores drop when benchmark logic is mutated","Three-level 'reasoning ladder' exposes LLMs' recall crutch","Symbolic mutations reveal LLMs lean on memorized answers","Benchmark mutations slash LLM scores, exposing recall reliance","RE-IMAGINE test: LLMs rely on memorized patterns, not logic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2805,"prompt_tokens":970,"completion_tokens":1835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":1745}},"tokens_in":586,"tokens_out":1835,"duration_ms":11570,"temperature":1.0,"reasoning_tokens":1745,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:34:22.270511+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the evaluation on mutations whose natural-language translations have been independently certified by multiple human raters as equivalent to the code, and look for any tested model whose accuracy stays flat or rises from level 1 to level 3; the paper's claim predicts a monotone drop across all four benchmarks for every model, so a single family with flat accuracy on certified mutations would undercut the universal reliance-on-recall conclusion. The paper's own reported invalid rates (up to 6.7% of reviewed examples) make this the natural decisive check.","supporting_citations":[{"cited_title":"Causality: Models, Reasoning, and Inference","cited_arxiv_id":null,"evidence_quote":"Supplies the ladder of causation (association, intervention, counterfactual) that the paper's three-level observe–mutate–imagine hierarchy is modeled on."},{"cited_title":"K., Lal, A., Rastogi, A., Roy, S., and Sharma, R","cited_arxiv_id":null,"evidence_quote":"Loop supplies the loop-invariant tasks, the 250-item evaluation subset, and the prompting and verification setup reused for the code mutations."}],"review_version":2}