Pith. sign in

REVIEW 3 major objections 7 minor 50 references

RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By mutating benchmark problems in an executable symbolic form, RE-IMAGINE shows that every tested LLM's accuracy falls as reasoning demands rise, indicating that high benchmark scores partly reflect memorization.

desk verdict Useful, reusable benchmark-synthesis pipeline with a real Level-3 confound: the accuracy drops are solid, but the 'statistical recall' interpretation outruns the evidence. read the letter →

arxiv 2506.15455 v1 pith:TPYHH5VB submitted 2025-06-18 cs.CL cs.AI

Xinnuo Xu , Rachel Lawrence , Kshitij Dubey , Atharva Pandey , Risa Ueno , Fabian Falck , Aditya V. Nori , Rahul Sharma
show 2 more authors
Amit Sharma Javier Gonzalez
This is my paper · ORCID
classification cs.CLcs.AI
keywords RE-IMAGINEreasoningevaluationbenchmarksynthesisstatisticalrecallmemorizationladderofcausationcounterfactuallargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RE-IMAGINE is a framework for deciding whether large language models solve reasoning benchmarks by reasoning or by recalling training data. The paper's claim is that translating benchmark questions into executable Python code, mutating that code, and translating the mutation back into natural language yields arbitrarily many new problems whose answers can be computed exactly and cannot be memorized. Tested on four benchmarks — GSM8K (math), CLadder (causal reasoning), CRUXEval and Loop (code) — every model in the GPT, Llama, and Phi families loses accuracy on the mutated test sets as problems move from unchanged (level 1, observe) to superficially altered (level 2, mutate) to logically amended (level 3, imagine). If correct, this means high scores on standard benchmarks overstate genuine reasoning, and the gap can be measured at scale rather than with hand-crafted probe questions.

What carries the argument

The machinery is the executable symbolic representation: each natural-language problem is converted into a Python snippet whose computational graph has variables as nodes and dependencies as edges. Six graph mutations are defined — SampleValues and UselessInfo at level 2, AddDependence, InsertConditional, CounterFactual, and Bi-CounterFactual at level 3 — and because the mutated code still runs, the ground-truth answer to every new problem is obtained by executing it, which guarantees correctness without manual annotation. Mutations are admitted only after a back-translation pass in which a second LLM rewrites the mutated question into code that must reproduce the execution answer, and a human review of 50 examples per mutation type found 0–6.7% invalid. For benchmarks that are already programs (CRUXEval and Loop), the mutation is applied directly to the code, with outputs checked by execution and by an automated program verifier respectively; the Loop mutations are restricted to level 2 because changing program values can make loop invariants cease to exist. The computational graph also plays the role of a structural causal model, which lets the Bi-CounterFactual mutation evaluate models with counterfactual metrics (probability of necessity and sufficiency, summarized as N-IR and S-IR).

What would settle it

Re-run the evaluation on mutations whose natural-language translations have been independently certified by multiple human raters as equivalent to the code, and look for any tested model whose accuracy stays flat or rises from level 1 to level 3; the paper's claim predicts a monotone drop across all four benchmarks for every model, so a single family with flat accuracy on certified mutations would undercut the universal reliance-on-recall conclusion. The paper's own reported invalid rates (up to 6.7% of reviewed examples) make this the natural decisive check.

Watch

Extended reading notes

Core claim

The paper's central claim is that part of LLMs' success on reasoning benchmarks is statistical recall of training data, and that this can be demonstrated and quantified with a three-level hierarchy modeled on Pearl's ladder of causation. Level 1 ('observe') is the original benchmark. Level 2 ('mutate') keeps the problem's logical core intact while changing surface content: new numbers, added irrelevant information, renamed variables, or small code-level edits. Level 3 ('imagine') rewrites the logic: an added dependency, a conditional branch, or a counterfactual assumption appended to the question, so that solving requires revising a previously stated fact. The framework renders each question as an executable Python snippet, applies one of six mutations to the snippet's computational graph, and translates the mutated code back to natural language with a second LLM, so every generated problem has an exact execution-computed answer. Across GSM8K, CLadder, CRUXEval, and Loop, overall performance degrades as ladder level increases, the decline persists when the number of reasoning steps is held constant, and composing mutations widens it — evidence, the paper argues, that the mutations themselves, not added complexity, reveal the reliance on recall.

Load-bearing premise

The whole measurement depends on each mutated natural-language question being a faithful, unambiguous rendering of the mutated code — a correspondence validated only by execution-matching, back-translation, and a 50-example human review per mutation type, so if the translations are systematically awkward or subtly wrong, the accuracy drops would be artifacts rather than evidence about reasoning.

Editorial extensions

If this is right

  • Raw benchmark scores should be reported together with scores on level-2 and level-3 variants, since the gap between them estimates how much of the original score came from recall rather than reasoning.
  • Any claim that a new model 'reasons' becomes testable at scale: the pipeline can generate an arbitrary number of unseen variants at each level, making memorization an insufficient route to a high score.
  • Level-3 questions — new logic, revisions, and counterfactuals — remain largely unsolved even by the strongest tested models, which identifies the next target for training and evaluation.
  • Composed mutations (for example, new values plus a counterfactual) enlarge the accuracy gap, so combining mutation types is an immediate way to build harder evaluation sets.
  • The pattern is cross-domain: the same hierarchy and pipeline apply to math, causal reasoning, and code, so the observed reliance on statistical recall is not specific to one benchmark format.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the same pipeline that produces evaluation variants also produces execution-verified training pairs, so level-2 and level-3 mutations could be used to fine-tune models on exactly the skills the hierarchy measures — a use the paper does not pursue.
  • My inference: the hierarchy implies a testable prediction — models trained with program-execution or causal supervision should show a smaller level-1-to-level-3 gap, and comparing such models with standard pretraining would separate training-data effects from architectural limits.
  • My inference: if the drops are genuine, leaderboard comparisons are partly confounded by test-set leakage, and a normalized 'reasoning gap' (level-1 accuracy minus level-3 accuracy) could become a standard complement to raw accuracy in future evaluations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces RE-IMAGINE, a framework that classifies LLM reasoning evaluation into three levels (Observe, Mutate, Imagine) inspired by Pearl's ladder of causation, and an automated pipeline that converts benchmark questions into executable symbolic (Python) representations, applies code-level mutations, and translates the mutated code back into natural language. The authors apply the pipeline to GSM8K, CLadder, CRUXEval, and Loop, and report that model accuracy declines monotonically from Level-1 to Level-2 to Level-3 across these benchmarks. They interpret this decline as evidence that current LLM performance on reasoning benchmarks reflects, at least in part, statistical recall of training data rather than robust reasoning. The paper includes multiple validity checks: execution-based ground truth, back-translation verification for GSM8K, Frama-C verification for Loop, human review of 50 examples per GSM8K mutation, matched factual accuracy for CRUXEval, and a reasoning-step control in Section 6.1.

Significance. If the central claim is supported, the paper makes a useful contribution: it provides a scalable, cross-benchmark methodology for generating benchmark variations that are not answerable by memorizing the original training answers, and it quantifies a hierarchy of reasoning difficulty. The paper's strengths include an automated pipeline with executable ground truth, conservative upper-bound adjustments for invalid GSM8K mutations, matched factual baselines for CRUXEval, Frama-C verification for Loop, and statistical accuracy over 10 random test-set samples. The three-level hierarchy is a clean organizing principle that could influence future benchmark design. However, the strength of the central interpretation—that the observed accuracy drops indicate statistical recall—depends on isolating the reasoning level from task-format confounds, which the current experiments do not fully establish.

major comments (3)
  1. [§6.1 and §4/§5.1] Section 6.1 controls for the number of code operations, but this does not isolate the reasoning level from surface-form difficulty. In GSM8K, Level-3 mutations append a 'Suppose/Assume ... instead' sentence that requires the model to detect a contradiction and override a previously stated fact; in CRUXEval, Level-3 mutations require the model to apply a diff to the original code. Both add parsing and instruction-following load beyond the single code operation controlled in §6.1. To support the claim that the L1→L2→L3 accuracy declines reflect the reasoning hierarchy and statistical recall, the authors should add a control that introduces the same added surface complexity (e.g., a 'Suppose' sentence or a diff-like format) without changing the underlying logic, and show that accuracy does not drop under that control.
  2. [§5.2, Appendix H, and Conclusion] The Loop benchmark contains only Level-1 and Level-2 mutations; Appendix H states, 'we limit ourselves to a category of Level-2 mutations that add useless information.' The abstract and conclusion, however, claim a consistent decline 'across all evaluated benchmarks' as mutations move from Level-1 to Level-2 to Level-3. This overstates the evidence: no Level-3 loop task is evaluated. The authors should either soften the cross-benchmark claim to reflect the absence of Level-3 Loop data or add Level-3 variants (e.g., counterfactual initializations for which a valid invariant still exists) to make the claim testable on all four benchmarks.
  3. [§4.1 and Figure 16] The back-translation check (Figure 16) verifies only that GPT-4o can find one code interpretation of the mutated natural language; it does not show that the mutated question is unambiguous or that the evaluated models parse it in the intended way. The human review of 50 examples per mutation type, with reported invalid rates of 0.00%–6.67%, is too small to rule out systematic ambiguity or awkward phrasing that could depress accuracy independently of reasoning ability. The authors should provide larger validity samples with confidence intervals, or filter the test set using stricter checks such as multiple independent back-translations or consistency across paraphrases.
minor comments (7)
  1. [Table 3] The row for 'Llama 3.3 (70B)' lists '#Parameters' as '8B', which appears to be a typo; it should be '70B'.
  2. [Figure 2] The hashed blocks above the accuracy bars represent the percentage of invalid mutated examples; the text should explicitly state that this percentage is added to the accuracy to form the upper bound, and clarify whether the percentage is computed over the same test-set size used for accuracy.
  3. [§4.3] The text says 50 questions are taken 'from the validation set of the benchmark,' while the rest of the paper refers to the GSM8K test set; please clarify which split is used and why.
  4. [Figure 7 and Appendix E.5] The aggregated accuracy in Figure 7 is averaged over all tested models; the authors should specify whether this is a macro-average or micro-average and report the number of models included.
  5. [§3.2] The distinction between SampleValues and CounterFactual is described only by how the change is presented in natural language; the authors should state explicitly that both mutations can affect the same nodes in the computational graph and that the difference lies in the presentation as an assumption statement.
  6. [Appendix F.2.2] The mutation name 'Unrelate-dIrrelevantInfo' contains a hyphen artifact; it should be 'UnrelatedIrrelevantInfo' consistently.
  7. [§4.1] The paper reports human-validated invalid rates for GSM8K mutations but does not report the coverage rate of the GSM8K transformation (i.e., the fraction of original problems for which all constants align with the code); a coverage statistic analogous to that in Appendix G.3 would help assess selection bias.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the measured declines are empirical, anchored to external benchmarks and executed code ground truth; author-overlap citations are used as external metrics/benchmarks, not as load-bearing derivations.

full rationale

The paper's derivation chain is: take existing external benchmarks (GSM8K, CLadder, CRUXEval, Loop); convert questions to Python where needed; apply explicitly defined code mutations; compute ground truth by executing the mutated code; translate mutations back to natural language with back-translation and human checks; then measure LLM accuracy. None of the headline results is a fitted parameter renamed as a prediction. The L1/L2/L3 ordering is a taxonomy that says what the levels mean, but it does not by itself force the observed accuracy decline; the decline is measured and could in principle have been flat or reversed. The 'not solvable by memorization alone' property of value-sampled mutations follows from construction, but the paper's empirical conclusion about model behavior is not equivalent to that construction. The self-citations (Gonzalez & Nori 2024; Huyuk et al. 2024; Kamath et al. 2024) overlap with the present authors, but they supply standard causal metrics (PN, PS, N-IR, S-IR) and an external code benchmark whose verification is checked by Frama-C with Z3/alt-ergo/CVC4; the central 'statistical recall' claim does not rest solely on those citations. The skeptic's points about GPT-4o translation ambiguity, small human review samples, and extra diff-application complexity in Level-3 CRUXEval are validity/confound concerns rather than cases where an output variable is identical to an input by definition. Accordingly, there is no exhibited circular reduction and the score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central claim rests on two families of assumptions: symbolic code representations and automated ground truth are faithful to the original and mutated questions, and the observed accuracy drops are caused by the mutations rather than by translation artifacts or generic distribution shift. Hand-chosen design parameters affect the difficulty of each mutation but are not fitted to any prediction target. No physical entities are introduced; the hierarchy is a conceptual construct with no independent falsifiable handle beyond the paper itself.

free parameters (3)
  • SampleValues sampling distribution for GSM8K = discrete uniform for integers; uniform centered on original value for floats
    Hand-chosen mutation ranges control how far Level-2 variants move from the original problem. Set in Section 4.1 without a stated difficulty-calibration principle.
  • CRUXEval MutateValue perturbation bound = ±10
    Chosen by hand; small perturbations are intended to keep mutated code difficulty close to the original while changing the output. Appears in Section 5.1 and Table 6.
  • Loop junk-variable count and polynomial identities = two junk variables; identically zero polynomials
    The number of added variables and the algebraic identities are hand-designed to be semantically invisible while syntactically distracting. Appears in Section 5.2 and Table 7.
assumptions (5)
  • domain assumption Python snippets in OpenMathInstruct faithfully represent the reasoning in GSM8K questions
    Filtering checks that execution matches ground truth and constants align with the text, but semantic equivalence between natural language and code is not fully proven. Invoked in Section 4.1.
  • domain assumption GPT-4o symbolic-to-NL translations produce unambiguous questions equivalent to the code mutation
    Validity is checked by back-translation and by 50 human examples per mutation type, not by exhaustive review. Invoked in Section 4.1 and Figures 16 and 17.
  • domain assumption Accuracy drops on mutated questions are attributable to statistical recall rather than generic distribution shift or question ambiguity
    The paper's main interpretation; no control condition separates memorization from other robustness effects. Stated in Sections 4.2 through 5.3 and the conclusion.
  • standard math Frama-C with SMT solvers reliably verifies loop invariants
    Trusted verification toolchain; used as the success criterion for Loop tasks. Described in Section 5 and Appendix H.
  • domain assumption Computational graph mutations map onto Pearl's ladder of causation levels
    Appendix C acknowledges the mapping is partial and that not all mutations have causal counterparts. Invoked in Sections 2 and 3.
invented entities (2)
  • RE-IMAGINE three-level reasoning hierarchy (Observe, Mutate, Imagine)
    purpose: Classifies benchmark problems and their variations into reasoning levels claimed to be increasingly hard for LLMs
    Conceptual construct without an external benchmark or falsifiable prediction beyond the paper's own experiments; Appendix C notes the causal mapping is incomplete.
  • Bi-Counterfactual binary computational graph abstraction
    purpose: Binarizes GSM8K computation graphs to compute necessity and sufficiency inconsistency rates
    An internal abstraction applied to only 50 validation questions; no independent dataset validates the abstraction. Appears in Section 4.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation." pith.science (2026). https://pith.science/paper/TPYHH5VB

@misc{pith2026250615455,
  author       = {Pith},
  title        = {Pith review of: RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TPYHH5VB}},
  note         = {Machine review of arXiv:2506.15455}
}
read the original abstract

Recent Large Language Models (LLMs) have reported high accuracy on reasoning benchmarks. However, it is still unclear whether the observed results arise from true reasoning or from statistical recall of the training set. Inspired by the ladder of causation (Pearl, 2009) and its three levels (associations, interventions and counterfactuals), this paper introduces RE-IMAGINE, a framework to characterize a hierarchy of reasoning ability in LLMs, alongside an automated pipeline to generate problem variations at different levels of the hierarchy. By altering problems in an intermediate symbolic representation, RE-IMAGINE generates arbitrarily many problems that are not solvable using memorization alone. Moreover, the framework is general and can work across reasoning domains, including math, code, and logic. We demonstrate our framework on four widely-used benchmarks to evaluate several families of LLMs, and observe reductions in performance when the models are queried with problem variations. These assessments indicate a degree of reliance on statistical recall for past performance, and open the door to further research targeting skills across the reasoning hierarchy.

Figures

Figures reproduced from arXiv: 2506.15455 by the authors.

Figure 1
Figure 1. Top: The benchmark transformation pipeline (Section 3) outlined with an example from GSM8K (Cobbe et al., 2021). This pipeline leverages the symbolic representation of the question (a Python snippet form) to automatically transform a math QA problem (leftmost) into a similar format with additional reasoning steps (rightmost). Bottom: To clearly define the mutations, we transform the symbolic representation into a co… view at source ↗
Figure 2
Figure 2. GSM8K results summary: model accuracy on numerical answer predictions across test set variations in different reasoning levels (see Section 3.2 and [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Sufficiency/necessity inconsistency rates (S-IR/N-IR) on GSM8K factual/counterfactual test set. Models located near the bottom-left corner are thought to predict the causal relationship between the cause and effect, i.e. sufficient and necessary, in a way that is consistent with the true causal relationship, as defined by the ground truth. and sufficiency inconsistency rates (N-IR, S-IR) introduced in Huy¨ uk et al.… view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Left: An example from CLadder test set and its Counter￾Factual mutation. Right: CLadder accuracy of the best-performing model in each family on causal question answering across test set variations at different reasoning levels. The full results can be found in [PITH_F…
Figure 6
Figure 6. Figure 6: Left: An example from Loop dataset and its Read Orig￾inal mutation. Right: Loop concise results – accuracy of the best-performing model in each family. Full results can be found in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: We analyze model performance across varying numbers of reasoning steps in the GSM8K benchmark. The x-axis rep￾resents the number of reasoning steps, defined as the number of operations in the code solution. The plot aggregates the perfor￾mance of all tested models. Det…
Figure 9
Figure 9. Figure 9: In addition to the models discussed in the main body of the paper, we also report results for several popular models on GSM8K, CruxEval, and Loop. The conclusions drawn in the main paper are found to generalize well to these additional models. E. Appendix: GSM8K Detail…
Figure 10
Figure 10. Figure 10: The statistical accuracy of models on GSM8K numerical answer predictions is evaluated across different mutated test sets at varying reasoning levels (see Section 3.2 and [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: GSM8K: Detailed accuracy for all models in the Phi, Llama, and GPT families. All models are prompted with 7 mutated in-context examples. Check [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: GSM8K: Detailed accuracy for all models in the Phi, Llama, and GPT families. All models are prompted with 7 paired raw and the corresponding mutated in-context examples. Check [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: GSM8K: Detailed accuracy for all models in the Phi, Llama, and GPT families. All models are prompted with 8 original in-context examples. However, with the exception of SampleValues, all other mutations are combined with SampleValues to create even more challenging te…
Figure 14
Figure 14. Figure 14: GSM8K: Visualization comparing model performance on test set variations with one or two mutations. We summarize model performance across two dimensions: the average accuracy across all six test sets, as in the left bar plot (x-axis), and the standard deviation in accu…
Figure 15
Figure 15. Figure 15: GSM8K: We analyze model performance across varying numbers of reasoning steps. We define the difficulty of a numerical reasoning question by the number of calculation steps (operations) in its code solution. For each model, we compute the average accuracy across examp…
Figure 16
Figure 16. Figure 16: Prompts used in GSM8K quality control. We prompt the LLM to back-translate the mutated math problem into Python by modifying the original question’s Python solution. The generated code must produce an execution result that matches the ground truth answer of the mutate…
Figure 17
Figure 17. Figure 17: Prompts used in GSM8K mutated symbolic representation to natural language. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: Prompts used in GSM8K testing. The prompt contains 8 raw in-context examples. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Prompts used in GSM8K testing. The prompt contains 7 in-context examples demonstrating different types of mutations. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_19.png]
Figure 20
Figure 20. Figure 20: Prompts used in GSM8K testing. The prompt contains 7 in-context examples. Each example contains a raw question and answer, with the corresponding mutated question and chain-of-thought fo the mutated question, with the final answer. The 7 examples are demonstrating dif…
Figure 21
Figure 21. Figure 21: CLadder: Detailed accuracy for models in the Phi, Llama, and GPT families. F.3. Prompts for mutated questions Below are illustration of NL prompts for our mutations on CLadder. We highlight parts of the original question in red, and the mutated natural language in blu…
Figure 22
Figure 22. Figure 22: Mutation Coverage Statistics (as a percentage of total CRUXEval data) G.4. Evaluation and Matched Factual Accuracy We plot the accuracy on the factual and mutated CRUXEval benchmark for ten language models in [PITH_FULL_IMAGE:figures/full_fig_p028_22.png]
Figure 23
Figure 23. Figure 23: CRUXEval: Detailed accuracy for models in the Phi, Llama, and GPT families. Because the coverage statistics for each mutation fall well below 100%, each mutation is tested on only a subset of the total CRUXEval benchmark problems. We therefore consider the possibility…
Figure 24
Figure 24. Figure 24: CRUXEval: Detailed accuracy and matched factual accuracy for models in the Phi, Llama, and GPT families. We observe that the overall trend of decreasing accuracy with Level-2 and Level-3 mutations is still evident with respect to matched accuracy scores. Nonetheless, …
Figure 25
Figure 25. Figure 25: Evaluation of various models on the Loop dataset and its mutated versions. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_25.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 22 canonical work pages

  1. [1]

    R., Bury, G., and de Oliveira, S

    Ait El Hara, H. R., Bury, G., and de Oliveira, S. Alt-Ergo-Fuzz: A fuzzer for the Alt-Ergo SMT solver . In Keller, C. and Bourke, T. (eds.), Journ \'e es Francophones des Langages Applicatifs , pp.\ 235--244, Saint-M \'e dard-d'Excideuil, France, June 2022. URL https://inria.hal.science/hal-03626861

    work page 2022

  2. [2]

    W., Conway, C

    Barrett, C. W., Conway, C. L., Deters, M., Hadarean, L., Jovanovic, D., King, T., Reynolds, A., and Tinelli, C. CVC4 . In Gopalakrishnan, G. and Qadeer, S. (eds.), Computer Aided Verification - 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14-20, 2011. Proceedings , volume 6806 of Lecture Notes in Computer Science, pp.\ 171--177. Spring...

    doi:10.1007/978-3-642-22110-1 2011

  3. [3]

    B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020

    arXiv 2005

  4. [5]

    Arc prize 2024: Technical report, 2025

    Chollet, F., Knoop, M., Kamradt, G., and Landers, B. Arc prize 2024: Technical report, 2025. URL https://arxiv.org/abs/2412.04604

    arXiv 2024

  5. [6]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Hilton, J., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  6. [7]

    Frama-C User Manual

    Correnson, L., Cuoq, P., Kirchner, F., Maroneze, A., Prevosto, V., Puccetti, A., Signoles, J., and Yakobowski, B. Frama-C User Manual. URL http://frama-c.com/download/frama-c-user-manual.pdf

    work page

  7. [8]

    and Bj rner, N

    de Moura, L. and Bj rner, N. Z3: An efficient smt solver. In Ramakrishnan, C. R. and Rehof, J. (eds.), Tools and Algorithms for the Construction and Analysis of Systems, pp.\ 337--340, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-78800-3

    work page 2008

  8. [9]

    Pal: Program-aided language models

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. Pal: Program-aided language models. In International Conference on Machine Learning, pp.\ 10764--10799. PMLR, 2023

    2023

Show all 50 references
  1. [10]

    Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., de Oliveira Santos, E., Järviniemi, O., Barnett, M., Sandler, R., Vrzala, M., Sevilla, J., Ren, Q., Pratt, E., Levine, L., Barkley, G., Stewart, N., Grechuk, B., G...

    2024 arXiv

  2. [11]

    and Nori, A

    González, J. and Nori, A. V. Does reasoning emerge? examining the probabilities of causation in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  3. [12]

    Gu, A., Rozi \`e re, B., Leather, H., Solar-Lezama, A., Synnaeve, G., and Wang, S. I. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024

    2024 arXiv

  4. [13]

    Halpern, J. Y. and Pearl, J. Causes and explanations: A structural-model approach. Part I : Causes. The British journal for the Philosophy of Science, 2005

    2005

  5. [14]

    Measuring massive multitask language understanding

    Hendrycks, D., Basart, S., Mazeika, M., Zou, A., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  6. [15]

    Holyoak, K. J. and Morrison, R. G. (eds.). The Cambridge Handbook of Thinking and Reasoning. Cambridge University Press, Cambridge, England, 2005. ISBN 9780521824170

    2005

  7. [16]

    and Chang, K

    Huang, J. and Chang, K. C.-C. Towards reasoning in large language models: A survey. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 1049--1065, Toronto, Canada, jul 2023. Association for Computat...

    2023

  8. [17]

    V., and González, J

    Hüyük, A., Xu, X., Maasch, J., Nori, A. V., and González, J. Reasoning elicitation in language models via counterfactual feedback, 2024. URL https://arxiv.org/abs/2410.03767

    2024 arXiv

  9. [18]

    Openai o1 system card

    Jaech, A., Kalai, A., Lerer, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  10. [19]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  11. [20]

    CL adder: A ssessing causal reasoning in language models

    Jin, Z., Chen, Y., Leeb, F., Gresele, L., Kamal, O., Lyu, Z., Blin, K., Gonzalez, F., Kleiman-Weiner, M., Sachan, M., and Sch \" o lkopf, B. CL adder: A ssessing causal reasoning in language models. In NeurIPS, 2023 a . URL https://openreview.net/forum?id=e2wtjx0Yqu

    2023

  12. [21]

    Cladder: Assessing causal reasoning in language models

    Jin, Z., Chen, Y., Leeb, F., Gresele, L., Kamal, O., Lyu, Z., Blin, K., Gonzalez Adauto, F., Kleiman-Weiner, M., Sachan, M., et al. Cladder: Assessing causal reasoning in language models. Advances in Neural Information Processing Systems, 36: 0 31038--31065, 2023 b

    2023

  13. [22]

    K., Lal, A., Rastogi, A., Roy, S., and Sharma, R

    Kamath, A., Mohammed, N., Senthilnathan, A., Chakraborty, S., Deligiannis, P., Lahiri, S. K., Lal, A., Rastogi, A., Roy, S., and Sharma, R. Leveraging llms for program verification. In FMCAD , 2024. URL http://hdl.handle.net/20.500.12708/200783

    2024

  14. [23]

    S., Iwafuchi, T., and Dasaka, S

    Kambhampati, R. S., Iwafuchi, T., and Dasaka, S. Phi-3: A family of small open models. Microsoft AI Research, 2024. Pre-release information

    2024

  15. [24]

    and Mitchell, M

    Lewis, M. and Mitchell, M. Evaluating the robustness of analogical reasoning in large language models. arXiv preprint arXiv:2411.14215, 2024

    2024 arXiv

  16. [25]

    Neuro-symbolic data generation for math reasoning, 2024

    Li, Z., Zhou, Z., Yao, Y., Li, Y.-F., Cao, C., Yang, F., Zhang, X., and Ma, X. Neuro-symbolic data generation for math reasoning, 2024. URL https://arxiv.org/abs/2412.04857

    2024 arXiv

  17. [26]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024

    2024 arXiv

  18. [27]

    and Krakauer, D

    Mitchell, M. and Krakauer, D. C. The debate over understanding in ai’s large language models. Proceedings of the National Academy of Sciences, 120 0 (13): 0 1--15, 2023. doi:10.1073/pnas.2215907120

    2023 doi

  19. [28]

    Neuberg, L. G. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000. Econometric Theory, 19 0 (4): 0 675--685, 2003

    2000

  20. [29]

    Early access for safety testing, December 2024

    OpenAI . Early access for safety testing, December 2024. https://openai.com/index/early-access-for-safety-testing/

    2024

  21. [30]

    Causality: Models, Reasoning, and Inference

    Pearl, J. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, England, 2nd edition, 2009. ISBN 9780521895606

    2009

  22. [31]

    Pearl, J. et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress, 19 0 (2): 0 3, 2000

    2000

  23. [32]

    H., Sch \"a rli, N., and Zhou, D

    Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Sch \"a rli, N., and Zhou, D. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp.\ 31210--31227. PMLR, 2023

    2023

  24. [33]

    Learning loop invariants for program verification

    Si, X., Dai, H., Raghothaman, M., Naik, M., and Song, L. Learning loop invariants for program verification. In NeurIPS 2018 , 2018

    2018

  25. [34]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models

    Srivastava, A., Wei, J., Jun, H., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022

    2022 arXiv

  26. [35]

    Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap

    Srivastava, S., PV, A., Menon, S., Sukumar, A., Philipose, A., Prince, S., Thomas, S., et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402.19450, 2024

    2024 arXiv

  27. [36]

    Team, D. A. Deepseek-r1: A comprehensive reasoning model. DeepSeek AI Research, 2025. URL https://github.com/deepseek-ai/DeepSeek-R1. Available on GitHub

    2025

  28. [37]

    Openmathinstruct-1: A 1.8 million math instruction tuning dataset

    Toshniwal, S., Moshkov, I., Narenthiran, S., Gitman, D., Jia, F., and Gitman, I. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv: Arxiv-2402.10176, 2024

    2024 arXiv

  29. [38]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Roziere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  30. [39]

    Large language models still can't plan (a benchmark for llms on planning and reasoning about change)

    Valmeekam, K., Olmo, A., Sreedharan, S., and Kambhampati, S. Large language models still can't plan (a benchmark for llms on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022

    2022

  31. [40]

    Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537, 2019

    1905 arXiv

  32. [41]

    S., and Momennejad, I

    Webb, T., Mondal, S. S., and Momennejad, I. Improving planning with large language models: A modular agentic architecture. arXiv preprint arXiv:2310.00194, 2024

    2024

  33. [42]

    Ethical and social risks of harm from language models

    Weidinger, L., Mellor, J., Rauh, M., et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021

    2021 arXiv

  34. [43]

    W., and Narodytska, N

    Wu, H., Barrett, C. W., and Narodytska, N. Lemur: Integrating large language models in automated program verification. In ICLR , 2024

    2024

  35. [44]

    Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks

    Wu, Z., Qiu, L., Ross, A., Aky \"u rek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv preprint arXiv:2307.02477, 2023

    2023 arXiv

  36. [45]

    Towards large reasoning models: A survey on scaling llm reasoning capabilities

    Xu, F., Hao, Q., Zong, Z., Wang, J., Zhang, Y., Wang, J., Lan, X., Gong, J., Ouyang, T., Meng, F., et al. Towards large reasoning models: A survey on scaling llm reasoning capabilities. arXiv preprint arXiv:2501.09686, 2025

    2025 arXiv

  37. [46]

    A critical review of causal reasoning benchmarks for large language models

    Yang, L., Shirvaikar, V., Clivio, O., and Falck, F. A critical review of causal reasoning benchmarks for large language models. In AAAI 2024 Workshop on''Are Large Language Models Simply Causal Parrots?'', 2024

    2024

  38. [47]

    Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process

    Ye, T., Xu, Z., Li, Y., and Allen-Zhu , Z. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process . In Proceedings of the 13th International Conference on Learning Representations, ICLR '25, April 2025. Full version available at https://ssrn.c...

    2025

  39. [48]

    Darg: Dynamic evaluation of large language models via adaptive reasoning graph, 2024

    Zhang, Z., Chen, J., and Yang, D. Darg: Dynamic evaluation of large language models via adaptive reasoning graph, 2024

    2024

  40. [49]

    X., Chen, X., Lin, Y., Wen, J.-R., and Han, J

    Zhou, K., Zhu, Y., Chen, Z., Chen, W., Zhao, W. X., Chen, X., Lin, Y., Wen, J.-R., and Han, J. Don't make your llm an evaluation benchmark cheater. arXiv preprint arXiv:2311.01964, 2023

    2023 arXiv

  41. [50]

    Z., Yang, D., and Xie, X

    Zhu, K., Chen, J., Wang, J., Gong, N. Z., Yang, D., and Xie, X. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2023

    2023

  42. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.