Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Code Simulation as a Proxy for High-order Tasks in Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that correctly simulating equivalent code is a scalable proxy for assessing core LLM reasoning, and it demonstrates this on five paired benchmarks.

desk verdict A useful, honest evaluation paper whose core proxy claim is plausible but not yet proven; the paired-task evidence is correlational and lacks per-instance checks. read the letter →

arxiv 2502.03568 v3 pith:KMA45QH5 submitted 2025-02-05 cs.LG cs.AI

Emanuele La Malfa , Christoph Weinhuber , Orazio Torre , Fangru Lin , X. Angelo Huang , Samuele Marro , Anthony Cohn , Nigel Shadbolt
show 1 more author
Michael Wooldridge
This is my paper · ORCID
classification cs.LGcs.AI
keywords codesimulationLLMreasoningevaluationsyntheticbenchmarkspairednaturalistictasksmemorisationchainofsortingalgorithmsentitytracking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Many reasoning, planning, and problem-solving tasks are algorithmic: if a model can correctly simulate each step, it can solve the task. This paper argues that code simulation is therefore a scalable proxy for assessing core LLM reasoning, and it builds five pairs of naturalistic and synthetic prompts that encode the same computation. Across GPT-4, GPT-4o, and Llama-3.1-405B, accuracy on the synthetic code tracks accuracy on the naturalistic task as difficulty grows, suggesting that cheaply generated Python programs can stand in for handcrafted reasoning benchmarks. The experiments also show that simulator performance is fragile, dropping sharply when slight algorithmic variations block memorisation, and that a minimal step-by-step 'Chain of Simulation' prompt partially restores it.

What carries the argument

The central object is the paired prompt: each naturalistic reasoning task is translated into an equivalent Python program whose ground-truth output is fixed by an interpreter, so the same question is asked in two surface forms. The pairs are built from programming constructs chosen to mirror cognitive-load building blocks: straight-line programs for sequential tracking, critical paths for ignoring irrelevant instructions, parallel independent paths for tracking multiple variables, nested loops for recurring calculations, and sorting algorithms for ranking. Difficulty is controlled by a single variable—number of instructions or exchanges, nesting depth, or vector length—so accuracy can be plotted against algorithmic load. The other load-bearing component is the Chain of Simulation (CoSm) prompt, which instructs the model to simulate the program instruction by instruction and report the trace, and is used to distinguish genuine execution from memorised or pattern-matched answers.

What would settle it

Take an independently authored naturalistic reasoning task with a code version encoding the same algorithm, vary difficulty, and check whether model accuracy on the two forms tracks each other for held-out instances; if code accuracy stays high while naturalistic accuracy collapses (or vice versa), the proxy claim fails. A second check: feed Bubble Sort inputs with controlled duplicate frequencies—if accuracy no longer drops when duplicates are repeated, the lazy-execution explanation is wrong.

Watch

Extended reading notes

Core claim

The paper's central claim is that 'code simulation is a scalable proxy for assessing some core reasoning capabilities of LLMs,' grounded in the observation that correctly simulating each step is sufficient to solve many high-order tasks. To support this, the authors pair five naturalistic settings—good exchange, critical good exchange, clique exchange, recurring calculation, and ranking objects—with synthetic Python programs that encode the same operations: straight-line code, code with critical paths, parallel paths, nested loops, and sorting routines. They report strong performance correlations between each naturalistic task and its code counterpart as the number of operations, nesting depth, or objects grows, with the naturalistic version usually noisier and sometimes harder. They further document a 'lazy execution regime' in which models such as GPT-3.5-Turbo stop simulating long sorting inputs and instead output the expected sorted sequence, and they show that memorisation, rather than execution, drives accuracy on familiar routines like Fibonacci and ascending Bubble Sort: slight anonymised variations collapse accuracy. A simple prompting extension, Chain of Simulation (CoSm), which explicitly asks the model to report the trace at each iteration, partially offsets these failures.

Load-bearing premise

Each paired naturalistic and synthetic prompt is assumed to be truly equivalent in the computation it requires and in difficulty, so that observed correlations reflect the proxy property rather than the way the pairs were constructed.

Editorial extensions

If this is right

  • Synthetic Python programs can be generated at scale to probe LLM reasoning, reducing the cost of handcrafted naturalistic benchmarks.
  • Performance on a code simulation task can be used to predict performance on the paired naturalistic task, making code a cheap screening tool for model capability.
  • Benchmark designers should anonymise and vary familiar algorithms, because accuracy on well-known routines can reflect memorisation rather than execution.
  • Adding an explicit trace instruction (CoSm) is a lightweight, model-agnostic way to reduce lazy execution and make code-simulation scores more faithful.
  • The failure modes identified—dropping repeated elements and pattern-matching known routines—point to concrete decoding and prompting interventions worth testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not stated in the paper but implied: the proxy will only hold for tasks whose reasoning is fully captured by an algorithm; tasks requiring world knowledge, pragmatics, or ambiguity will not have faithful code equivalents.
  • Because five of the six paired tasks were authored in-house, an independent third-party set of naturalistic–code pairs is needed to rule out that the correlation is an artifact of task construction.
  • The repeated-element sorting failure suggests a concrete testable intervention: decoding with duplicate-aware constraints or presence-penalty tuning should recover the dropped items if the explanation is probabilistic.
  • If the proxy generalizes, the same correlation should appear across programming languages or pseudocode formulations; if it is Python-specific, the mechanism is closer to memorised syntax than to general simulation ability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that code simulation can serve as a scalable proxy for evaluating LLM reasoning on naturalistic tasks. It introduces five paired benchmark families (straight-line/good exchange, critical path/critical good exchange, parallel path/clique exchange, nested loops/recurring calculation, sorting/ranking objects) plus a pairing of an external entity-tracking benchmark with code, and evaluates GPT-4, GPT-4o, and Llama-3.1-405B with Chain of Thought. Additional experiments on pure code simulation across open and closed models identify memorisation and 'lazy execution' as failure modes and propose Chain of Simulation (CoSm) prompting. The central claim is that performance on synthetic code correlates with performance on naturalistic tasks, so synthetic data can complement handcrafted benchmarks.

Significance. If substantiated, the proxy claim is practically valuable: synthetic code generation is cheaper and more scalable than human-authored naturalistic prompts, and the paper's detailed public code and appendices support reproducibility. The paper also makes a useful empirical observation that LLM code execution is fragile and affected by memorisation, with a proposed mitigation (CoSm). However, the central evidence is currently qualitative and correlational: no statistical tests, no per-instance error correspondence, and several divergences that complicate the 'sufficient condition' framing. The contribution is therefore promising but not yet rigorous enough to support the strongest claims.

major comments (4)
  1. [Section 3.1, Figures 3-4 and 8] The central proxy claim rests on the assumption that each synthetic/naturalistic pair encodes the same computation and is of comparable difficulty, but this equivalence is never validated per instance. Five of the six paired tasks were authored in-house for this paper; only the object-tracking pair uses an external source (Kim & Schuster, 2023). The reported evidence consists of aggregate accuracy trends against a control variable, so the observed correlation could be produced by shared surface features (e.g., number of steps, agent-variable mapping, arithmetic operations) rather than by the model executing the same algorithm in both modalities. The paper should report a per-instance measure of correspondence, such as the conditional probability of correct synthetic execution given correct naturalistic performance (or a confusion matrix over instances), and should include at least one held-out or externally sourced pair beyond object tracking.
  2. [Section 4.1 and Appendix A, Figures 4 and 8] Several reported results conflict with the 'sufficient condition' reading of the proxy claim. In the sorting tasks, GPT-4 and GPT-4o find the synthetic condition easier for longer inputs while their naturalistic ranking performance behaves differently (Section 4.1, Figure 4). In the only externally sourced pair, Llama-3.1-405B shows a large synthetic advantage on object tracking (Appendix A, Figure 8). Moreover, the paper itself presents evidence that the object-tracking dataset has been memorised verbatim by GPT models (Appendix A, Figure 9), so this external benchmark cannot serve as a clean validation of the proxy. The authors should either explain these divergences within the proposed framework or explicitly moderate the claim to a task- and model-specific correlation, rather than a general sufficient condition.
  3. [Section 4.1, Figures 3-4] The paper repeatedly describes the relationship between synthetic and naturalistic performance as 'strongly correlated,' but no correlation coefficient, significance test, or confidence interval is reported; the claim is based on visual inspection of trends over a small number of control points. Given that some trends are non-monotonic (e.g., sorting for GPT-4/GPT-4o), the paper should quantify the association (e.g., Spearman's rho with uncertainty) and, ideally, report per-instance agreement between the two conditions. At minimum, standard errors or error bars for the three runs should be shown in Figures 3 and 4.
  4. [Appendix B.2] The manuscript explicitly states that logical-{and,or} instructions are excluded after observing that performance 'considerably drops' for them. This is a post hoc exclusion of an experimental condition and should be presented as a limitation in the main text, with the excluded results shown or linked so readers can assess whether the scope of the proxy claim is narrower than suggested. The abstract speaks of 'common constructs in programming,' which could include logical operations; the paper should clarify that the current evidence covers add/sub/mov and not logical operations.
minor comments (6)
  1. [Section 3, metrics] The formula for Levenshtein-based prediction error, 1/N Σ|ψ(y_i|x,p_i) ∩ Ω(p_i)|, is not standard notation; please define the distance explicitly (e.g., Levenshtein distance divided by length) and avoid the set-intersection notation.
  2. [Section 4.2] The sentence 'We study GPT-3.5-Turbo, GPT-4, and Llama-3-70B and run 3 independent runs of 30 experiments each are shown in Figure 5' is grammatically incomplete; please revise.
  3. [Section 4.1] The claim that the Straight line result 'proves' the synthetic task a faithful proxy is too strong for correlational evidence; suggest 'is consistent with' or 'suggests.'
  4. [Figure 2] The text refers to 'Figure 2 (left)' for both the Straight line/Good exchange and the Critical path tasks; please clarify the panel labels so readers can locate the referenced examples.
  5. [Appendix A.2] The sentence 'which can be found in the code material and in particular in which can be found in the code material' contains a duplicated phrase; please correct.
  6. [Benchmark prompts] The full naturalistic prompts are not included in the appendices; to support the equivalence assumption, the exact prompts (or a representative set) should be made available in the repository or an appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the proxy claim is an empirical generalization from independently measured benchmark scores, not a derivation that reduces to its inputs.

full rationale

The paper does not derive its central claim from an equation whose terms already contain the conclusion. The claim that code simulation is a scalable proxy is supported by measuring LLM accuracy on paired naturalistic and synthetic prompts and comparing the resulting performance curves. Neither side is fitted to the other; no parameter is estimated from the naturalistic data and then renamed as a synthetic prediction. The paired benchmarks do encode the same computations by design, and five of the six pairs are authored in-house, but this is an experimental-design limitation (a potential shared-template confound), not a circular derivation: the observed correlations are not forced, as the paper's own results show (e.g., the ambivalent sorting trends in Section 4.1 and the large object-tracking gap for Llama-3.1-405B in Figure 8). The sufficient-condition statement in the Abstract is an analytic premise about algorithmic tasks, not a derived result. Self-citations such as La Malfa et al. (2024), Lin et al. (2024), and Huang et al. (2024) are historical or contextual and are not used to justify the load-bearing empirical claim; the object-tracking pair is taken from external work (Kim & Schuster, 2023). No uniqueness theorem, imported ansatz, or renamed known result carries the argument. The main risk of the paper is the external validity of hand-crafted pairs, which belongs under correctness risk rather than circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The central claim rests on the assumption that code constructs and naturalistic tasks share algorithmic content, and on Python execution as ground truth. No numbers are fitted to data; the experimental design choices (30 samples, 3 runs, critical path length 5) are controls rather than free parameters.

assumptions (3)
  • domain assumption Naturalistic reasoning tasks of interest can be faithfully represented as Python programs with equivalent algorithmic structure.
    Section 3.1 builds all five paired benchmarks on this premise; if code and English versions differ in difficulty or content, the correlation cannot validate the proxy.
  • domain assumption Python interpreter output is the correct ground truth for both synthetic and naturalistic problems.
    Section 3 defines correctness as matching the interpreter; this assumes the English prompt's intended computation is fully captured by the code translation.
  • domain assumption Correlated performance across the five task families is evidence that code simulation transfers to naturalistic reasoning, rather than shared surface patterns.
    Section 4.1 interprets the trends as proxy evidence; no control for shared vocabulary or memorized templates is applied.
invented entities (1)
  • Lazy execution regime independent evidence
    purpose: Explains why LLMs return correct-looking sorted outputs on long inputs without tracing the algorithm; invoked in Section 4.2 for sorting failures.
    The paper provides indirect evidence in Figure 5: token output drops sharply for long inputs and models output ordered sequences. However the construct is an interpretation of behavior, not a measured quantity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code Simulation as a Proxy for High-order Tasks in Large Language Models." pith.science (2026). https://pith.science/paper/KMA45QH5

@misc{pith2026250203568,
  author       = {Pith},
  title        = {Pith review of: Code Simulation as a Proxy for High-order Tasks in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KMA45QH5}},
  note         = {Machine review of arXiv:2502.03568}
}
read the original abstract

Many reasoning, planning, and problem-solving tasks share an intrinsic algorithmic nature: correctly simulating each step is a sufficient condition to solve them correctly. We collect pairs of naturalistic and synthetic reasoning tasks to assess the capabilities of Large Language Models (LLM). While naturalistic tasks often require careful human handcrafting, we show that synthetic data is, in many cases, a good proxy that is much easier to collect at scale. We leverage common constructs in programming as the counterpart of the building blocks of naturalistic reasoning tasks, such as straight-line programs, code that contains critical paths, and approximate and redundant instructions. We further assess the capabilities of LLMs on sorting problems and repeated operations via sorting algorithms and nested loops. Our synthetic datasets further reveal that while the most powerful LLMs exhibit relatively strong execution capabilities, the process is fragile: it is negatively affected by memorisation and seems to rely heavily on pattern recognition. Our contribution builds upon synthetically testing the reasoning capabilities of LLMs as a scalable complement to handcrafted human-annotated problems.

Figures

Figures reproduced from arXiv: 2502.03568 by the authors.

Figure 1
Figure 1. Left: an example of the naturalistic vs. synthetic good exchange settings. The former describes, in natural language, two agents who exchange goods; the latter is an equivalent formulation in code. While GPT-3.5-Turbo performs better on the synthetic task (a “simulation gap”), performance in the synthetic and naturalistic tasks is strongly correlated with respect to the control variable, i.e., the number of operatio… view at source ↗
Figure 2
Figure 2. On the left, examples of Straight line and Good exchange tasks (top), Critical path and Critical good exchange (middle), and Parallel path and Clique good exchange (bottom). On the right, examples of synthetic and naturalistic Sorting and Ranking objects (top) and Nested loops and Recurring calculation (bottom). III. Parallel paths and Clique good exchange. Another aspect of reasoning that complements the “extraneou… view at source ↗
Figure 3
Figure 3. Top: Accuracy of different models on the Straight line and Good exchange. Middle: Accuracy on the Critical path and Critical good exchange. The critical path length is 5. Bottom: Levenshtein similarity of the ground truth and the prediction on the Parallel path and Clique exchange. The control variable for each problem is the number of operations, that spans from 10 to 50 with granularity 10. 5 [PITH_FULL_IMAGE:fig… view at source ↗
Figures from the paper (17 more)
Figure 4
Figure 4. Figure 4: Top: Accuracy of different models on the Nested loops: the control variable is the max computational complexity of the task (synthetic) or, equivalently, the number of recurring operations (naturalistic). Middle and bottom: Accuracy on the Sorting and Ranking objects t…
Figure 5
Figure 5. Figure 5: GPT-3.5-Turbo on sorting algorithms with varying complexity (O(n 2 ) and O(n log n)), both in their recursive (top) and iterative (bottom) versions. For long inputs, GPT-3.5 switches to a “lazy execution regime” (in magenta, right) where a model no longer simulates but…
Figure 6
Figure 6. Figure 6: Results of GPT-3.5-Turbo, GPT-4 and Llama-3-70B on 50 independent simulations of classic algorithms and their vari￾ations. Top-left: Performance on the vanilla implementation of each algorithm with CoT. Top-right and bottom: Performance on the variations for each model…
Figure 8
Figure 8. Figure 8: Accuracy of different models on the Object tracking task introduced in (Kim & Schuster, 2023). We pair the naturalistic task with an equivalent coding task. A.1. Straight Line and Good Exchange A linguistic analysis of the simulation gap. Across the incorrect solutions…
Figure 9
Figure 9. Figure 9: Memorisation of the Object tracking dataset (Kim & Schuster, 2023). When GPT-3.5-Turbo-Instruct is fed with some inputs from the dataset, it can recover the continuation almost verbatim (highlighted in yellow), a hint the model has memorised the dataset. For this examp…
Figure 11
Figure 11. Figure 11: Accuracy on 3 independent runs of 30 experiments each of different LLMs on code snippets with solely {and, or}, {add, sub} or {mov} instructions. We group results by codes of varying number of instructions (x-axis), namely {1, 10, 30}. B. An Analysis of the Code Simul…
Figure 10
Figure 10. Figure 10: Straight-line code. We first assess the simulation capabilities of different LLMs on code that contains only {add,sub}, {mov}, or logical-{and,or} instructions [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 12
Figure 12. Figure 12: Accuracy and Mean Absolute Error of different LLMs on code of varying length with only {add,sub} and {mov} instructions (out of 3 independent runs of 30 experiments each) [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 14
Figure 14. Figure 14: Accuracy of different LLMs on 3 runs of 30 experiments each on programs with varying critical path lengths, for snippets of 20 and 30 lines of code respectively. a0 = a1 = a2 = 1 a3 = a4 = a5 = -1 a0 -= a1 a3 -= a4 a5 &= a3 a3 |= a5 a0 += a1 a1 -= a3 [PITH_FULL_IMAGE…
Figure 13
Figure 13. Figure 13: Code with critical path. Some sequential problems can be solved without executing all the instruc￾tions in a program. For instance, consider the code in [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 15
Figure 15. Figure 15: On the left, an example of an algorithm to test the approximation capabilities of a model. On the right, the Levenshtein similarity between the ground truth and an LLM’s output measures the performance of different models (the higher, the better) [PITH_FULL_IMAGE:fig…
Figure 16
Figure 16. Figure 16: Left: an example of a fault-tolerant algorithm. We feed an LLM with a few equivalent programs and instruct it to execute all of them to return the same result. Right: how redundancy affects the performances of GPT-3.5-Turbo, GPT-4 and Llama-3-70B on multiple equivalen…
Figure 18
Figure 18. Figure 18: Performances of different LLMs on nested loops with increasing computational complexity. On the right, the number of input tokens per complexity class grows linearly. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_18.png]
Figure 19
Figure 19. Figure 19: On sorting algorithms, the number of input tokens grows linearly (left). At the same time, GPT-4 outputs fewer tokens than Llama-3-70B, especially for complexity larger than O(n 2 ). The number is approximately the same for linear complexity. Both the graphs report th…
Figure 17
Figure 17. Figure 17: Examples of programs with varying com￾putational complexity: on the left, linear (O(n)), on the right, cubic (O(n 3 )). Results in [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 20
Figure 20. Figure 20: On top, results of GPT-4-Turbo with Chain of Thought prompting technique on different sorting algorithms, both in their recursive (top) and iterative (bottom) versions. Differently from GPT-3.5-Turbo, GPT-4 forces a model to simulate a routine and does not suffer from…
Figure 21
Figure 21. Figure 21: Results of Llama-3-70B with CoT prompting technique on different sorting algorithms, both in their recursive (top) and iterative (bottom) versions. Llama-3-70B is not a good simulator for sorting algorithms and, in general, does not understand the underlying task is s…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Large Language Models Generalize Procedures Across Representations?

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Post-training on graph or code versions of a planning task does not transfer to natural-language versions, but a symbolic-then-natural-language RL curriculum achieves strong transfer.

Reference graph

Works this paper leans on

56 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    a is b" fail to learn

    Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., and Evans, O. The reversal curse: Llms trained on" a is b" fail to learn" b is a". arXiv preprint arXiv:2309.12288, 2023. URL https://arxiv.org/pdf/2309.12288.pdf

    arXiv 2023

  3. [3]

    S., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raf, E

    Biderman, S., Prashanth, U. S., Sutawika, L., Schoelkopf, H., Anthony, Q., Purohit, S., and Raf, E. Emergent and predictable memorization in large language models. arXiv preprint arXiv:2304.11158, 2023

    arXiv 2023

  4. [4]

    Evaluating large language models with runtime behavior of program execution, 2024 a

    Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., and Xia, X. Evaluating large language models with runtime behavior of program execution, 2024 a

    work page 2024

  5. [5]

    Can language models pretend solvers? logic code simulation with llms, 2024 b

    Chen, M., Li, G., Wu, L.-I., Liu, R., Su, Y., Chang, X., and Xue, J. Can language models pretend solvers? logic code simulation with llms, 2024 b

    work page 2024

  6. [6]

    Training verifiers to solve math word problems, 2021

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    arXiv 2021

  7. [7]

    Position information in transformers: An overview

    Dufter, P., Schmitt, M., and Sch \"u tze, H. Position information in transformers: An overview. Computational Linguistics, 48 0 (3): 0 733--763, 2022

    work page 2022

  8. [8]

    L., Jian, L., Lin, B

    Dziri, N., Lu, X., Sclar, M., Li, X. L., Jian, L., Lin, B. Y., West, P., Bhagavatula, C., Bras, R. L., Hwang, J. D., et al. Faith and fate: Limits of transformers on compositionality. arXiv preprint arXiv:2305.18654, 2023

    arXiv 2023

Show all 56 references
  1. [9]

    and Russinovich, M

    Eldan, R. and Russinovich, M. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023. URL https://www.thetalkingmachines.com/sites/default/files/2023-10/2310.02238.pdf

    2023 arXiv

  2. [10]

    C., and Berner, J

    Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., and Berner, J. Mathematical capabilities of ChatGPT . ArXiv preprint, abs/2301.13867, 2023. URL https://arxiv.org/abs/2301.13867

    2023 arXiv

  3. [11]

    The pile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  4. [12]

    D., and Papailiopoulos, D

    Giannou, A., Rajput, S., Sohn, J.-y., Lee, K., Lee, J. D., and Papailiopoulos, D. Looped transformers as programmable computers. arXiv preprint arXiv:2301.13196, 2023

    2023 arXiv

  5. [13]

    The llama 3 herd of models, 2024

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., and et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  6. [14]

    Gu, A., Rozière, B., Leather, H., Solar-Lezama, A., Synnaeve, G., and Wang, S. I. Cruxeval: A benchmark for code reasoning, understanding and execution, 2024

    2024

  7. [15]

    J., Wang, Z., Wang, D

    Hao, S., Gu, Y., Ma, H., Hong, J. J., Wang, Z., Wang, D. Z., and Hu, Z. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023. URL https://arxiv.org/abs/2305.14992

    2023 arXiv

  8. [16]

    Large language models for software engineering: A systematic literature review

    Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., and Wang, H. Large language models for software engineering: A systematic literature review. arXiv preprint arXiv:2308.10620, 2023

    2023 arXiv

  9. [17]

    A., La Malfa, E., Marro, S., Asperti, A., Cohn, A., and Wooldridge, M

    Huang, X. A., La Malfa, E., Marro, S., Asperti, A., Cohn, A., and Wooldridge, M. A notion of complexity for theory of mind via discrete world models. arXiv preprint arXiv:2406.11911, 2024

    2024 arXiv

  10. [18]

    Gpt is becoming a turing machine: Here are some ways to program it

    Jojic, A., Wang, Z., and Jojic, N. Gpt is becoming a turing machine: Here are some ways to program it. arXiv preprint arXiv:2303.14310, 2023

    2023 arXiv

  11. [19]

    L., Kim, G., Choi, Y., and Sap, M

    Kim, H., Sclar, M., Zhou, X., Bras, R. L., Kim, G., Choi, Y., and Sap, M. Fantom: A benchmark for stress-testing machine theory of mind in interactions, 2023. URL https://arxiv.org/abs/2310.15421

    2023 arXiv

  12. [20]

    and Schuster, S

    Kim, N. and Schuster, S. Entity tracking in language models, 2023. URL https://arxiv.org/abs/2305.02363

    2023 arXiv

  13. [21]

    G., Shadbolt, N., and Wooldridge, M

    La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Cohn, A. G., Shadbolt, N., and Wooldridge, M. The arrt of language-models-as-a-service: Overview of a new paradigm and its challenges. arXiv preprint arXiv:2309.16573, 2023. URL https://arxiv.org/pdf/2309.16573.pdf

    2023 arXiv

  14. [22]

    Code simulation challenges for large language models

    La Malfa, E., Weinhuber, C., Torre, O., Lin, F., Cohn, A., Shadbolt, N., and Wooldridge, M. Code simulation challenges for large language models. arXiv preprint arXiv:2401.09074, 2024

    2024 arXiv

  15. [23]

    Code as policies: Language model programs for embodied control

    Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 9493--9500. IEEE, 2023

    2023

  16. [24]

    L., Hofmann, V., Yang, E

    Lin, F., Malfa, E. L., Hofmann, V., Yang, E. M., Cohn, A., and Pierrehumbert, J. B. Graph-enhanced large language models in asynchronous plan reasoning, 2024

    2024

  17. [25]

    Code execution with pre-trained language models

    Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., and Duan, N. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05383, 2023

    2023 arXiv

  18. [26]

    D., Ibrahimzada, A

    Liu, C., Zhang, S. D., Ibrahimzada, A. R., and Jabbarvand, R. Codemind: A framework to challenge large language models for code reasoning, 2024

    2024

  19. [27]

    Large language models as code executors: An exploratory study, 2024

    Lyu, C., Yan, L., Xing, R., Li, W., Samih, Y., Ji, T., and Wang, L. Large language models as code executors: An exploratory study, 2024. URL https://arxiv.org/abs/2410.06667

    2024 arXiv

  20. [28]

    T., Smolensky, P., Linzen, T., Gao, J., and Celikyilmaz, A

    McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., and Celikyilmaz, A. How much do language models copy from their training data? E valuating linguistic novelty in text generation using RAVEN . Transactions of the Association for Computational Linguistics, 11: 0 652--670, 2023 ...

    2023

  21. [29]

    T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T

    McCoy, R. T., Yao, S., Friedman, D., Hardy, M., and Griffiths, T. L. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.13638, 2023 b . URL https://arxiv.org/pdf/2309.13638.pdf

    2023 arXiv

  22. [30]

    J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al

    Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021

    2021 arXiv

  23. [31]

    In-context learning and induction heads

    Olsson, C., Elhage, N., Nanda, N., et al. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html

    2022

  24. [32]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. ArXiv preprint, abs/2303.08774, 2023. URL https://arxiv.org/abs/2303.08774

    2023 arXiv

  25. [33]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 27730--27744, 2022

    2022

  26. [34]

    Attention is turing complete

    P \'e rez, J., Barcel \'o , P., and Marinkovic, J. Attention is turing complete. The Journal of Machine Learning Research, 22 0 (1): 0 3463--3497, 2021

    2021

  27. [35]

    Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. M. A., and Botvinick, M. Machine theory of mind. In Dy, J. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.\ 4218-...

    2018

  28. [36]

    E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al

    Rozi \`e re, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al. Code Llama : Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. URL https://arxiv.org/abs/2308.12950

    2023 arXiv

  29. [37]

    A., Prasad, P., and Becker, B

    Santos, E. A., Prasad, P., and Becker, B. A. Always provide context: The effects of code context on programming error message enhancement. In Proceedings of the ACM Conference on Global Computing Education Vol 1, pp.\ 147--153, 2023

    2023

  30. [38]

    L., Fan, A., Akiki, C., Pavlick, E., Ili \'c , S., Hesslow, D., Castagn \'e , R., Luccioni, A

    Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ili \'c , S., Hesslow, D., Castagn \'e , R., Luccioni, A. S., Yvon, F., Gall \'e , M., et al. BLOOM : A 176B -parameter open-access multilingual language model. ArXiv preprint, abs/2211.05100, 2022. URL https://arxiv.org/abs/2211.05100

    2022 arXiv

  31. [39]

    Memory augmented large language models are computationally universal

    Schuurmans, D. Memory augmented large language models are computationally universal. arXiv preprint arXiv:2301.04589, 2023

    2023 arXiv

  32. [40]

    Detecting pretraining data from large language models

    Shi, W., Ajith, A., Xia, M., Huang, Y., Liu, D., Blevins, T., Chen, D., and Zettlemoyer, L. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789, 2023. URL https://arxiv.org/pdf/2310.16789.pdf

    2023 arXiv

  33. [41]

    and Chandler, P

    Sweller, J. and Chandler, P. Evidence for cognitive load theory. Cognition and instruction, 8 0 (4): 0 351--362, 1991

    1991

  34. [42]

    Predicting code coverage without execution

    Tufano, M., Chandel, S., Agarwal, A., Sundaresan, N., and Clement, C. Predicting code coverage without execution. arXiv preprint arXiv:2307.13383, 2023

    2023 arXiv

  35. [43]

    Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting

    Turpin, M., Michael, J., Perez, E., and Bowman, S. Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36, 2024

    2024

  36. [44]

    Vaithilingam, P., Zhang, T., and Glassman, E. L. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In CHI '22: CHI Conference on Human Factors in Computing Systems - 5 May 2022, Extended Abstracts , pp.\ 332:1--332:...

    2022

  37. [45]

    J., and Lu, H

    Webb, T., Holyoak, K. J., and Lu, H. Emergent analogical reasoning in large language models, 2023

    2023

  38. [46]

    Statistically meaningful approximation: a case study on approximating turing machines with transformers

    Wei, C., Chen, Y., and Ma, T. Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems, 35: 0 12071--12083, 2022 a . URL https://proceedings.neurips.cc/paper_files/paper/2022/file...

    2022

  39. [47]

    Chain of thought prompting elicits reasoning in large language models

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. ArXiv preprint, abs/2201.11903, 2022 b . URL https://arxiv.org/abs/2201.11903

    2022 arXiv

  40. [48]

    Thinking like transformers

    Weiss, G., Goldberg, Y., and Yahav, E. Thinking like transformers. In International Conference on Machine Learning, pp.\ 11080--11090. PMLR, 2021. URL https://proceedings.mlr.press/v139/weiss21a/weiss21a.pdf

    2021

  41. [49]

    what it can create, it may not understand

    West, P., Lu, X., Dziri, N., Brahman, F., Li, L., Hwang, J. D., Jiang, L., Fisher, J., Ravichander, A., Chandu, K., et al. The generative ai paradox:" what it can create, it may not understand". arXiv preprint arXiv:2311.00059, 2023

    2023 arXiv

  42. [50]

    and Treude, C

    Widjojo, P. and Treude, C. Addressing compiler errors: Stack overflow or large language models? arXiv preprint arXiv:2307.10793, 2023

    2023 arXiv

  43. [51]

    What do code models memorize? an empirical study on large language models of code

    Yang, Z., Zhao, Z., Wang, C., Shi, J., Kim, D., Han, D., and Lo, D. What do code models memorize? an empirical study on large language models of code. arXiv preprint arXiv:2308.09932, 2023

    2023 arXiv

  44. [52]

    How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015, 2023

    Yuan, Z., Yuan, H., Tan, C., Wang, W., and Huang, S. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015, 2023. URL https://arxiv.org/pdf/2304.02015.pdf

    2023 arXiv

  45. [53]

    Large language models meet nl2code: A survey

    Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Yongji, W., and Lou, J.-G. Large language models meet nl2code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 7443--7464, 2023

    2023

  46. [54]

    Transformer-based models are not yet perfect at learning to emulate structural recursion, 2024

    Zhang, D., Tigges, C., Zhang, Z., Biderman, S., Raginsky, M., and Ringer, T. Transformer-based models are not yet perfect at learning to emulate structural recursion, 2024

    2024

  47. [55]

    D., Tigges, C., Biderman, S., Raginsky, M., and Ringer, T

    Zhang, S. D., Tigges, C., Biderman, S., Raginsky, M., and Ringer, T. Can transformers learn to solve problems recursively? arXiv preprint arXiv:2305.14699, 2023. URL https://arxiv.org/pdf/2305.14699.pdf

    2023 arXiv

  48. [56]

    What algorithms can transformers learn? a study in length generalization

    Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., and Nakkiran, P. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.