Pith. sign in

REVIEW 3 major objections 4 minor 9 cited by

CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training models to predict code inputs and outputs in natural language improves reasoning across symbolic, math, logic, and commonsense benchmarks.

desk verdict Solid empirical data-recipe paper with a credible main result; the 'reasoning primitives in code' mechanism is underdetermined by the teacher-distillation confound. read the letter →

arxiv 2502.07316 v4 pith:JA3QDJGV submitted 2025-02-11 cs.CL cs.AI

Junlong Li , Daya Guo , Dejian Yang , Runxin Xu , Yu Wu , Junxian He This is my paper · ORCID
classification cs.CLcs.AI
keywords codeinput-outputpredictionchain-of-thoughtreasoningsynthetictrainingdataprimitivesinstructiontuningexecutionverificationmulti-turnrevisionLLMpost-training
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CodeI/O claims that real-world code is a dense and scalable source of universal reasoning patterns that can be decoupled from programming syntax. The method converts raw code into executable functions with textual queries, then trains models to predict either the output for a given input or a feasible input for a given output, writing all reasoning as natural-language chain-of-thought. On four base models from 7B to 30B and 14 benchmarks spanning symbolic, scientific, logic, math, and commonsense reasoning, training on 3.5M such samples consistently improves scores relative to instruction tuning alone and to larger alternative datasets. The authors argue the gains come from repeated exposure to structured reasoning primitives rather than from data size or from imitating the synthesis model, and they support this with ablations, a synthesis-model comparison, and leakage checks.

What carries the argument

The central machinery is the code-to-I/O transformation pipeline, which refactors raw code into a cleaned reference function, a main entrypoint with JSON-serializable inputs and outputs, a rule-based input generator, and a problem query. The training task itself—given a function and query, predict the output for a supplied input or a feasible input for a supplied output, with responses written entirely as natural-language chain-of-thought—is what separates the reasoning flow from code syntax. Because every prediction can be verified by executing the code, the pipeline also produces execution feedback, which CodeI/O++ feeds back into a second generation turn to revise incorrect rationales.

What would settle it

Re-train on code-derived samples whose CoT rationales are replaced by execution traces or by rationales written in an artificial language (so no natural-language reasoning style is distilled), and check whether the cross-domain gains persist; if the improvements vanish, the transfer comes from the synthesis model's reasoning style, not from code-embedded primitives. Alternatively, train on functions whose logic is deliberately randomized but still well-formed, and show that symbolic and logic gains disappear.

Watch

Extended reading notes

Core claim

CodeI/O is a data-construction pipeline that turns raw code files into a unified format: a cleaned reference function, a main entrypoint with input/output descriptions, a rule-based input generator, and a textual query. Inputs are sampled from the generator and executed to obtain ground-truth outputs, and a strong open model writes free-form chain-of-thought rationales that either derive the output from a given input or construct a feasible input from a given output. Training on the resulting 3.5M samples as a separate first stage before general instruction tuning yields higher average scores and more balanced improvements than instruction tuning alone or than matching-size samples from WebInstruct, OpenMathInstruct2, OpenCoder-SFT-Stage1, and raw PythonEdu code. CodeI/O++ appends execution-verification feedback to initially wrong predictions and asks the model to revise, converting roughly a tenth of errors into correct answers and improving average scores further. The paper's central claim is that predicting code inputs and outputs in natural language exposes models to transferable reasoning primitives—logic flow planning, state-space searching, decision-tree traversal, modular decomposition—that generalize beyond code.

Load-bearing premise

The pipeline assumes DeepSeek-V2.5's rewrites and generated rationales faithfully preserve the original code's reasoning content; if the refactored functions, queries, or CoTs are noisy or substantively altered, the measured gains could come from distilling the synthesis model's reasoning style rather than from universal reasoning primitives embedded in code.

Editorial extensions

If this is right

  • Adding a code-I/O prediction stage before instruction tuning is a transferable post-training recipe: it lifts average scores on nearly every evaluated benchmark, not just code ones, across four base models of different architectures and sizes.
  • The data is highly scalable: any executable function with an input sampler yields verified I/O pairs, so the 3.5M-sample corpus can be expanded by adding more code sources and more test cases per function.
  • Keeping incorrect predictions and revising them with execution feedback helps more than discarding them: rejection sampling and replacing wrong answers with ground-truth outputs both reduce average performance.
  • Two-stage training (CodeI/O first, instruction tuning second) beats single-stage mixing, although the optimal mixing ratio in the second stage is model-dependent.
  • The performance gains are not explained by 13-gram leakage: accuracy gains persist on non-leaked subsets of the two benchmarks with the highest overlap ratios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the decoupling claim is right, the same I/O-prediction format should transfer to other executable artifacts—SQL queries, spreadsheet formulas, state-machine specifications—extending the approach beyond Python code.
  • The multi-turn revision results suggest the synthesis model's self-correction saturates after one turn; pairing the pipeline with a stronger or more diverse synthesis model, or with reinforcement learning on verified I/O, is a natural next step.
  • The model-dependent benefit of mixing instruction data in stage two implies that a fixed two-stage recipe is not optimal; tuning the mixture per base model could yield further gains.
  • A sharper test of the code-embedded reasoning primitives claim would be to train on functions whose internal logic is randomized but well-formed: if symbolic and logic gains collapse, the signal is structural reasoning rather than code semantics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CODE I/O, a two-stage training approach that first fine-tunes a base LLM on 3.5M synthetic input/output prediction tasks derived from Python code files, with natural-language chain-of-thought rationales generated by DeepSeek-V2.5, before standard instruction tuning. A variant, CODE I/O++, adds multi-turn revision of incorrect predictions using execution feedback. The authors evaluate on 14 reasoning benchmarks across four base models and report higher average scores than instruction tuning alone and than several alternative first-stage datasets (WebInstruct, OpenMathInstruct-2, OpenCoder-SFT, Python-Edu). They also present ablations on input vs. output prediction, rejection sampling, scaling, data format, revision turns, and two-stage training, along with a leakage analysis.

Significance. The paper offers a scalable, verifiable pipeline for synthesizing diverse reasoning data from code, and the public release of data and models is a concrete strength. The evaluation spans multiple model families and includes a leakage analysis, which is a good practice. If the mechanistic claim were fully supported, the notion of 'universal reasoning primitives' distilled from code would be an important conceptual contribution. At present, the evidence does not cleanly separate the contribution of code structure from that of the teacher model's reasoning style, and several empirical claims in the text are stronger than the data in Table 1 justify.

major comments (3)
  1. [Sections 2.2–2.4 and Figure 3] The central mechanistic claim that the gains arise from 'universal reasoning primitives' embedded in code is not cleanly isolated from teacher-distillation effects. Every natural-language CoT in CODE I/O and CODE I/O++ is produced by DeepSeek-V2.5, and the same model also performs the code refactoring and query generation in Section 2.2. The comparison with WebInstruct-DS25 (Figure 3) holds the teacher fixed but changes the source text from web instructions to code-derived I/O tasks; it cannot rule out the possibility that the gains come from DeepSeek-V2.5's reasoning style being expressed more effectively in the code-I/O prompt format rather than from the original code's inherent structure. Please add a same-code/different-teacher ablation (e.g., generating CoTs for a subset of the same transformed functions with a different strong model) and, if feasible, a control that uses untransformed raw code in the same I/O-prediction format, to support the attribution of the improvement to code-derived reasoning primitives.
  2. [Section 3.2, Table 1] The claim in Section 3.2 that 'CODE I/O++ systematically outperforms CODE I/O... without trade-offs on individual tasks' is directly contradicted by Table 1. For Qwen 2.5 Coder 7B, CODE I/O++ is lower than CODE I/O on WinoGrande (66.9 vs. 67.9), GSM8K (85.7 vs. 86.4), GPQA (40.6 vs. 43.3), and CRUXEval-I (62.5 vs. 63.6); for LLaMA 3.1 8B it is lower on MATH (53.2 vs. 53.8), GPQA (40.9 vs. 43.5), MMLU-STEM (68.4 vs. 69.0), and CRUXEval-O (53.1 vs. 53.3). Please either soften the claim to 'improves average performance with some per-benchmark regressions' or provide a statistical analysis demonstrating that the regressions are within noise.
  3. [Section 3 (all experiments)] All results in Tables 1–4 and Figures 3–5 come from single runs without error bars or multiple seeds. Given that the reported average gains are often small (e.g., 1–3 points) and that several individual benchmarks decline, the robustness of the central claim is not fully established. Please report standard deviations or run at least 2–3 seeds for the main comparison on one base model (e.g., Qwen 2.5 Coder 7B) to show that the improvements are reproducible and not noise.
minor comments (4)
  1. [Abstract and Conclusion] The phrases 'consistent improvements' and 'without sacrificing performance in any domain' overstate the results; Table 1 shows several per-benchmark regressions for most base models (e.g., LLaMA 3.1 8B MMLU-STEM 70.0 to 69.0, LiveBench 25.7 to 24.7). Recommend phrasing such as 'improves average performance' or 'improves on most benchmarks.'
  2. [Table 4] The paper chooses fully separated data for simplicity, but for LLaMA 3.1 8B the best result is with mixed data (CODE I/O + CODE I/O(10%)+IT, 52.7 vs. 51.2 for fully separated). The choice is acknowledged as a simplification, but it would be helpful to mention explicitly that the optimal mixing strategy is model-dependent.
  3. [Table 1 caption] The abbreviation 'LC' appears in the table header without being defined in the caption or text; it is presumably 'LeetCode-O'. Please define it for readers.
  4. [Related Work] The sentence 'we are the first to train LLMs on large-scale, diverse code input-output predictions and demonstrate its efficacy in improving general reasoning ability beyond code' may be too strong given earlier work such as LogicPro (Jiang et al., 2024b) and execution-aware pre-training methods (TRACED, NExT). Please qualify the novelty claim to avoid overclaiming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training-data pipeline and evaluation are externally grounded; self-citations are operational tools, not load-bearing premises.

full rationale

The paper's central chain is empirical rather than derivational: it constructs 3.5M code input-output prediction samples with natural-language CoTs generated by DeepSeek-V2.5, trains several base models in a two-stage procedure, and evaluates on external benchmarks. No fitted parameter is later renamed as a prediction, no benchmark score is used to define the training objective, and no uniqueness theorem or prior result by the authors is invoked to force the method's choice. The self-citations to DeepSeek models are operational (using DeepSeek-V2.5 as a data generator and DeepSeek Coder as one testbed model), and the main gains are replicated across Qwen, LLaMA, and Gemma, so the citation is not load-bearing. Section 4.2's WebInstruct-DS25 comparison and Section 4.7's leakage analysis further address alternative explanations. The concern that the gains could partly reflect teacher distillation rather than code-embedded reasoning primitives is a substantive interpretation/correctness risk, but it is not a circularity: the paper's reported improvements are measured independently of that attribution.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method adds no fitted constants or post-hoc physical parameters; its assumptions concern data fidelity and the validity of the synthetic CoTs and benchmarks. The hand-set filtering thresholds and per-source I/O counts are the main free choices, but they are not fitted to the evaluation.

free parameters (2)
  • CodeMix filtering success-rate thresholds = 10%-90%
    Files with DeepSeek-Coder-V2-Lite-Inst success rates between 10% and 90% are retained; the bounds are hand-selected and not swept in the paper, but they shape the diversity of CodeMix (Appendix C.1).
  • I/O pairs per source = 3 (CodeMix), 6 (PyEdu-R), 10 (Other)
    The number of input-output pairs sampled per function is chosen per source and not optimized; it affects the total dataset size (Appendix C.2).
assumptions (4)
  • standard math Executing the rewritten reference code yields correct ground-truth outputs for the generated inputs.
    I/O pairs are obtained by running the transformed functions; the paper skips functions with randomness and imposes time and size limits (Section 2.3).
  • domain assumption DeepSeek-V2.5 produces faithful natural-language CoTs that capture the function's reasoning process.
    All CoT responses are synthesized by DeepSeek-V2.5 and used verbatim as training targets, with no human validation (Section 2.4).
  • ad hoc to paper The transformed functions and generated queries preserve the reasoning patterns of the original raw code.
    Raw files are refactored by DeepSeek-V2.5; if refactoring changes semantics or drops reasoning content, the dataset no longer represents the claimed source (Section 2.2).
  • domain assumption The evaluation benchmarks are not meaningfully contaminated by the training data.
    A 13-gram leakage check is provided; for LeetCode-O and KorBench the authors show similar gains on non-leaked subsets (Section 4.7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction." pith.science (2026). https://pith.science/paper/JA3QDJGV

@misc{pith2026250207316,
  author       = {Pith},
  title        = {Pith review of: CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JA3QDJGV}},
  note         = {Machine review of arXiv:2502.07316}
}
read the original abstract

Reasoning is a fundamental capability of Large Language Models. While prior research predominantly focuses on enhancing narrow skills like math or code generation, improving performance on many other reasoning tasks remains challenging due to sparse and fragmented training data. To address this issue, we propose CodeI/O, a novel approach that systematically condenses diverse reasoning patterns inherently embedded in contextually-grounded codes, through transforming the original code into a code input-output prediction format. By training models to predict inputs/outputs given code and test cases entirely in natural language as Chain-of-Thought (CoT) rationales, we expose them to universal reasoning primitives -- like logic flow planning, state-space searching, decision tree traversal, and modular decomposition -- while decoupling structured reasoning from code-specific syntax and preserving procedural rigor. Experimental results demonstrate CodeI/O leads to consistent improvements across symbolic, scientific, logic, math & numerical, and commonsense reasoning tasks. By matching the existing ground-truth outputs or re-executing the code with predicted inputs, we can verify each prediction and further enhance the CoTs through multi-turn revision, resulting in CodeI/O++ and achieving higher performance. Our data and models are available at https://github.com/hkust-nlp/CodeIO.

Figures

Figures reproduced from arXiv: 2502.07316 by the authors.

Figure 1
Figure 1. Overview of our training data construction: Raw code files are gathered from various sources and converted into a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Two examples for the collected responses for input and output prediction respectively. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Average scores of Stage 1 training on CODEI/O, a 3.5M WebInstruct subset (WI) and an enhanced version distilled from DeepSeek-V2.5 Directly (WI-DS25). pair dataset synthesized from general code data, covering diverse computer science domains. Python-Edu (Ben Allal et al., 2024): Following findings that continued pre-training on code tends to enhance reasoning, we adopt its full 7.7M code corpus and train on it using… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The scaling effect of CODEI/O in the first stage training. I and -O also favor input and output prediction, respectively. Rejection Sampling We explore filtering incorrect re￾sponses using rejection sampling, which removes 50% of the training data. However, this result…
Figure 5
Figure 5. Figure 5: Average benchmark scores from training on data [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: An example in the constructed Leetcode-O benchmark. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: In multi-turn revision, we track the percentage (%) of each response type across the entire dataset after each [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: An example prompt we used for collecting responses and training. Most parts of the prompt for input and output [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Can Large Language Models Generalize Procedures Across Representations?

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Post-training on graph or code versions of a planning task does not transfer to natural-language versions, but a symbolic-then-natural-language RL curriculum achieves strong transfer.

  2. Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL

    cs.AI 2025-09 conditional novelty 6.0 of 10

    DRER rewards CoT trajectories that increase the model's likelihood of the correct answer, plus a length penalty, and the new LogicTree benchmark reportedly lifts a 7B model's average accuracy from 0.13 to 0.60.

  3. CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning

    cs.SE 2025-07 conditional novelty 6.0 of 10

    CodeReasoner combines a concise execution-focused dataset, instruction tuning, and GRPO RL to make 7B/14B models match or beat GPT-4o on code reasoning benchmarks.

  4. Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A minimal-prior pipeline with automated data curation and verifier-driven RL lets small LLMs generate verifiable Dafny specifications and beat larger proprietary models on a synthetic compositional benchmark.

  5. Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code

    cs.CL 2025-07 conditional novelty 6.0 of 10

    TeaR uses GRPO reinforcement learning on test-case output prediction for algorithmic problems, with no code shown, and reports broad reasoning gains across 17 benchmarks.

  6. Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Influence-function attribution shows high-difficulty math and low-difficulty code training data best improve math and code reasoning, and difficulty-based reweighting improves benchmark performance.

  7. O$^2$-Searcher: A Searching-based Agent Model for Open-Domain Open-Ended Question Answering

    cs.CL 2025-05 conditional novelty 6.0 of 10

    O2-Searcher uses GRPO reinforcement learning to teach a 3B LLM to search a local corpus and answer open-ended and closed-ended questions, and introduces the O2-QA benchmark.

  8. Dream-Coder 7B: An Open Diffusion Language Model for Code

    cs.CL 2025-09 conditional novelty 5.0 of 10

    An open 7B diffusion language model for code, adapted from Qwen2.5-Coder, reaches 21.4% pass@1 on LiveCodeBench and shows adaptive, non-left-to-right generation patterns.

  9. Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models

    cs.AI 2025-08 reject novelty 4.0 of 10

    A reinforcement-learning pipeline for predicting macro-actions in Honor of Kings improves action prediction accuracy, but the method is imitation of human replay labels, not the claimed environmental interaction.

Reference graph

Works this paper leans on

53 extracted references · 16 canonical work pages · cited by 9 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  3. [3]

    Smollm-corpus, 2024

    Ben Allal, L., Lozhkov, A., Penedo, G., Wolf, T., and von Werra, L. Smollm-corpus, 2024. URL https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus

    work page 2024

  4. [4]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv 2021

  5. [5]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    DeepSeek-AI, Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024

    arXiv 2024

  6. [6]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., ...

    arXiv 2025

  7. [7]

    Dehaene, S., Molko, N., Cohen, L., and Wilson, A. J. Arithmetic and the brain. Current opinion in neurobiology, 14 0 (2): 0 218--224, 2004

    work page 2004

  8. [8]

    J., Kaiser, G., and Ray, B

    Ding, Y., Min, M. J., Kaiser, G., and Ray, B. Cycle: Learning to self-refine the code generation. Proceedings of the ACM on Programming Languages, 8 0 (OOPSLA1): 0 392--418, 2024 a

    work page 2024

Show all 53 references
  1. [9]

    J., Kaiser, G., Yang, J., and Ray, B

    Ding, Y., Peng, J., Min, M. J., Kaiser, G., Yang, J., and Ray, B. Semcoder: Training code language models with comprehensive semantics reasoning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b . URL https://openreview.net/forum?id=PnlCHQrM69

    2024

  2. [10]

    Traced: Execution-aware pre-training for source code

    Ding, Y., Steenhoek, B., Pei, K., Kaiser, G., Le, W., and Ray, B. Traced: Execution-aware pre-training for source code. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp.\ 1--12, 2024 c

    2024

  3. [11]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2019

  4. [12]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al

    GemmaTeam, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ram \'e , A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  6. [14]

    Neural turing machines

    Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014

    2014 arXiv

  7. [15]

    J., Solar-Lezama, A., Synnaeve, G., and Wang, S

    Gu, A., Roziere, B., Leather, H. J., Solar-Lezama, A., Synnaeve, G., and Wang, S. CRUXE val: A benchmark for code reasoning, understanding and execution. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings ...

    2024

  8. [16]

    Measuring massive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021 a . URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  9. [17]

    Measuring mathematical problem solving with the MATH dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021 b

    2021

  10. [18]

    and Chang, K

    Huang, J. and Chang, K. C.-C. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022

    2022 arXiv

  11. [19]

    K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J., Zhang, C., Chai, L., et al

    Huang, S., Cheng, T., Liu, J. K., Hao, J., Song, L., Xu, Y., Yang, J., Liu, J., Zhang, C., Chai, L., et al. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905, 2024

    2024 arXiv

  12. [20]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Dang, K., et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186, 2024

    2024 arXiv

  13. [21]

    Openai o1 system card

    Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Carney, A., et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024

    2024 arXiv

  14. [22]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  15. [23]

    Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. l., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024 a

    2024 arXiv

  16. [24]

    Logicpro: Improving complex logical reasoning via program-guided learning

    Jiang, J., Yan, Y., Liu, Y., Jin, Y., Peng, S., Zhang, M., Cai, X., Cao, Y., Gao, L., and Tang, Z. Logicpro: Improving complex logical reasoning via program-guided learning. arXiv preprint arXiv:2409.12929, 2024 b

    2024 arXiv

  17. [25]

    and Wolf, A

    Knauff, M. and Wolf, A. G. Complex cognition: the science of human reasoning, problem-solving, and decision-making, 2010

    2010

  18. [26]

    Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L. J. V., Liu, A., Dziri, N., Lyu, S., et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  19. [27]

    Y., Bras, R

    Lin, B. Y., Bras, R. L., Richardson, K., Sabharwal, A., Poovendran, R., Clark, P., and Choi, Y. Zebralogic: On the scaling limits of llms for logical reasoning. arXiv preprint arXiv:2502.01100, 2025

    2025 arXiv

  20. [28]

    Code execution with pre-trained language models

    Liu, C., Lu, S., Chen, W., Jiang, D., Svyatkovskiy, A., Fu, S., Sundaresan, N., and Duan, N. Code execution with pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ 4984--4999, 2023

    2023

  21. [29]

    B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al

    Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173, 2024

    2024 arXiv

  22. [30]

    Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks

    Ma, K., Du, X., Wang, Y., Zhang, H., Wen, Z., Qu, X., Yang, J., Liu, J., Liu, M., Yue, X., et al. Kor-bench: Benchmarking language models on knowledge-orthogonal reasoning tasks. arXiv preprint arXiv:2410.06526, 2024

    2024 arXiv

  23. [31]

    Codestral, 2024

    Mistral-AI. Codestral, 2024. URL https://mistral.ai/news/codestral/

    2024

  24. [32]

    NE xt: Teaching large language models to reason about code execution

    Ni, A., Allamanis, M., Cohan, A., Deng, Y., Shi, K., Sutton, C., and Yin, P. NE xt: Teaching large language models to reason about code execution. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=B1W712hMBi

    2024

  25. [33]

    J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al

    Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021

    2021 arXiv

  26. [34]

    Reasoning with language model prompting: A survey

    Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., and Chen, H. Reasoning with language model prompting: A survey. arXiv preprint arXiv:2212.09597, 2022

    2022 arXiv

  27. [35]

    L., Stickland, A

    Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. GPQA : A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=Ti67584b98

    2024

  28. [36]

    E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023

    2023 arXiv

  29. [37]

    Winogrande: An adversarial winograd schema challenge at scale

    Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 8732--8740, 2020

    2020

  30. [38]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  31. [39]

    K., Kocyigit, M

    Singh, A. K., Kocyigit, M. Y., Poulton, A., Esiobu, D., Lomeli, M., Szilvasy, G., and Hupkes, D. Evaluation data contamination in llms: how do we measure it and (when) does it matter? arXiv preprint arXiv:2411.03923, 2024

    2024 arXiv

  32. [40]

    Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022

    2022 arXiv

  33. [41]

    W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al

    Suzgun, M., Scales, N., Sch \"a rli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp.\ ...

    2023

  34. [42]

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data

    Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., and Gitman, I. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. arXiv preprint arXiv:2410.01560, 2024

    2024 arXiv

  35. [43]

    Leti: Learning to generate from textual interactions

    Wang, X., Peng, H., Jabbarvand, R., and Ji, H. Leti: Learning to generate from textual interactions. In Findings of the Association for Computational Linguistics: NAACL 2024, pp.\ 223--239, 2024

    2024

  36. [44]

    and Chiew, V

    Wang, Y. and Chiew, V. On the cognitive process of human problem solving. Cognitive systems research, 11 0 (1): 0 81--92, 2010

    2010

  37. [45]

    Livebench: A challenging, contamination-free llm benchmark

    White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314, 2024

    2024 arXiv

  38. [46]

    Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though

    Xiang, V., Snell, C., Gandhi, K., Albalak, A., Singh, A., Blagden, C., Phung, D., Rafailov, R., Lile, N., Mahan, D., et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though. arXiv preprint arXiv:2501.04682, 2025

    2025 arXiv

  39. [47]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement

    Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  40. [48]

    Internlm-math: Open math large language models toward verifiable reasoning

    Ying, H., Zhang, S., Li, L., Zhou, Z., Shao, Y., Fei, Z., Ma, Y., Hong, J., Liu, K., Wang, Z., et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024

    2024 arXiv

  41. [49]

    Scaling relationship on learning mathematical reasoning with large language models

    Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023

    2023 arXiv

  42. [50]

    Mammoth2: Scaling instructions from the web

    Yue, X., Zheng, T., Zhang, G., and Chen, W. Mammoth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548, 2024

    2024 arXiv

  43. [51]

    and Sutskever, I

    Zaremba, W. and Sutskever, I. Learning to execute. arXiv preprint arXiv:1410.4615, 2014

    2014 arXiv

  44. [52]

    Skywork-math: Data scaling laws for mathematical reasoning in large language models--the story goes on

    Zeng, L., Zhong, L., Zhao, L., Wei, T., Yang, L., He, J., Cheng, C., Hu, R., Liu, Y., Yan, S., et al. Skywork-math: Data scaling laws for mathematical reasoning in large language models--the story goes on. arXiv preprint arXiv:2407.08348, 2024

    2024 arXiv

  45. [53]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence

    Zhu, Q., Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.