Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Having an LLM judge and repair each code logic unit lifts its reasoning scores.

desk verdict A genuinely new combination—CFG-driven logic-unit extraction with a same-LLM rewind-and-correct dialogue—delivers modest but consistent gains; worth reviewing, but the judge-reliability assumption is the load-bearing soft spot. read the letter →

arxiv 2502.07803 v1 pith:ZV4CEH2N submitted 2025-02-05 cs.AI cs.LG

Cheryl Li , Tianyuan Xu , Yiwen Guo This is my paper · ORCID
classification cs.AIcs.LG
keywords reasoninghallucinationstest-timescalinglogicunitscontrolflowgraphself-correctionLLM-as-judgemathematicalcodegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a large fraction of LLM reasoning error is "reasoning hallucination": the natural-language steps a model reports and the logic of the program it writes do not actually line up. It proposes RaLU, a test-time framework that takes a generated program as the logical skeleton, cuts it into small self-contained logic units with static analysis, and then asks the same model to judge, explain, and repair each unit in turn, rewinding to recheck any unit it changes. The accepted units form a verified reasoning path from which the model writes its final program or answer. On GSM8K, MATH-np, HumanEval(+), and MBPP(+), RaLU reports higher accuracy or pass@1 than chain-of-thought, program-of-thought, self-consistency, tree-of-thought, and self-correction baselines across three open LLMs, with per-benchmark gains from 0.31 to 14.46 percentage points over the best baseline.

What carries the argument

The central object is the logic unit: a small block of statements carved from an initial program by traversing its control flow graph at branches, loops, and function entries, each paired with a natural-language explanation tying the code to the problem. The alignment loop asks the same LLM for a judgment ($J_i \in \{\mathrm{OK}, \mathrm{WRONG}\}$), a repair when the judgment is WRONG, and a re-judgment after repair, so corrections are always checked against the earlier accepted units. The formal condition carrying the argument is that this loop raises the probability a unit is correct exactly when the repair success rate exceeds the probability that a judged-wrong unit was actually correct, $\gamma_{\mathrm{repair}} > P(U \text{ correct} \mid J = \mathrm{WRONG})$.

What would settle it

Take a held-out sample of logic units from the four benchmarks, label each unit correct or incorrect by human agreement, and compare those labels with the model's OK/WRONG judgments and with whether its repaired units actually fix the error; if judge accuracy is near chance or repaired units are no more often correct than the units they replace, the inequality $\gamma_{\mathrm{repair}} > P(U \text{ correct} \mid J = \mathrm{WRONG})$ fails and RaLU's advantage over single-shot prompting should disappear.

Watch

Extended reading notes

Core claim

RaLU's central claim is that reasoning hallucinations between natural-language explanations and executable code can be reduced by making the code itself the unit of reasoning. An initial LLM program is decomposed by a control flow graph into logic units; an iterative dialogue then has the same LLM issue an OK/WRONG judgment for each unit, explain how the unit maps to the task specification, and repair it when needed, with a rewind step that re-judges the repaired unit against previously accepted units. The paper reports that this process raises final-answer accuracy on GSM8K and MATH-np and pass@1 on HumanEval, HumanEval+, MBPP, and MBPP+ across DeepSeek-V3, Qwen2.5-72B, and Llama3.3-70B, without fine-tuning, and that it exceeds reported closed-model scores on the extended code benchmarks.

Load-bearing premise

The load-bearing premise is that the same model can reliably judge whether each code unit is correct; the paper cites judge accuracy from preference-judgment studies, not unit-level code checks, and does not calibrate that accuracy on the three models it tests.

Editorial extensions

If this is right

  • On GSM8K, MATH-np, HumanEval(+), and MBPP(+), RaLU reports per-benchmark gains over the best baseline of 0.31 to 14.46 percentage points across the three LLM backbones.
  • Because each unit is checked before the next one is processed, a repaired unit becomes context for later units, so localized errors do not cascade into the final program.
  • The loop stops either when every unit is judged OK or when the iteration limit is reached, and the final synthesis reads from the verified unit path, giving the answer a step-by-step provenance.
  • RaLU needs at most three correction turns per unit, compared with up to ten sampled paths for self-consistency and tree-of-thought, so its gains do not depend on open-ended sampling.
  • On the extended EvalPlus code suites, RaLU's open-model results surpass reported o1-preview and GPT-4-class scores, despite trailing slightly on the original HumanEval and MBPP benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's margin should track the model's unit-level judge accuracy; measuring that accuracy directly on code units would tell whether the gains generalize to models not tested in the paper.
  • Because only prompts are task-specific, the same logic-unit loop could be ported to SQL queries, spreadsheet formulas, or formal proof steps; such experiments would show whether program-guided verification is the active ingredient.
  • The larger gains on the plus code suites suggest unit alignment pays off most when hidden tests impose several competing constraints; constructing adversarial test suites with deliberately entangled constraints could test that reading.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RaLU, a test-time scaling framework that prompts an LLM to write a program, decomposes the program via a control-flow graph into logic units, and then uses the same LLM in a multi-turn dialogue to judge, explain, and repair each unit, rewinding to revalidate corrected units before synthesizing a final solution. Experiments on GSM8K, a MATH subset (MATH-np), HumanEval(+), and Mbpp(+) with DeepSeek-V3, Qwen2.5-72B, and Llama3.3-70B report consistent gains over several prompting and self-correction baselines and claim superiority over some closed-source models. The paper also includes a Bayesian-style argument intended to show that unit alignment raises the probability of a correct final program.

Significance. RaLU is a practical and interpretable test-time method: it is training-free, combines natural language explanations with executable code, and the reported improvements are consistent across three open-weight backbones and across math and code benchmarks. The ablation studies (CFG vs. line-by-line, program units vs. NL steps) support the design choice of CFG-based units, and the code release plus the clear taxonomy of 'reasoning hallucinations' are useful contributions. However, the significance is currently limited by the absence of statistical calibration, by an unmeasured judge-reliability assumption on which the theoretical argument depends, and by protocol incompatibilities in the closed-source comparisons; these issues need to be resolved before the central claims can be fully credited.

major comments (4)
  1. [Section 3.3, Eqs. (8)-(10)] The Bayesian derivation is circular. O_i is defined as 'a verified reasoning step aligned with both program execution and problem semantics', so the inequality P(O_1,...,O_n|C) >> P(O_1,...,O_n|not C) is an assumption about the output of the verification process, not a consequence of RaLU's operations. Eq. (10) then concludes P(C|O_1,...,O_n) > P(C) from that same assumption. No model or measurement links the likelihood asymmetry to the actual judge/repair parameters alpha, beta, and gamma_repair introduced in Section 3.2. I suggest either deleting this formalization or re-deriving it from measured quantities.
  2. [Section 3.2, Eqs. (6)-(7), Appendix A.3] The condition gamma_repair > P(U correct | J=WRONG) is a correct sufficient condition, but the paper never instantiates it for the three backbones. alpha, beta, and gamma_repair are not measured; the sentence 'Empirical studies show that modern LLMs can achieve high accuracies when serving as a judge (Thakur et al., 2024) (where alpha can reach 0.9+)' refers to work on preference judgments, not to unit-level classification of generated code. The correctness of RaLU's rewind-and-correct loop therefore rests on an unverified premise. Please report judge and repair accuracies on a sample of units for each backbone and show that the inequality in Eq. (7) holds.
  3. [Table 1] The paper uses the word 'significant' without any statistical support. No error bars, confidence intervals, or significance tests are given, and several Delta_Gain values are below one percentage point (GSM8K DeepSeek +0.310; Mbpp Qwen +0.543; Mbpp Llama +0.598). For datasets of this size, those differences are within sampling noise. At minimum, report multiple independent runs with standard deviations, or paired per-problem tests for the small-gain cells.
  4. [Section 4.1 and Table 2] The abstract and introduction claim evaluation on MATH, but the experiments actually use the MATH-np subset (Section 4.1), and the closed-source comparisons in Table 2 are taken from external leaderboards with different decoding and evaluation protocols. The 'Ours (RaLU)' column does not identify which backbone produced each number. Please clarify the exact protocol, use the same evaluation harness for all compared models, and correct the MATH claim, or the headline SoTA statements are not directly interpretable.
minor comments (5)
  1. [Appendix B.2] 'Thee-of-Thought' should be 'Tree-of-Thought'; Section B.1 contains 'HumenEval' instead of 'HumanEval'; Section 2.1 contains the phrase 'remains under explosion', which appears to be a typo.
  2. [Equations (4)-(5)] The summation bound is written ambiguously as 'j = 0n-1' and the definition of n (the number of tokens in the response) is missing; please clarify the notation.
  3. [Section 4.2] The claim that RaLU reduces LLM calls by about 60+% is stated without any supporting table or measurement; please add the cost comparison.
  4. [Appendix A.1] The code 'def eulerian num(n, m):' is not valid Python as written; either the underscores were lost in rendering or the CFG extraction stage would fail on this example. Please correct the example.
  5. [Section 3.2] The notation 'P(˜U) is correct' has a tilde-placement typo; the intended statement should be P(tilde U is correct) > P(U is correct).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical benchmark evaluation is independent of the paper's theoretical framing, and no fitted quantity is renamed as a prediction.

full rationale

RaLU's central claim is the measured improvement on GSM8K, MATH-np, HumanEval(+), and MBPP(+) over reproduced baselines and ablations; this evidence does not presuppose the correctness of the method's self-judging loop. The Section 3.2 derivation (Eqs. 6-7 and Appendix A.3) is a standard probability identity relating alpha, beta, and gamma_repair to the condition p' > p; it introduces no fitted parameter and does not define the conclusion in terms of the inputs. The paper's reliance on Thakur et al. (2024) for high judge accuracy is an external citation, not a self-citation, and whether that result transfers to unit-level code correctness is an unmeasured assumption rather than a circular reduction. The Bayesian section (Eqs. 8-10) likewise takes P(O|C) >> P(O|not C) as a stated premise and derives the posterior comparison by Bayes' theorem; this is an assumption about the evidence, not a conclusion that has been built into the definition of O_i. The self-referential design (the same LLM judges its own units) is a methodological robustness concern, but the reported gains are measured outcomes, and no load-bearing step reduces by construction to its own inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small number of hand-chosen hyperparameters and on assumptions about the reliability of LLM self-judging and CFG extraction. No new physical entities or fitted constants are introduced; the main burden is the domain assumption that the same model can accurately validate its own units.

free parameters (3)
  • Confidence mapping constants in Eq. (5) = 0.005 and 0.01
    The clamping function sigma maps log-probabilities to [0,1] using an offset of 0.005 and a scale of 10^-2; these constants are chosen by hand and affect which candidate unit is selected when no unit is judged OK.
  • Maximum self-correction turns = 3
    Set as a stopping condition in all RaLU runs; the value is not justified by data and directly caps the number of repair attempts.
  • Temperature and frequency penalty = 0.7 and 0.3
    Decoding hyperparameters used for all experiments; standard choices but not fitted.
assumptions (4)
  • standard math Bayes' theorem and standard probability calculus
    Used in Section 3.3 and Appendix A.3 to derive posterior correctness and the repair condition.
  • domain assumption P(O|C) >> P(O|not C): a correct program's logic units are far more likely to be judged valid
    Assumed in Section 3.3 without measurement; this is the core of the Bayesian argument.
  • domain assumption The LLM judge has high true-positive rate alpha for unit correctness
    Stated in Section 3.2 with a citation to Thakur et al. that does not directly support code-unit judging.
  • domain assumption CFG construction and static analysis faithfully represent program logic
    The entire logic-unit extraction in Section 3.1 depends on this; no validation of CFG fidelity is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment." pith.science (2026). https://pith.science/paper/ZV4CEH2N

@misc{pith2026250207803,
  author       = {Pith},
  title        = {Pith review of: Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZV4CEH2N}},
  note         = {Machine review of arXiv:2502.07803}
}
read the original abstract

Chain-of-Thought (CoT) prompting has shown promise in enhancing the reasoning capabilities of large language models (LLMs) by generating natural language (NL) rationales that lead to the final answer. However, it struggles with numerical computation, which has somehow led to the development of program-aided techniques. Despite their potential, a persistent challenge remains: inconsistencies between LLM-reported reasoning steps and the logic in generated programs, which we term ``reasoning hallucinations." This stems from the inherent ambiguities of NL and the statistical nature of LLMs, which often lack rigorous logical coherence. To address this challenge, we propose a novel test-time scaling framework, Reasoning-as-Logic-Units (RaLU), which constructs a more reliable reasoning path by aligning logical units between the generated program and their corresponding NL descriptions. By decomposing the initially generated program into discrete units using static analysis, RaLU engages in an iterative dialogue with the LLM to judge, refine, and explain each unit. A rewind-and-correct mechanism ensures alignment between code statements and task requirements in each unit, ultimately forming a cohesive reasoning path under the program's logic, from which the model reaches a final solution. Our experiments demonstrate that RaLU significantly outperforms existing baselines in mathematical reasoning (GSM8K, MATH) and algorithmic reasoning (HumanEval+, MBPP+), underscoring its potential to advance LLM reasoning and programming by offering enhanced accuracy and interpretability.

Figures

Figures reproduced from arXiv: 2502.07803 by the authors.

Figure 1
Figure 1. Motivating examples reflecting the reasoning hallucinations: In Example 1, the LLM attempts to factor out the negative sign as stated yet wrongly neglects to reverse the positions of x and y in the expression. In Example 2, the LLM overlooks the NL step that indicates using the leftover to calculate the money earned; instead, it is misled by the question description to sell all the sets, thus omitting this crucial s… view at source ↗
Figure 2
Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustrating the three-stage process of RaLU: Logic Unit Extraction, Logic Unit Alignment, and Solution Synthesis for operationalizing synergy in reasoning tasks. OK ∧ U˜ i = Ui , indicating that no further are refinements needed; and 2) a predefined iteration limit or confidence threshold is reached. Upon triggering the second condition, multiple candidate units will exist, and we select the optimal version U˜∗ i u… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Ablation on unit abstraction: 5.52% accuracy drop (Math) and 4.35% score decline (Code) when replacing program￾guided logic units with NL steps. Performance deterioration stems from reasoning hallucinations exacerbated by NL’s lack of opera￾tional specificity and weak …
Figure 6
Figure 6. Figure 6: displays an example of transforming a program into a CFG and how RaLU organizes the graph to obtain a linear sequence of logic units. To enhance interpretability, units are annotated with human-readable descriptors (e.g., “LOOP BEGIN: Process each item in list X”). The…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning

    cs.AI 2026-08 conditional novelty 5.0 of 10

    Sampling multiple reasoning paths, refining each with self-critique and self-correction, then majority voting improves math reasoning accuracy over width-only or verifier-based test-time scaling on several open-weight LLMs.

  2. Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models

    cs.AI 2025-05 reject novelty 2.0 of 10

    A probabilistic saturation model for test-time scaling is proposed and fitted to reasoning benchmarks, but the plateau 'prediction' is computed from the same per-problem data used to measure it.

Reference graph

Works this paper leans on

47 extracted references · 22 canonical work pages · cited by 2 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C

    Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C. Program synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  3. [3]

    Graph of thoughts: Solving elaborate problems with large language models

    Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., and Hoefler, T. Graph of thoughts: Solving elaborate problems with large language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of A...

    2024

  4. [4]

    Codet: Code generation with generated tests

    Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J., and Chen, W. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023 a . URL https://openreview.net/forum?id=ktrw68Cmu9c

    work page 2023

  5. [5]

    Divide-and-conquer meets consensus: Unleashing the power of functions in code generation

    Chen, J., Tang, H., Chu, Z., Chen, Q., Wang, Z., Liu, M., and Qin, B. Divide-and-conquer meets consensus: Unleashing the power of functions in code generation. CoRR, abs/2405.20092, 2024 a . URL https://doi.org/10.48550/arXiv.2405.20092

    arXiv doi:10.48550/arxiv.2405.20092 2024

  6. [6]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Ch...

    arXiv 2021

  7. [7]

    Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Trans. Mach. Learn. Res., 2023, 2023 b . URL https://openreview.net/forum?id=YfZ4ZPt8zd

    work page 2023

  8. [8]

    Teaching large language models to self-debug

    Chen, X., Lin, M., Sch \" a rli, N., and Zhou, D. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 b . URL https://openreview.net/forum?id=KuPixIqPiq

    work page 2024

Show all 47 references
  1. [10]

    Training verifiers to solve math word problems

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. CoRR, abs/2110.14168, 2021 b . URL https://arxiv.org/abs/2110.14168

    2021 arXiv

  2. [11]

    J., Kaiser, G

    Ding, Y., Min, M. J., Kaiser, G. E., and Ray, B. CYCLE: learning to self-refine the code generation. Proc. ACM Program. Lang. , 8 0 ( OOPSLA1 ): 0 392--418, 2024. URL https://doi.org/10.1145/3649825

    2024 doi

  3. [12]

    Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., and Tan, S. H. Automated repair of programs from large language models. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 , pp.\ 1469--1481. IEEE , 2023. URL http...

    2023

  4. [13]

    Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance

    Fu, Y., Ou, L., Chen, M., Wan, Y., Peng, H., and Khot, T. Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance. CoRR, abs/2305.17306, 2023. URL https://doi.org/10.48550/arXiv.2305.17306

    2023 doi arXiv

  5. [14]

    PAL: program-aided language models

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., and Neubig, G. PAL: program-aided language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Researc...

    2023

  6. [15]

    CRITIC: large language models can self-correct with tool-interactive critiquing

    Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., and Chen, W. CRITIC: large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview...

    2024

  7. [16]

    Measuring mathematical problem solving with the MATH dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets ...

    2021

  8. [17]

    Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., and Schmidhuber, J. Metagpt: Meta programming for A multi-agent collaborative framework. In The Twelfth International Conference on ...

    2024

  9. [18]

    Codecot: Tackling code syntax errors in cot reasoning for code generation, 2024 a

    Huang, D., Bu, Q., Qing, Y., and Cui, H. Codecot: Tackling code syntax errors in cot reasoning for code generation, 2024 a . URL https://arxiv.org/abs/2308.08784

    2024 arXiv

  10. [19]

    S., Yu, A

    Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024 b . ...

    2024

  11. [20]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield - Dodds, Z., DasSarma, N., Tran - Johnson, E., Johnston, S., Showk, S. E., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Ja...

    2022 doi arXiv

  12. [21]

    When can llms actually correct their own mistakes? A critical survey of self-correction of llms

    Kamoi, R., Zhang, Y., Zhang, N., Han, J., and Zhang, R. When can llms actually correct their own mistakes? A critical survey of self-correction of llms. CoRR, abs/2406.01297, 2024. URL https://doi.org/10.48550/arXiv.2406.01297

    2024 doi arXiv

  13. [22]

    S., Yang, L., tse Huang, J., Zhu, Z., Zhang, L., and Lyu, M

    Lee, C., Xia, C. S., Yang, L., tse Huang, J., Zhu, Z., Zhang, L., and Lyu, M. R. A unified debugging approach via llm-based multi-agent synergy, 2024. URL https://arxiv.org/abs/2404.17153

    2024

  14. [23]

    and Marcus, G

    Lenat, D. and Marcus, G. Getting from generative AI to trustworthy AI: what llms might learn from cyc. CoRR, abs/2308.04445, 2023. URL https://doi.org/10.48550/arXiv.2308.04445

    2023 doi arXiv

  15. [24]

    How do humans write code? large models do it the same way too

    Li, L., He, X., Wang, H., Wang, L., and He, L. How do humans write code? large models do it the same way too. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , pp.\ 4638--4649. Associat...

    2024

  16. [25]

    Deductive verification of chain-of-thought reasoning

    Ling, Z., Fang, Y., Li, X., Huang, Z., Lee, M., Memisevic, R., and Su, H. Deductive verification of chain-of-thought reasoning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, ...

    2023

  17. [26]

    Large language models have intrinsic self-correction ability

    Liu, D., Nassereldine, A., Yang, Z., Xu, C., Hu, Y., Li, J., Kumar, U., Lee, C., and Xiong, J. Large language models have intrinsic self-correction ability. CoRR, abs/2406.15673, 2024. URL https://doi.org/10.48550/arXiv.2406.15673

    2024 doi arXiv

  18. [27]

    Evaluating the logical reasoning ability of chatgpt and GPT-4

    Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., and Zhang, Y. Evaluating the logical reasoning ability of chatgpt and GPT-4 . CoRR, abs/2304.03439, 2023 a . URL https://doi.org/10.48550/arXiv.2304.03439

    2023 doi arXiv

  19. [28]

    S., Wang, Y., and Zhang, L

    Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems ...

    2023

  20. [29]

    At which training stage does code data help llms reasoning? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Ma, Y., Liu, Y., Yu, Y., Zhang, Y., Jiang, Y., Wang, C., and Li, S. At which training stage does code data help llms reasoning? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://o...

    2024

  21. [30]

    W., and Rainforth, T

    Miao, N., Teh, Y. W., and Rainforth, T. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net...

    2024

  22. [31]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models

    Mirzadeh, S., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. CoRR, abs/2410.05229, 2024. URL https://doi.org/10.48550/arXiv.2410.05229

    2024 doi arXiv

  23. [32]

    and Plank, B

    Mondorf, P. and Plank, B. Beyond accuracy: Evaluating the reasoning behavior of large language models - A survey. CoRR, abs/2404.01869, 2024. URL https://doi.org/10.48550/arXiv.2404.01869

    2024 doi arXiv

  24. [33]

    I., and Lin, X

    Ni, A., Iyer, S., Radev, D., Stoyanov, V., Yih, W., Wang, S. I., and Lin, X. V. LEVER: learning to verify language-to-code generation with execution. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings ...

    2023

  25. [34]

    Gsm8k, 2025

    Paperwithcode. Gsm8k, 2025. URL https://paperswithcode.com/dataset/gsm8k

    2025

  26. [35]

    and Freitas, A

    Ranaldi, L. and Freitas, A. Self-refine instruction-tuning for aligning reasoning in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024 , pp.\ 2325--2347. Association for ...

    2024

  27. [36]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters

    Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. CoRR, abs/2408.03314, 2024. URL https://doi.org/10.48550/arXiv.2408.03314

    2024 doi arXiv

  28. [37]

    S., Choudhary, K., Ramayapally, V

    Thakur, A. S., Choudhary, K., Ramayapally, V. S., Vaidyanathan, S., and Hupkes, D. Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. CoRR, abs/2406.12624, 2024. URL https://doi.org/10.48550/arXiv.2406.12624

    2024 doi arXiv

  29. [38]

    V., Chi, E

    Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, ...

    2023

  30. [39]

    H., Le, Q

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Proces...

    2022

  31. [40]

    Enhancing mathematical reasoning in llms by stepwise correction

    Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., and Jiang, M. Enhancing mathematical reasoning in llms by stepwise correction. CoRR, abs/2410.12934, 2024. URL https://doi.org/10.48550/arXiv.2410.12934

    2024 doi arXiv

  32. [41]

    Xia, C. S. and Zhang, L. Keep the conversation going: Fixing 162 out of 337 bugs for 0.42 each using chatgpt. CoRR, abs/2304.00385, 2023. URL https://doi.org/10.48550/arXiv.2304.00385

    2023 doi arXiv

  33. [42]

    Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle

    Xue, S., Huang, Z., Liu, J., Lin, X., Ning, Y., Jin, B., Li, X., and Liu, Q. Decompose, analyze and rethink: Solving intricate problems with human-like reasoning cycle. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openrevie...

    2024

  34. [43]

    Tree of thoughts: Deliberate problem solving with large language models

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023,...

    2023

  35. [44]

    Natural language reasoning, A survey

    Yu, F., Zhang, H., Tiwari, P., and Wang, B. Natural language reasoning, A survey. ACM Comput. Surv. , 56 0 (12): 0 304:1--304:39, 2024. URL https://doi.org/10.1145/3664194

    2024 doi

  36. [45]

    Automatic chain of thought prompting in large language models

    Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/forum?id...

    2023

  37. [46]

    Y., Fu, J., Chen, W., and Yue, X

    Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterpreter: Integrating code generation with execution and refinement. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, Aug...

    2024 doi

  38. [47]

    Zhong, T., Liu, Z., Pan, Y., Zhang, Y., Zhou, Y., Liang, S., Wu, Z., Lyu, Y., Shu, P., Yu, X., Cao, C., Jiang, H., Chen, H., Li, Y., Chen, J., Hu, H., Liu, Y., Zhao, H., Xu, S., Dai, H., Zhao, L., Zhang, R., Zhao, W., Yang, Z., Chen, J., Wang, P., Ruan, W., Wang, H., Zhao, H.,...

    2024 doi

  39. [48]

    V., and Chi, E

    Zhou, D., Sch \" a rli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q. V., and Chi, E. H. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations...

    2023

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.