CoCoNUT: Structural Code Understanding does not fall out of a tree

Claas Beger; Saikat Dutta

REVIEW 3 major objections 5 minor 25 references

CoCoNUT: Structural Code Understanding does not fall out of a tree

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read High scores on code-generation benchmarks do not imply the ability to trace which lines execute; the best model fully traces only 47% of HumanEval tasks.

desk verdict A solid new control-flow tracing benchmark whose headline dissociation claim overstates what the data actually measure. read the letter →

arxiv 2501.16456 v3 pith:FTWHTYKS submitted 2025-01-27 cs.LG cs.SE

Claas Beger , Saikat Dutta This is my paper · ORCID

classification cs.LGcs.SE

keywords codeunderstandinglargelanguagemodelsexecutionbenchmarkstracingcontrolflowHumanEvalrecursion

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the ability to write correct code implies the ability to say what code does when run. To test this, the authors build CoCoNUT, a benchmark that asks seven large language models to reproduce the exact ordered list of line numbers a program executes for a given input, starting from HumanEval problems and adding recursion, concurrency, and object-oriented tasks. They find a wide gap: the best model, Gemini 1.5 Pro, fully traces only 47% of HumanEval tasks, and on recursion and concurrency no model exceeds roughly 5% task accuracy. They also report that trace difficulty is uncorrelated with generation difficulty, and that chain-of-thought prompting does not close the gap. If this holds, benchmark scores on code generation cannot be taken as evidence that models understand control flow, which matters for any use of LLMs in debugging, testing, or automated repair.

What carries the argument

The load-bearing object is the line-number execution trace: for a program and a given input, the ground-truth answer is the ordered list of executed source-line indices, produced automatically with Python's `sys.settrace` machinery. The benchmark annotates every source line with its index, asks the model to output the trace as a list, and scores responses three ways: exact match per trace, hard accuracy over all of a task's test calls, and a Gestalt-pattern similarity that rewards matching contiguous subsequences. For concurrency the ground truth is handled leniently by sorting overlapping thread segments after isolating the concurrent block, which still requires the model to identify where the parallel region begins and ends. This trace sequence is what carries the argument: it turns 'understanding code' into a concrete observable that is distinct from producing code.

What would settle it

Re-run the advanced tasks with test inputs generated independently of any LLM, such as property-based inputs whose traces provably exercise deep recursion or real thread interleaving, and compare the fraction of tasks solved completely: if a model's hard accuracy rises above the reported 5% ceiling, the paper's advanced-topic conclusion is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that execution tracing is not a by-product of code generation: current LLMs can produce semantically identical code yet cannot predict which lines execute. The ground truth is produced by running Python's tracer over the canonical HumanEval solution for each call in the test suite, yielding an ordered sequence of line indices; the model is shown the annotated source and must emit that sequence. On this task, Gemini 1.5 Pro reaches 47.2% hard accuracy (all test calls traced correctly) and GPT-4o 21.2% with direct prompting, despite strong generation performance. On the advanced subset, only object-oriented programs show measurable success (best 20.0%), while recursion and concurrency stay near or below 3% for every model. The authors conclude that structural control-flow understanding, as measured by tracing, is largely missing from current models even where generative performance is strong.

Load-bearing premise

For the advanced structural subset, the ground-truth test inputs were drafted by GPT-4o and then manually expanded, with no independent verification that those inputs truly exercise the intended recursion, concurrency, or OOP behavior; if those tests are noisy or unrepresentative, the near-zero accuracy on those topics could understate the models' real tracing ability.

Editorial extensions

If this is right

Tracing and generation have no meaningful correlation in the paper's data, with Spearman's rank correlation between -0.09 and 0.06, so raising scores on HumanEval-style benchmarks will not automatically improve control-flow reasoning.
Trace length dominates: even the strongest models fall off sharply after about 25 executed lines and are near zero beyond 40, so long-horizon execution reasoning remains unsolved.
Chain-of-thought prompting helps smaller models but leaves large ones flat or slightly worse, meaning the deficit is not merely a prompt-elicitation problem.
Dedicated code training is not decisive: code-specific models such as CodeLlama and Codestral do not outperform comparably sized general models on tracing.
The advanced structural tasks are near-failing for all models, so claims that LLMs understand recursive or concurrent programs need separate, trace-level evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next experiment is to re-run the advanced subset with independently verified test inputs, such as property-based or coverage-guided inputs, since the paper's own curation relies on GPT-4o-generated tests with manual expansion and no independent check.
If the deficit generalizes across languages, extending the same traced-line protocol to non-Python code would test whether the failure is about simulating execution generally rather than Python-specific surface patterns.
The near-zero recursion and concurrency numbers suggest that agentic coding tools should not trust an LLM's mental simulation for programs with loops that nest or threads that interleave; an external interpreter or formal checker is the safer fallback.
The correlation result implies that fine-tuning on traces, or training a separate execution-prediction head, may be a more direct route to control-flow understanding than further scaling on generation data alone.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: CoCoNUT is a genuinely useful new benchmark for line-level execution tracing, and the headline numbers—top model fully traces 47% of HumanEval tasks, near-zero on recursion and concurrency—are probably correct for the HumanEval subset. But the central framing, that strong code generation does not imply execution tracing ability, is only partially supported because models are never asked to trace code they generated themselves; they trace canonical reference solutions.

The stress-test note you passed along lands. Section II-A explicitly says the trace tasks are built from the canonical solution merged with the signature and docstring. The correlation experiment in IV-B uses Llama 3.1 70B to generate solutions and measures pass rate, but the tracing difficulty ranking is still based on canonical reference traces. So the dissociation is between generating a correct program and tracing someone else's correct program. That is a meaningful ability to measure, and the benchmark stands on its own as a control-flow reasoning test. But the title and abstract claim more: they say models can generate semantically identical code yet cannot trace execution paths. That would require asking the model to trace its own output. The fix is straightforward: have the same models trace their own generated solutions, at least the ones that pass tests, and compare those scores to reference-trace scores. I expect the gap would shrink but not disappear, and the paper would be more honest.

Other soft spots, in proportion: no human baseline is measured, yet the paper leans on "humans' innate ability" and says models lack human-like ability; that framing should be cut or backed by data. The advanced subset uses GPT4o-generated tests with manual expansion and no independent validation; given the near-zero scores, the tests should be validated or the claims softened. The Spearman correlations are reported without significance testing, so "no significant correlation" is not established. Reproducibility: the code/dataset link is not visible in the arXiv text; make it a real URL and include the artifact.

Where the paper is strong: ground truth for HumanEval traces comes from Python's sys tracer, which eliminates the circularity concern. The error analysis (else blocks, definition calls, statement skips) is concrete and useful. The trace-length decay is clean and consistent across models. The CoT finding—that it helps smaller models but not larger ones—is a useful data point.

Who it's for: anyone working on code LLM evaluation or training for execution reasoning. It deserves a serious referee. I'd send it to review with a request for revision, focusing on the overclaim and test validation, not a desk rejection.

Referee Report

3 major / 5 minor

Summary. CoCoNUT introduces a benchmark for line-level execution tracing in Python. It uses the 164 HumanEval tasks to create HumanEval-Trace (161 tasks, 1083 execution traces extracted with Python's sys tracer) and augments it with an Advanced-Trace subset (124 tasks, 620 traces) covering recursion, concurrency, and object-oriented programming, with test inputs generated by GPT-4o and manually expanded. The paper evaluates seven state-of-the-art LLMs under direct and chain-of-thought prompting, reporting that the best model, Gemini 1.5 Pro, fully traces only about 47% of HumanEval tasks, that performance degrades sharply for traces longer than about 25 lines, and that none of the models exceeds roughly 5% accuracy on recursion and concurrency traces. The authors also report a small correlation experiment using Llama 3.1 70B suggesting that code-generation difficulty and tracing difficulty are not significantly correlated across HumanEval tasks.

Significance. If the results hold, CoCoNUT is a useful and timely addition to code-reasoning evaluation: the HumanEval-Trace ground truth is produced by an external Python interpreter, avoiding circularity with the models under test, and the benchmark spans short and long traces, multiple control structures, and standard prompting variations. The paper explicitly ships the dataset and code, evaluates open and closed models, and includes a prompt-ablation study, all of which are strengths. The headline separation between code generation performance and execution-tracing ability is important for claims about LLM code competence and for the design of code reasoning benchmarks. However, the central dissociation claim is currently stated more strongly than the evidence supports: the tracing tasks use canonical HumanEval solutions rather than the code each model generates, and the advanced-topic ground truth lacks independent validation. Both issues are fixable, but they affect the interpretation of the main result and the reliability of the advanced-topic numbers.

major comments (3)

[Section IV-B and Conclusion] The central claim that strong code generation does not imply execution tracing is tested on canonical reference solutions, not on the code the models themselves generate. A model could produce a correct but structurally different solution and be able to trace that solution while failing to trace the canonical implementation. The correlation experiment in Section IV-B ranks generated-solution difficulty against tracing difficulty of the reference traces, so it does not address this confound. The Conclusion's statement that "models lack the human-like ability to trace code they generate" is therefore not supported by the reported experiments. Please either add an experiment in which each model traces its own generated, test-passing solutions and compare those trace accuracies with the reference-trace accuracies, or explicitly reframe RQ1 and the abstract/conclusion claims as being about tracing canonical HumanEval solutions.
[Section III-B and Table II] The ground-truth test inputs for the advanced subset are generated by GPT-4o and then manually expanded, but no independent validation is reported. The near-zero accuracies on recursion and concurrency, and the benchmark's value for future use, depend on these tests correctly invoking the intended behavior rather than, say, only exercising shallow parts of the programs. Please report a validation procedure such as independent human verification of a sample, a second-model cross-check, or execution-based checks that each test exercises the targeted construct, and release the exact test inputs with the benchmark. Without this, the advanced-topic results and the associated claims about control-flow structures remain difficult to interpret.
[Abstract and Introduction] The paper repeatedly frames the results against "humans' innate ability" to understand structural control flow, but no human baseline or human-subject data are presented. The measured quantities are LLM trace accuracies, and claims such as "high performance on such benchmarks does not correlate to humans' innate ability" are not empirically supported. Please either remove or carefully hedge this framing, or add a small human study on the same traces. This is not merely a wording issue, because the stated significance of the benchmark is tied to the human-comparison claim.

minor comments (5)

[Section II-B] The definition of "Accuracy Mean" is ambiguous: it is described as both a trace-level accuracy metric and as an aggregation of task-level accuracies. Please state precisely whether each task contributes equally or whether each trace contributes equally, and define "False Similarity" explicitly for incorrect traces.
[Section III-B] The concurrency evaluation sorts overlapping segments before applying the similarity metric, but the prompt says models may use parentheses to mark concurrent execution. Please clarify with an example how parenthesized predicted traces are normalized, sorted, and scored, so that readers can assess the "valid trade-off" claim.
[Appendix A, Tables IV and V] The trace-length bucket labels overlap at the boundaries ("1-3, 3-5, 5-10"), making it unclear which bucket contains traces of length 3 or 5. Please use non-overlapping intervals such as [1,3], [4,5], [6,10], or state the inclusive/exclusive convention.
[Table I] The table reports "Acc Hard" and "Acc Mean" without a full derivation of how the two differ. A sentence defining both metrics, with a small worked example, would improve reproducibility.
[References] Reference [4] and reference [9] are the same paper and should be merged; reference [3] lacks author names and a formal citation. Also, the model names are written inconsistently ("LLama" vs. "Llama", "CodeLLama" vs. "CodeLlama").

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoCoNUT's ground-truth traces are produced by an external Python interpreter, and the evaluated models' outputs are not used to define the benchmark.

full rationale

The paper's central claim is an empirical measurement: state-of-the-art LLMs trace execution paths poorly (Gemini 1.5 Pro fully traces 47% of HumanEval tasks; advanced recursion/concurrency below 5%). The ground-truth traces are generated by running the canonical HumanEval solutions through Python's sys tracer with test arguments (Section II-A), so the labels are external to the models being evaluated. No parameter is fitted to maximize model scores: the 1024-token filter and one-shot prompting are fixed methodological choices, and the paper explicitly tests alternatives (Section V-B) without tuning to improve results. The correlation experiment (Section IV-B) compares difficulty rankings of generation and tracing; Spearman values are reported, not optimized. There is no derivation chain in which an output is obtained from the same data used to construct it. A reviewer concern that models trace canonical reference solutions rather than their own generated code is a benchmark-validity/scope threat, not circularity: it does not entail that any claimed result is true by definition or by fitted construction. The advanced-subset tests are generated by GPT-4o and manually expanded, but the traces themselves are obtained by executing code, so label generation remains independent of the models under test. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation patterns appear. The paper is a self-contained empirical evaluation; circularity score 0.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central result does not depend on fitted free parameters. It rests on the assumption that Python tracer output is the correct ground truth, that GPT4o-generated advanced tests are valid, and that exact line-sequence matching is the right evaluation. The 1024-token filter and concurrency sorting are author-defined choices that could bias results toward shorter or leniently scored traces.

assumptions (6)

domain assumption Python sys.settrace output corresponds to the intended notion of execution path for each evaluated program.
Section II-A: all HumanEval traces are generated with the sys library tracer; if tracer line coverage differs from what a human would consider the execution path, the benchmark measures something else.
domain assumption The canonical HumanEval solutions are representative of correct implementations that exercise meaningful control flow.
Section II-A: HumanEval-Trace is built from canonical solutions; this assumes these solutions are the right reference for testing structural understanding.
domain assumption Advanced-topic test inputs generated with GPT4o and expanded manually are correct and appropriately difficult.
Section III-B: the paper states 'We employ GPT4o to generate simple base test cases which we expand manually'; no independent verification of these tests is reported.
domain assumption Exact match on the ordered list of line numbers is a valid measure of trace understanding.
Section II-B defines Exact Match and hard accuracy as the primary metrics; this assumes that a single wrong line should fail a trace and that partial credit via Gestalt similarity is meaningful.
ad hoc to paper The 1024-token trace filter and the concurrency sorting rule do not materially change the conclusions.
Section II-A filters out traces requiring more than 1024 tokens, and Section II-B sorts concurrent segments before computing similarity; both choices affect the measured accuracies and are not sensitivity-analyzed.
domain assumption The evaluated models have not memorized the exact CoCoNUT trace outputs.
No contamination analysis is provided; HumanEval code is widely present in training data, and the paper does not test whether the line-annotation trace format appears in pretraining.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CoCoNUT: Structural Code Understanding does not fall out of a tree." pith.science (2026). https://pith.science/paper/FTWHTYKS

@misc{pith2026250116456,
  author       = {Pith},
  title        = {Pith review of: CoCoNUT: Structural Code Understanding does not fall out of a tree},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FTWHTYKS}},
  note         = {Machine review of arXiv:2501.16456}
}

read the original abstract

Large Language Models (LLMs) have shown impressive performance across a wide array of tasks involving both structured and unstructured textual data. Recent results on various benchmarks for code generation, repair, or completion suggest that certain models have programming abilities comparable to or even surpass humans. In this work, we demonstrate that high performance on such benchmarks does not correlate to humans' innate ability to understand structural control flow in code. To this end, we extract solutions from the HumanEval benchmark, which the relevant models perform strongly on, and trace their execution path using function calls sampled from the respective test set. Using this dataset, we investigate the ability of seven state-of-the-art LLMs to match the execution trace and find that, despite their ability to generate semantically identical code, they possess limited ability to trace execution paths, especially for longer traces and specific control structures. We find that even the top-performing model, Gemini, can fully and correctly generate only 47% of HumanEval task traces. Additionally, we introduce a subset for three key structures not contained in HumanEval: Recursion, Parallel Processing, and Object-Oriented Programming, including concepts like Inheritance and Polymorphism. Besides OOP, we show that none of the investigated models achieve an accuracy over 5% on the relevant traces. Aggregating these specialized parts with HumanEval tasks, we present CoCoNUT: Code Control Flow for Navigation Understanding and Testing, which measures a model's ability to trace execution of code upon relevant calls, including advanced structural components. We conclude that current LLMs need significant improvement to enhance code reasoning abilities. We hope our dataset helps researchers bridge this gap.

Figures

Figures reproduced from arXiv: 2501.16456 by the authors.

**Figure 1.** Tracing the execution flow requires a diverse understanding of code structures, even for shorter sequences like HumanEval Tasks. We list a couple [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Model Accuracy by Buckets of Trace Length using Direct Prompting. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Model Accuracy by Buckets of Trace Length using Chain-of-Thought [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Examples of three of the most common error types encountered in HumanEval tracing. The subtitle denotes the number of models that exhibited this [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Length distribution across HumanEval solutions. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 10 canonical work pages

[2]

The future landscape of large language models in medicine,

J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N. Eckardt, N. G. Laleh, C. M. L. L ¨offler, S.-C. Schwarzkopf, M. Unger, G. P. Veldhuizen et al. , “The future landscape of large language models in medicine,” Communications medicine, vol. 3, no. 1, p. 141, 2023

work page 2023
[3]

Introducing bloomberggpt, bloomberg’s 50-billion param- eter large language model, purpose-built from scratch for finance,

“Introducing bloomberggpt, bloomberg’s 50-billion param- eter large language model, purpose-built from scratch for finance,” https://www.bloomberg.com/company/press/ bloomberggpt-50-billion-parameter-llm-tuned-finance

work page
[5]

Program synthesis with large language models,

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

arXiv 2021
[6]

EvalPlus Leaderboard — evalplus.github.io,

“EvalPlus Leaderboard — evalplus.github.io,” https://evalplus.github.io/ leaderboard.html, [Accessed 19-11-2024]

work page 2024
[7]

Cruxeval: A benchmark for code reasoning, understanding and execution,

A. Gu, B. Rozi `ere, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang, “Cruxeval: A benchmark for code reasoning, understanding and execution,” arXiv preprint arXiv:2401.03065 , 2024

arXiv 2024
[8]

Livecodebench: Holistic and contamination free evaluation of large language models for code,

N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024

arXiv 2024
[9]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

arXiv 2021
[10]

Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,

J. Liu, C. S. Xia, Y . Wang, and L. ZHANG, “Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=1qvx610Cu7

work page 2023

Show all 25 references

[11]

Pattern matching: The gestalt approach,

J. W. Ratcliff, D. E. Metzener et al. , “Pattern matching: The gestalt approach,” Dr. Dobb’s Journal, vol. 13, no. 7, p. 46, 1988

1988
[12]

LeetCode Solutions — walkccc.me,

P.-Y . Chen, “LeetCode Solutions — walkccc.me,” https://walkccc.me/ LeetCode/, [Accessed 19-11-2024]

2024
[13]

GitHub - neetcode-gh/leetcode: Leetcode solutions — github.com,

“GitHub - neetcode-gh/leetcode: Leetcode solutions — github.com,” https://github.com/neetcode-gh/leetcode, [Accessed 12-11-2024]

2024
[14]

Rosetta Code — rosettacode.org,

“Rosetta Code — rosettacode.org,” https://rosettacode.org/wiki/Rosetta Code, 2007, [Accessed 26-01-2025]

2007
[15]

RosettaCodeData/Task at main · acmeism/RosettaCodeData — github.com,

“RosettaCodeData/Task at main · acmeism/RosettaCodeData — github.com,” https://github.com/acmeism/RosettaCodeData/tree/main/ Task, [Accessed 12-11-2024]

2024
[16]

Next: Teaching large language models to reason about code execution,

A. Ni, M. Allamanis, A. Cohan, Y . Deng, K. Shi, C. Sutton, and P. Yin, “Next: Teaching large language models to reason about code execution,” 2024. [Online]. Available: https://arxiv.org/abs/2404.14662

2024 arXiv
[17]

Large language models as code executors: An exploratory study,

C. Lyu, L. Yan, R. Xing, W. Li, Y . Samih, T. Ji, and L. Wang, “Large language models as code executors: An exploratory study,”
[18]

Chain-of-thought prompting elicits reasoning in large language models,

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2201.11903

2023 arXiv
[19]

Tree of thoughts: Deliberate problem solving with large language models,

S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y . Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” 2023. [Online]. Available: https: //arxiv.org/abs/2305.10601

2023 arXiv
[20]

Show your work: Scratchpads for intermediate computation with language models,

M. Nye, A. J. Andreassen, G. Gur-Ari, H. Michalewski, J. Austin, D. Bieber, D. Dohan, A. Lewkowycz, M. Bosma, D. Luan, C. Sutton, and A. Odena, “Show your work: Scratchpads for intermediate computation with language models,” 2021. [Online]. Available: https://arxiv.org/abs/2112.00114

2021 arXiv
[21]

Codemind: A framework to challenge large language models for code reasoning,

C. Liu, S. D. Zhang, A. R. Ibrahimzada, and R. Jabbarvand, “Codemind: A framework to challenge large language models for code reasoning,” arXiv preprint arXiv:2402.09664 , 2024

2024 arXiv
[22]

Codexglue: A machine learning benchmark dataset for code understanding and generation,

S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al., “Codexglue: A machine learning benchmark dataset for code understanding and generation,” arXiv preprint arXiv:2102.04664 , 2021

2021 arXiv
[23]

Reasoning runtime behavior of a program with llm: How far are we?

J. Chen, Z. Pan, X. Hu, Z. Li, G. Li, and X. Xia, “Reasoning runtime behavior of a program with llm: How far are we?” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) . IEEE Computer Society, 2024, pp. 140–152

2025
[24]

Lms: Understanding code syntax and semantics for code analysis,

W. Ma, S. Liu, Z. Lin, W. Wang, Q. Hu, Y . Liu, C. Zhang, L. Nie, L. Li, and Y . Liu, “Lms: Understanding code syntax and semantics for code analysis,” 2024. [Online]. Available: https://arxiv.org/abs/2305.12138

2024 arXiv
[25]

Do large code models understand programming concepts? a black-box approach,

A. Hooda, M. Christodorescu, M. Allamanis, A. Wilson, K. Fawaz, and S. Jha, “Do large code models understand programming concepts? a black-box approach,” 2024. [Online]. Available: https: //arxiv.org/abs/2402.05980

2024 arXiv
[26]

Semcoder: Training code language models with comprehensive semantics,

Y . Ding, J. Peng, M. J. Min, G. Kaiser, J. Yang, and B. Ray, “Semcoder: Training code language models with comprehensive semantics,” arXiv preprint arXiv:2406.01006, 2024. APPENDIX A. Trace Length Buckets In the following, we display the measured accuracies by trace-lengths p...

2024 arXiv
[2024]

Available: https://arxiv.org/abs/2410.06667

[Online]. Available: https://arxiv.org/abs/2410.06667

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [2]

The future landscape of large language models in medicine,

J. Clusmann, F. R. Kolbinger, H. S. Muti, Z. I. Carrero, J.-N. Eckardt, N. G. Laleh, C. M. L. L ¨offler, S.-C. Schwarzkopf, M. Unger, G. P. Veldhuizen et al. , “The future landscape of large language models in medicine,” Communications medicine, vol. 3, no. 1, p. 141, 2023

work page 2023

[2] [3]

Introducing bloomberggpt, bloomberg’s 50-billion param- eter large language model, purpose-built from scratch for finance,

“Introducing bloomberggpt, bloomberg’s 50-billion param- eter large language model, purpose-built from scratch for finance,” https://www.bloomberg.com/company/press/ bloomberggpt-50-billion-parameter-llm-tuned-finance

work page

[3] [5]

Program synthesis with large language models,

J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “Program synthesis with large language models,” 2021. [Online]. Available: https://arxiv.org/abs/2108.07732

arXiv 2021

[4] [6]

EvalPlus Leaderboard — evalplus.github.io,

“EvalPlus Leaderboard — evalplus.github.io,” https://evalplus.github.io/ leaderboard.html, [Accessed 19-11-2024]

work page 2024

[5] [7]

Cruxeval: A benchmark for code reasoning, understanding and execution,

A. Gu, B. Rozi `ere, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang, “Cruxeval: A benchmark for code reasoning, understanding and execution,” arXiv preprint arXiv:2401.03065 , 2024

arXiv 2024

[6] [8]

Livecodebench: Holistic and contamination free evaluation of large language models for code,

N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica, “Livecodebench: Holistic and contamination free evaluation of large language models for code,” arXiv preprint arXiv:2403.07974, 2024

arXiv 2024

[7] [9]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Her...

arXiv 2021

[8] [10]

Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,

J. Liu, C. S. Xia, Y . Wang, and L. ZHANG, “Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation,” in Thirty-seventh Conference on Neural Information Processing Systems , 2023. [Online]. Available: https://openreview.net/forum?id=1qvx610Cu7

work page 2023