Pith. sign in

REVIEW 8 cited by

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.16437 v3 pith:LZHPLYYR submitted 2024-03-25 cs.SE cs.CL

Junkai Chen , Zhiyuan Pan , Xing Hu , Zhenhao Li , Ge Li , Xin Xia This is my paper · ORCID
classification cs.SEcs.CL
keywords codellmsreasoningbenchmarksprogrambehaviorconsistencyevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data, and \newname leaderboard are available at https://r-eval.github.io.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

    cs.SE 2026-07 conditional novelty 7.0 of 10

    RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.

  2. Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Veritas detects out-of-bounds vulnerabilities in stripped binaries at 90% recall by grounding LLM reasoning in static witness-backed flows and runtime validation.

  3. Is "Knowing It's Malicious Enough?" Evaluating LLMs for Fine-Grained Malware Behavior Auditing

    cs.CR 2025-09 conditional novelty 6.0 of 10

    A new evaluation framework shows current LLMs are not yet reliable at fine-grained Android malware behavior auditing, with the best model scoring 50.67% on the composite workload metric.

  4. CodeReasoner: Enhancing the Code Reasoning Ability with Reinforcement Learning

    cs.SE 2025-07 conditional novelty 6.0 of 10

    CodeReasoner combines a concise execution-focused dataset, instruction tuning, and GRPO RL to make 7B/14B models match or beat GPT-4o on code reasoning benchmarks.

  5. Large Language Models for Validating Network Protocol Parsers

    cs.SE 2025-04 conditional novelty 6.0 of 10

    PARVAL uses multiple LLM agents to extract format specifications from both RFC 5880 and the FRRouting BFD parser and flags 34 true inconsistencies, including seven unique bugs and two RFC documentation issues.

  6. Correctness Assessment of Code Generated by Large Language Models Using Internal Representations

    cs.SE 2025-01 conditional novelty 6.0 of 10

    Internal hidden states of code LLMs carry a usable signal about whether the generated code is correct, and a lightweight probing classifier on those states predicts test-passing with F1 around 0.7 to 0.8 across benchmarks.

  7. Themisto: Jupyter-Based Runtime Benchmark

    cs.SE 2025-04 conditional novelty 5.0 of 10

    A new benchmark for Jupyter runtime-aware code prediction shows five frontier LLMs get under 19% exact match on output prediction and under 14% on next cell prediction, with runtime context not helping.

  8. Revisit Self-Debugging with Self-Generated Tests for Code Generation

    cs.SE 2025-01 conditional novelty 5.0 of 10

    Post-execution self-debugging with self-generated tests degrades performance on basic code tasks due to biased test labels, while in-execution trace-based self-debugging mitigates the bias and improves pass rates.

Pith tools