Pith. sign in

REVIEW 3 major objections 4 minor 78 references

Position: Intelligent Coding Systems Should Write Programs with Justifications

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Intelligent coding systems should generate justifications alongside code, and the justifications must be both cognitively aligned and semantically faithful.

desk verdict A clear, honest position paper on justification generation for code LLMs; the framing is useful, but the concrete inference-time verification pipeline rests on an unvalidated post-hoc explainability assumption. read the letter →

arxiv 2508.06017 v1 pith:NYSXXYWM submitted 2025-08-08 cs.SE cs.CLcs.LG

Xiangzhe Xu , Shiwei Feng , Zian Su , Chengpeng Wang , Xiangyu Zhang This is my paper
classification cs.SEcs.CLcs.LG
keywords intelligentcodingsystemscodegenerationjustificationscognitivealignmentsemanticfaithfulnessneuro-symbolicmethodspost-hocexplainabilityformalprogramsemanticschain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Intelligent coding systems should not simply emit code from a natural-language prompt; they should also produce a justification that explains why the generated program satisfies the request. This paper argues that a real justification must be cognitively aligned—readable and followable by a non-expert—and semantically faithful—consistent with what the code actually does and checkable against program semantics. The authors maintain that chain-of-thought traces, static analysis, formal verification, and post-hoc explainability each fall short on at least one of these properties. They recommend a neuro-symbolic design where domain knowledge and semantic constraints guide training rewards, and an inference-time pipeline verifies that the model's explanation matches the code's formal semantics. If accepted, justification generation becomes a first-class requirement for AI code assistants, not a cosmetic add-on.

What carries the argument

The carrying mechanism is the neuro-symbolic consistency check. A post-hoc explainability step (attention attribution or counterfactual perturbation) maps each element of the natural-language justification to concrete program entities; the model's reasoning is then expressed as 'neural semantic rules' whose premises refer to those entities; finally those rules are compared against the program's formal semantics. Supporting this at training time are reward signals built from domain knowledge (e.g., CWE-434 file-upload constraints) and a metamorphic mutation scheme that rewards justifications that stay consistent when code is rewritten into semantically equivalent forms.

What would settle it

Apply the proposed pipeline to the avatar-upload example: a justification says 'the code ensures only .png files are allowed,' while the generated code is `file = requests.files['file']; file.save(file.name)`. Use attention attribution or counterfactual perturbation to map the justification phrase to code entities. If the mapping does not connect the phrase to the missing `.png` check on the executed path—or if the neural semantic rule says 'Safe' when the formal semantics say 'Unsafe'—the pipeline fails on the very case it was designed to catch. A benchmark of deliberately mismatched justific

Watch

Extended reading notes

Core claim

On the paper's own terms, an intelligent coding system should be judged deficient if it generates correct-looking code without a justification a user can evaluate. Justifications must satisfy two properties: cognitive alignment (natural language a non-expert can follow) and semantic faithfulness (consistency with the actual program semantics, checkable against code entities). The paper argues existing routes fail on at least one property: chain-of-thought can be unfaithful, static analysis is too low-level, formal verification needs expert-written specifications, and post-hoc explainability is not tied to formal semantics. The recommended design is neuro-symbolic: domain knowledge and metamo

Load-bearing premise

The load-bearing premise is that post-hoc explainability can reliably attach each phrase of a generated justification to the specific program entities that actually determine the code's behavior; if that attachment is noisy, the semantic-faithfulness check cannot distinguish a true justification from a plausible-sounding one.

Editorial extensions

If this is right

  • A program that passes tests but comes with a misleading explanation will count as a system failure, changing how code assistants are benchmarked and released.
  • Training pipelines will incorporate explanation-consistency rewards—using domain knowledge and metamorphic rewriting—alongside code-correctness rewards, so explanations are constrained before generation.
  • Inference-time verification will become an automated safety layer that rejects any code/justification pair whose neural semantic rules contradict the formal semantics.
  • Justification generation will be designed jointly with code generation, since the same program semantics must constrain both artifacts.
  • Non-expert users will get an auditable artifact written in their own vocabulary, making it possible to check generated code without reading the implementation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to reuse the consistency check for other paired natural-language/structured outputs, such as commit messages, API documentation, and security-audit summaries, where a plausible but unfaithful sentence can be as dangerous as a wrong program.
  • The two properties are necessary but not obviously sufficient; a third property such as coverage—whether the justification mentions every behaviorally relevant code entity—may be required before a consistency check can certify faithfulness. This is my inference, as the paper does not discuss it.
  • A testable extension follows from the cognitive-alignment claim: in a controlled study, non-expert users should be measurably better at spotting injected bugs when explanations are trained with the proposed reward than with plain chain-of-thought explanations.
  • If post-hoc attribution turns out to be brittle, an inverse design would be to derive the formal semantics first and constrain the natural-language justification to be a faithful description of it, moving verification from post-hoc association to generation time.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This position paper argues that intelligent coding systems should not only generate code but also produce natural-language justifications that are cognitively aligned with human reasoning and semantically faithful to the generated program. The authors identify two properties—cognitive alignment and semantic faithfulness—and argue that existing approaches (static analysis, formal verification, code-comment consistency checking, and post-hoc explainability) each fall short. They recommend a neuro-symbolic design: use domain knowledge and metamorphic rewriting to craft reward signals that guide model training, and at inference time use post-hoc explainability to associate justification elements with program entities, express the model's reasoning as "neural semantic rules," and compare these against formal program semantics. The paper is a vision/position paper with illustrative figures and no empirical validation.

Significance. If the central recommendation is adopted, justification generation could become a first-class requirement for AI coding assistants, with potential benefits for trust, transparency, and non-expert usability. The paper usefully identifies a conceptual gap: existing justifications are either too formal (proof assistants) or too shallow (comments, static-analysis reports), and CoT-style traces can be unfaithful. The two target properties provide a reasonable starting vocabulary for a research agenda. The paper also grounds its proposal in prior neuro-symbolic program-analysis work (e.g., LLMSA, LLMDFA, ProSec), which lends some plausibility to the overall direction. However, the proposal is at a very early stage: the core mechanisms are under-specified, and the inference-time verification pipeline depends on an unvalidated assumption about the reliability of post-hoc explanation-to-entity association.

major comments (3)
  1. [Section 3, Figure 5] The proposed inference-time verification pipeline has as its first step the assumption that post-hoc explainability techniques (attention attribution, counterfactual perturbations) can reliably associate each element of a natural-language justification with specific program entities. This assumption is load-bearing: if the association is brittle, the subsequent construction of "neural semantic rules" and their comparison with formal semantics cannot be trusted, and the automated semantic-faithfulness check fails. The paper provides no evidence for this reliability, and Figure 5 is explicitly hypothetical ("suppose that a post-hoc explainability indicates..."). Given known limitations of attention- and perturbation-based explanations, the authors should either provide a proof-of-concept demonstration on a small set of examples or reframe this step as an open challenge with concrete evalua
  2. [Section 3] The paper introduces "neural semantic rules" as the medium for expressing the model's reasoning, but never defines what these rules are. They are described as "akin to inference rules in programming languages," but no syntax, semantics, or extraction procedure is given. Without a concrete representation, the proposed consistency check against formal semantics is not implementable or falsifiable. The manuscript should specify at least a minimal representation (e.g., a small DSL or a structured form of attention/feature attribution) and show how it is derived from model internals. This is a central conceptual gap because the entire inference-time verification concept depends on the existence and computability of such rules.
  3. [Section 1] The two properties that motivate the paper—cognitive alignment and semantic faithfulness—are defined only at an intuitive level. No operational metrics or evaluation protocols are offered for determining whether a given justification satisfies them. Since these properties are the yardsticks against which existing methods are judged and the new neuro-symbolic approach is proposed, the central claim remains untestable without at least a provisional operationalization. For a position paper this need not be a full empirical study, but the authors should propose a concrete measurement approach (e.g., a human study protocol for cognitive alignment, or a formal consistency check for semantic faithfulness) so that the research agenda is actionable.
minor comments (4)
  1. [Figure 2] The figure contains the placeholder text "Todo: examples" in place of an actual illustration. This gives the impression that the taxonomy of existing justification efforts is unfinished. The placeholder should be replaced with a concrete example or removed.
  2. [Figures 3 and 4] In the submitted version, Figures 3 and 4 appear to contain the same graphic. If this is not a rendering artifact, the two proposed reward-design mechanisms (domain-knowledge reward and metamorphic rewarding) are not actually illustrated separately, which makes the exposition confusing. The authors should ensure that the two figures show distinct content.
  3. [Figure 5 caption] The caption contains a grammatical error: "The gray boxes describes a program path" should be "The gray boxes describe a program path."
  4. [Section 3] The paper does not discuss the potential costs of generating justifications, such as increased latency and token usage, nor the risk that users might over-trust a fluent but incorrect justification. Adding a short discussion of these trade-offs would strengthen the position statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: position paper makes a normative recommendation, not a derived prediction; self-citations are external engineering evidence.

full rationale

The paper is a position paper whose central claim is normative: intelligent coding systems 'should not only generate code but also produce clear, consistent justifications' (Abstract). It proposes two properties and recommends a neuro-symbolic design. There is no derivation chain in which an output quantity is defined in terms of an input quantity, no fitted parameter is renamed as a prediction, and no uniqueness theorem is invoked to force a choice. The self-citations (e.g., [16, 51-53, 60, 70]) are used to support the premise that LLM-based tools can reason about program behavior; they are prior empirical/engineering results, not the conclusion of this paper, so they are not load-bearing in a circular sense. The inference-time verification proposal (Section 3) rests on an unvalidated assumption that post-hoc explainability can reliably associate justification elements with program entities, and Figure 5 is explicitly hypothetical ('suppose that a post-hoc explainability indicates...'). That is a feasibility/robustness limitation, not a circularity: the paper does not claim to have derived this association from the property it is meant to verify. Figure 2 even contains a 'Todo: examples' placeholder, underscoring the proposal's illustrative status. Under the requirement to exhibit a specific reduction between the paper's own equations or citations, no circular step can be identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The paper contributes no free parameters and no formal derivation. It rests on domain assumptions about the feasibility of training and verification. The main invented concept is 'neural semantic rules', which lacks a concrete definition.

assumptions (3)
  • domain assumption Language models can produce unfaithful explanations that do not match their actual behavior.
    Invoked in Section 1 as a motivation: 'language models can produce unfaithful explanations that do not match their actual behavior.' This is cited from prior work but taken as given.
  • domain assumption Post-hoc explainability techniques can reliably associate each element of a natural language justification with specific program entities.
    Section 3's inference-time pipeline begins with this step. Without reliable association, the proposed verification cannot work. No evidence is provided in the paper.
  • domain assumption Reinforcement learning with a reward function can optimize for both cognitive alignment and semantic faithfulness.
    Section 3 recommends training via reward functions. The feasibility of designing such rewards is asserted but not demonstrated.
invented entities (1)
  • Neural semantic rules
    purpose: To express the model's reasoning over program entities in a form comparable to formal program semantics, enabling automated consistency checks.
    Introduced in Section 3 as 'neural semantic rules, akin to inference rules in programming languages'. No concrete formalization, examples, or implementation is given, and no independent validation exists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position: Intelligent Coding Systems Should Write Programs with Justifications." pith.science (2026). https://pith.science/paper/NYSXXYWM

@misc{pith2026250806017,
  author       = {Pith},
  title        = {Pith review of: Position: Intelligent Coding Systems Should Write Programs with Justifications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NYSXXYWM}},
  note         = {Machine review of arXiv:2508.06017}
}
read the original abstract

Intelligent coding systems are transforming software development by enabling users to specify code behavior in natural language. However, the opaque decision-making of AI-driven coders raises trust and usability concerns, particularly for non-expert users who cannot inspect low-level implementations. We argue that these systems should not only generate code but also produce clear, consistent justifications that bridge model reasoning and user understanding. To this end, we identify two critical justification properties-cognitive alignment and semantic faithfulness-and highlight the limitations of existing methods, including formal verification, static analysis, and post-hoc explainability. We advocate exploring neuro-symbolic approaches for justification generation, where symbolic constraints guide model behavior during training and program semantics are enriched through neural representations, enabling automated consistency checks at inference time.

Figures

Figures reproduced from arXiv: 2508.06017 by the authors.

Figure 1
Figure 1. Intelligent coding systems play a similar role as the traditional compilers. Traditional compilers empower devel￾opers by converting higher-level source code to executable machine code. Intelligent systems empower non-expert users by converting natural language requests to source code. Similarly, AI-driven coding systems require human supervi￾sion in critical scenarios: their probabilistic models can yield variable … view at source ↗
Figure 2
Figure 2. Existing efforts on generating justification for code artifacts. A technique closer to the left denotes the technique works closer to the natural language space, and vice versa, a technique closer to the right denotes it works at the level closer to the programming language space. Coding Request Help me write a server that allows user to upload their avatar. Domain Knowledge CWE-434: Unrestricted File Upload. A secu… view at source ↗
Figure 3
Figure 3. Reward signal for domain-aligned justification. For example, given the domain knowledge that avatar uploads should be restricted to .png files (CWE-434), a coding agent tasked with building a user-avatar upload server must enforce file extension validation. The reward function grants positive feedback only if the justification explicitly discusses the .png-only constraint, ensuring consistency with domain knowledge.… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Metamorphic rewarding for justification consistency. By applying semantic-equivalent rewriting to a code snippet (e.g., one without any file-extension check and another that only logs a warning), our metamorphic rewarding mechanism guides the two corresponding justific…
Figure 5
Figure 5. Figure 5: Neural-symbolic consistency check. The gray boxes describes a program path in which an uploaded file’s name fails the .png check yet is still saved, which is an unsafe behavior under formal semantics. In contrast, suppose that a post-hoc explainability indicates the mo…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

78 extracted references · 53 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905 (2024)

    arXiv 2024

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al . 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Paul Ammann and Jeff Offutt. 2017. Introduction to software testing . Cambridge University Press

    work page 2017

  4. [4]

    Iván Arcuschin, Jett Janiak, Robert Krzyzanowski, Senthooran Ra- jamanoharan, Neel Nanda, and Arthur Conmy. 2025. Chain-of- thought reasoning in the wild is not always faithful. arXiv preprint arXiv:2503.08679 (2025)

    arXiv 2025

  5. [5]

    Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation.arXiv preprint arXiv:2503.11926 (2025)

    arXiv 2025

  6. [6]

    Gareth Bennett, Tracy Hall, Emily Winter, and Steve Counsell. 2024. Semgrep*: Improving the limited performance of static application security testing (sast) tools. In Proceedings of the 28th International Position: Intelligent Coding Systems Should Write Programs with Justifications Post-hoc Explainability Natural Language Programming Language Todo: exam...

    work page 2024

  7. [7]

    Milan Bhan, Nina Achache, Victor Legrand, Annabelle Blangero, and Nicolas Chesneau. 2023. Evaluating self-attention interpretability through human-grounded experimental protocol. In World Conference on Explainable Artificial Intelligence. Springer, 26–46

    work page 2023

  8. [8]

    Guanxu Chen, Dongrui Liu, Tao Luo, Lijie Hu, and Jing Shao. 2025. Beyond External Monitors: Enhancing Transparency of Large Lan- guage Models for Easier Monitoring. arXiv:2502.05242 [cs.CL] https: //arxiv.org/abs/2502.05242

    work page Pith review arXiv 2025

Show all 78 references
  1. [9]

    Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Car- son Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wag- ner, Fabien Roger, et al. 2025. Reasoning Models Don’t Always Say Xiangzhe Xu, Shiwei Feng, Zian Su, Chengpeng Wang, and Xiangyu Zhang Coding Req...

    2025 arXiv

  2. [10]

    Claude Sonnet 4

    Claude 2025. Claude Sonnet 4. https://www.anthropic.com/claude/ sonnet#benchmarks

    2025

  3. [11]

    Benjamin Delaware, Sorawit Suriyakarn, Clément Pit-Claudel, Qianchuan Ye, and Adam Chlipala. 2019. Narcissus: Correct-by- construction derivation of decoders and encoders from binary formats. Proceedings of the ACM on Programming Languages 3, ICFP (2019), 1–29

    2019

  4. [12]

    Mahdi Dhaini, Ege Erdogan, Smarth Bakshi, and Gjergji Kasneci. 2024. Explainability meets text summarization: A survey. In Proceedings of the 17th International Natural Language Generation Conference . 631– 645

    2024

  5. [13]

    Yangruibo Ding, Jinjun Peng, Marcus Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024. Semcoder: Training code language mod- els with comprehensive semantics reasoning. Advances in Neural Information Processing Systems 37 (2024), 60275–60308

    2024

  6. [14]

    Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. 2024. What did I do wrong? quantifying LLMs’ sensitivity and consistency to prompt engineering. arXiv preprint arXiv:2406.12334 (2024)

    2024 arXiv

  7. [15]

    Sarah Fakhoury, Markus Kuppe, Shuvendu K Lahiri, Tahina Ra- mananandro, and Nikhil Swamy. 2024. 3DGen: AI-Assisted Gen- eration of Provably Correct Binary Format Parsers. arXiv preprint arXiv:2404.10362 (2024)

    2024 arXiv

  8. [16]

    Jinyao Guo, Chengpeng Wang, Xiangzhe Xu, Zian Su, and Xiangyu Zhang. 2025. RepoAudit: An Autonomous LLM-Agent for Repository- Level Code Auditing. arXiv preprint arXiv:2501.18160 (2025)

    2025 arXiv

  9. [17]

    Sai Gurrapu, Ajay Kulkarni, Lifu Huang, Ismini Lourentzou, and Feras A Batarseh. 2023. Rationalization for explainable NLP: a survey. Frontiers in artificial intelligence 6 (2023), 1225093

    2023

  10. [18]

    Jónathan Heras and Ekaterina Komendantskaya. 2014. ML4PG: Ma- chine learning for Proof General. (2014)

    2014

  11. [19]

    Andrei Kozyrev, Gleb Solovev, Nikita Khramov, and Anton Podkopaev

  12. [20]

    Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021. Qed: A framework and dataset for explanations in question answering. Transactions of the Association for computational Linguistics 9 (2021), 790–806

    2021

  13. [21]

    Zhong Qiu Lin, Mohammad Javad Shafiee, Stanislav Bochkarev, Michael St Jules, Xiao Yu Wang, and Alexander Wong. 2019. Do explanations reflect decisions? A machine-centric strategy to quan- tify the performance of explainability algorithms. arXiv preprint arXiv:1910.07387 (2019)

    2019 arXiv

  14. [22]

    Jiawei Liu, Thanh Nguyen, Mingyue Shang, Hantian Ding, Xiaopeng Li, Yu Yu, Varun Kumar, and Zijian Wang. 2024. Learning code prefer- ence via synthetic evolution. arXiv preprint arXiv:2410.03837 (2024)

    2024 arXiv

  15. [23]

    Minghai Lu, Benjamin Delaware, and Tianyi Zhang. 2024. Proof automation with large language models. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1509–1520

    2024

  16. [24]

    Yingwei Ma, Rongyu Cao, Yongchang Cao, Yue Zhang, Jue Chen, Yibo Liu, Yuchen Liu, Binhua Li, Fei Huang, and Yongbin Li. 2024. Lingma swe-gpt: An open development-process-centric language model for automated software improvement. arXiv preprint arXiv:2411.00622 (2024)

    2024 arXiv

  17. [25]

    Anders Møller and Michael I Schwartzbach. 2012. Static program analysis. Notes. Feb (2012)

    2012

  18. [26]

    Glenford J Myers, Corey Sandler, and Tom Badgett. 2011. The art of software testing. John Wiley & Sons

    2011

  19. [27]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wain- wright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al . 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems...

    2022

  20. [28]

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. 2025. An empirical study of the non-determinism of chatgpt in code genera- tion. ACM Transactions on Software Engineering and Methodology 34, 2 (2025), 1–28

    2025

  21. [29]

    Bc ONDŘEJ PAVELA. 2023. Advanced Static Performance Analysis Using Meta Infer

    2023

  22. [30]

    Kexin Pei, David Bieber, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2023. Can large language models reason about program invari- ants?. In International Conference on Machine Learning . PMLR, 27496– 27520

    2023

  23. [31]

    Benjamin C Pierce. 2002. Types and programming languages . MIT press

    2002

  24. [32]

    Gordon D Plotkin. 1981. A structural approach to operational seman- tics. (1981)

    1981

  25. [33]

    Vaughan R Pratt. 1976. Semantical considerations on Floyd-Hoare logic. In 17th Annual Symposium on Foundations of Computer Science (sfcs 1976). IEEE, 109–121

    1976

  26. [34]

    Fazle Rabbi and Md Saeed Siddik. 2020. Detecting code comment inconsistency using siamese recurrent network. In Proceedings of the 28th international conference on program comprehension . 371–375

    2020

  27. [35]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023). Position: Intelligent Coding Syste...

    2023 arXiv

  28. [36]

    Ahmed M Salih, Zahra Raisi-Estabragh, Ilaria Boscolo Galazzo, Petia Radeva, Steffen E Petersen, Karim Lekadir, and Gloria Menegaz. 2025. A perspective on explainable artificial intelligence methods: SHAP and LIME. Advanced Intelligent Systems 7, 1 (2025), 2400304

    2025

  29. [37]

    Amir Samadi, Konstantinos Koufos, Kurt Debattista, and Mehrdad Dianati. 2024. SAFE-RL: Saliency-aware counterfactual explainer for deep reinforcement learning policies. IEEE Robotics and Automation Letters (2024)

    2024

  30. [38]

    Alex Sanchez-Stern, Emily First, Timothy Zhou, Zhanna Kaufman, Yuriy Brun, and Talia Ringer. 2023. Passport: Improving automated for- mal verification using identifiers. ACM Transactions on Programming Languages and Systems 45, 2 (2023), 1–30

    2023

  31. [39]

    Neela Sawant and Srinivasan H Sengamedu. 2022. Learning-based identification of coding best practices from software documentation. In 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 533–542

    2022

  32. [40]

    Pratinav Seth, Yashwardhan Rathore, Neeraj Kumar Singh, Chintan Chitroda, and Vinay Kumar Sankarapu. 2025. xai_evals: A Framework for Evaluating Post-Hoc Local Explanation Methods. arXiv preprint arXiv:2502.03014 (2025)

    2025 arXiv

  33. [41]

    Qingkai Shi, Xiao Xiao, Rongxin Wu, Jinguo Zhou, Gang Fan, and Charles Zhang. 2018. Pinpoint: Fast and precise sparse value flow anal- ysis for million lines of code. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation . 693–706

    2018

  34. [42]

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2024. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism. arXiv preprint arXiv:2407.10457 (2024)

    2024 arXiv

  35. [43]

    Nataliia Stulova, Arianna Blasi, Alessandra Gorla, and Oscar Nier- strasz. 2020. Towards detecting inconsistent comments in java source code automatically. In 2020 IEEE 20th international working conference on source code analysis and manipulation (SCAM) . IEEE, 65–69

    2020

  36. [44]

    Alona Sydorova, Nina Poerner, and Benjamin Roth. 2019. Interpretable question answering on knowledge bases and text. arXiv preprint arXiv:1906.10924 (2019)

    2019 arXiv

  37. [45]

    Lin Tan, Ding Yuan, Gopal Krishna, and Yuanyuan Zhou. 2007. /* icomment: Bugs or bad comments?*. In Proceedings of twenty-first ACM SIGOPS symposium on Operating systems principles . 145–158

    2007

  38. [46]

    Tian Tan and Yue Li. 2022. Tai-e: a static analysis framework for java by harnessing the best designs of classics. arXiv preprint arXiv:2208.00337 (2022)

    2022 arXiv

  39. [47]

    Tian Tan and Yue Li. 2023. Tai-e: A developer-friendly static analysis framework for java by harnessing the good designs of classics. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1093–1105

    2023

  40. [48]

    Wensheng Tang, Dejun Dong, Shijie Li, Chengpeng Wang, Peisen Yao, Jinguo Zhou, and Charles Zhang. 2024. Octopus: Scaling Value-Flow Analysis via Parallel Collection of Realizable Path Conditions. ACM Transactions on Software Engineering and Methodology 33, 3 (2024), 1–33

    2024

  41. [49]

    CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A Choquette-Choo, Jingyue Shen, Joe Kelley, et al. 2024. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409 (2024)

    2024 arXiv

  42. [50]

    Marcos V Treviso and André FT Martins. 2020. The explanation game: Towards prediction explainability through sparse communication. arXiv preprint arXiv:2004.13876 (2020)

    2020 arXiv

  43. [51]

    Chengpeng Wang, Yifei Gao, Wuqi Zhang, Xuwei Liu, Qingkai Shi, and Xiangyu Zhang. 2024. LLMSA: A Compositional Neuro-Symbolic Approach to Compilation-free and Customizable Static Analysis.arXiv preprint arXiv:2412.14399 (2024)

    2024 arXiv

  44. [52]

    Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, Xiaoheng Xie, and Xiangyu Zhang. 2024. LLMDFA: analyzing dataflow in code with large language models. Advances in Neural Information Processing Systems 37 (2024), 131545–131574

    2024

  45. [53]

    Chengpeng Wang, Wuqi Zhang, Zian Su, Xiangzhe Xu, and Xiangyu Zhang. 2024. Sanitizing Large Language Models in Bug Detection with Data-Flow. In Findings of the Association for Computational Linguistics: EMNLP 2024. 3790–3805

    2024

  46. [54]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al

  47. [55]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  48. [56]

    arXiv preprint arXiv:2407.16741 (2024)

    Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741 (2024)

    2024 arXiv

  49. [57]

    Rongxin Wu, Yuxuan He, Jiafeng Huang, Chengpeng Wang, Wensheng Tang, Qingkai Shi, Xiao Xiao, and Charles Zhang. 2024. Libalchemy: A two-layer persistent summary design for taming third-party libraries in static bug-finding systems. In Proceedings of the IEEE/ACM 46th Internati...

    2024

  50. [58]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang

  51. [59]

    Xiangzhe Xu, Shiwei Feng, Yapeng Ye, Guangyu Shen, Zian Su, Siyuan Cheng, Guanhong Tao, Qingkai Shi, Zhuo Zhang, and Xi- angyu Zhang. 2023. Improving binary code similarity transformer models by semantics-driven instruction deemphasis. In Proceedings of the 32nd ACM SIGSOFT In...

    2023

  52. [60]

    Xiangzhe Xu, Zian Su, Jinyao Guo, Kaiyuan Zhang, Zhenting Wang, and Xiangyu Zhang. 2024. ProSec: Fortifying Code LLMs with Proac- tive Security Alignment. arXiv preprint arXiv:2411.12882 (2024)

    2024 arXiv

  53. [61]

    Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang

  54. [62]

    arXiv preprint arXiv:2407.01489 (2024)

    Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489 (2024)

    2024 arXiv

  55. [63]

    Chenyang Yang, Yike Shi, Qianou Ma, Michael Xieyang Liu, Christian Kästner, and Tongshuang Wu. 2025. What Prompts Don’t Say: Under- standing and Managing Underspecification in LLM Prompts. arXiv preprint arXiv:2505.13360 (2025)

    2025 arXiv

  56. [64]

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent- computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems 37 (2024), 50528–50652

    2024

  57. [65]

    Xiangzhe Xu, Jinhua Wu, Yuting Wang, Zhenguo Yin, and Pengfei Li

  58. [66]

    Boyu Zhang, Tianyu Du, Junkai Tong, Xuhong Zhang, Kingsum Chow, Sheng Cheng, Xun Wang, and Jianwei Yin. 2024. SecCoder: Towards Generalizable and Robust Secure Code Generation. arXiv preprint arXiv:2410.01488 (2024)

    2024 arXiv

  59. [67]

    Xiangzhe Xu, Zhuo Zhang, Zian Su, Ziyang Huang, Shiwei Feng, Yapeng Ye, Nan Jiang, Danning Xie, Siyuan Cheng, Lin Tan, et al. 2023. Leveraging generative models to recover variable names from stripped binary. arXiv e-prints (2023), arXiv–2306

    2023

  60. [68]

    Yichi Zhang, Zixi Liu, Yang Feng, and Baowen Xu. 2024. Leveraging Large Language Model to Assist Detecting Rust Code Comment Incon- sistency. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 356–366

    2024

  61. [69]

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2024. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology 15, 2 (2024), 1–38

    2024

  62. [70]

    Juan Zhai, Xiangzhe Xu, Yu Shi, Guanhong Tao, Minxue Pan, Shiqing Ma, Lei Xu, Weifeng Zhang, Lin Tan, and Xiangyu Zhang. 2020. CPC: Automatically classifying and propagating natural language comments via program analysis. In Proceedings of the ACM/IEEE 42nd Interna- tional con...

    2020

  63. [71]

    Hao Zhong and Zhendong Su. 2013. Detecting API documentation errors. In Proceedings of the 2013 ACM SIGPLAN international confer- ence on Object oriented programming systems languages & applications . 803–816

    2013

  64. [72]

    Qirun Zhang, Chengnian Sun, and Zhendong Su. 2017. Skeletal pro- gram enumeration for rigorous compiler testing. In Proceedings of the 38th ACM SIGPLAN conference on programming language design and implementation. 347–361. Xiangzhe Xu, Shiwei Feng, Zian Su, Chengpeng Wang, and...

    2017

  65. [73]

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. ProSA: Assessing and understanding the prompt sensitivity of LLMs. arXiv preprint arXiv:2410.12405 (2024)

    2024 arXiv

  66. [75]

    Mingwei Zheng, Danning Xie, Qingkai Shi, Chengpeng Wang, and Xiangyu Zhang. 2025. Validating network protocol parsers with trace- able rfc document interpretation. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1772–1794

    2025

  67. [77]

    Yu Zhou, Ruihang Gu, Taolue Chen, Zhiqiu Huang, Sebastiano Panichella, and Harald Gall. 2017. Analyzing APIs documentation and code to detect directive defects. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) . IEEE, 27–37

    2017

  68. [2021]

    In International Conference on Computer Aided Verification

    Automatic generation and validation of instruction encoders and decoders. In International Conference on Computer Aided Verification . Springer, 728–751

  69. [2023]

    arXiv preprint arXiv:2312.02120 (2023)

    Magicoder: Empowering code generation with oss-instruct. arXiv preprint arXiv:2312.02120 (2023)

    2023 arXiv

  70. [2024]

    In Pro- ceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering

    CoqPilot, a plugin for LLM-based generation of proofs. In Pro- ceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2382–2385

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.