pith. sign in

arxiv: 2606.25356 · v1 · pith:UNC6YQVCnew · submitted 2026-06-24 · 💻 cs.CR · cs.SE

Representation Matters: An Empirical Study of Program Representations for LLM Vulnerability Reasoning

Andrew Stoltman , Johnathan Tang , Haipeng Cai This is my paper

Pith reviewed 2026-06-25 21:12 UTC · model grok-4.3

classification 💻 cs.CR cs.SE
keywords vulnerability detectionlarge language modelsprogram representationsabstract syntax treeprogram dependence graphstatic analysisprompt engineeringsecurity
0
0 comments X

The pith

Choosing structured program representations such as AST combined with PDG raises LLM accuracy on vulnerability reasoning from 53.5 percent with raw source to 83.2 percent.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests the assumption that feeding LLMs more raw source code improves their ability to detect vulnerabilities in C and C++ programs. It builds RepBench to convert the same set of 107 real vulnerability cases into ten different representations, including trees, graphs, and combinations, then runs each through an identical Chain-of-Thought prompting setup. Graph-only versions consistently beat both raw source and source-plus-graph versions while using far shorter prompts. The results point to a dilution effect in which extra source text makes the key structural signals less noticeable to the model. If the pattern holds, security tools that rely on LLMs would benefit from preprocessing code with static analysis rather than simply increasing the volume of source sent to the model.

Core claim

Representation choice substantially affects LLM vulnerability reasoning. The strongest variant, AST+PDG, achieves 83.2% accuracy, compared with 53.5% for raw source. Graph-only prompts outperform both source-only and source-plus-graph prompts while requiring far less prompt overhead. These results reveal a context dilution effect: adding raw source code to compact structural graph evidence can degrade reasoning by making vulnerability-relevant evidence less salient.

What carries the argument

RepBench benchmark that converts the same vulnerability testcases into multiple representations (raw source, AST, CFG, PDG, their combinations, and ePDGs) and evaluates them under a fixed Chain-of-Thought and structured-output protocol.

If this is right

  • Graph-only prompts deliver a better accuracy-overhead tradeoff than prompts containing raw source code.
  • Static analysis can function as an effective preprocessing layer for constructing prompts in LLM-based security reasoning.
  • Adding raw source code to graph representations reduces the salience of vulnerability evidence.
  • The observed performance differences hold across the five CWE categories in the evaluated corpus.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same representation experiments could be applied to other LLM code-reasoning tasks such as repair or summarization to test whether graph forms help there as well.
  • Security tools built around LLMs could incorporate static-analysis front ends to produce shorter, more targeted prompts before model inference.
  • Prompt design for code might shift emphasis from expanding context volume toward selecting compact structural signals.

Load-bearing premise

The 107 Joern-based testcases drawn from PrimeVul across five CWE categories are representative of real-world C/C++ vulnerabilities and the fixed prompting protocol provides an unbiased comparison across representation variants.

What would settle it

Re-running the identical comparison on a substantially larger or more varied collection of vulnerabilities and finding that raw source or source-plus-graph inputs achieve equal or higher accuracy than AST+PDG.

Figures

Figures reproduced from arXiv: 2606.25356 by Andrew Stoltman, Haipeng Cai, Johnathan Tang.

Figure 1
Figure 1. Figure 1: Overview of the RepBench benchmark pipeline across three stages. Stage 1 (Inputs): PrimeVul vulnerability testcases are curated and checked out from source repositories; testcase metadata (CWE type, CVE identifier, source location) is extracted and passed directly to the prompt constructor (dashed arrow). Stage 2 (Code Representation Generation): Source code is processed through two parallel tracks—the sta… view at source ↗
Figure 2
Figure 2. Figure 2: Curated accuracy versus average input-token count for each representation variant. Colors [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Curated accuracy versus CWE category grouped by prompt family. Colors indicate [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Curated accuracy by CWE category and representation variant. Cell color indicates [PITH_FULL_IMAGE:figures/full_fig_p023_4.png] view at source ↗
read the original abstract

Large Language Models (LLMs) are increasingly used for automated vulnerability detection, but it remains unclear how program structure and semantics should be represented for LLM-based reasoning. Most prompting-based approaches provide raw source code, implicitly assuming that more source-level context gives the model better evidence. This paper challenges that assumption through RepBench, an empirical benchmark comparing raw source code with static-analysis-based program representations. RepBench converts real-world C/C++ vulnerability testcases into multiple representations: raw source, Abstract Syntax Trees (ASTs), Control-Flow Graphs (CFGs), Program Dependence Graphs (PDGs), their combinations, and an auxiliary track of enriched PDGs (ePDGs). Using a curated PrimeVul-derived corpus of 107 Joern-based testcases across five CWE categories, we evaluate ten representation variants under a fixed Chain-of-Thought and structured-output protocol, plus 19 additional ePDG cases generated through VulChecker/Hector. Representation choice substantially affects LLM vulnerability reasoning. The strongest variant, AST+PDG, achieves 83.2% accuracy, compared with 53.5% for raw source. At the family level, graph-only prompts outperform both source-only and source-plus-graph prompts while requiring far less prompt overhead. These results reveal a context dilution effect: adding raw source code to compact structural graph evidence can degrade reasoning by making vulnerability-relevant evidence less salient. Overall, our findings show that carefully selected structural representations offer a better accuracy-overhead tradeoff than simply giving LLMs more raw input, and suggest that static analysis can serve as an effective prompt-construction layer for security-focused LLM reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript presents RepBench, a benchmark comparing ten program representation variants (raw source, AST, CFG, PDG, combinations, and ePDG) for LLM vulnerability reasoning on a curated PrimeVul-derived corpus of 107 Joern-extracted C/C++ functions across five CWE categories, plus 19 additional ePDG cases. Under a fixed Chain-of-Thought and structured-output protocol, it reports that AST+PDG achieves 83.2% accuracy versus 53.5% for raw source, that graph-only prompts outperform source-inclusive variants with lower overhead, and that adding raw source can cause context dilution.

Significance. If the results hold under a more transparent and representative protocol, the work supplies concrete evidence that static-analysis-derived graph representations can improve LLM accuracy on vulnerability tasks while reducing prompt size, challenging the default of providing raw source code. The multi-variant comparison and identification of the dilution effect offer actionable guidance for prompt construction in security applications of LLMs.

major comments (2)
  1. [Abstract / Dataset] Abstract and § on dataset construction: the headline accuracy gap (AST+PDG 83.2% vs. raw source 53.5%) rests on a manually curated 107-case subset. No selection protocol, no comparison of function-size or vulnerability-location statistics to the parent PrimeVul distribution, and no error bars or cross-validation are supplied; this selection could preferentially retain cases whose signals are compactly captured by PDG edges, making the observed gap an artifact rather than a general property.
  2. [Abstract / Methods] Experimental protocol (abstract and methods): the manuscript supplies no information on the LLM family, number of trials per representation, temperature/sampling settings, or exact prompt templates. Without these details the controlled comparison across variants cannot be reproduced or assessed for statistical significance, leaving the central claim only moderately supported.
minor comments (2)
  1. [Abstract] Clarify how the 19 additional ePDG cases generated via VulChecker/Hector are integrated with the main 107-case results and whether they alter the family-level conclusions.
  2. Add a table or appendix listing the exact ten representation variants with token counts or prompt-length statistics to support the overhead claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and indicate the revisions we will make to improve transparency, reproducibility, and the strength of our claims.

read point-by-point responses

  1. Referee: [Abstract / Dataset] Abstract and § on dataset construction: the headline accuracy gap (AST+PDG 83.2% vs. raw source 53.5%) rests on a manually curated 107-case subset. No selection protocol, no comparison of function-size or vulnerability-location statistics to the parent PrimeVul distribution, and no error bars or cross-validation are supplied; this selection could preferentially retain cases whose signals are compactly captured by PDG edges, making the observed gap an artifact rather than a general property.

    Authors: We agree that the curation details require clarification. The 107 functions were drawn from PrimeVul and retained only if Joern successfully extracted the full set of graphs (AST, CFG, PDG) needed for all ten representation variants; this was a practical filter to ensure a uniform extraction pipeline rather than an attempt to favor PDG-friendly cases. We will revise the dataset section to state this criterion explicitly and add available descriptive statistics on function lengths and vulnerability locations. A formal statistical comparison against the full PrimeVul distribution was not performed and cannot be added retrospectively without re-running the entire extraction pipeline; we will note this as a limitation. Error bars and cross-validation were omitted because the study used a single fixed test set under identical prompting conditions to isolate representation effects; we will add a short discussion of this design choice and, if space permits, report variance from repeated LLM calls on the same set. revision: partial

  2. Referee: [Abstract / Methods] Experimental protocol (abstract and methods): the manuscript supplies no information on the LLM family, number of trials per representation, temperature/sampling settings, or exact prompt templates. Without these details the controlled comparison across variants cannot be reproduced or assessed for statistical significance, leaving the central claim only moderately supported.

    Authors: The referee is correct that these parameters were not reported. The experiments used a single commercial LLM family with temperature set to 0 for determinism, one trial per representation, and a fixed Chain-of-Thought template that requested structured JSON output. We will add a new subsection in Methods that names the exact model, lists all sampling settings, states the number of trials, and reproduces the prompt templates (or places them in an appendix). This will allow readers to assess statistical significance and fully reproduce the comparison. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark reports direct measurements with no derivation chain

full rationale

The paper's central results are accuracy figures (AST+PDG 83.2%, raw source 53.5%) obtained by running a fixed LLM prompting protocol on an external, curated corpus of 107 PrimeVul-derived test cases. No equations, fitted parameters, or self-citations are used to derive these numbers; they are direct experimental outputs. The protocol and corpus selection are described as fixed inputs rather than outputs of the analysis itself. No self-definitional, fitted-input, or self-citation-load-bearing steps appear in the reported chain.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on empirical performance differences observed across representation variants; no free parameters are fitted to produce the accuracies, and no new entities are postulated beyond standard static-analysis outputs.

axioms (2)
  • domain assumption Static analysis tools such as Joern can accurately extract AST, CFG, and PDG representations from C/C++ vulnerability testcases.
    Invoked when the paper converts the corpus into the evaluated representations.
  • domain assumption A single fixed Chain-of-Thought and structured-output protocol yields comparable reasoning conditions across all representation variants.
    Invoked by the statement that all variants were evaluated under the same protocol.

pith-pipeline@v0.9.1-grok · 5824 in / 1554 out tokens · 36786 ms · 2026-06-25T21:12:44.621578+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 3 canonical work pages

  1. [1]

    Compilers: Principles, techniques, and tools, 2006

    Alfred V Aho, Monica S Lam, Ravi Sethi, and Jeffrey D Ullman. Compilers: Principles, techniques, and tools, 2006

    2006

  2. [2]

    Vulnerability detection with code language models: How far are we?arXiv preprint arXiv:2403.18624, 2024

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. Vulnerability detection with code language models: How far are we?arXiv preprint arXiv:2403.18624, 2024

    work page arXiv 2024

  3. [3]

    Detecting kernel memory leaks in specialized modules with ownership reasoning

    Navid Emamdoost, Qiushi Wu, Kangjie Lu, and Stephen McCamant. Detecting kernel memory leaks in specialized modules with ownership reasoning. InThe 2021 Annual Network and Distributed System Security Symposium (NDSS’21), pages 1–15, 2021

    2021

  4. [4]

    Ottenstein, and Joe D

    Jeanne Ferrante, Karl J. Ottenstein, and Joe D. Warren. The program dependence graph and its use in optimization.ACM Trans. Program. Lang. Syst., 9(3):319–349, July 1987

    1987

  5. [5]

    FreeRDP: A remote desktop protocol implementation.https: //github.com/FreeRDP/FreeRDP

    FreeRDP and contributors. FreeRDP: A remote desktop protocol implementation.https: //github.com/FreeRDP/FreeRDP. 31

  6. [6]

    Jiawei Guo, Yu Nong, Zhiqiang Lin, and Haipeng Cai. FreeLens: Revealing mobile app feature differences and their security and privacy implications across geographical regions.IEEE Transactions on Dependable and Secure Computing, 2026

    2026

  7. [7]

    VerLog: Enhancing release note generation for android apps using large language models.Proceedings of the ACM on Software Engineering, 2(ISSTA):1910–1932, 2025

    Jiawei Guo, Haoran Yang, and Haipeng Cai. VerLog: Enhancing release note generation for android apps using large language models.Proceedings of the ACM on Software Engineering, 2(ISSTA):1910–1932, 2025

    1910

  8. [8]

    CBMC–c bounded model checker

    Daniel Kroening and Michael Tautschnig. CBMC–c bounded model checker. InInternational Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pages 389–391, 2014

    2014

  9. [9]

    {LLMxCPG}:{Context-Aware}vulnerability detection through code property{Graph-Guided} large language models

    Ahmed Lekssays, Hamza Mouhcine, Khang Tran, Ting Yu, and Issa Khalil. {LLMxCPG}:{Context-Aware}vulnerability detection through code property{Graph-Guided} large language models. In34th USENIX Security Symposium (USENIX Security 25), pages 489–507, 2025

    2025

  10. [10]

    A practical fuzzer for the python runtime system

    Wen Li and Haipeng Cai. A practical fuzzer for the python runtime system. InACM International Conference on the Foundations of Software Engineering (FSE), Tool Demos, pages 1–5, 2026

    2026

  11. [11]

    SV-TrustEval-C: Evaluating structure and semantic reasoning in large language models for source code vulnerability analysis

    Yansong Li, Paula Branco, Alexander M Hoole, Manish Marwah, Hari Manassery Koduvely, Guy-Vincent Jourdan, and Stephan Jou. SV-TrustEval-C: Evaluating structure and semantic reasoning in large language models for source code vulnerability analysis. In2025 IEEE Symposium on Security and Privacy (SP), pages 3014–3032. IEEE, 2025

    2025

  12. [12]

    Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics, 12:157–173, 2024

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the association for computational linguistics, 12:157–173, 2024

    2024

  13. [13]

    Pre-training by predicting program dependencies for vulnerability analysis tasks

    Zhongxin Liu, Zhijie Tang, Junwei Zhang, Xin Xia, and Xiaohu Yang. Pre-training by predicting program dependencies for vulnerability analysis tasks. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13, 2024

    2024

  14. [14]

    In32nd USENIX Security Symposium (USENIX Security 23), pages 6557–6574, 2023

    Yisroel Mirsky, George Macon, Michael Brown, Carter Yagemann, Matthew Pruett, Evan Downing, Sukarno Mertoguno, and Wenke Lee.{VulChecker}: Graph-based vulnerability localization in source code. In32nd USENIX Security Symposium (USENIX Security 23), pages 6557–6574, 2023

    2023

  15. [15]

    Common Weakness Enumeration (CWE).https://cwe.mitre.org/,

    MITRE Corporation. Common Weakness Enumeration (CWE).https://cwe.mitre.org/,

  16. [16]

    Accessed: June 8, 2026

    2026

  17. [17]

    A controlled experiment of different code representations for learning-based bug repair

    Marjane Namavar, Noor Nashid, and Ali Mesbah. A controlled experiment of different code representations for learning-based bug repair. corr abs/2110.14081 (2021).arXiv preprint arXiv:2110.14081, 2021

    work page arXiv 2021

  18. [18]

    Chain- of-thought prompting of large language models for discovering and fixing software vulnerabilities

    Yu Nong, Mohammed Aldeen, Long Cheng, Hongxin Hu, Feng Chen, and Haipeng Cai. Chain- of-thought prompting of large language models for discovering and fixing software vulnerabilities. arXiv preprint arXiv:2402.17230, 2024

    work page arXiv 2024

  19. [19]

    Evaluating and comparing memory error vulnerability detectors.Information and Software Technology, 137:106614, 2021

    Yu Nong, Haipeng Cai, Pengfei Ye, Li Li, and Feng Chen. Evaluating and comparing memory error vulnerability detectors.Information and Software Technology, 137:106614, 2021. 32

    2021

  20. [20]

    VGX: Large-scale sample generation for boosting learning-based software vulnerability analyses

    Yu Nong, Richard Fang, Guangbei Yi, Kunsong Zhao, Xiapu Luo, Feng Chen, and Haipeng Cai. VGX: Large-scale sample generation for boosting learning-based software vulnerability analyses. InIEEE/ACM International Conference on Software Engineering (ICSE), pages 1837–1849, 2024

    2024

  21. [21]

    VulGen: Realistic vulnerable sample generation via pattern mining and deep learning

    Yu Nong, Yuzhe Ou, Michael Pradel, Feng Chen, and Haipeng Cai. VulGen: Realistic vulnerable sample generation via pattern mining and deep learning. InIEEE/ACM International Conference on Software Engineering (ICSE), pages 2527–2539, 2023

    2023

  22. [22]

    Open science in software engineering: A study on deep learning-based vulnerability detection.IEEE Transactions on Software Engineering (TSE), 49(4):1983–2005, 2022

    Yu Nong, Rainy Sharma, Abdelwahab Hamou-Lhadj, Xiapu Luo, and Haipeng Cai. Open science in software engineering: A study on deep learning-based vulnerability detection.IEEE Transactions on Software Engineering (TSE), 49(4):1983–2005, 2022

    1983

  23. [23]

    In34th USENIX Security Symposium (USENIX Security 25), pages 4481–4500, 2025

    Yu Nong, Haoran Yang, Long Cheng, Hongxin Hu, and Haipeng Cai.{APPATCH}: Automated adaptive prompting large language models for{Real-World}software vulnerability patching. In34th USENIX Security Symposium (USENIX Security 25), pages 4481–4500, 2025

    2025

  24. [24]

    Assessing and improving prompting large language models for software vulnerability analysis

    Yu Nong, Guangbei Yi, Mohammed Aldeen, Long Cheng, Hongxin Hu, and Haipeng Cai. Assessing and improving prompting large language models for software vulnerability analysis. ACM Transactions on Software Engineering and Methodology (TOSEM), pages 1–53, 2026

    2026

  25. [25]

    Ptlvd:program slicing and transformer-based line-level vulnerability detection system

    Tao Peng, Shixu Chen, Fei Zhu, Junwei Tang, Junping Liu, and Xinrong Hu. Ptlvd:program slicing and transformer-based line-level vulnerability detection system. In2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM), pages 162–173, 2023

    2023

  26. [26]

    Openexr project

    OpenEXR project and contributors. Openexr project. https://github.com/ AcademySoftwareFoundation/openexr. Academy Software Foundation

  27. [27]

    Qwen3.6-Plus: Towards real world agents.https://qwen.ai/blog?id=qwen3.6, 2026

    Qwen Team. Qwen3.6-Plus: Towards real world agents.https://qwen.ai/blog?id=qwen3.6, 2026

    2026

  28. [28]

    Deep learning representations of programs: A systematic literature review.ACM Computing Surveys, 58(5):1–37, 2025

    Deepika Shanmugasundaram, Pallavi Arivukkarasu, Huaming Chen, and Haipeng Cai. Deep learning representations of programs: A systematic literature review.ACM Computing Surveys, 58(5):1–37, 2025

    2025

  29. [29]

    LLMs in software security: A survey of vulnerability detection techniques and insights.ACM Computing Surveys, 58(5):1–35, 2025

    Ze Sheng, Zhicheng Chen, Shuning Gu, Heqing Huang, Guofei Gu, and Jeff Huang. LLMs in software security: A survey of vulnerability detection techniques and insights.ACM Computing Surveys, 58(5):1–35, 2025

    2025

  30. [30]

    Repbench

    SRestLabUB. Repbench. https://github.com/SRestLabUB/RepBench, 2026. GitHub reposi- tory

    2026

  31. [31]

    Abstract syntax tree for programming language understanding and representation.ACM Transactions on Software Engineering and Methodology, 2026

    Weisong Sun, Chunrong Fang, Yun Miao, Mengzhe Yuan, Zhenpeng Chen, Yuchen Chen, Quanjun Zhang, An Guo, Xiang Chen, and Zhenyu Chen. Abstract syntax tree for programming language understanding and representation.ACM Transactions on Software Engineering and Methodology, 2026

    2026

  32. [32]

    On the impact of multiple source code representations on software engineering tasks—an empirical study.Journal of Systems and Software, 210:111941, 2024

    Karthik Chandra Swarna, Noble Saji Mathews, Dheeraj Vagavolu, and Sridhar Chimalakonda. On the impact of multiple source code representations on software engineering tasks—an empirical study.Journal of Systems and Software, 210:111941, 2024. 33

    2024

  33. [33]

    Modeling and discovering vulnerabilities with code property graphs

    Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. Modeling and discovering vulnerabilities with code property graphs. In2014 IEEE Symposium on Security and Privacy, pages 590–604, 2014

    2014

  34. [34]

    Learning to detect and localize multilingual bugs.Proceedings of the ACM on Software Engineering, 1(FSE):2190–2213, 2024

    Haoran Yang, Yu Nong, Tao Zhang, Xiapu Luo, and Haipeng Cai. Learning to detect and localize multilingual bugs.Proceedings of the ACM on Software Engineering, 1(FSE):2190–2213, 2024

    2024

  35. [35]

    Exploring and improving real-world vulnerability data generation via prompting large language models

    Guangbei Yi, Yu Nong, Minzhang Li, and Haipeng Cai. Exploring and improving real-world vulnerability data generation via prompting large language models. InIEEE/ACM International Conference on Software Engineering (ICSE), pages 2044–2056, 2026

    2044

  36. [36]

    Large language model for vulnerability detection and repair: Literature review and the road ahead.ACM Transactions on Software Engineering and Methodology, 34(5):1–31, 2025

    Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. Large language model for vulnerability detection and repair: Literature review and the road ahead.ACM Transactions on Software Engineering and Methodology, 34(5):1–31, 2025

    2025

  37. [37]

    Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks.Advances in Neural Information Processing Systems (NeurIPS), 32:10197–10207, 2019. 34

    2019