Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An automated LLM pipeline turns real-world audit reports into a large standardized smart contract vulnerability dataset, labeled by CWE category.

desk verdict A genuinely large and useful dataset, but the quality claims rest on a small self-annotated sample; worth refereeing with demand for external validation. read the letter →

arxiv 2506.18795 v1 pith:H5WQLUMX submitted 2025-06-23 cs.CR cs.SE

Jiachi Chen , Yiming Shen , Jiashuo Zhang , Zihao Li , John Grundy , Zhenzhe Shao , Yanlin Wang , Jiashui Wang
show 2 more authors
Ting Chen Zibin Zheng
This is my paper · ORCID
classification cs.CRcs.SE
keywords smartcontractvulnerabilitydatasetLLMCWEauditreportsdetectionconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large smart contract vulnerability datasets no longer have to be built by hand. It introduces an automated, LLM-driven pipeline that reads real-world audit reports, extracts structured information about each identified vulnerability, and labels that information with standard Common Weakness Enumeration (CWE) categories. Run over 6,454 reports from 47 professional audit teams, the pipeline produced 81,390 Solidity files and 27,497 labeled vulnerability findings in 229.5 hours, a scale that would have required many person-months of manual effort on older datasets. The dataset is offered as a standardized benchmark for evaluating detection tools and for aligning research priorities with real-world security concerns.

What carries the argument

The framework pipeline is the paper's central object, combining three LLM techniques: a Semantic Chunker that splits long audit reports into self-contained chunks; a MapReduce Extractor that maps each chunk into structured JSON of project metadata and vulnerability fields and then reduces and merges them; and a Tree-of-Thoughts Hierarchical Classifier that walks the CWE hierarchy level by level, selecting the k most relevant child nodes at each step and falling back to a higher node when that is more appropriate. A Code Fetcher then retrieves the audited contract source code at the pinned commit to complete each entry.

What would settle it

Have an independent team of auditors, different from the authors and the report's original authors, manually re-annotate a new random sample of, say, 300 findings drawn uniformly from the 296 CWE categories and compute precision and Krippendorff's alpha against forge's labels; if the precision falls well below 95.6% or the agreement falls below the 0.80 threshold, the dataset quality claim fails for the broader corpus.

Watch

Extended reading notes

Core claim

The central claim is that an LLM-driven pipeline can construct a large-scale, standardized smart contract vulnerability dataset directly from real-world audit reports, with quality close to that of human annotation. The paper reports extraction precision of 95.6%, macro-F1 of 86.1%, and Krippendorff's alpha of 0.87 for CWE classification against two expert annotators on a 96-sample evaluation. The dataset covers 296 CWE categories, uses recent Solidity compiler versions in 59.0% of projects, and shows that existing tools perform poorly on it: the best F1 among 13 detection tools was 18.59%, with most below 1%.

Load-bearing premise

The 96-sample manual evaluation is representative of all 27,497 findings, so the measured 95.6% extraction precision and 0.87 classification agreement apply to the whole dataset, including the findings used as ground truth when benchmarking the 13 tools.

Editorial extensions

If this is right

  • A standardized CWE-labeled dataset of 27,497 real-world findings gives security tool researchers a common benchmark with severity and frequency information attached, so tool evaluations become comparable across the field.
  • The reported F1 scores below 19% for existing tools imply current detection technology is far from production-ready on realistic contracts averaging 2,575 lines of code, not just on small synthetic samples.
  • Comparing forge's top-severity CWE categories with the research community's favorite target vulnerabilities identifies concrete under-studied areas such as CWE-940 and CWE-347.
  • Because the pipeline is automated, the dataset can be re-generated on newly published audit reports, allowing the benchmark to track the evolving vulnerability landscape rather than becoming stale.
  • Mapping smart contract findings to CWE for the first time at this scale draws a quantitative contrast with traditional software security concerns and with the OWASP Top 10, showing where web-application lessons do not transfer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable implication the paper leaves implicit: if the pipeline is rerun with a different base LLM or with future longer-context models, the quality statistics should remain stable or improve, since the framework is model-agnostic.
  • The severity-frequency analysis suggests a practical target for the community: developing tools that detect high-severity but uncommon business-logic vulnerabilities, since frequency-based benchmarks alone would reward tools that find only common low-severity issues.
  • The claimed quality rests on the 96-sample manual evaluation; a larger independent sample, sampled per audit team rather than uniformly, would strengthen or bound the generalization of the 95.6% precision figure.
  • The paper's extraction recall of 78.4% implies the underlying audit report corpus still contains roughly one in five findings that the pipeline missed, so the dataset is a lower bound on the real vulnerability population.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents FORGE, an LLM-driven framework that automatically constructs smart contract vulnerability datasets from professional audit reports. The pipeline comprises a semantic chunker, a map-reduce extractor, a tree-of-thoughts hierarchical CWE classifier, and a code fetcher. Running on 6,454 real-world audit reports, the authors report 81,390 Solidity files, 27,497 vulnerability findings, and 296 CWE categories; they claim high extraction precision (95.6%), substantial classification agreement with human experts (Krippendorff's alpha = 0.87), and benchmark 13 tools to show that existing detectors perform poorly (max F1 18.59%). They also analyze severity-frequency distributions to argue that academic research priorities misalign with real-world smart contract vulnerabilities.

Significance. If validated, this work would provide a substantial community resource: a large, real-world-sourced, CWE-standardized vulnerability dataset constructed at far lower cost than manual annotation (229.5 hours vs. 44 person-months for DAppSCAN). The authors creditably release the dataset, code, and experimental results, and the pipeline design is well motivated by concrete challenges of audit-report complexity and CWE hierarchy navigation. The comparison with prior datasets (SmartBugs, DAppSCAN) and the tool benchmark address practical needs. However, the headline quality claims rest on a single 96-sample manual evaluation with author-provided labels, and the RQ4 benchmark treats FORGE-generated labels as ground truth, so the evidence currently underdetermines the paper's central assertions.

major comments (4)
  1. [§4.2-§4.3] The 96-sample manual evaluation (95% confidence, ±10) is insufficient to support the claim of high precision and expert-level consistency across the dataset's 296 CWE categories and 27,497 findings. A simple random sample of 96 findings provides essentially no coverage of categories that occur only a handful of times, so the long tail of the taxonomy—including low-frequency/high-severity classes that drive the RQ4 benchmark and the Figure 6 risk-priority analysis—is unvalidated. The paper should stratify sampling by CWE category or pillar, report per-category (or per-pillar) agreement, and justify that the sampled reports are representative of the 6,454-report corpus.
  2. [§4.3] The human reference labels were produced by two authors of this paper, who also defined the CWE mapping-allowed notes and the ToT prompts used by FORGE (§3.4.1). This creates a substantial risk that FORGE-human agreement reflects shared labeling conventions rather than independent expert correctness. No human-human inter-rater agreement is reported, so the k-alpha = 0.87 does not by itself license the conclusion of 'expert-level classification consistency' without an external annotation study.
  3. [§4.4] The RQ4 tool benchmark treats every FORGE-generated dataset label as ground truth when computing tool F1 scores. Given that extraction precision for vulnerability findings is 91.7% (Table 2) and classification k-alpha is 0.87, label noise will propagate directly into the TP and FN counts for all 13 tools. The benchmark conclusion that existing tools are severely limited (max F1 = 18.59%) needs a sensitivity analysis, such as recomputing the scores on a manually verified subset or perturbing labels according to the measured error rates, before it can be taken as a reliable statement about tool capability.
  4. [§4.4] The contract-level true-positive definition ('if a tool detects any vulnerability within the contract that contains labeled vulnerabilities, we count it as a true positive') is ambiguous and likely over-counts true positives when a contract contains multiple labeled vulnerabilities but the tool detects only one. The paper should clarify whether TP, FP, and FN are counted per finding, per contract, or per alert, and should report a location-matched evaluation so that Table 3's precision and recall values are interpretable.
minor comments (5)
  1. [§4.1] The sentence 'each project contains an average of n Solidity smart contract files' should state the actual value from Table 1 (12 files), not a placeholder 'n'.
  2. [§4.2] The text first calls the aggregated F1 a 'Macro-F1' but then says it is a weighted average across entity types; these are different aggregations and the terminology should be reconciled.
  3. [§3.4] Algorithm 1's notation is loose: 'selected_node(s)' can be a list, and the recursion depth l should be explicitly bounded to prevent uncontrolled descent; also clarify the termination condition when multiple children are selected.
  4. [General] The framework name is typeset inconsistently as 'FORGE' in the title and 'forge' throughout the body; use one spelling consistently.
  5. [§5.2] The threats-to-validity section acknowledges LLM imprecision but does not connect this to the concrete sampling and independence limitations of §4.2-§4.4; adding such a discussion would help readers judge the dataset's reliability.

Circularity Check

0 steps flagged · score 2.0 of 10

No definitional circularity; the extraction, classification, and benchmarking claims are grounded in external audit reports and external tools, with only minor self-citations and non-independent human checks that weaken but do not define the result.

full rationale

FORGE's central derivation chain is: (1) extract vulnerability findings from externally sourced audit reports (Section 3.3), (2) classify those findings into the externally maintained CWE taxonomy using ToT (Section 3.4), and (3) evaluate extraction and classification against human judgment on a 96-sample (Sections 4.2-4.3). Step (1) is an information-extraction task whose ground truth is the audit report itself, so precision/recall are not defined in terms of FORGE's own outputs. Step (2) uses a public taxonomy (MITRE CWE) and the ToT method of Yao et al., not an ansatz imported from self-citation. Step (3) compares FORGE labels to labels assigned independently by two authors; this is not a definitional reduction, though the annotators' involvement in designing the CWE_s mapping notes is a validity threat to the 'expert-level' wording. RQ4 benchmarks 13 external tools against FORGE labels; because tool outputs are generated independently of FORGE's construction, this is a standard benchmark use, not a circular derivation, although imperfect labels could bias F1 values. The paper acknowledges the LLM imprecision threat in Section 5.2. The only self-citations, e.g., DAppSCAN [88] for comparison statistics, are descriptive and not load-bearing for FORGE's validity. Therefore no step reduces by construction to its own input.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The framework rests on the trustworthiness of the audit reports, the adequacy of the hand-filtered CWE hierarchy, and the validity of the authors' own manual labels as a gold standard. The hyperparameters (chunk length, k, temperature) and the hand-authored mapping-allowance labels are choices that shape the output. No new physical or formal entities are introduced.

free parameters (4)
  • chunk_length = 4096 tokens
    Semantic chunking threshold; controls how much report text reaches the LLM at once and can affect extraction completeness.
  • ToT branching factor k = 1
    Number of most relevant child CWE nodes selected at each hierarchy level; with k=1 the classifier commits to one path, which may misclassify multi-category vulnerabilities.
  • LLM sampling temperature = 0.8
    Temperature for Llama3 in extraction and classification; affects output consistency and reproducibility.
  • CWE mapping-allowance labels = Per-entry boolean labels
    The authors labeled whether each CWE_s entry is a valid final category for smart contract vulnerabilities; these hand-authored labels steer the ToT fallback strategy (Section 3.4.1).
assumptions (3)
  • domain assumption Audit reports from 47 Etherscan-verified security teams are accurate, expert-validated ground truth for vulnerabilities (Section 2.2).
    If audit reports contain false positives or miss vulnerabilities, the dataset inherits those errors; the paper does not independently verify report content.
  • domain assumption The CWE-1000 Research Concepts view, after filtering 108 hardware entries, adequately covers smart contract vulnerabilities (Section 3.4.1).
    CWE was not designed for smart contracts; the paper acknowledges this and relies on its mapping notes and the ToT fallback to mitigate mismatches.
  • domain assumption The two authors' manual labels are a reliable gold standard for evaluating extraction and classification consistency (Sections 4.2 and 4.3).
    The inter-rater agreement k-alpha is computed between FORGE and two co-authors who designed the system, so the comparison may overstate agreement relative to independent experts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction." pith.science (2026). https://pith.science/paper/H5WQLUMX

@misc{pith2026250618795,
  author       = {Pith},
  title        = {Pith review of: FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H5WQLUMX}},
  note         = {Machine review of arXiv:2506.18795}
}
abstract

High-quality smart contract vulnerability datasets are critical for evaluating security tools and advancing smart contract security research. Two major limitations of current manual dataset construction are (1) labor-intensive and error-prone annotation processes limiting the scale, quality, and evolution of the dataset, and (2) absence of standardized classification rules results in inconsistent vulnerability categories and labeling results across different datasets. To address these limitations, we present FORGE, the first automated approach for constructing smart contract vulnerability datasets. FORGE leverages an LLM-driven pipeline to extract high-quality vulnerabilities from real-world audit reports and classify them according to the CWE, the most widely recognized classification in software security. FORGE employs a divide-and-conquer strategy to extract structured and self-contained vulnerability information from these reports. Additionally, it uses a tree-of-thoughts technique to classify the vulnerability information into the hierarchical CWE classification. To evaluate FORGE's effectiveness, we run FORGE on 6,454 real-world audit reports and generate a dataset comprising 81,390 solidity files and 27,497 vulnerability findings across 296 CWE categories. Manual assessment of the dataset demonstrates high extraction precision and classification consistency with human experts (precision of 95.6% and inter-rater agreement k-$\alpha$ of 0.87). We further validate the practicality of our dataset by benchmarking 13 existing security tools on our dataset. The results reveal the significant limitations in current detection capabilities. Furthermore, by analyzing the severity-frequency distribution patterns through a unified CWE perspective in our dataset, we highlight inconsistency between current smart contract research focus and priorities identified from real-world vulnerabilities...

Figures

Figures reproduced from arXiv: 2506.18795 by the authors.

Figure 1
Figure 1. Overview of forge framework. 2.2 Smart Contract Audit Reports Smart contract auditing is a rigorous security assessment process conducted by professional audit teams (e.g., Etherscan-verified audit teams [19]) to identify security issues and document vulnerabilities in detailed audit reports [24]. These expert-validated reports not only detail vulnerability attack vectors but also provide essential context about the… view at source ↗
Figure 2
Figure 2. Structured information structured result containing project metadata and vulnerability information. ⊕ URL Commit Chain Reduce result 1 Reduce result 2 Merged result Finding 1 Finding 2 Finding 3 Commit Finding 1 Finding 2 URL Chain Finding 1 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A diagram of JSON merge operation To merge these 𝐾 partial results into a final output, we define an operator ⊕ that combines both project metadata (P) and vulner￾ability information (V) from two reduced JSON results in 𝑎𝑖 , 𝑎𝑗 : 𝑎𝑖 ⊕ 𝑎𝑗 = {P𝑖 ∪ P𝑗 , V𝑖 ∪ V𝑗 } (3) As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: An example of ToT workflow [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: An example record of forge dataset Answer to RQ1: forge approach has built the most com￾prehensive smart contract vulnerability dataset compared to previous attempts, such as Smartbugs and DAppSCAN, in both scope and completeness. 4.2 RQ2: Performance of Information Ex…
Figure 6
Figure 6. Figure 6: Risk priority visualization of smart contract from [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: forge Top 10 vs. OWASP Top 10 2 contract vulnerability rankings and the widely recognized tradi￾tional software security risks. The OWASP Top 10 represents a broad consensus on the most critical security risks to web applications in the traditional software domain [56]…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RISKTAGGER: Evidence-Guided LLM Agent for Post-Incident Forensic Analysis of Money Laundering in Web3

    cs.CR 2025-10 conditional novelty 5.0 of 10

    An LLM-driven agent automatically extracts clues from incident reports, traces laundering paths on-chain, and labels accounts as high/medium/low risk, with evaluation on the Bybit hack.

Reference graph

Works this paper leans on

102 extracted references · 72 canonical work pages · cited by 1 Pith paper

  1. [1]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel

  2. [2]

    Meta AI. 2024. Introducing Meta Llama 3: The Most Capable Openly Available LLM to Date. https://ai.meta.com/blog/meta-llama-3/

    2024

  3. [3]

    Andrei. 2024. llama-cpp-python. https://github.com/abetlen/llama-cpp-python/ blob/main/llama_cpp/server/types.py#L25

    2024

  4. [4]

    Anthropic. 2024. Introducing the next Generation of Claude. https://www. anthropic.com/news/claude-3-family

    2024

  5. [5]

    BscScan. 2025. BNB Smart Chain (BNB) Blockchain Explorer. https://bscscan. com/

    2025

  6. [6]

    Vitalik Buterin. 2014. A Next-Generation Smart Contract and Decentralized Application Platform. whitepaper (2014), 3(37):2–1

    2014

  7. [7]

    Calculator.net. 2025. Sample Size Calculator. https://www.calculator.net/sample- size-calculator.html

    2025

  8. [8]

    CertiK. 2025. CertiK - Hack3d: The Web3 Security Report 2024. https://certik. com/resources/blog/hack3d-the-web3-security-report-2024

    2025

Show all 102 references
  1. [9]

    Chainsulting. 2020. 1inch v2 Audit Report . Technical Report. Chainsulting. 16 pages

    2020

  2. [10]

    Stefanos Chaliasos, Marcos Antonios Charalambous, Liyi Zhou, Rafaila Galanopoulou, Arthur Gervais, Dimitris Mitropoulos, and Benjamin Livshits

  3. [11]

    Jiachi Chen, Zhenzhe Shao, Shuo Yang, Yiming Shen, Yanlin Wang, Ting Chen, Zhenyu Shan, and Zibin Zheng. 2025. NumScout: Unveiling Numerical Defects in Smart Contracts Using LLM-Pruning Symbolic Execution. IEEE Trans. Softw. Eng. 51, 5 (March 2025), 1538–1553. https://doi.org/...

    2025

  4. [12]

    conkas. 2021. Nveloso/Conkas: Ethereum Virtual Machine (EVM) Bytecode or Solidity Smart Contract Static Analysis Tool Based on Symbolic Execution. https://github.com/nveloso/conkas

    2021

  5. [13]

    Consensys. 2018. Consensys/Mythril: Mythril Is a Symbolic-Execution-Based Securty Analysis Tool for EVM Bytecode. It Detects Security Vulnerabilities in Smart Contracts Built for Ethereum and Other EVM-compatible Blockchains. https://github.com/Consensys/mythril

    2018

  6. [14]

    Rosen, Gerbrand Ceder, Kristin A

    John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S. Rosen, Gerbrand Ceder, Kristin A. Persson, and Anubhav Jain. 2024. Structured Information Extraction from Scientific Text with Large Language Models. Nature Communications 15, 1 (2024), 1418

    2024

  7. [15]

    DefiLlama. 2025. DefiLlama. https://defillama.com/

    2025

  8. [16]

    Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2024. Vulnerability Detection with Code Language Models: How Far Are We?. In 2025 IEEE/ACM 47th International Conference on Software Engine...

    2024

  9. [17]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, Xu Sun, Lei Li, and Zhifang Sui

  10. [18]

    Ferreira, Rui Abreu, and Pedro Cruz

    Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz. 2020. Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE) . ACM, 530–541

    2020

  11. [19]

    etherscan.io. 2024. Smart Contracts Audit. https://etherscan.io/directory/Smart_ Contracts/Smart_Contracts_Audit_And_Security

    2024

  12. [20]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing

    A Survey on In-context Learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1107–1128. https://doi.org/10.18653/v1/2024.emnlp-main.64

    2024 doi

  13. [21]

    Josselin Feist, Gustavo Greico, Alex Groce, and ACM. 2019. Slither: A Static Analysis Framework For Smart Contracts. In 2019 IEEE/ACM 2ND INTERNA- TIONAL WORKSHOP ON EMERGING TRENDS IN SOFTW ARE ENGINEERING FOR BLOCKCHAIN (WETSEB 2019). 8–15

    2019

  14. [22]

    NCC Group. 2021. DASP - TOP 10. https://dasp.co/

    2021

  15. [23]

    etherscan.io. 2025. Ethereum (ETH) Blockchain Explorer. https://etherscan.io/

    2025

  16. [24]

    Hedera. 2025. What Is a Smart Contract Audit? https://hedera.com/learning/ smart-contracts/smart-contract-audit

    2025

  17. [25]

    Mingyuan Huang, Jiachi Chen, Zigui Jiang, and Zibin Zheng. 2024. Revealing Hidden Threats: An Empirical Study of Library Misuse in Smart Contracts. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Association for Computing Machin...

    2024

  18. [26]

    Zhuobing Han, Xiaohong Li, Hongtao Liu, Zhenchang Xing, and Zhiyong Feng

  19. [27]

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models Are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems , Vol. 35. 22199–22213

    2022

  20. [28]

    Klaus Krippendorff. 2019. Content Analysis: An Introduction to Its Methodology . SAGE Publications, Inc

    2019

  21. [29]

    Kaixuan Li, Yue Xue, Sen Chen, Han Liu, Kairan Sun, Ming Hu, Haijun Wang, Yang Liu, and Yixiang Chen. 2024. Static Application Security Testing (SAST) Tools for Smart Contracts: How Far Are We?. In Proceedings of the ACM on Software Engineering, Vol. 1. 1447–1470. https://doi....

    2024 doi

  22. [30]

    Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen

  23. [31]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  24. [32]

    Peiyu Liu, Junming Liu, Lirong Fu, Kangjie Lu, Yifan Xia, Xuhong Zhang, Wenzhi Chen, Haiqin Weng, Shouling Ji, and Wenhai Wang. 2024. Exploring ChatGPT’s Capabilities on Vulnerability Management. In Proceedings of the 33rd USENIX Conference on Security Symposium (SEC ’24) . US...

    2024

  25. [33]

    Xin Liu, Yuan Tan, Zhenghang Xiao, Jianwei Zhuge, and Rui Zhou. 2023. Not The End of Story: An Evaluation of ChatGPT-Driven Vulnerability Description Mappings. In Findings of the Association for Computational Linguistics: ACL 2023 , Anna Rogers, Jordan Boyd-Graber, and Naoaki ...

    2023

  26. [34]

    Loi Luu, Duc-Hiep Chu, Hrishi Olickel, Prateek Saxena, and Aquinas Hobor. 2016. Making Smart Contracts Smarter. In CCS’16: 2016 ACM SIGSAC Conference on Computer and Communications Security . ACM, 254–269

    2016

  27. [35]

    Lu Liu, Lili Wei, Wuqi Zhang, Ming Wen, Yepang Liu, and Shing-Chi Cheung

  28. [36]

    Pascal C Meunier and Eugene H Spafford. 1999. Final Report of the 2nd Workshop on Research with Security Vulnerability Databases, January 1999. (1999)

    1999

  29. [37]

    MITRE. 2024. 2024 CWE Top 25 Most Dangerous Software Weaknesses. https: //cwe.mitre.org/top25/archive/2024/2024_methodology.html

    2024

  30. [38]

    MITRE. 2025. CWE - About CWE. https://cwe.mitre.org/about/index.html

    2025

  31. [39]

    Root Cause Mapping

    MITRE. 2025. CWE - CVE→ CWE Mapping "Root Cause Mapping" Guidance. https://cwe.mitre.org/documents/cwe_usage/guidance.html

    2025

  32. [40]

    MITRE. 2025. CWE - CWE-1000: Research Concepts (4.16). https://cwe.mitre. org/data/definitions/1000.html

    2025

  33. [41]

    Giacomo Marzi, Marco Balzano, and Davide Marchiori. 2024. K-Alpha Calculator– Krippendorff’s Alpha Calculator: A User-Friendly Tool for Computing Krippen- dorff’s Alpha Inter-Rater Reliability Coefficient. MethodsX 12 (2024), 102545

    2024

  34. [42]

    MITRE. 2025. CWE - CWE-1344: Weaknesses in OWASP Top Ten (2021) (4.16). https://cwe.mitre.org/data/definitions/1344.html

    2021

  35. [43]

    MITRE. 2025. CWE - CWE-284: Improper Access Control. https://cwe.mitre. org/data/definitions/284.html

    2025

  36. [44]

    MITRE. 2025. CWE - CWE-287: Improper Authentication. https://cwe.mitre. org/data/definitions/287.html

    2025

  37. [45]

    MITRE. 2025. CWE - CWE-295: Improper Certificate Validation. https://cwe. mitre.org/data/definitions/295.html

    2025

  38. [46]

    MITRE. 2025. CWE - CWE-298: Improper Validation of Certificate Expiration. https://cwe.mitre.org/data/definitions/298.html

    2025

  39. [47]

    MITRE. 2025. CWE - CWE-1192: Improper Identifier for IP Block Used in System- On-Chip (SOC) (4.16). https://cwe.mitre.org/data/definitions/1192.html

    2025

  40. [48]

    MITRE. 2025. CWE - CWE-477: Use of Obsolete Function. https://cwe.mitre. org/data/definitions/477.html

    2025

  41. [49]

    MITRE. 2025. CWE - CWE-691: Insufficient Control Flow Management. https: //cwe.mitre.org/data/definitions/691.html

    2025

  42. [50]

    MITRE. 2025. CWE - CWE-755: Improper Handling of Exceptional Conditions. https://cwe.mitre.org/data/definitions/755.html

    2025

  43. [51]

    Mark Mossberg, Felipe Manzano, Eric Hennenfent, Alex Groce, Gustavo Grieco, Josselin Feist, Trent Brunson, and Artem Dinaburg. 2019. Manticore: A User- Friendly Symbolic Execution Framework for Binaries and Smart Contracts. In 2019 34th IEEE/ACM International Conference on Aut...

    2019

  44. [52]

    Chaals Nevile. 2023. EEA EthTrust Security Levels Specification V2. https: //entethalliance.org/specs/ethtrust-sl/#sec-3-access-control

    2023

  45. [53]

    MITRE. 2025. CWE - CWE-362: Concurrent Execution Using Shared Resource with Improper Synchronization (‘Race Condition’). https://cwe.mitre.org/data/ definitions/362.html

    2025

  46. [54]

    NVD. 2025. NVD - Vulnerability Metrics. https://nvd.nist.gov/vuln-metrics/cvss

    2025

  47. [55]

    OpenAI. 2023. GPT-4. https://openai.com/index/gpt-4/

    2023

  48. [56]

    OWASP. 2024. OWASP Top Ten | OWASP Foundation. https://owasp.org/www- project-top-ten/

    2024

  49. [57]

    Shengyi Pan, Lingfeng Bao, Xin Xia, David Lo, and Shanping Li. 2023. Fine- Grained Commit-Level Vulnerability Type Prediction by CWE Tree Structure. In Proceedings of the 45th International Conference on Software Engineering (ICSE ’23). IEEE Press, 957–969

    2023

  50. [58]

    Protofire. 2025. Solhint. https://github.com/protofire/solhint

    2025

  51. [59]

    Ivica Nikolić, Aashish Kolluri, Ilya Sergey, Prateek Saxena, and Aquinas Hobor

  52. [60]

    In Proceed- ings of the 34th Annual Computer Security Applications Conference (ACSAC ’18)

    Finding The Greedy, Prodigal, and Suicidal Contracts at Scale. In Proceed- ings of the 34th Annual Computer Security Applications Conference (ACSAC ’18) . Association for Computing Machinery, 653–663

  53. [61]

    Claudia Ruggiero, Pietro Mazzini, Emilio Coppa, Simone Lenti, and Silvia Bonomi

  54. [62]

    Damian Rusinek and Paweł Kuryłowicz. 2021. Smart Contract Security Verifica- tion Standard. https://securing.github.io/SCSVS/

    2021

  55. [63]

    Semgrep. 2025. Semgrep. https://semgrep.dev/p/smart-contracts

    2025

  56. [64]

    Christoph Sendner, Lukas Petzi, Jasper Stang, and Alexandra Dmitrienko. 2024. Large-Scale Study of Vulnerability Scanners for Ethereum Smart Contracts. In 2024 IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society, 2273– 2290

    2024

  57. [65]

    Şevval Şimşek, Zhenpeng Shi, Howell Xia, David Sastre Medina, and David Starobinski. 2024. Poster: Analyzing and Correcting Inaccurate CVE-CWE Map- pings in the National Vulnerability Database. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications ...

    2024

  58. [66]

    pymupdf. 2025. PyMuPDF. https://github.com/pymupdf/PyMuPDF

    2025

  59. [67]

    Alexander Remie. 2024. Uniswap v4 Core Security Assessment . Technical Report. Trail of Bits

    2024

  60. [68]

    Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, and Yingjiu Li. 2024. LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs’ Vulnerability Reasoning. arXiv:2401.16185

    2024 arXiv

  61. [69]

    In Proceedings of the 19th International Conference on A vailability, Reliability and Security

    SoK: A Unified Data Model for Smart Contract Vulnerability Taxonomies. In Proceedings of the 19th International Conference on A vailability, Reliability and Security. ACM, 1–13

  62. [70]

    SWC. 2020. SWC-114 - Smart Contract Weakness Classification (SWC). https: //swcregistry.io/docs/SWC-114/

    2020

  63. [71]

    SWC. 2020. SWC-115 - Smart Contract Weakness Classification (SWC). https: //swcregistry.io/docs/SWC-115/

    2020

  64. [72]

    Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large Language Models for Data Annotation and Synthesis: A Survey. InProceed- ings of the 2024 Conference on Empirical Methods in Na...

    2024

  65. [73]

    Sergei Tikhomirov, Ekaterina Voskresenskaya, Ivan Ivanitskiy, Ramil Takhaviev, Evgeny Marchenko, and Yaroslav Alexandrov. 2018. SmartCheck: Static Analysis of Ethereum Smart Contracts. In Proceedings of the 1st International Workshop on Emerging Trends in Software Engineering ...

    2018

  66. [74]

    SmartBugs. 2020. Vulnerabilities Mapping. https://github.com/smartbugs/ smartbugs/wiki/Vulnerabilities-mapping

    2020

  67. [75]

    Majd Soud, Grischa Liebel, and Mohammad Hamdaqa. 2023. A Fly in the Oint- ment: An Empirical Study on the Characteristics of Ethereum Smart Contracts Code Weaknesses and Vulnerabilities. Empirical Software Engineering 29, 1 (2023)

    2023

  68. [76]

    Christof Ferreira Torres, Mathis Steichen, and Radu State. 2019. The Art of The Scam: Demystifying Honeypots in Ethereum Smart Contracts. In Proceedings of the 28th USENIX Conference on Security Symposium (SEC’19). USENIX Association, 1591–1607

    2019

  69. [77]

    SWC. 2020. Smart Contract Weakness Classification (SWC). https://swcregistry. io/

    2020

  70. [78]

    Uniswap. 2024. Uniswap/v4-Core at Audit/Trail-of-Bits. https://github.com/ Uniswap/v4-core/tree/audit/trail-of-bits

    2024

  71. [79]

    Zhiyuan Wan, Xin Xia, David Lo, Jiachi Chen, Xiapu Luo, and Xiaohu Yang

  72. [80]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT a Good NLG Evaluator? A Preliminary Study. In Proceedings of the 4th New Frontiers in Summarization Workshop, Yue Dong, Wen Xiao, Lu Wang, Fei Li...

    2023

  73. [81]

    Shuo Yang, Jiachi Chen, and Zibin Zheng. 2023. Definition and Detection of Defects in NFT Smart Contracts. In Proceedings of the 32nd ACM SIGSOFT Inter- national Symposium on Software Testing and Analysis (ISSTA 2023) . Association for Computing Machinery, 373–384

    2023

  74. [82]

    Christof Ferreira Torres, Antonio Ken Iannillo, Arthur Gervais, and Radu State

  75. [83]

    Chavhan Sujeet Yashavant. 2021. Sujeetc/ScrawlD

    2021

  76. [84]

    Christof Ferreira Torres, Julian Schütte, and Radu State. 2018. Osiris: Hunting for Integer Bugs in Ethereum Smart Contracts. In Proceedings of the 34th An- nual Computer Security Applications Conference (ACSAC ’18) . Association for Computing Machinery, 664–676

    2018

  77. [85]

    Jiashuo Zhang, Yiming Shen, Jiachi Chen, Jianzhong Su, Yanlin Wang, Ting Chen, Jianbo Gao, and Zhong Chen. 2024. Demystifying and Detecting Cryptographic Defects in Ethereum Smart Contracts. In 2025 IEEE/ACM 47th International Con- ference on Software Engineering (ICSE) . IEEE...

    2024

  78. [86]

    Petar Tsankov, Andrei Dan, Dana Drachsler-Cohen, Arthur Gervais, Florian Bünzli, and Martin Vechev. 2018. Securify: Practical Security Analysis of Smart Contracts. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS ’18) . Association ...

    2018

  79. [87]

    Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2024. Towards an Un- derstanding of Large Language Models in Software Engineering Tasks. Empirical Software Engineering 30, 2 (2024), 50

    2024

  80. [88]

    Zibin Zheng, Jianzhong Su, Jiachi Chen, David Lo, Zhijie Zhong, and Mingxi Ye

  81. [89]

    In Proceedings of the 43rd International Conference on Software Engineering (ICSE ’21)

    Smart Contract Security: A Practitioners’ Perspective. In Proceedings of the 43rd International Conference on Software Engineering (ICSE ’21) . IEEE Press, 1410–1422

  82. [90]

    Haozhe Zhou, Amin Milani Fard, and Adetokunbo Makanju. 2022. The State of Ethereum Smart Contracts Security: Vulnerabilities, Countermeasures, and Tool Support. Journal of Cybersecurity and Privacy 2, 2 (2022), 358–378

    2022

  83. [91]

    Liyi Zhou, Xihan Xiong, Jens Ernstberger, Stefanos Chaliasos, Zhipeng Wang, Ye Wang, Kaihua Qin, Roger Wattenhofer, Dawn Song, and Arthur Gervais. 2023. SoK: Decentralized Finance (DeFi) Attacks. In 2023 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society, 2444–2461

    2023

  84. [92]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, ...

    2023

  85. [94]

    Jiashuo Zhang, Jiachi Chen, Zhiyuan Wan, Ting Chen, Jianbo Gao, and Zhong Chen. 2024. When Contracts Meets Crypto: Exploring Developers’ Struggles with Ethereum Cryptographic APIs. In Proceedings of the IEEE/ACM 46th Interna- tional Conference on Software Engineering (ICSE ’24...

    2024

  86. [96]

    Zhuo Zhang, Brian Zhang, Wen Xu, and Zhiqiang Lin. 2023. Demystifying Ex- ploitable Bugs in Smart Contracts. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 615–627

    2023

  87. [99]

    IEEE Trans

    DAppSCAN: Building Large-Scale Datasets for Smart Contract Weaknesses in DApp Projects. IEEE Trans. Softw. Eng. 50, 6 (2024), 1360–1373

    2024

  88. [100]

    Zibin Zheng, Shaoan Xie, Hong-Ning Dai, Weili Chen, Xiangping Chen, Jian Weng, and Muhammad Imran. 2020. An Overview on Smart Contracts: Chal- lenges, Advances and Platforms. Future Generation Computer Systems 105 (2020), 475–491

    2020

  89. [2018]

    Graph Embedding

    DeepWeak: Reasoning Common Software Weaknesses via Knowledge ICSE ’26, April 12-18, 2026, Rio de Janeiro, Brazil Chen et al. Graph Embedding. In2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER) . 456–466

    2026

  90. [2021]

    In 2021 IEEE European Symposium on Security and Privacy (EuroS&P)

    ConFuzzius: A Data Dependency-Aware Hybrid Fuzzer for Smart Contracts. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P) . 103–119

    2021

  91. [2022]

    In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE ’21)

    Characterizing Transaction-Reverting Statements in Ethereum Smart Con- tracts. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (ASE ’21) . IEEE Press, 630–641

  92. [2023]

    In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.)

    StructGPT: A General Framework for Large Language Model to Reason over Structured Data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 9237–9251

    2023

  93. [2024]

    In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24)

    Smart Contract and DeFi Security Tools: Do They Meet the Needs of Practitioners?. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24) . Association for Computing Machinery, 1–13

  94. [2025]

    In MSR ’25: 22nd International Conference on Mining Software Repositories

    Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. In MSR ’25: 22nd International Conference on Mining Software Repositories

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.