{"id":"d61562bc-a72c-4a35-9bdc-ba43a9446225","arxiv_id":"2505.15756","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Empirical evaluation of 20 vulnerability detection tools for Solidity smart contracts on a new line-level annotated dataset of 2,182 instances identifies a 3-tool combination achieving 76.78% detection in under one minute on average.","lead":"This paper empirically evaluates 20 automated vulnerability detection tools for Solidity smart contracts on a manually annotated dataset of 2,182 contracts using the SmartBugs 2.0 framework. A smart generalist might read it to identify practical tool combinations for securing blockchain applications and access a new public dataset for further security research.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Manual line-level annotations for 2,182 contracts lack reported inter-annotator agreement or external validation","rationale":"The reader's weakest assumption matches the central dependency: all quantitative results rest on the correctness of the manual ground truth. Because the paper positions the dataset as a major contribution, the absence of reliability metrics is the single most load-bearing gap. This does not invalidate the work but conditions any acceptance on verification of annotation quality. No other internal inconsistency is visible from the abstract and claim description.","tokens_in":1723,"tokens_out":349,"duration_ms":16300,"concrete_test":"Randomly sample 150 contracts from the released dataset; have two independent Solidity auditors (not involved in the original labeling) re-annotate them for the same DASP categories at line level; compute Cohen's kappa and per-category F1 agreement with the published labels. If kappa < 0.7 or average agreement < 75%, the ground-truth reliability is insufficient to support the 76.78% claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline performance numbers (76.78% vulnerabilities found by the best 3-tool combination) are computed against the authors' own manual annotations. The abstract states these are line-level labels for DASP TOP 10 categories, yet no details are given on annotation protocol, number of annotators, resolution of disagreements, or any cross-check against known vulnerable contracts from other sources (e.g., SWC registry or prior SmartBugs labels). If systematic annotation errors exist—especially for subtle reentrancy or access-control issues—the reported recall figures become unreliable and the claim that the combination is practically useful cannot be assessed.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper empirically evaluates 20 vulnerability detection tools for Solidity smart contracts via the SmartBugs 2.0 framework on a new manually annotated dataset of 2,182 contracts with line-level labels for DASP TOP 10 categories. It also tests an LLM-based detector on two datasets and reports inconsistent results on real-world contracts. A key result is that a combination of three tools detects up to 76.78% of vulnerabilities while averaging under one minute of runtime. The authors release the annotated dataset as the largest such resource.","tokens_in":1832,"tokens_out":548,"duration_ms":26066,"significance":"If the ground-truth annotations hold, the work is significant for releasing the largest manually labeled smart-contract vulnerability dataset to date and for providing the broadest empirical comparison (20 tools) of detection effectiveness. The practical finding that a small tool ensemble reaches substantial coverage quickly could inform developer workflows, and the LLM inconsistency result highlights limitations of current generative approaches on real contracts.","major_comments":[{"comment":"Abstract and dataset-construction section: the headline metrics (including the 76.78% coverage by the best three-tool combination) rest entirely on the authors' line-level manual annotations of 2,182 contracts. No information is supplied on annotator count, inter-rater agreement, disagreement-resolution protocol, or external validation against known-vulnerable contracts (e.g., SWC registry or prior SmartBugs labels). Without these details, systematic labeling errors for subtle issues such as reentrancy or access control cannot be ruled out, rendering recall figures and the practical-utility claim difficult to assess.","section":"Abstract / Dataset construction"},{"comment":"LLM-evaluation section: the claim of 'inconsistent results' and 'unreliable detection' on real-world contracts is load-bearing for the paper's broader conclusions about automated methods. The manuscript does not specify the two datasets, the exact prompting or few-shot setup, how inconsistencies were quantified, or any statistical test used to compare performance across datasets.","section":"LLM-based detection evaluation"}],"minor_comments":[{"comment":"Add a table that lists every one of the 20 tools together with version, configuration flags, and average runtime on the dataset to improve reproducibility.","section":null},{"comment":"Clarify in the abstract and introduction whether the 2,182 contracts are distinct from or overlap with existing SmartBugs corpora; this affects the novelty claim of 'the largest dataset'.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript accordingly to improve clarity and reproducibility.","responses":[{"response":"We acknowledge that the manuscript does not currently provide these details on the annotation process. In the revised version, we will add a new subsection in the dataset construction section that specifies the number of annotators (two authors with prior Solidity experience), the inter-rater agreement (Cohen's kappa), the disagreement resolution protocol (discussion until consensus), and external validation steps against a sample of contracts cross-checked with the SWC registry and prior SmartBugs labels. These additions will directly address concerns about potential systematic errors in labeling subtle vulnerabilities.","revision_made":"yes","referee_comment":"[Abstract / Dataset construction] Abstract and dataset-construction section: the headline metrics (including the 76.78% coverage by the best three-tool combination) rest entirely on the authors' line-level manual annotations of 2,182 contracts. No information is supplied on annotator count, inter-rater agreement, disagreement-resolution protocol, or external validation against known-vulnerable contracts (e.g., SWC registry or prior SmartBugs labels). Without these details, systematic labeling errors for subtle issues such as reentrancy or access control cannot be ruled out, rendering recall figures and the practical-utility claim difficult to assess."},{"response":"We agree that additional methodological details are required for the LLM evaluation to support the claims of inconsistent results. In the revision, we will explicitly name the two datasets, describe the prompting templates and few-shot examples used, explain how inconsistencies were quantified (via detection overlap and performance deltas), and include statistical comparisons (e.g., McNemar's test) between the datasets. This will strengthen the reproducibility and evidential basis of the LLM findings.","revision_made":"yes","referee_comment":"[LLM-based detection evaluation] LLM-evaluation section: the claim of 'inconsistent results' and 'unreliable detection' on real-world contracts is load-bearing for the paper's broader conclusions about automated methods. The manuscript does not specify the two datasets, the exact prompting or few-shot setup, how inconsistencies were quantified, or any statistical test used to compare performance across datasets."}],"tokens_in":1455,"tokens_out":486,"duration_ms":35447,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main things to know are that the authors release the largest manually labeled line-level dataset for Solidity vulnerabilities so far and show that a simple combination of three existing tools catches most issues quickly on their data. The work stays empirical and avoids overclaiming broader theory.","headline":"The paper's real contribution is the released line-level dataset of 2,182 contracts plus the practical result that three tools together reach 76.78% coverage in under a minute, but the annotation quality is under-documented.","tokens_in":2340,"tokens_out":142,"would_cite":true,"duration_ms":18007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"Empirical SE study on smart-contract vulnerability tools; no structural overlap with RS forcing chain","alignment":"orthogonal","rationale":"The paper performs tool benchmarking, manual line-level annotation of 2,182 Solidity contracts against DASP TOP 10, and combination analysis (Conkas+Slither+Smartcheck reaching 76.78 %). Its machinery is standard empirical software-engineering evaluation with no reference to recognition cost J(x), φ-ladder, 8-tick periodicity, absolute-floor distinguishability, or any RS theorem. RS framework (reality_from_one_distinction, J-cost uniqueness, AlexanderDuality D=3 forcing, etc.) has no opinion on this domain.","tokens_in":57970,"confidence":"high","tokens_out":160,"duration_ms":7531,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Combining three vulnerability detection tools for Solidity smart contracts reaches 76.78% coverage in under one minute on average.","keywords":["smart contracts","solidity","vulnerability detection","empirical study","blockchain security","tool combination","DASP taxonomy","LLM evaluation"],"falsifier":"Independent re-labeling of a random sample of the 2,182 contracts that changes the vulnerability status of a large fraction of instances, or discovery of a real deployed contract containing a vulnerability missed by the three-tool combination.","tokens_in":2630,"feed_emoji":"🔐","tokens_out":464,"duration_ms":30903,"temperature":0.7,"pith_summary":"The paper evaluates twenty automated tools for finding vulnerabilities in Solidity smart contracts against a new set of 2,182 manually annotated contracts. It reports that single tools vary widely in coverage and that large-language-model methods produce inconsistent results on real-world contracts. A particular trio of tools together identifies 76.78 percent of the labeled issues while averaging less than one minute of runtime. The authors also release the full annotated dataset for use in future comparisons.","feed_headline":"Three tools catch 76.78% of smart contract bugs under one minute","feed_subtitle":"Evaluation on 2,182 annotated contracts shows the trio outperforms single tools and LLMs on real Solidity code.","key_machinery":"The manually annotated dataset of 2,182 smart contracts with line-level vulnerability labels serving as ground truth to compare the twenty tools inside the SmartBugs 2.0 framework.","core_discovery":"Leveraging the SmartBugs 2.0 framework and a dataset of 2,182 Solidity contracts manually labeled at the line level with DASP TOP 10 vulnerability categories, the study measures detection rates across twenty tools. Different tools catch largely non-overlapping subsets of issues. The combination of three specific tools detects up to 76.78 percent of known vulnerabilities with average runtime below one minute, while LLM-based detection yields unreliable outcomes on real contracts.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Three tools detect 76.78% Solidity vulnerabilities under one minute","Trio catches 76.78% smart contract bugs in under one minute","76.78% of Solidity vulnerabilities caught by three tools in under one minute","Three tools combine for 76.78% smart contract bug detection under one minute"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The manual annotations of the 2,182 contracts correctly and completely identify the true vulnerabilities without systematic bias or missed cases.","fun_headline_variants_meta":{"raw":{"variants":["Three tools detect 76.78% Solidity vulnerabilities under one minute","Trio catches 76.78% smart contract bugs in under one minute","76.78% of Solidity vulnerabilities caught by three tools in under one minute","Three tools combine for 76.78% smart contract bug detection under one minute"]},"model":"grok-4.3","cost_usd":0.014042,"raw_usage":{"total_tokens":5990,"prompt_tokens":693,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":140415500,"prompt_tokens_details":{"text_tokens":693,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5217,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":693,"tokens_out":80,"duration_ms":52565,"temperature":1.0,"reasoning_tokens":5217,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-22T13:38:52.235950+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Independent re-labeling of a random sample of the 2,182 contracts that changes the vulnerability status of a large fraction of instances, or discovery of a real deployed contract containing a vulnerability missed by the three-tool combination.","supporting_citations":[],"review_version":1}