{"id":"ef901b95-83a1-4eba-9235-a96f4496e89c","arxiv_id":"2412.18225","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SimilarGPT pairs GPT-4 with embedding-based code similarity and a three-role debate to detect smart contract vulnerabilities, claiming improved recall and lower false positives.","lead":"SimilarGPT is a proposed smart contract auditing tool that combines GPT-4 with code-similarity search, comparing each function against known third-party libraries before asking an LLM to explain any differences. The authors report higher recall and lower false-positive rates than existing LLM and static-analysis baselines, but the evidence is hampered by small datasets and an internal inconsistency in the main results table.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table II's 'with SimilarChecking' row sums to 143, not the stated 138, so the headline CBSC recall gain (20→38) is not supported by the data as reported.","rationale":"The reader's verdict of REJECT is appropriate, and this stress-test identifies the same overall conclusion but via a different load-bearing concern. The reader's weakest_assumption focuses on the correctness of reference-code labels and the embedding similarity metric; those are legitimate threats to validity but are not directly falsifiable from the paper's reported numbers. A more decisive and concrete problem is the sample-sum inconsistency in Table II, the very table that supports the paper's headline quantitative claim of recall doubling. The 'with SimilarChecking' row sums to 143 while the dataset is stated as 138, and the class counts do not match the stated 67 positive and 71 negative samples. This is not a matter of interpretation or consensus; it is an arithmetic inconsistency that makes the central result unverifiable. The paper contains no artifact, no error bars, and no per-sample outputs, so the reader cannot distinguish a typo from a methodological flaw. Because the empirical foundation for both the CBSC recall gain and the Socratic false-positive reduction is compromised, the paper should not be accepted as validated. The verdict remains REJECT, hence UNCHANGED relative to the reader's decision.","tokens_in":15241,"tokens_out":2906,"duration_ms":26990,"concrete_test":"Request the per-sample prediction table for both conditions on the 138-sample Solodit set and recompute the confusion matrices. Verify that the 'with SimilarChecking' row sums to 138, that TP+FN=67, and that TN+FP=71. If the released predictions still produce TP=38, TN=63, FP=12, FN=30, identify which 5 samples are extra relative to the dataset description and confirm they belong to the same 138-sample set. Also recompute the false-positive rate from the released matrix to check whether it equals 12% (or 16%, 16.9%, or another value), and compare against the claimed 57% for the one-stage baseline using its released confusion matrix. If no artifact is supplied, an independent re-run of the described pipeline on the same Solodit data is required to establish the recall and FPR effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that code-based similarity checking more than doubles true positives rests on Table II. The 'with SimilarChecking' row reports TP=38, TN=63, FP=12, FN=30, which sums to 143 samples, while the dataset is explicitly described in Sections IV-A and IV-D as 138 samples (67 positive, 71 negative). The same row gives TP+FN=68 and FP+TN=75, matching neither the stated positive nor negative counts; only the 'without SimilarChecking' row (20+61+10+47=138) is consistent with the described dataset. This internal inconsistency means the two conditions were not evaluated on the same labeled sample set, or some counts are misreported, or the dataset description is wrong. Since RQ3's conclusion of a recall improvement from 20 to 38 is derived entirely from this table, the central empirical claim cannot be verified from the paper as written. The reported 12% false-positive rate for SimilarGPT also does not match Table II: 12 FP out of 63 TN + 12 FP is 16%, and 12 out of the stated 71 negatives is 16.9%. The one-stage 57% FPR is not backed by any confusion matrix in the paper, so the Socratic method's headline FPR reduction likewise lacks a checkable numerical basis. The threats-to-validity section claims multiple runs were averaged, but no error bars, per-run results, or artifacts are provided, making it impossible to tell whether the discrepancies are typographical or reflect a different evaluation protocol. Additionally, the data-collection section contains contradictory counts (35,705 files versus 46,918 .sol files; 357,050 functions versus 766,505 functions before filtering), further undermining confidence in the pipeline. The load-bearing evidence for both main contributions is therefore unreliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SimilarGPT, a smart-contract vulnerability detection tool that combines GPT-4 with code-based similarity checking (CBSC). The tool vectorizes Solidity functions using all-MiniLM-L6-v2, retrieves similar code from a reference codebase of third-party libraries, and asks GPT-4 to detect vulnerabilities by comparing the target function with the reference. To manage contextual information, functions are analyzed in a topological order of call dependencies, and a Socratic debate framework (Critic, Supporter, Judge) is used to filter false positives. The evaluation claims that on a Solodit dataset of 138 samples, CBSC raises true positives from 20 to 38, and the Socratic method reduces the false-positive rate from 57% to 12%, with additional results on 13 real-world exploits compared against Slither, Mythril, and GptLens.","tokens_in":15553,"tokens_out":6509,"duration_ms":58959,"significance":"If the results held, the paper would make a useful practical contribution by showing that LLM-based auditing can be made both more sensitive and more precise through retrieval of reference implementations and a multi-stage debate, while the topological ordering idea addresses an acknowledged weakness of LLM context handling. The core motivation, that Ethereum smart contracts reuse third-party code heavily, is well supported by prior work. However, the paper provides no artifact, code, or dataset release, and its headline quantitative claims are contradicted by the paper's own tables and text, so the significance as currently demonstrated is limited and the empirical contribution is not verifiable.","major_comments":[{"comment":"The central RQ3 result is not supported by the table as reported. The 'with SimilarChecking' row gives TP=38, TN=63, FP=12, FN=30, which sums to 143 samples, while the Solodit dataset is explicitly defined in §IV-A and §IV-D as 138 samples with 67 positive and 71 negative. The row's TP+FN=68 and FP+TN=75 match neither the stated positive count (67) nor the negative count (71); only the 'without SimilarChecking' row (20+61+10+47=138) is consistent with the described dataset. Since the abstract's claim that CBSC 'more than doubles' true positives rests entirely on this table, the headline result cannot be verified from the manuscript as written.","section":"§IV-D, Table II"},{"comment":"The false-positive-rate claims are inconsistent with Table II. The paper states that the Socratic method reduces the false-positive rate to 12%, but Table II's with-CBSC row yields 12 FP / (12 FP + 63 TN) = 16%, or 12/71 = 16.9% if computed against the stated negative set. The 57% one-stage false-positive rate is asserted in §IV-C but no confusion matrix or per-condition counts for the one-stage framework are provided anywhere in the paper, so the RQ2 improvement cannot be independently checked.","section":"§IV-C and §IV-D"},{"comment":"The reported sizes of the reference codebase are contradictory. The Introduction states '35705 files and 357,050 functions in total' and '83,321 selected functions' after filtering, whereas §III-D reports 'around 46,918 .sol files,' '766,505 functions before the hash match and only 35,709 functions following the hash-based filtering.' These are irreconcilable differences in both file and function counts, and they make the data-collection pipeline described in §III-D not reproducible from the text.","section":"§III-D vs. Introduction"},{"comment":"The construction of the reference labels introduces a circularity risk that is not quantified. The authors label vulnerable functions in the third-party codebase using 'the GPT-4-based method outlined in [16]' and then use GPT-4 as the detector in SimilarGPT. If the reference labels are wrong, the 'secure' code presented to the detector is not actually secure, and the similarity-based comparison loses its ground truth. The manuscript does not report any manual validation rate, inter-annotator agreement, or accuracy measure for this labeling step, so the reliability of the reference codebase is unknown.","section":"§III-D, 'Filter vulnerability code'"},{"comment":"The similarity threshold δ=0.65 is introduced as 'Based on our observations' with no derivation, sensitivity analysis, or validation against other thresholds or embedding models. The distance measure itself, Euclidean(e1,e2)/(||e1||+||e2||), is non-standard and the text does not state whether the embeddings are L2-normalized before computing it. Because the entire CBSC component and therefore the RQ3 recall gain depend on this hand-set threshold, the generalizability of the reported improvement is unsubstantiated.","section":"§III-B, Eq. (1) and threshold δ"}],"minor_comments":[{"comment":"The phrase 'on Ethernet' should be 'on Ethereum' in the Introduction and in §V's related-work discussion of code-reuse rates.","section":"§I, §V"},{"comment":"The caption says 'green blocks indicating GPT works and green blocks suggesting code similar analysis'; the second reference should presumably be to the blue blocks, matching the text that describes green as LLM-based and blue as data-processing components.","section":"Fig. 1 caption"},{"comment":"The sentence 'if LLM thinks that function B, called by function A, has a vulnerability, we can test function B first. Then, we can test function B first and then function A' contains a redundant and garbled repetition; also, the formalization f: V → V is misleading because f describes a one-to-one function, whereas the calling relation is a directed graph edge set.","section":"§III-C"},{"comment":"The text refers to Figure 5 and states that the three frameworks differ little in recall, but no figure image or numeric table is actually included in the manuscript body, so the comparison cannot be inspected; the authors should provide the data.","section":"§IV-C"},{"comment":"In §III-E the text has '[46] [46]' at the end of a sentence, and §V contains the incomplete citation 'Manticore citemanticore'; both need correction.","section":"References"},{"comment":"The description states that negative samples (71 non-vulnerable functions) are collected using methods 'similar to those employed to address vulnerable code from third-party libraries,' but it does not specify the origin, selection criteria, or validation process for these negative samples, which is essential for interpreting the false-positive rates.","section":"§IV-A"}],"recommendation":"major_revision","confidential_remarks":"The Table II inconsistency is the most serious issue: the with-SimilarChecking row sums to 143 while the dataset is 138, and the FPR claim in the text does not match the table. If the authors can supply the corrected raw confusion matrices, per-run results, and a consistent description of the reference-codebase construction, the central claims may become verifiable. I would encourage the editor to require the data artifact as a condition of any revision. The contradictory dataset counts between the Introduction and §III-D, and the lack of any validation of the GPT-4-based labeling step, are additional reasons to treat the current empirical evidence with caution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the integration idea is sensible, but the central numbers don't add up, so the headline claims are not currently supportable.\n\nWhat's new: the specific combination of GPT-4 prompting with embedding-based retrieval of third-party reference code, topological ordering of the call graph, and a Socratic multi-agent filter is not something I've seen in the cited literature. The motivation is real: smart contracts reuse a lot of library code, and comparing against known-good implementations is a reasonable way to ground an LLM. The design choices are defensible.\n\nThe soft spots are in the evaluation, and they're load-bearing. Table II's \"with SimilarChecking\" row sums to 143 samples when the dataset is described as 138 (67 positive, 71 negative). The without row is consistent, the with row is not. So the 20-to-38 recall gain cannot be verified from the paper as written. The reported 12% FPR doesn't match either: 12 FP out of 63 TN + 12 FP is 16%, and 12 out of 71 negatives is 16.9%. The 57% FPR for the one-stage baseline has no confusion matrix behind it. The dataset counts also contradict each other (35,705 files vs 46,918 .sol files; 357,050 functions vs 766,505 functions). The threats-to-validity section says multiple runs were averaged, but no error bars or per-run results appear. No artifact is provided. The 0.65 similarity threshold is hand-set \"based on observations\", and the reference labels come from a GPT-4-assisted pipeline without reported accuracy, which adds a mild circularity.\n\nThat's a lot of problems, but they are fixable. The core idea is worth testing. A corrected evaluation with reproducible artifact, per-run results, and consistent counts would be a legitimate contribution.\n\nWho this is for: people working on LLM-assisted smart contract auditing. If you're building such a tool, the paper gives you a useful design sketch, but don't rely on its numbers.\n\nRecommendation: send it to peer review, but with the expectation of heavy revision. It deserves referee time because the idea is novel and the field needs better empirical evidence, but as-is the empirical claims should not be accepted.","headline":"The SimilarGPT integration idea is worth a look, but the central empirical claims are undermined by internal inconsistencies in the reported numbers, so the paper needs a corrected evaluation before its results can be trusted.","tokens_in":16117,"tokens_out":2004,"would_cite":false,"duration_ms":16677,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SimilarGPT pairs GPT with trusted-code comparison to double recall and cut false alarms to 12 percent.","keywords":["smart contracts","vulnerability detection","large language models","code similarity","Socratic debate","topological ordering","Solidity","LLM auditing"],"falsifier":"Take a random sample of the functions the GPT-assisted filter marked vulnerable in the reference codebase, have independent human auditors relabel them, and compare the model's 0.65 'similar' pairings with expert judgments; if labels or similarity judgments disagree at a nontrivial rate, the reported 20-to-38 true-positive jump loses its ground truth.","tokens_in":15020,"feed_emoji":"🛡️","tokens_out":10148,"duration_ms":83530,"temperature":0.7,"pith_summary":"SimilarGPT is a smart-contract auditing pipeline that tries to make large language models detect vulnerabilities by giving them a reference: the closest safe function from popular third-party libraries, matched by code embedding similarity. The paper claims this retrieval step is what lifts recall, raising true positives from 20 to 38 on a 138-function test set, and that a three-role Socratic debate over the detector's findings cuts the false-positive rate from 57% to 12%. It also reports that on 13 real-world exploits the tool finds 8, against 4 for a leading LLM baseline and one each for two static and dynamic analyzers. If these numbers hold, auditors could catch a substantially larger share of copied-and-tweaked contract bugs without drowning in false alarms.","feed_headline":"Similarity-checked GPT doubles smart contract bug detection","feed_subtitle":"Reference library code lifts true positives from 20 to 38; a debate step cuts false alarms to 12 percent.","key_machinery":"The load-bearing machinery is the combination of four components: a reference codebase of 83,321 deduplicated functions from the 150 most-used third-party Solidity packages; a similarity checker using normalized Euclidean distance between 384-dimensional sentence embeddings, with a hand-set threshold of 0.65, to retrieve the closest 'safe' function; a topological ordering of the contract's call graph that fixes the order in which functions are presented to the model; and a Socratic debate among detector, critic, supporter, and judge that filters false positives. The similarity check is what turns a generic LLM audit into a comparison against a concrete correct implementation.","core_discovery":"The paper's central claim is that a smart-contract vulnerability detector becomes substantially more sensitive and more precise when the large language model is shown, alongside each function under test, the most similar function from a curated reference codebase of trusted third-party contracts, and when its verdict is then challenged by a critic, a supporter, and a judge. The reference match is computed as normalized Euclidean distance between 384-dimensional embeddings of normalized Solidity functions, with 0.65 as the threshold for treating a retrieved function as close enough to use as reference; exact clones are resolved directly, near-clones are fed into the model as comparison material, and distant functions get no augmentation. Detection order follows a topological ordering of the call graph so that called functions are audited before their callers, which the paper argues stops the model from blaming a safe function for a vulnerability in a function it calls. The evidence offered is a controlled ablation in which adding similarity checking roughly doubled true positives on the 138-function set, and a comparison in which the Socratic debate cut false positives from 57% to 12%.","pith_inferences":["The paper never reports the accuracy of the GPT-assisted labeling that marks reference functions as secure or vulnerable; we infer that measuring that label accuracy against human experts is the fastest way to know how much of the true-positive jump is real.","The 0.65 threshold is hand-set; we infer that an automatic threshold search or a learned similarity function could change the results substantially on other contract families.","The false-positive reduction was measured on one 138-function set; we infer that the Socratic benefit could shrink on datasets with more diverse vulnerability types, and stress-testing that is a natural next step.","The repeated fork exploits the paper describes suggest the same retrieval mechanism could also serve as a triage pre-filter that flags near-clones of known-vulnerable code for human auditors; the paper does not draw this conclusion."],"forward_implications":["If the central claim is right, an LLM-based auditor can roughly double its hit rate simply by retrieving a trusted reference implementation before asking the model to judge the code.","The Socratic debate step offers a concrete way to cut LLM hallucination false positives in security settings, beyond prompt engineering.","Topological ordering of the call graph is a reusable recipe for preventing an auditing model from blaming a caller for a flaw in a function it calls.","Because the reference codebase is refreshed from current third-party packages, the approach can track newly introduced vulnerable patterns without retraining the model.","The reported results imply that code reuse is not just a supply-chain risk but an audit lever: the same cloning that spreads bugs can be used to expose them."],"supporting_citations":[{"why":"Supplies the code-reuse statistics and third-party package catalog that justify building the reference codebase.","marker":"[21]"},{"why":"Provides the Euclidean-distance code similarity metric that SimilarGPT adapts for retrieving similar functions.","marker":"[24]"},{"why":"Is the embedding model used to vectorize normalized Solidity functions for similarity matching.","marker":"[36]"},{"why":"Provides the structured vulnerability-knowledge method and the GPT-based extraction and filtering procedure used to label the reference data.","marker":"[16]"},{"why":"Supplies the motivating exploit example of a cloned library function that manual audits missed.","marker":"[4]"},{"why":"Supplies the Socratic critic-supporter-judge debate protocol used to filter false positives.","marker":"[47]"},{"why":"Serves as the LLM baseline for the real-world detection comparison and for the two-stage framework.","marker":"[15]"},{"why":"Supplies the 138-function audit dataset on which the true-positive and false-positive ablations are run.","marker":"[50]"}],"fun_headline_variants":["Reference code doubles GPT smart contract vulnerability finds","Smart contract bugs: GPT with similar-code reference doubles detections","Debate refines GPT similarity detection, false positives fall to 12%","Similarity-checked GPT finds twice the smart contract bugs","Topological order plus GPT similarity: double hits, fewer false alarms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything hinges on the reference labels: the third-party functions must really be safe or really be vulnerable, and a 0.65 embedding-similarity score must mark the code pairs whose differences a security expert would care about.","fun_headline_variants_meta":{"raw":{"variants":["Reference code doubles GPT smart contract vulnerability finds","Smart contract bugs: GPT with similar-code reference doubles detections","Debate refines GPT similarity detection, false positives fall to 12%","Similarity-checked GPT finds twice the smart contract bugs","Topological order plus GPT similarity: double hits, fewer false alarms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3391,"prompt_tokens":963,"completion_tokens":2428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2343}},"tokens_in":579,"tokens_out":2428,"duration_ms":15943,"temperature":1.0,"reasoning_tokens":2343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:54:41.856030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the functions the GPT-assisted filter marked vulnerable in the reference codebase, have independent human auditors relabel them, and compare the model's 0.65 'similar' pairings with expert judgments; if labels or similarity judgments disagree at a nontrivial rate, the reported 20-to-38 true-positive jump loses its ground truth.","supporting_citations":[{"cited_title":"Demystifying the composition and code reuse in solidity smart contracts,","cited_arxiv_id":null,"evidence_quote":"Supplies the code-reuse statistics and third-party package catalog that justify building the reference codebase."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the embedding model used to vectorize normalized Solidity functions for similarity matching."},{"cited_title":"Demystifying ex- ploitable bugs in smart contracts,","cited_arxiv_id":null,"evidence_quote":"Supplies the motivating exploit example of a cloned library function that manual audits missed."},{"cited_title":"Prompting large language models with the so- cratic method,","cited_arxiv_id":null,"evidence_quote":"Supplies the Socratic critic-supporter-judge debate protocol used to filter false positives."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 138-function audit dataset on which the true-positive and false-positive ablations are run."}],"review_version":1}