{"id":"2b0543c7-1cbd-44cf-9ad1-8e6966820b2d","arxiv_id":"2508.19472","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"SIExVulTS combines sentence embeddings, CodeQL dataflow queries, and GraphCodeBERT verification to detect CWE-200 sensitive-information-exposure vulnerabilities in Java, reporting an F1 of 85.71 percent on a synthetic benchmark and six new CVEs.","lead":"A new three-stage tool, SIExVulTS, uses transformer models plus CodeQL static analysis to find Java code that leaks passwords, API keys, and other sensitive data. The paper reports that it caught six real, previously unknown vulnerabilities in Apache and other open-source projects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 87.23% flow-verification precision in Table V may be an in-sample result: Section V.A.3 says the same 31 projects were used to train and evaluate the Flow Verification Engine, and no held-out split is reported for Table V.","rationale":"I read the paper in good faith: the system architecture is coherent, the benchmark and CVE datasets are reasonable, and the six newly assigned CVEs provide real independent evidence that the tool can find true positives. However, the strongest quantitative claim, the precision improvement of the Flow Verification Engine, is the linchpin of the paper's headline. The reader correctly identified that Section V.A.3 says the SIEx-Flow dataset is used to train and evaluate the FVE, while Table V reports precision across those same 31 projects without specifying any held-out split. Section IV.C mentions only stratified splits, not a test set. This is a load-bearing ambiguity, not a stylistic issue: if the same flows are in both training and evaluation, 87.23% precision is a measure of fit, not generalization. The six CVEs show the tool can discover real vulnerabilities, but they do not quantify precision or false-positive effort. The concern is testable and addressable: a leave-one-project-out evaluation, or even a disclosed 70/15/15 split, would settle it. I therefore agree with the reader's conditional verdict and recommend no change: the paper should be conditionally accepted pending a held-out evaluation or explicit split disclosure.","tokens_in":16535,"tokens_out":4948,"duration_ms":47255,"concrete_test":"Use the released repository (github.com/MSUSEL/hawaii-pique-cwe200) to rerun the Flow Verification Engine evaluation as a leave-one-project-out cross-validation: train on 30 projects' flows, test on the remaining project, repeat for all 31, and report macro-averaged precision and recall on the held-out folds. If the held-out precision is materially below 87.23% (e.g., below 60%), the Table V improvement is attributable to training/evaluation overlap and the precision claim should not be stated without a caveat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim rests on Table V, where the Flow Verification Engine raises precision from 22.61% to 87.23% on 31 open-source projects. The paper does not give a train/test split for this evaluation. Section V.A.3 states the SIEx-Flow dataset (2,555 labeled flows from these 31 projects) is used to train and evaluate the Flow Verification Engine, and Section IV.C only says training used stratified splits without naming a held-out test set. Unlike the Attack Surface Detection Engine, which has an explicit 70:15:15 split (Section IV.A.2), the FVE has no stated test protocol. If the same labeled flows appear in training and in Table V, the 87.23% precision is an in-sample fit, not a prediction; it could be much lower on unseen projects. The six assigned CVEs are genuine independent evidence of useful true-positive discovery, but they do not measure false-positive rate, so they do not substitute for a held-out precision estimate. Thus the headline precision improvement, which is the main quantitative support for high accuracy, is currently unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SIExVulTS, a three-stage system for detecting CWE-200 sensitive-information-exposure vulnerabilities in Java: an Attack Surface Detection Engine using sentence embeddings and classifiers; an Exposure Analysis Engine that instantiates CodeQL queries aligned with a CWE-200 hierarchy; and a Flow Verification Engine that uses GraphCodeBERT embeddings and a Transformer aggregator to filter false-positive data flows. The authors evaluate the system on three self-constructed datasets: a CVE-derived dataset with 40 real vulnerabilities, a synthetic benchmark of 300 labeled code samples, and a set of 2,555 manually labeled data flows from 31 open-source Java projects. Headline results are F1 scores above 93% for attack-surface classification, an F1 of 85.71% on the benchmark, a precision improvement from 22.61% to 87.23% attributable to the Flow Verification Engine, and the discovery of six previously unknown CVEs in Apache IoTDB, Apache Pulsar, Jenkins, and Para. The central claim is that SIExVulTS is both accurate and practically useful for CWE-200 detection.","tokens_in":16797,"tokens_out":5871,"duration_ms":53754,"significance":"If the reported quantitative results hold under out-of-sample evaluation, the paper addresses a genuinely under-served vulnerability class (CWE-200) with a thoughtfully designed multi-stage architecture, and it contributes three labeled datasets plus a public implementation. The six assigned CVEs are an unusually strong form of real-world validation; they demonstrate that the system can surface true positives that matter to maintainers. However, the paper's quantitative accuracy claims are currently not substantiated because the evaluation protocols described in Sections V.A and V.C do not establish that the Flow Verification Engine and the Attack Surface Detection classifiers are evaluated on data they did not see during training. The benchmark and flow-level metrics may therefore be in-sample fits rather than predictions. The practical CVE discoveries cannot substitute for a held-out precision estimate, because they measure only true positives and not the false-positive rate.","major_comments":[{"comment":"Table V reports a precision improvement from 22.61% to 87.23% across 31 projects, but the Flow Verification Engine was trained on the 2,555 labeled flows extracted from those same 31 projects (Section V.A.3). Section IV.C states only that training used 'stratified splits' and gives no ratio or project-disjoint split, in contrast to the explicit 70:15:15 split for the Attack Surface Detection Engine in Section IV.A.2. If the flows used for Table V overlap with the training flows, the 87.23% precision is an in-sample fit and does not support the claim that the engine reduces false positives on unseen projects. Please specify the exact train/validation/test protocol for the Flow Verification Engine, hold out entire projects for testing, and report Table V on the held-out test set.","section":"Section V.C.1, V.A.3, IV.C"},{"comment":"The benchmark dataset used for RQ2 appears to have been part of the training data for the Attack Surface Detection classifiers. Table I aggregates samples from the 'CVE' and 'Bench' columns into a single total, and Section V.A.4 refers to the combined 'collected data' when describing training for the Attack Surface Detection models. Unless the benchmark samples were exclusively placed in the 15% test split mentioned in Section IV.A.2, the 85.71% F1 reported in Table III is contaminated by training data. Please clarify whether the benchmark was used to train the classifiers, and if so, re-evaluate the benchmark using a model trained only on non-benchmark data.","section":"Section V.A.4 and Table III"},{"comment":"No confidence intervals or per-project variability measures are reported for the precision, recall, F1, or accuracy figures in Table V. The text states that 28 of 31 projects improved, 3 stayed the same, and none decreased, but the table omits this per-project breakdown. Given the relatively small number of projects and the likelihood of project-specific effects, please provide a per-project analysis with confidence intervals (e.g., bootstrap) and clearly state the number of flows per project.","section":"Section V.C.1, Table V"},{"comment":"The Flow Verification Dataset labels (Yes/No for true and false positive flows) are manually assigned, but the paper reports inter-rater agreement only for variables, strings, and comments (Table VI), not for the flow-level labels. Since Table V's ground truth consists entirely of these flow labels, the reliability of the flow labeling is load-bearing for the precision measurement. Please report Cohen's Kappa (or equivalent) for the flow-level labeling process and provide the labeling instructions in an appendix or repository.","section":"Section VI.D, V.A.3"}],"minor_comments":[{"comment":"The Flow Verification Engine training details are underspecified; please state the split ratio, number of hyperparameter search trials, dropout, learning rate, and early-stopping criteria, as was done for the Attack Surface Detection Engine.","section":"Section IV.C"},{"comment":"The CWE-538 row contains only 5 total samples; an F1 of 100% on this row is unstable. Consider reporting bootstrapped confidence intervals or excluding such small categories from the aggregate F1 calculation.","section":"Table III"},{"comment":"The text says 22.6% and 87.2% while Table V reports 0.226 and 0.872; please keep the notation consistent throughout.","section":"Section V.C.1, Table V"},{"comment":"The heading 'Weighed performance' should read 'Weighted performance'.","section":"Table II"},{"comment":"The abstract reports 'an average F1 score greater than 93%' without the exact value; please report the exact average and, if possible, the standard deviation across the four attack-surface categories.","section":"Abstract"},{"comment":"The paper mentions 'Section VIII concludes' but the conclusion section is numbered VIII after an ethics section VII; the cross-reference is correct, but the prose in Section I says 'Finally, Section VIII concludes the paper' while the actual numbering may confuse readers; please verify the section numbering.","section":"Section I"},{"comment":"The hierarchical model in Fig. 1 is described as 'adapted' from the CWE repository; please state explicitly which of the shown CWE subtypes are actually covered by the implemented CodeQL queries and which are aspirational.","section":"Section II.B"}],"recommendation":"major_revision","confidential_remarks":"The core system design and the discovery of six previously unknown CVEs are promising contributions, but the evaluation methodology is not yet at the standard expected for publication. The two most important fixes are: (1) a clear, project-disjoint train/test protocol for the Flow Verification Engine with results reported on the held-out test set, and (2) clarification or correction of the benchmark dataset's overlap with the Attack Surface Detection training data. Both issues are fixable within the manuscript's scope. I would not recommend rejection because the practical CVE findings and the public repository are valuable, but the paper should not be accepted with the current quantitative claims. The editor may wish to ask the authors to provide the exact labeling instructions for the Flow Verification Dataset and to confirm that the listed CVEs are publicly assigned."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know upfront. First, this is the first systematic treatment of CWE-200 detection I know of, and the six newly assigned CVEs in Apache projects are concrete, independent evidence that the tool finds real vulnerabilities. Second, the paper's main accuracy claim—that the Flow Verification Engine raises precision from 22.6% to 87.2%—is not backed by a clear held-out evaluation. The text says the flow dataset 'is used to train and evaluate' the engine and reports results on the same 31 projects, with no train/test split described. That number is very likely in-sample, so the central precision improvement is currently unsubstantiated.\n\nWhat's genuinely new: the three-stage pipeline combining sentence-embedding attack surface detection, instantiated CodeQL queries aligned with a CWE-200 hierarchy, and a GraphCodeBERT-based flow verifier. The literature review supports the claim that prior work went after SQLi, XSS, memory bugs, or general vulnerabilities, not CWE-200 systematically. The attack surface detection evaluation has an explicit 70:15:15 split and reports F1 >93%, which is plausible. The CodeQL benchmark on 300 synthetic samples shows F1 85.71%, and the per-CWE breakdown is honest about weak spots (CWE-535, CWE-615). The six CVEs are the strongest part of the paper: IoTDB, Pulsar, Jenkins, and Para are real, and the descriptions match the tool's logic.\n\nSoft spots, in proportion. The load-bearing one is the missing held-out evaluation for the Flow Verification Engine. The paper never states that Table V is out-of-sample, and Section V.A.3 explicitly says the same 31 projects were used to train and evaluate the engine. That makes the 87.2% precision a fit to the labels, not a prediction. The CVEs demonstrate true-positive discovery but don't measure false-positive rate, so they don't fix the precision claim. Also, there are no confidence intervals anywhere; the benchmark is small and author-created; and the comparison to existing tools is thin, though the SonarQube check is a reasonable sanity test.\n\nWho should read this: people building vulnerability detectors and security teams looking for tools in this niche. It deserves a serious referee. The gap is real, the CVEs are meaningful, and the evaluation flaw is addressable. A careful revision needs either a proper held-out split for the FVE or a clear statement that the 87.2% is on training-domain flows, plus artifact release. I'd cite it for the CWE-200 datasets and the CVE discoveries, not for the precision number until that's resolved.","headline":"A real gap and six real CVEs, but the headline 87.2% precision figure for the flow verifier is likely in-sample and needs a held-out evaluation before it can be believed.","tokens_in":17314,"tokens_out":2694,"would_cite":true,"duration_ms":24037,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a three-stage pipeline combining transformer classifiers, CodeQL data-flow analysis, and a GraphCodeBERT-based verifier detects CWE-200 sensitive-information-exposure vulnerabilities in Java with enough precision to…","keywords":["CWE-200","sensitive information exposure","vulnerability detection","static analysis","CodeQL","transformer models","GraphCodeBERT","Java security"],"falsifier":"Re-run the pipeline while holding out one or more entire projects from the 31 used to train the Flow Verification Engine, and measure precision on those held-out projects. If precision stays near 22.6% instead of 87.2%, the verifier is memorizing labels rather than generalizing.","tokens_in":16365,"feed_emoji":"🛡️","tokens_out":6121,"duration_ms":51462,"temperature":0.7,"pith_summary":"The paper sets out to show that sensitive-information-exposure vulnerabilities (CWE-200) in Java, a category most detectors ignore, can be found reliably by chaining transformer-based classification, CodeQL static data-flow analysis, and a semantic flow-verification step. The three-stage system first labels variables, strings, comments, and API sinks as sensitive or not, then traces source-to-sink paths with CodeQL queries built around an eight-category model of sensitive data, and finally uses a GraphCodeBERT-based classifier to discard flow paths that do not correspond to real exposures. On a benchmark of 300 synthetic samples the full pipeline reports an F1 score of 85.71%, and on 2,555 labeled flows from 31 open-source projects the verification step raises precision from 22.61% to 87.23% while keeping recall at 93.8%. The authors also report six previously unknown CVEs found in Apache IoTDB, Apache Pulsar, a Jenkins plugin, and the Para platform, which is what a practical detector must do to earn trust.","feed_headline":"CWE-200 detector finds six new CVEs, lifts precision to 87%","feed_subtitle":"Three-stage Java pipeline pairs transformer models with CodeQL to find exposed sensitive data in real projects.","key_machinery":"The load-bearing mechanism is the pairing of the CWE-200 hierarchy with a flow-verification classifier. The hierarchy is re-expressed as eight concrete categories of sensitive data (credentials, PII, financial information, sensitive file paths, configuration information, encryption material, application-specific data, and query parameters); these categories define what the attack-surface classifiers label and which CodeQL source/sink rules are instantiated. Attack surface detection pairs name and method/line context embeddings from Sentence-BERT, concatenates them, and feeds a residual network classifier. The Flow Verification Engine is the second critical mechanism: it parses source code around each CodeQL flow node, serializes deduplicated flows with their CWE identifier and propagation path, embeds them with GraphCodeBERT (a transformer pretrained for code semantics), and runs a transformer-based aggregator plus residual classifier to label each flow as true or false positive. That mechanism is what converts high-recall static analysis into high-precision findings.","core_discovery":"The central claim is that CWE-200 can be treated as a structured detection problem: an eight-category taxonomy of sensitive data derived from the CWE hierarchy, plus transformer classifiers for attack surface, plus customizable CodeQL rules, plus a learned verifier on enriched data-flow paths. The paper claims this combination detects a broad set of CWE-200 subtypes and, crucially, that the verifier removes most static-analysis false positives: global precision on labeled flows from 31 projects rises from 0.226 to 0.872, with recall dropping only from 1.000 to 0.938, and F1 rising from 0.369 to 0.904. The discovery of six confirmed CVEs is presented as evidence that the pipeline finds real, previously unknown exposures in mature projects rather than only synthetic patterns.","pith_inferences":["If the flow-verification numbers were measured on training data, a held-out project evaluation would likely show lower precision; testing on one or more projects excluded from the 31 would settle this directly.","The eight-category sensitive-data taxonomy is language-neutral in principle; porting the same pipeline to Python or JavaScript is plausible, but the sink catalog and CodeQL rules would need rework, and REST/API sinks are the known weak point.","The six findings cluster around logging of credentials and configuration values, suggesting that logging-related exposure patterns are a high-yield target and that a specialized logging-sink detector could raise recall quickly.","A natural extension is to use the flow verifier's rejected paths as active-learning examples for the attack-surface classifier, since the paper shows false positives concentrate in flows through complex objects such as maps."],"forward_implications":["On the 31-project flow dataset, every project either improved or stayed the same in precision after verification (28 improved, 3 unchanged), so the precision gain is not driven by a single outlier.","Real-world recall is bounded by sink coverage: missed Jenkins CVEs such as CVE-2017-2606 are attributed to unrecognized REST-API sinks, so improving sink detection is the clearest path to finding more vulnerabilities.","For CWE subtypes without data flow, such as CWE-615 (sensitive comments), the pipeline reduces to attack-surface detection alone and the benchmark F1 drops to 66.67%, meaning the verification stage cannot help where no flow exists.","The six new CVEs show the pipeline can be deployed as a proactive audit tool; the authors report twelve additional reports under review, so the count of confirmed finds may grow.","The design instantiates queries only for detected sources and sinks, so it avoids scanning all possible paths and stays efficient on large codebases, an explicit design goal for practical use."],"supporting_citations":[{"why":"Supplies the CodeQL data-flow engine and query language used by the Exposure Analysis Engine to trace source-to-sink paths.","marker":"[13]"},{"why":"Provides Sentence-BERT embeddings for the Attack Surface Detection Engine's sensitive-element classifiers.","marker":"[33]"},{"why":"Defines CWE-200, the vulnerability class that the whole system targets.","marker":"[24]"},{"why":"Provides the CWE hierarchy that is re-expressed into the eight sensitive-data categories and CodeQL rules.","marker":"[23]"},{"why":"Establishes that sensitive data exposure is a top web application risk, motivating the need for this detection system.","marker":"[30]"},{"why":"Contrasting prior deep-learning vulnerability detector; the paper positions SIExVulTS against this style of source-sink slice learning.","marker":"[19]"},{"why":"Closest existing vulnerability prediction tool; cited to show the current CWE-200 coverage gap, since it covers only C/C++.","marker":"[12]"},{"why":"SMOTE is applied to balance the rare sensitive labels in training the attack-surface and flow-verification classifiers.","marker":"[8]"}],"fun_headline_variants":["Transformer+CodeQL finds 6 new CVEs, precision jumps to 87%","SIExVulTS: 87% precision on CWE-200, 6 real CVEs","CWE-200 detector uses transformers, raises precision to 87%","Six new CVEs from transformer-based CWE-200 detector","Precision to 87% from 23%: transformer static CWE-200 tool"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline precision gain assumes the Flow Verification Engine was evaluated on flows not seen during training; the paper reports the 87.23% figure on the same 31 projects that supplied the 2,555 labeled flows and does not state a train/test split.","fun_headline_variants_meta":{"raw":{"variants":["Transformer+CodeQL finds 6 new CVEs, precision jumps to 87%","SIExVulTS: 87% precision on CWE-200, 6 real CVEs","CWE-200 detector uses transformers, raises precision to 87%","Six new CVEs from transformer-based CWE-200 detector","Precision to 87% from 23%: transformer static CWE-200 tool"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1668,"prompt_tokens":1021,"completion_tokens":647,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":539}},"tokens_in":637,"tokens_out":647,"duration_ms":5428,"temperature":1.0,"reasoning_tokens":539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:52:04.027178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline while holding out one or more entire projects from the 31 used to train the Flow Verification Engine, and measure precision on those held-out projects. If precision stays near 22.6% instead of 87.2%, the verifier is memorizing labels rather than generalizing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CodeQL data-flow engine and query language used by the Exposure Analysis Engine to trace source-to-sink paths."},{"cited_title":"Cwe-200: Exposure of sensitive information to an unautho- rized actor","cited_arxiv_id":null,"evidence_quote":"Defines CWE-200, the vulnerability class that the whole system targets."},{"cited_title":"Common weakness enumeration (cwe)","cited_arxiv_id":null,"evidence_quote":"Provides the CWE hierarchy that is re-expressed into the eight sensitive-data categories and CodeQL rules."},{"cited_title":"Owasp top ten","cited_arxiv_id":null,"evidence_quote":"Establishes that sensitive data exposure is a top web application risk, motivating the need for this detection system."},{"cited_title":"LineVul: A Transformer-based Line-level Vulnerability Prediction","cited_arxiv_id":null,"evidence_quote":"Closest existing vulnerability prediction tool; cited to show the current CWE-200 coverage gap, since it covers only C/C++."},{"cited_title":"V., B OWYER , K","cited_arxiv_id":null,"evidence_quote":"SMOTE is applied to balance the rare sensitive labels in training the attack-surface and flow-verification classifiers."}],"review_version":1}