{"id":"3f8c68ff-28fd-489d-a4ca-31ba26e089cd","arxiv_id":"2508.20911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Convul combines CodeBERT, RGCN, and SubgraphX on a new Concurrency-Aware Code Property Graph, reporting detection F1 of 86% and localization IoU of 15% on the studied benchmarks.","lead":"A deep learning pipeline called Convul detects and localizes concurrency bugs in C/C++ code by combining a pre-trained code model with a graph neural network over a new concurrency-aware program graph. Its reported gains over eight detection and five localization baselines are real but currently hard to verify because no code or dataset is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim's 10% precision improvement is contradicted by Table 2: Convul's precision (78.32) is below GraphCodeBERT's (84.89).","rationale":"The reader's weakest assumption focused on dataset label correctness, which is a valid external validity concern. However, the most load-bearing issue is internal inconsistency: the paper's headline quantitative claim is contradicted by its own reported results. This is not a matter of missing artifacts or potential label noise, but a verifiable failure of the argument as written. The abstract promises 10% improvement in precision, yet Table 2 shows Convul's precision is lower than a baseline (GraphCodeBERT). No other detection experiment in the paper reports precision, so the claim is unsupported. Similarly, the 26% recall improvement is not reflected in any metric gap in Table 2. The model selection choice (CodeBERT instead of the better-performing GraphCodeBERT from Table 4) compounds the issue, suggesting the reported results are cherry-picked or the aggregation was flawed. These internal inconsistencies directly undermine the central claim and cannot be fixed by releasing data or adding error bars alone. Therefore the verdict should move from CONDITIONAL to REJECT, or at minimum require a major revision that either corrects the claims or provides a reproducible aggregation that matches the tables.","tokens_in":13196,"tokens_out":7009,"duration_ms":69672,"concrete_test":"Independently recompute mean accuracy, precision, and recall improvements from Tables 2 and 6, using any explicit aggregation rule (e.g., average over baselines, or best-baseline differences, or relative improvements). If the resulting improvements do not reproduce the claimed 10%/26%, or if the precision deficit of Convul vs GraphCodeBERT in Table 2 is confirmed, the central claim is refuted. Also re-run the detection experiment with GraphCodeBERT as embedding in Convul; if its F1 exceeds 86%, the paper's model choice is unjustified.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim—'improvement of 10% in accuracy and precision and 26% in recall'—is internally contradicted by its main detection table. In Table 2, on DeepRace, Convul w/ CCPG achieves Precision=78.32, while baseline GraphCodeBERT achieves Precision=84.89. Thus Convul's precision is 6.57 percentage points lower than a baseline's, so the claimed precision improvement of 10% is false for the primary detection evaluation. For recall, Convul's 94.60 is only 1.85 points above LineVul's 92.75 (≈2% relative), not 26%. The only other experimental table reporting detection comparisons, Table 6, uses localization accuracy/IoU and does not report precision or recall. Therefore the aggregate numbers in the abstract are not traceable to any reported experiment. Additionally, Table 4 shows that replacing CodeBERT with GraphCodeBERT as the node embedding improves F1 from 84.03 to 89.91, yet the method uses CodeBERT; this unexplained model selection, combined with the precision deficit, suggests the reported configuration is not the best found and the headline claim is not reproducible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Convul, a deep-learning pipeline for concurrency bug detection and line-level localization. The authors build a new dataset of 41,317 concurrency bug instances by filtering and manually reviewing samples from DiverseVul, BigVul, SARD, and DeepRace. They introduce the Concurrency-Aware Code Property Graph (CCPG), which augments a standard CPG with synchronization and blocking nodes/edges, and combine CodeBERT node embeddings with an RGCN classifier. For localization, they adapt SubgraphX using MCTS and Shapley values to identify bug-related subgraphs. Experiments compare Convul with detection baselines on DeepRace (Table 2) and localization baselines on BigVul and SARD (Table 6), reporting a detection F1 of about 86% and localization IoU of 14.50/12.57. The abstract and introduction additionally claim average improvements of 10% in accuracy/precision and 26% in recall over state-of-the-art methods.","tokens_in":13557,"tokens_out":5784,"duration_ms":60952,"significance":"If the results hold, the CCPG representation and the concurrency-specific dataset would be useful contributions: the paper directly addresses a real gap in modeling synchronization semantics and in locating bug lines rather than only classifying functions. The use of SubgraphX for subgraph-level localization is a plausible and relevant design choice. However, at present the headline empirical claims are not traceable to the reported tables, and the dataset—one of the three stated contributions—is neither released nor validated beyond a passing mention of manual review. No code, configuration details, or reproducibility artifacts are provided. The significance is therefore conditional on a substantial revision of the reporting and on making the data/experimental protocol verifiable.","major_comments":[{"comment":"The abstract's claim of an average 10% improvement in accuracy and precision and 26% improvement in recall is inconsistent with the paper's own main detection table. In Table 2, Convul w/ CCPG has Precision 78.32, below GraphCodeBERT's 84.89; Accuracy is 75.68 vs. DeepRace's 70.38 (5.30 points, not 10); Recall is 94.60 vs. LineVul's 92.75 (1.85 points) and DeepRace's 76.97 (17.63 points). No detection table reports precision/recall for the merged DiverseVul/BigVul/SARD data, so the aggregate numbers in the abstract are not reproducible. Please either remove the aggregate claim or give an explicit per-table breakdown that justifies each percentage.","section":"Abstract; Experiments, Table 2"},{"comment":"The proposed 41,317-instance concurrency dataset is a central contribution, but its construction is not verifiable. The text says records were filtered by keywords and 'manually reviewed,' yet no inter-rater agreement, adjudication protocol, label-error analysis, or release of the filtered instances is provided; the Data Availability section states only that data will be shared 'on reasonable request.' Because every detection and localization metric in Tables 2 and 6 is computed on these labels, systematic label noise or selection bias would invalidate all comparative conclusions. A validation study or public release of the extraction pipeline is needed.","section":"Dataset"},{"comment":"Table 4 reports that RGCN with GraphCodeBERT embeddings achieves F1=89.91, while RGCN with CodeBERT achieves F1=84.03. The paper does not state the dataset or split used for Tables 4–5, nor does it explain why the final model nonetheless uses CodeBERT when the reported numbers suggest GraphCodeBERT is better. Without this information, the chosen configuration appears not to be the best configuration found, which undermines the main claim and the reproducibility of the results. Please specify the evaluation protocol for RQ2 and justify the final model selection.","section":"RQ2, Tables 4 and 5"},{"comment":"The localization accuracy metric is defined very leniently: a subgraph is counted as accurate if it contains at least one bug node. This, together with the low IoU values (14.50 on Big-Vul, 12.57 on SARD), makes the practical debugging utility difficult to assess. The paper should report stricter metrics, such as exact-match accuracy or top-k line accuracy, and discuss the trade-off between the loose accuracy definition and the reported IoU.","section":"Experiments, RQ4, Table 6"}],"minor_comments":[{"comment":"Algorithm 3 uses SNmax on line 5, but only Nmin is defined in Eq. (5). Also, the hyperparameters α, β, λ, subgraph size limits, RGCN layer count, hidden dimensions, and MCTS iteration count B are never given, so the proposed method cannot be instantiated from the paper.","section":"Algorithm 3 / Eqs. (5)–(9)"},{"comment":"The VulDeeLocator row contains '\\ \\' for Big-Vul Accuracy/IoU instead of numeric values. Please report the missing numbers or state explicitly why they are unavailable.","section":"Table 6"},{"comment":"The sentence 'We thoroughly' at the end of the Dataset section appears incomplete; it does not continue into the next subsection.","section":"Dataset, last paragraph"},{"comment":"The term 'accuracy' is used both for detection accuracy and for localization accuracy in Table 6. These are different metrics and should be named distinctly (e.g., 'localization hit-rate') to avoid ambiguity.","section":"Throughout"},{"comment":"Sharing data only 'on reasonable request' is not sufficient for reproducibility, especially since the construction of the dataset is a core contribution. A public artifact or a detailed release plan should be provided.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The central idea is publishable in principle, but the reporting of the empirical claims needs serious correction. The strongest issue is the mismatch between the abstract and Table 2; this is not a matter of style but of correctness of the stated contribution. I would also check whether the journal's scope and standards permit a dataset contribution that is not publicly released. The self-citations in the introduction are background references and do not, by themselves, create a conflict."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has a real contribution in the CCPG representation and the concurrency bug dataset, and the within-table comparisons mostly support the claim that Convul beats the baselines. But the abstract and conclusion overstate the gains. The 10% precision improvement is contradicted by Table 2, where GraphCodeBERT has higher precision (84.89 vs 78.32). The 26% recall improvement is also off; Convul's recall on DeepRace is 94.60 vs LineVul's 92.75, roughly 2 points. These aggregate numbers are not traceable to the experiments.\n\nWhat is genuinely new: the dedicated dataset (41k instances from four sources), CCPG with synchronization edges and blocking nodes, and the use of SubgraphX for localization. The ablation shows CCPG helps, and the localization results improve IoU from ~10 to ~14.5 on Big-Vul. That is a meaningful step.\n\nThe soft spots are mostly in reporting and reproducibility. Table 4 shows RGCN+GraphCodeBERT gives F1 89.91 while RGCN+CodeBERT gives 84.03, yet the method uses CodeBERT. The paper justifies by saying CodeBERT has higher precision, but F1 is the headline metric; that choice needs an explanation. The dataset is not released, and the manual review has no inter-rater agreement or validation details. No error bars across runs. The data availability statement says 'on reasonable request' which is effectively no release.\n\nThe core idea is sound and the limitations (C/C++ only) are honestly stated. The paper deserves a serious referee, but the authors should fix the claims, justify the model choice, and release artifacts.","headline":"Genuinely new dataset and graph representation, but the headline performance gaps are not in the tables.","tokens_in":13999,"tokens_out":3284,"would_cite":false,"duration_ms":30825,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding synchronization edges and blocking nodes to a code property graph lets a CodeBERT–RGCN model detect concurrency bugs with 86% F1 and localize them to subgraphs, beating eight detection and five localization base","keywords":["concurrency bug detection","bug localization","code property graph","graph neural network","relational graph convolution","SubgraphX","CodeBERT","data race"],"falsifier":"Run Convul on a fresh, independently labeled set of C/C++ concurrent programs with ground-truth race and deadlock lines, with the labelers blind to the model's output. If predicted subgraphs overlap ground truth no better than a random subgraph of the same size, or if the original keyword-filtered labels cannot be reproduced by a second independent review, the central claim is undercut.","tokens_in":13155,"feed_emoji":"🐛","tokens_out":5741,"duration_ms":55453,"temperature":0.7,"pith_summary":"Concurrency bugs are hard to find because the relevant information spans threads, locks, and interleavings that ordinary code representations do not show. This paper tries to make that information explicit by extending the code property graph with synchronization edges and blocking nodes, producing what it calls the Concurrency-Aware Code Property Graph (CCPG). It then trains a heterogeneous graph network on CodeBERT embeddings of CCPG nodes and uses a subgraph-explanation method to localize bugs to source lines. The authors report that Convul, their pipeline, reaches an F1 score of 86% for detection and an Intersection over Union of 15% for localization, outperforming eight detection and five localization baselines. They also assemble 41,317 labeled concurrency bug instances from four existing datasets, which they say is the first dataset built specifically for concurrency bugs.","feed_headline":"Concurrency-aware code graph detects bugs at 86 percent F1","feed_subtitle":"Synchronization edges plus GNN localization beat eight detection and five localization baselines.","key_machinery":"CCPG: Concurrency-Aware Code Property Graph, a property graph obtained by taking a code property graph (merged AST, CFG, and PDG) and adding blocking nodes and synchronization edges: thread-create edges from caller to thread entry, join edges from thread exit to caller, and lock-to-unlock edges spanning critical sections. It carries the argument by making thread interactions and critical sections explicit so the RGCN's relation-specific weights can learn them, and by giving SubgraphX a concurrency-weighted prior for search.","core_discovery":"The paper's central claim is that concurrency bugs can be detected and localized by a deep learning pipeline whose graph representation explicitly encodes synchronization. It defines the Concurrency-Aware Code Property Graph (CCPG), which extends the standard code property graph with blocking nodes and synchronization edges for lock/unlock and thread create/join operations. Node embeddings come from CodeBERT, a pretrained code model; a relational graph convolution network (RGCN) classifies each program; and SubgraphX, with a concurrency-weighted search, extracts connected subgraphs whose Shapley values identify the lines involved in the bug. On the DeepRace dataset Convul reaches 85.69 F1 ve","pith_inferences":["The CCPG construction is tied to POSIX primitives (pthread_mutex, pthread_create/join); extending the same idea to Java monitors, std::thread, OpenMP, or async/await is a natural test, but is not implemented here.","The reported IoU of 15% means most predicted subgraph nodes are still outside the ground-truth bug nodes; a follow-up could combine subgraph explanations with line-level ranking to make localization usable in practice.","The θ term that biases MCTS toward subgraphs with synchronization edges and blocking nodes is a domain prior; ablating α and β would isolate how much of the localization gain comes from this prior versus from generic Shapley scoring.","Releasing the filtered 41,317 instances with line-level labels would let others reproduce the metrics and check label quality; the paper currently offers the data 'on reasonable request'."],"forward_implications":["A representation that encodes lock/unlock and thread create/join structure can replace hand-written race or deadlock heuristics for a learned detector.","Detection and localization share one graph representation, so a single pass can return both a verdict and candidate source lines rather than just a binary label.","The same CCPG pipeline can consume slice-, function-, or file-level inputs, which removes the function-level restriction that limits several graph baselines.","The concurrency-extraction procedure can be rerun on updated CVE/CWE records to expand the dataset as new concurrency bugs are disclosed.","Using pretrained code embeddings inside a heterogeneous GNN improves over either component alone, since the two encode complementary semantic and relational information."],"supporting_citations":[{"why":"Supplies the POSIX/OpenMP data race dataset used for detection evaluation and the DeepRace baseline that Convul must beat.","marker":"[6]"},{"why":"Defines the code property graph that CCPG extends with concurrency edges and blocking nodes.","marker":"[11]"},{"why":"Provides the relational graph convolution network used to learn from the heterogeneous CCPG.","marker":"[16]"},{"why":"Supplies the SubgraphX MCTS/Shapley method adapted for subgraph-based bug localization.","marker":"[17]"},{"why":"Provides CodeBERT, the pretrained model used for node embeddings in the graph.","marker":"[18]"},{"why":"One of the four source datasets from which concurrency bug instances are extracted.","marker":"[21]"},{"why":"Source of labeled vulnerability instances with location information used for detection and localization.","marker":"[22]"},{"why":"Source of labeled functions used for localization evaluation and dataset construction.","marker":"[25]"},{"why":"GNNExplainer, the interpretation method used as a localization baseline through IVDetect and compared against SubgraphX.","marker":"[30]"}],"fun_headline_variants":["Concurrency-aware graph lifts bug detection recall 26% and pinpoints lines","Sync-edge code graph localizes concurrency bugs to specific source lines","CCPG + SubgraphX localize concurrency bugs precisely","GNN with synchronization edges finds concurrency bugs and maps them to lines","Concurrency-aware graph with GNN pinpoints bug lines via explainability"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the 41,317 labels produced by keyword filtering plus manual review are correct; if the labels are noisy or systematically biased, every reported detection and localization number inherits that error.","fun_headline_variants_meta":{"raw":{"variants":["Concurrency-aware graph lifts bug detection recall 26% and pinpoints lines","Sync-edge code graph localizes concurrency bugs to specific source lines","CCPG + SubgraphX localize concurrency bugs precisely","GNN with synchronization edges finds concurrency bugs and maps them to lines","Concurrency-aware graph with GNN pinpoints bug lines via explainability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001789,"raw_usage":{"total_tokens":6881,"prompt_tokens":731,"completion_tokens":6150,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":6070}},"tokens_in":475,"tokens_out":6150,"duration_ms":44108,"temperature":1.0,"reasoning_tokens":6070,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:42:23.067666+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Convul on a fresh, independently labeled set of C/C++ concurrent programs with ground-truth race and deadlock lines, with the labelers blind to the model's output. If predicted subgraphs overlap ground truth no better than a random subgraph of the same size, or if the original keyword-filtered labels cannot be reproduced by a second independent review, the central claim is undercut.","supporting_citations":[{"cited_title":"Deeprace: A learning-based data race detector","cited_arxiv_id":null,"evidence_quote":"Supplies the POSIX/OpenMP data race dataset used for detection evaluation and the DeepRace baseline that Convul must beat."},{"cited_title":"Yamaguchi, N","cited_arxiv_id":null,"evidence_quote":"Defines the code property graph that CCPG extends with concurrency edges and blocking nodes."},{"cited_title":"Modeling relational data with graph convolutional networks","cited_arxiv_id":null,"evidence_quote":"Provides the relational graph convolution network used to learn from the heterogeneous CCPG."},{"cited_title":"On explainability of graph neural networks via subgraph explorations","cited_arxiv_id":null,"evidence_quote":"Supplies the SubgraphX MCTS/Shapley method adapted for subgraph-based bug localization."},{"cited_title":"Codebert: A pre-trained model for programming and natural languages","cited_arxiv_id":null,"evidence_quote":"Provides CodeBERT, the pretrained model used for node embeddings in the graph."},{"cited_title":"Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection","cited_arxiv_id":null,"evidence_quote":"One of the four source datasets from which concurrency bug instances are extracted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of labeled vulnerability instances with location information used for detection and localization."},{"cited_title":"GNNExplainer: generating explanations for graph neural networks","cited_arxiv_id":null,"evidence_quote":"GNNExplainer, the interpretation method used as a localization baseline through IVDetect and compared against SubgraphX."}],"review_version":1}