{"id":"37007311-630d-418d-aa82-4da6d3b64583","arxiv_id":"2507.18957","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SliceMate, an LLM multi-agent pipeline with synthesis, verification, and refinement agents, reports large gains over existing slicing tools on a new manually annotated Java and Python benchmark, SliceBench.","lead":"SliceMate is a multi-agent LLM system for static program slicing that expands search scope across functions and files, verifies, and refines candidate slices, reporting higher accuracy and F1 than traditional slicers and learned baselines on a new 2,200-program benchmark. Generalists should read it to see how LLM agents are being aimed at precise code analysis tasks that usually require expensive dependency graphs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gains may reflect GPT-4o memorization of public benchmark programs (CodeNet, Defects4J, BugsInPy) rather than generalizable slicing; the paper reports no leakage controls or private holdout evaluation.","rationale":"The strongest claim is an empirical comparison, and for that comparison to be meaningful the evaluation must isolate slicing ability. The benchmark and downstream tasks are drawn from public datasets that are standard LLM training material, and the paper reports no attempt to detect or mitigate overlap with GPT-4o's pretraining data. Since GPT-4o is a closed model with undisclosed training data, this is not a minor methodological preference but a necessary control for any LLM benchmark claim. A semantics-preserving renaming test would settle it: static slicing is invariant under consistent renaming, so a method that genuinely computes dependencies should maintain performance, and a large drop would indicate memorization. This concern is more load-bearing than the reader's verifier-reliability concern, because the verifier could be reliable and the headline claim could still be an artifact of data exposure. The appropriate verdict remains conditional: the architecture and benchmark are promising, but the central comparison needs a leakage-controlled evaluation before the performance claims can be accepted.","tokens_in":20136,"tokens_out":10143,"duration_ms":99958,"concrete_test":"Run SliceMate on a private or post-cutoff holdout set: collect about 100 Java and 100 Python programs from repositories created after GPT-4o's knowledge cutoff (or otherwise not publicly indexed), annotate ground-truth slices with the same dual-annotation protocol, and compare F1 and Accuracy-EM to SliceBench. As a cheaper first check, apply a semantics-preserving transformation (systematic identifier renaming plus function reordering) to a random 100-program subset of SliceBench and rerun the pipeline at temperature 0; static slicing is invariant under consistent renaming, so if F1 drops materially (e.g., more than 10 percentage points) relative to the original programs, memorization of benchmark code, rather than generalized slicing, is implicated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that SliceMate's multi-agent pipeline performs accurate static slicing on large real-world programs. The load-bearing condition is that SliceMate's outputs measure slicing competence rather than recall of the evaluation programs. SliceBench is built from CodeNet and GitHub (Defects4J/BugsInPy), all public corpora almost certainly present in GPT-4o's pretraining data; downstream evaluation also uses Defects4J/BugsInPy. The paper explicitly uses GPT-4o (Section 6.1.1) but provides no membership check, no knowledge-cutoff filter, no private holdout, and no comparison on semantics-preserving variants. Because the best baselines are classical tools or models fine-tuned without access to these exact programs, any memorization advantage transfers directly to the reported 63.1% accuracy and 62.5% F1 gaps and to the Top-10 localization improvements. The ablation in Table 3 shows the verification and refinement agents help, but that only shows the pipeline fits the benchmark distribution; it does not distinguish semantic verification from alignment with annotation conventions or memorized ground truth. Without controlling for pretraining overlap, the claim that LLM agents significantly outperform slicing tools on arbitrary large programs is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SliceMate is an LLM-based multi-agent system for static program slicing that avoids explicit dependency graph construction. It uses a synthesis agent to generate candidate slices, a verification agent to check completeness and conciseness, and a refinement agent to repair slices, controlled by a module that limits iterations. The paper introduces SliceBench, a manually annotated benchmark of 2,200 Java and Python slicing instances, and reports that SliceMate outperforms traditional tools (Joern, JavaSlicer, TyperSlicer) and learning-based baselines (NS-Slicer, NS-Slicer Pro) on both CodeNet-scale and real-world GitHub-scale programs. It also demonstrates downstream improvements in debugging and bug localization on Defects4J and BugsInPy.","tokens_in":20398,"tokens_out":6027,"duration_ms":58112,"significance":"If validated, SliceMate would be a significant advance: it shows that a prompt-based LLM pipeline can perform inter-procedural slicing on large multi-file programs without constructing a program dependence graph, and the manually annotated SliceBench with high inter-annotator agreement is a valuable resource for the community. The strength of the paper lies in the concrete agent design, the replication of the full pipeline description, and the substantial annotation effort. However, the empirical claims are currently not fully supported because of the lack of control for training-data leakage, test-set tuning of a hyperparameter, and absence of statistical validation. The core results could still hold, but the evidence as presented is not yet convincing.","major_comments":[{"comment":"The evaluation uses only public programs from CodeNet, Defects4J, and BugsInPy, all of which are very likely to be present in GPT-4o's pretraining data. No membership check, knowledge-cutoff filter, private holdout, or semantics-preserving variant evaluation is reported. Since the baselines do not have access to these exact programs, any memorization advantage transfers directly to the reported gains of up to 63.1% in accuracy and 62.5% in F1. Please provide evidence that the results reflect slicing competence rather than memorization, for example by evaluating on newly collected or modified programs, or by reporting a leakage analysis.","section":"§4.1, §6.1.1"},{"comment":"The maximum number of verification-refinement iterations (set to 5) is chosen by measuring F1 on SliceBench-GitHub, which is the same benchmark used for the headline RQ1 results in Table 1. This constitutes tuning a free parameter on the test set and can inflate reported performance. Use a separate validation set, nested cross-validation, or report RQ1 results across different iteration limits to show that the conclusion is robust.","section":"§3.4, §6.3.3"},{"comment":"No repeated runs or statistical significance tests are reported for SliceMate. LLM outputs are stochastic even at temperature 0, and the paper claims SliceMate 'significantly outperforms' baselines without quantifying variance. Report mean and standard deviation over multiple runs, and perform appropriate significance tests (e.g., paired bootstrap or Wilcoxon) against the best baseline for the headline metrics.","section":"§6.1.1"},{"comment":"The verification agent is load-bearing for the claim that the refinement loop converges to the human-defined slice, yet its completeness and conciseness judgments are never evaluated in isolation. The paper only reports end-to-end F1. An evaluation of the verifier's precision/recall on artificially corrupted slices, or an error analysis of the refinement iterations, would clarify whether the gains come from genuine semantic verification or from aligning with annotation conventions or memorized ground truth.","section":"§3.2"},{"comment":"The contributions state that SliceBench is 'publicly available', but the manuscript contains no link, repository identifier, or data availability statement. Since SliceBench is a central contribution and the experiments depend on it, please provide the actual availability information and ensure code and data are accessible for reproducibility.","section":"§1, §7"}],"minor_comments":[{"comment":"The description of the CodeNet sampling is internally inconsistent: it first says 'select 1,000 representative programs' and then says 'we further sample 1,000 Python and 1,000 Java programs from CodeNet, yielding 2,000 slicing instances.' Please clarify the exact composition of the CodeNet subset.","section":"§4.1"},{"comment":"The abstract reports improvements of 'up to 22% accuracy and 28% F1' while the introduction reports 'up to 63.1% accuracy and 62.5% F1' for large programs. Please reconcile these numbers or clearly specify the setting for each.","section":"Abstract/§1"},{"comment":"The text uses 'BugsInPY' instead of the correct 'BugsInPy' in several places.","section":"§6.2.2"},{"comment":"The sentence 'The bolded value in the table indicate the highest value' should be 'values indicate' or 'value indicates'.","section":"§6.1.2"},{"comment":"Several instances of 'SliceMate' and other words are missing spaces in the text, e.g., 'AlthoughSliceMatedoes not need' in §3.1.","section":"§3.1, §6.1.1"},{"comment":"The legend relies on red/blue colors that are not distinguishable in grayscale print; add markers or textual labels for the Python and Java curves.","section":"Figure 12"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a plausible fit for a software engineering venue and the benchmark is a valuable contribution. The main gate is the contamination risk: the evaluation uses public programs almost certainly seen by GPT-4o, and the paper does not control for this. I recommend major revision rather than rejection because the issue is addressable with additional experiments or a careful leakage analysis, and the pipeline itself is described in enough detail to be reproduced."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. The SliceMate pipeline is a real technical step beyond prompt-based LLM slicing: the ACI-mediated scope expansion plus the verify-refine loop is a sensible way to do inter-procedural slicing without a dependency graph. And SliceBench, the 2,200-instance hand-annotated benchmark, is a useful asset even if you never touch the agent. That said, the headline numbers should not be taken at face value: the evaluation is not yet rigorous enough to rule out leakage, and the paper is missing artifacts.\n\nWhat is genuinely new is the three-agent architecture. Prior LLM works either prompted a single model or trained a classifier over statements. SliceMate adds incremental scope expansion via Tree-sitter-based interfaces and an iterative verification/refinement loop. The ablation shows that loop earns its keep: removing it drops F1 by 5.5-29.1%, and exact-match accuracy collapses. That is meaningful evidence the architecture matters, not just the base model.\n\nThe soft spots, in order. First, contamination. CodeNet, Defects4J, and BugsInPy are all public corpora, and GPT-4o was almost certainly trained on them. The paper has no membership test, no private holdout, and no variant-based check. The biggest gains appear on the large GitHub programs-exactly where memorized code could most plausibly help. So the claim that SliceMate generalizes to arbitrary large programs is not yet supported. Second, the evaluation is a single run with no significance tests; temperature 0 is not a guarantee of determinism. Third, the max-iteration cutoff was chosen on SliceBench-GitHub, the same slice of data used for the RQ1 headline numbers. That is a relatively minor form of tuning, but it should be disclosed or avoided. Fourth, the paper says the dataset is public but gives no link; code and data are not accessible, which would block reproduction.\n\nNone of this is a load-bearing flaw in the method itself. The pairwise agreement on the benchmark is high, the design is coherent, and the downstream localization gains are consistent across two tools. The issue is that the empirical claims outrun the experimental controls.\n\nI'd send this to serious peer review. It is a legitimate new technique and benchmark, and the right reviewers will push the authors to add leakage controls, repeated runs, and artifacts. It probably needs major revision, not a desk reject.","headline":"A genuinely new LLM-agent slicing pipeline with a valuable hand-annotated benchmark, but the headline gains are weakened by leakage risk, single-run evaluation, and a hyperparameter tuned on the same benchmark.","tokens_in":20909,"tokens_out":4061,"would_cite":true,"duration_ms":41168,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM agents beat dependency-graph slicing on large programs.","keywords":["static program slicing","LLM agents","multi-agent systems","program dependence graph","bug localization","code analysis benchmark","verification refinement loop","Java and Python"],"falsifier":"Record the verification agent's flagged missing and irrelevant statements on the 2,200 SliceBench instances and compare them with the human-annotated ground truth. If the verifier's per-judgment precision and recall are near chance while end-to-end F1 remains high, the reported gains do not come from the claimed verification mechanism. A direct oracle experiment—feeding the true missing statements into the refinement agent instead of the verifier's output—should leave end-to-end F1 unchanged if the verifier is already supplying exactly that information.","tokens_in":19979,"feed_emoji":"✂️","tokens_out":7650,"duration_ms":75308,"temperature":0.7,"pith_summary":"SliceMate is a proposal to replace the standard machinery of static program slicing—dependency-graph reachability and per-statement learned classification—with an orchestrated team of three large-language-model agents: one that synthesizes candidate slices, one that verifies them for completeness and conciseness, and one that repairs them. The paper claims this agent loop beats all five existing slicing tools it compares against, with the largest gains on large multi-file programs: up to 63.1% higher accuracy and 62.5% higher F1 than the best baseline, and up to 34% better Top-10 bug-localization accuracy downstream. It also contributes SliceBench, 2,200 manually annotated slicing instances in Java and Python, with programs up to 8,577 lines. A sympathetic reader would care because the result suggests that a core code-analysis primitive, one traditionally tied to expensive graph infrastructure and language-specific parsers, can be carried by LLM reasoning with no dependency graph and no task-specific training.","feed_headline":"Three LLM agents beat graph-based slicing by up to 63%","feed_subtitle":"On 2,200 annotated programs, it beats dependency-graph and learned baselines without building dependency graphs.","key_machinery":"The load-bearing mechanism is the four-interface, three-agent loop. A criterion-scope search establishes the starting function; invocation extraction and project-function retrieval let the synthesis agent expand across functions using LLM-inferred dependencies; module-information extraction pulls in globals and structural context. The verification agent then runs conciseness analysis, which drops irrelevant statements, and completeness analysis, which reports missing data, control, or inter-procedural dependencies in a fixed format, and the refinement agent repairs the slice using only verbatim code from the search scope. A control module halts at a valid verdict or after five iterations. The structured check-then-repair cycle is what the paper claims converts a strong but unreliable single-pass generator into a high-accuracy slicer.","core_discovery":"The paper's central claim is that a fixed-pipeline trio of LLM agents can compute static program slices more accurately than both dependency-graph reachability tools and models trained on those tools' output. The synthesis agent starts from the function containing the slicing criterion and expands its search scope one function or file at a time; the verification agent separates conciseness from completeness and reports missing dependencies or structures in a structured format; the refinement agent adds only missing code copied verbatim from the already-seen scope. The authors report that this loop outperforms all five baselines on every metric for Java and Python, with the largest wins on large multi-file programs, and that the resulting slices improve downstream bug localization by up to 34% in Top-10 accuracy. The claim is backed by a newly constructed manually annotated benchmark of 2,200 instances, with programs up to 8,577 lines and near-perfect annotator agreement.","pith_inferences":["If the mechanism generalizes, the same synthesis–verify–repair architecture could be applied to other graph-reachability analyses such as taint tracking or change-impact analysis, with the verifier redesigned around each analysis's own correctness condition.","The headline gains are measured against human-annotated slices rather than the classical reachability-defined slice, so whether those gains transfer to consumers that expect exact graph-theoretic slices remains untested.","A cheap mechanism check is to replace the LLM verification agent with a deterministic under-approximation, such as flagging only variables used before definition within the slice, and measure how much of the F1 and localization gains survive; this isolates the verifier's contribution.","Because refinement may only add code verbatim from the already-searched scope, the system's ceiling is set by the synthesis agent's scope expansion, so measuring scope-expansion recall separately from slice F1 would show where failures originate."],"forward_implications":["Static slicing no longer requires building and traversing a program dependence graph, since the same LLM front-end handles Java and Python and is claimed to be language-agnostic with suitable parsing support.","The advantage over existing tools grows with program size: on multi-file programs averaging over 2,100 statements, SliceMate reports up to 63.1% higher accuracy and 62.5% higher F1 than the best baseline, where the paper's analysis says both traditional reachability and per-statement classification degrade.","Downstream consumers benefit: slices from SliceMate raise Top-10 bug-localization accuracy of existing localizers by up to 34% and improve bug-coverage ratios in debugging, because the slices are both precise and concise.","The verification–refinement loop is load-bearing: removing it lowers F1 by 5.5% to 29.1% across settings, and each additional iteration yields diminishing returns beyond five."],"supporting_citations":[{"why":"Defines the program dependence graph and the reachability-based slicing formulation that SliceMate claims to bypass.","marker":"[15]"},{"why":"Provides a traditional Java slicing baseline and the training-data source for the learning-based baselines.","marker":"[19]"},{"why":"Represents the state-of-the-art learning-based slicing approach whose training-data ceiling motivates SliceMate's design and which is used as the main comparison.","marker":"[21]"},{"why":"Shows that straightforward LLM prompting gives suboptimal slices, motivating the multi-agent orchestration.","marker":"[23]"},{"why":"Introduces interprocedural slicing via system dependence graphs, the scalability bottleneck SliceMate targets.","marker":"[29]"},{"why":"Supplies the two thousand small single-function programs that make up the bulk of SliceBench.","marker":"[48]"},{"why":"Supplies real-world Java programs used for the large-scale and downstream bug-localization evaluations.","marker":"[49]"},{"why":"Supplies real-world Python programs used for the large-scale and downstream bug-localization evaluations.","marker":"[50]"},{"why":"Provides a dependency-graph slicing baseline used for both Java and Python.","marker":"[52]"}],"fun_headline_variants":["LLM agents slice programs without dependency graphs","SliceMate: agent pipeline beats graph-based slicing","No dependency graphs, just LLM agents for slicing","Agent trio tops traditional slicing accuracy","SliceMate: LLM agents redefine static slicing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the verification agent's judgments about missing and irrelevant statements align with the human slicing semantics used to build the ground truth, since every repair step obeys those judgments and the paper only validates the end-to-end slice, never that judgment directly.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents slice programs without dependency graphs","SliceMate: agent pipeline beats graph-based slicing","No dependency graphs, just LLM agents for slicing","Agent trio tops traditional slicing accuracy","SliceMate: LLM agents redefine static slicing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000817,"raw_usage":{"total_tokens":3616,"prompt_tokens":1023,"completion_tokens":2593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":2525}},"tokens_in":639,"tokens_out":2593,"duration_ms":19873,"temperature":1.0,"reasoning_tokens":2525,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:04:20.826716+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the verification agent's flagged missing and irrelevant statements on the 2,200 SliceBench instances and compare them with the human-annotated ground truth. If the verifier's per-judgment precision and recall are near chance while end-to-end F1 remains high, the reported gains do not come from the claimed verification mechanism. A direct oracle experiment—feeding the true missing statements into the refinement agent instead of the verifier's output—should leave end-to-end F1 unchanged if the verifier is already supplying exactly that information.","supporting_citations":[{"cited_title":"Ottenstein, and Joe D","cited_arxiv_id":null,"evidence_quote":"Defines the program dependence graph and the reachability-based slicing formulation that SliceMate claims to bypass."},{"cited_title":"A program slicer for java","cited_arxiv_id":null,"evidence_quote":"Provides a traditional Java slicing baseline and the training-data source for the learning-based baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the state-of-the-art learning-based slicing approach whose training-data ceiling motivates SliceMate's design and which is used as the main comparison."},{"cited_title":"Program slicing in the era of large language models, 2024","cited_arxiv_id":null,"evidence_quote":"Shows that straightforward LLM prompting gives suboptimal slices, motivating the multi-agent orchestration."},{"cited_title":"Interprocedural slicing using dependence graphs","cited_arxiv_id":null,"evidence_quote":"Introduces interprocedural slicing via system dependence graphs, the scalability bottleneck SliceMate targets."},{"cited_title":"Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R","cited_arxiv_id":null,"evidence_quote":"Supplies the two thousand small single-function programs that make up the bulk of SliceBench."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies real-world Java programs used for the large-scale and downstream bug-localization evaluations."},{"cited_title":"Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies","cited_arxiv_id":null,"evidence_quote":"Supplies real-world Python programs used for the large-scale and downstream bug-localization evaluations."},{"cited_title":"Modeling and discovering vulnerabilities with code property graphs","cited_arxiv_id":null,"evidence_quote":"Provides a dependency-graph slicing baseline used for both Java and Python."}],"review_version":2}