{"id":"d98c7d55-2513-44a7-a878-a7c3185214ee","arxiv_id":"2607.05888","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"i-EXAM combines PDDL-based AI planning with LLM-generated explanations to help sysadmins visualize attack paths and evaluate network hardening strategies with formal guarantees.","lead":"i-EXAM is an interactive tool that uses AI planning to model network attack paths, suggest hardening strategies, and explain them in natural language via LLMs. A smart generalist might read it to see how formal automated planning can be made accessible to sysadmins for practical security analysis.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The formal guarantees (Theorem 1) apply to the ACG/PDDL model, but i-EXAM's new contribution — the automated scanner-to-PDDL pipeline — has no specified or validated translation rules, so the guarantees may not transfer to the automatically generated models.","rationale":"The reader correctly identified the scanner-to-PDDL fidelity gap as the most load-bearing concern. I agree: the formal guarantees from Theorem 1 are defined over the ACG/PDDL model, and i-EXAM's new contribution — the automated translation from scanner outputs to PDDL — is the link where those guarantees could break. The reader framed this primarily as a scanner limitation (zero-days, undetected misconfigurations), which is a real but generic issue affecting any scanner-based tool. The more precise concern is about the JSON-to-PDDL translation itself: even for correctly detected vulnerabilities and connectivity, there is no specification or validation that the generated PDDL faithfully encodes ACG semantics (action preconditions, effects, connectivity constraints, F1_C preservation). This is the paper's primary technical contribution beyond SPEAR, and it is entirely unvalidated. The paper is a demo-track paper and is honest about lacking empirical evaluation, so CONDITIONAL remains appropriate. The concern does not warrant REJECT because the tool concept is sound, the underlying planning theory is from peer-reviewed prior work, and demo papers are not expected to include full evaluations. But it does warrant keeping the verdict conditional rather than ACCEPT because the central claim of 'formal guarantees' for the automated pipeline is unsubstantiated. The LLM explanation accuracy concern raised by the reader is secondary — the LLM is a UI layer, and its correctness does not affect the formal guarantees of the planning component, though it does affect the tool's 'Explainable' value proposition.","tokens_in":4570,"tokens_out":2527,"duration_ms":269166,"concrete_test":"Construct a small test network (5-10 hosts with known vulnerabilities and connectivity). Manually build the ACG/PDDL model following the SPEAR formalism. Then run i-EXAM's automated pipeline (Nmap/Wazuh/OpenVAS → JSON → PDDL) on the same network. Diff the two PDDL files: compare the action set, preconditions, effects, and initial state. If the automated pipeline produces actions with missing preconditions (e.g., omitting network reachability), incorrect effects, or spurious/missing host-attribute fluents, the Theorem 1 guarantees do not transfer to the automated pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that i-EXAM produces correct, optimal hardening strategies with formal guarantees inherited from SPEAR's Theorem 1 (Podder et al. 2025). Theorem 1 establishes a bidirectional correspondence between valid plans in the PDDL model and valid attack/connectivity paths in the ACG. But this guarantee is defined over a correctly constructed ACG/PDDL model. i-EXAM's new contribution is the automated pipeline: Nmap/Wazuh/OpenVAS → JSON database → PDDL generator → planning model. The paper provides no formal specification of the JSON-to-PDDL translation rules, no example PDDL output for a known configuration, and no comparison between manually-constructed and automatically-generated models. If the translation misrepresents host connectivity, vulnerability preconditions (e.g., requires both network reachability AND a specific vulnerable service), or service dependency constraints (the F1_C connectivity preservation requirement), then Theorem 1's guarantees apply to a model that does not correspond to the real network — making them vacuous for the automated pipeline. This is distinct from the general limitation that scanners miss zero-days; it is about whether the translation of *detected* information into PDDL preserves ACG semantics. The paper states the pipeline 'constructs planning models for any network without manual modeling effort' (Generalization section) but provides no evidence that the constructed models are semantically correct. Since this translation is the paper's primary technical contribution beyond SPEAR, the absence of any validation is the most load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This demo-track paper presents i-EXAM, an interactive tool built on the SPEAR framework (Podder et al. 2025) that automates the pipeline from network scanner outputs (Nmap, Wazuh, OpenVAS) through JSON representation to PDDL planning models, and uses LLMs to generate natural-language explanations of hardening strategies. The tool supports visualization of attack paths, computation of two security metrics (impenetrability M1 and attack difficulty M2), diverse hardening strategy enumeration, and counterfactual explanations. The formal soundness and completeness guarantees are inherited from SPEAR's Theorem 1 and Proposition 4.","tokens_in":4708,"tokens_out":1744,"duration_ms":114864,"significance":"The integration of formal AI planning with practical network security tooling is a worthwhile contribution, and the tool addresses a genuine practitioner need. The use of diverse solution selection and LLM-based explanation of counterfactual plan failures is a reasonable design choice for accessibility. The demo video and case-study screenshot provide concrete evidence of a working system. However, the paper's new contribution over SPEAR—the automated scanner-to-PDDL pipeline—lacks the specification and validation needed to support the claims made about it.","major_comments":[{"comment":"Generalization section: The paper claims the automated pipeline 'constructs planning models for any network without manual modeling effort' and that i-EXAM 'inherits SPEAR's soundness and completeness guarantees.' However, Theorem 1 (Podder et al. 2025) establishes a correspondence between plans and ACG paths over a correctly constructed model. The new contribution of i-EXAM is the automated JSON-to-PDDL translation, but the paper provides no specification of the translation rules, no example PDDL output for a known configuration, and no comparison between manually-constructed and automatically-generated models. Without any evidence that the translation preserves ACG semantics (e.g., vulnerability preconditions requiring both network reachability and a specific vulnerable service, or the F1_C connectivity preservation requirement), the claim that formal guarantees transfer to the auto-g ","section":null},{"comment":"Explanation Generation section: The LLM component (llama-3.1-nemotron-70b-instruct) is presented as converting 'plan failure information into natural language,' but no example input/output pair is shown, and there is no discussion of how the LLM is prompted or what happens when the LLM misrepresents the planner's counterfactual information. For a tool whose name includes 'Explainable,' this component is underspecified. Even a single worked example from the case study in Fig. 1 would substantially strengthen the contribution.","section":null},{"comment":"Scalability section: The scalability claims reference empirical results from Podder et al. 2025 on 30-node networks, but it is unclear whether these results were obtained with i-EXAM's automated pipeline or with manually constructed models. If the latter, the overhead of the automated pipeline (scanner data collection, JSON processing, PDDL generation) is not discussed. The claim that 'the top-k planner enables enumeration of diverse attack paths even in larger networks' is stated without any evidence.","section":null}],"minor_comments":[{"comment":"The impenetrability metric M1 is written as F^I_A(E_A) = 1 iff |E_A| = 0, but the notation F^I_A is not defined beyond this equation; clarifying what the superscript I and subscript A denote would help readers unfamiliar with SPEAR.","section":null},{"comment":"The M2 compilation mentions auxiliary fluents (init_change_mode, act_mode, goal_reached) but these are not formally defined; a brief description of their role would improve self-containedness.","section":null},{"comment":"The reference to 'Dubey et al. 2024' for the Llama 3 model is cited, but the specific model used (llama-3.1-nemotron-70b-instruct) appears to be an NVIDIA variant; the citation may need updating to the correct source.","section":null},{"comment":"The paper mentions 'other scanners' and 'documented data' in the Building Planning Models section without specifying what these are; this vagueness makes the pipeline hard to reproduce.","section":null},{"comment":"Fig. 1 is referenced but the screenshot is too small to read in the provided text; if accepted, ensure the figure is legible in the camera-ready.","section":null}],"recommendation":"major_revision","confidential_remarks":"This is a demo-track paper, so the absence of a full empirical evaluation is expected. However, even for a demo paper, the central new contribution (automated scanner-to-PDDL translation) needs at least a specification of translation rules and one validated example to support the claim that formal guarantees transfer. The overlap with Podder et al. 2025 is significant (4 of 5 authors overlap); the authors should clearly delineate what is novel in i-EXAM versus SPEAR. The demo video may partially address some concerns about the working system, but the semantic correctness of the translation pipeline is not something a video can establish."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee correctly identifies that the automated scanner-to-PDDL pipeline, the LLM explanation component, and the scalability claims are underspecified in the current manuscript. We agree with all three major comments and will revise the paper accordingly. Below we address each point and describe the specific revisions we will make.","responses":[{"response":"The referee is correct. The current manuscript does not specify the translation rules from JSON to PDDL, does not provide example PDDL output, and does not demonstrate semantic preservation between automatically generated and manually constructed models. We agree that the claim about inheriting SPEAR's formal guarantees cannot be made without evidence that the automated translation preserves ACG semantics. In the revision we will: (1) include a specification of the JSON-to-PDDL translation rules, covering how vulnerability preconditions (requiring both network reachability and a specific vulnerable service) and connectivity preservation constraints (F1_C) are encoded; (2) add a worked example showing the PDDL domain and problem files generated from a known small network configuration; and (3) include a comparison between manually constructed and automatically generated models on the same network to demonstrate semantic equivalence. We will also qualify the generalization claim to state that the pipeline constructs models for networks whose scanner outputs cover the attribute types required by the ACG formalism, rather than claiming coverage of 'any network' without qualification.","revision_made":"yes","referee_comment":"Generalization section: The paper claims the automated pipeline 'constructs planning models for any network without manual modeling effort' and that i-EXAM 'inherits SPEAR's soundness and completeness guarantees.' However, Theorem 1 (Podder et al. 2025) establishes a correspondence between plans and ACG paths over a correctly constructed model. The new contribution of i-EXAM is the automated JSON-to-PDDL translation, but the paper provides no specification of the translation rules, no example PDDL output for a known configuration, and no comparison between manually-constructed and automatically-generated models. Without any evidence that the translation preserves ACG semantics (e.g., vulnerability preconditions requiring both network reachability and a specific vulnerable service, or the F1_C connectivity preservation requirement), the claim that formal guarantees transfer to the auto-g"},{"response":"We agree. The LLM explanation component is underspecified for a tool whose name includes 'Explainable.' In the revision we will add: (1) the prompt template used to query the LLM, showing how structured counterfactual plan traces from the planner are formatted as input; (2) at least one concrete input/output pair from the case study shown in Fig. 1, including the planner's counterfactual trace and the LLM's generated natural-language explanation; and (3) a discussion of failure modes when the LLM misrepresents the planner's output, including the mitigation strategy of presenting the raw planner trace alongside the LLM-generated explanation so the sysadmin can verify the explanation against the formal output. We acknowledge that without these details the reader cannot evaluate the reliability or usefulness of the explanation component.","revision_made":"yes","referee_comment":"Explanation Generation section: The LLM component (llama-3.1-nemotron-70b-instruct) is presented as converting 'plan failure information into natural language,' but no example input/output pair is shown, and there is no discussion of how the LLM is prompted or what happens when the LLM misrepresents the planner's counterfactual information. For a tool whose name includes 'Explainable,' this component is underspecified. Even a single worked example from the case study in Fig. 1 would substantially strengthen the contribution."},{"response":"The referee is correct on both points. The empirical results cited from Podder et al. 2025 were obtained with manually constructed models, not with i-EXAM's automated pipeline. The current manuscript does not discuss the overhead of the automated pipeline (scanner data collection, JSON processing, PDDL generation), and the claim about the top-k planner enabling enumeration in larger networks is stated without supporting evidence. In the revision we will: (1) clarify that the cited 30-node results used manually constructed models; (2) report the overhead introduced by the automated pipeline stages (scanner data collection, JSON processing, PDDL generation) measured on our case-study network; and (3) either provide empirical evidence for the top-k planner claim on larger networks or remove the unsupported claim and note it as future work. We agree that the scalability section as written overstates what has been demonstrated.","revision_made":"yes","referee_comment":"Scalability section: The scalability claims reference empirical results from Podder et al. 2025 on 30-node networks, but it is unclear whether these results were obtained with i-EXAM's automated pipeline or with manually constructed models. If the latter, the overhead of the automated pipeline (scanner data collection, JSON processing, PDDL generation) is not discussed. The claim that 'the top-k planner enables enumeration of diverse attack paths even in larger networks' is stated without any evidence."}],"tokens_in":4279,"tokens_out":1054,"duration_ms":118230,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Bottom line: i-EXAM is a demo-track paper describing an interactive tool that wraps the SPEAR framework (Podder et al. 2025) with an automated scanner-to-PDDL pipeline, a visualization UI, what-if hardening analysis, and LLM-based natural language explanations. The integration is new as a software artifact. The individual components — ACG formalism, planning compilation, security metrics (M1, M2), diverse solution selection, explanation by model restriction — are all inherited from prior work and cited appropriately. The tool concept is reasonable and the paper is honest about being a demo with empirical evaluation left to future work. Credit where it's due: the idea of combining formal planning guarantees with LLM-generated explanations for sysadmins is a sensible engineering direction, and the what-if hardening workflow with diverse solutions is a genuinely useful design choice. The paper does not overclaim what the formal guarantees cover — it correctly attributes Theorem 1 and Proposition 4 to the prior work. The main soft spot is the one the stress-test flags, and I think it lands. The paper's primary technical contribution beyond SPEAR is the automated pipeline: Nmap/Wazuh/OpenVAS → JSON → PDDL. But there is no specification of the translation rules, no example PDDL output for a known configuration, and no comparison between manually-constructed and automatically-generated models. The formal guarantees from Theorem 1 hold over a correctly constructed ACG/PDDL model — they say nothing about whether the automated translation produces such a model. If the JSON-to-PDDL mapping mishandles host connectivity, vulnerability preconditions, or the F1_C connectivity preservation requirement, the guarantees become vacuous for the automated pipeline. This is the load-bearing gap, and it's proportionate to the paper's scope: for a demo, you'd want at least one worked example showing the translation preserves ACG semantics. Secondary concerns are minor by comparison: no LLM explanation accuracy assessment, no code release, and the 30-node scalability figure is borrowed from prior work. None of these are surprising for a demo paper, but they do limit what the paper substantiates. This paper is for readers interested in the intersection of AI planning and network security tooling. It deserves a serious referee — the tool is plausible and the integration is real, but the referee should push hard on whether the automated pipeline is validated enough to support the generalization claims.","headline":"Demo-track tool integrating SPEAR planning with LLM explanations; the new automated pipeline is unvalidated","tokens_in":5374,"tokens_out":1111,"would_cite":false,"duration_ms":128251,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"AI planner turns network scans into guaranteed hardening strategies","keywords":["network security","automated planning","PDDL","attack graph","attack connectivity graph","hardening strategies","explainable AI","LLM explanation"],"falsifier":"If the JSON-to-PDDL translation loses security-relevant state present in the scanner outputs, or if the ACG model cannot represent attack vectors that real adversaries use, then the optimal hardening strategies would be formally correct but practically incomplete.","tokens_in":4613,"feed_emoji":"🛡️","tokens_out":565,"duration_ms":69154,"temperature":0.7,"pith_summary":"i-EXAM is an interactive tool that bridges automated AI planning and network security administration. Its central claim is that by compiling network scanner data into formal planning problems, it can automatically identify attack paths, compute security metrics, generate diverse hardening strategies with optimality guarantees, and explain those strategies in natural language. The tool builds on the SPEAR framework's Attack Connectivity Graph (ACG), a hypergraph that simultaneously models attack paths and service connectivity. i-EXAM automates the entire pipeline: it ingests data from standard network scanners, converts it to PDDL planning models, solves them using optimal search, and uses an LLM to translate the results into plain-English explanations for sysadmins who need no expertise in formal planning.","feed_headline":"AI planner turns network scans into guaranteed hardening strategies","feed_subtitle":"i-EXAM compiles scanner data into formal planning models, finds optimal defenses with soundness guarantees, and explains them in plain","key_machinery":"Attack Connectivity Graph (ACG): a hypergraph capturing both attack paths and service connectivity in a network, compiled into PDDL planning problems where host-attribute pairs become fluents and attack/connectivity edges become actions.","core_discovery":"The paper demonstrates that the gap between formal AI planning theory and practical network security administration can be closed by an automated pipeline. The key mechanism is the compilation of network state into PDDL, which allows security questions (can an attacker reach this target? what is the cheapest attack? what changes block all attacks?) to be answered with formal soundness and completeness guarantees inherited from the underlying planning framework. The LLM explanation layer converts structured counterfactual plan traces into natural language, making the formal reasoning accessible to administrators without planning expertise.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Formal planning pipeline generates verified network defenses","AI planning translates network scans into sound hardening","PDDL compilation maps network state to guaranteed attack paths","i-EXAM converts network scans into formally verified defenses","Automated planning finds and explains sound network hardening"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The formal guarantees hold only over the planning model, not over the actual network. If the automated scanner-to-PDDL pipeline misses vulnerabilities, misconfigurations, or zero-days that the scanners cannot detect, the soundness and completeness guarantees become vacuous with respect to the real system being defended.","fun_headline_variants_meta":{"raw":{"variants":["Formal planning pipeline generates verified network defenses","AI planning translates network scans into sound hardening","PDDL compilation maps network state to guaranteed attack paths","i-EXAM converts network scans into formally verified defenses","Automated planning finds and explains sound network hardening"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":868,"prompt_tokens":337,"completion_tokens":531,"prompt_tokens_details":null},"tokens_in":337,"tokens_out":531,"duration_ms":28599,"temperature":1.0,"reasoning_tokens":550,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T21:25:03.509643+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If the JSON-to-PDDL translation loses security-relevant state present in the scanner outputs, or if the ACG model cannot represent attack vectors that real adversaries use, then the optimal hardening strategies would be formally correct but practically incomplete.","supporting_citations":[],"review_version":1}