{"id":"ea478c81-72e4-4b25-ab27-909ce67ec33b","arxiv_id":"2608.09524","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"STAIR, a graph-state, stage-routed, experience-reusing agentic framework, reports an overall defense score of 0.94 on 100 Docker cyber ranges, beating the strongest LLM baseline by 9.5%.","lead":"This paper introduces STAIR, an automated framework that plans cyber incident responses by keeping a graph model of the incident, choosing the right response stage, and reusing validated past actions. It reports a normalized defense score of 0.94 on 100 Docker-based cyber ranges, a 9.5% relative improvement over the strongest LLM-agent baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 9.5% gain is carried by the Experience Database, whose effect labels derive from the same evaluation profiles used to compute ODS; this creates a structural information advantage over baselines.","rationale":"The reader's weakest assumption identifies the same structural dependency: effect labels in the Experience Database are generated from incident-specific evaluation profiles that also define the final ODS metric. I agree with that identification and find it load-bearing. The additional evidence from Table III strengthens the concern: the full system without the Experience Database scores 0.8250, lower than the strongest baseline (0.8589), so the entire reported headline gain of 9.5% is attributable to the component that has privileged access to scoring-profile-derived labels. This does not by itself show that the framework is useless, but it means the paper's central quantitative claim is not yet supported as an evaluation of general incident-response reasoning. The CONDITIONAL verdict remains appropriate; the condition should be that the experience channel be re-evaluated with labels independent of the final scoring function, or the paper should explicitly justify why profile-derived labels are a legitimate form of training signal rather than test-time leakage. No ad hominem is intended; the issue is in the evaluation design, not the authors' conduct.","tokens_in":15981,"tokens_out":3503,"duration_ms":36270,"concrete_test":"Rebuild the Experience Database with effect labels y_i computed only from raw execution feedback (command exit status, service reachability, artifact presence/absence) and without using the incident-specific evaluation profile, while keeping final ODS scoring unchanged. Rerun RQ1 and RQ3 on the same held-out ranges. If Full STAIR's ODS stays near 0.94 and systematically beats CC-DeepSeek V4 Pro, the profile-derived labels are not the source of the gain; if Full ODS falls toward the 0.8250 w/o-E level, the headline 9.5% improvement is an artifact of leakage from the scoring profiles into the experience database.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim rests on the Experience Database. In Table III, removing only the Experience Database (w/o E) drops ODS from 0.9409 to 0.8250, which is below the strongest baseline, CC-DeepSeek V4 Pro at 0.8589. Thus the entire headline advantage is attributable to component E, exactly the component that receives profile-derived labels. In Sec. VI-A, the evaluation profile is stated to be used both for effect labeling and final scoring. In Eq. (4), each experience record stores an effect label y_i produced by the Execution Harness from that incident-specific profile. Although the profile is not exposed to the planner as text, the Experience Database is built from these labels, and retrieval at test time returns stage-compatible records ranked by state similarity. For held-out variants of training families, the retrieved records therefore encode what the hidden scoring profile rewards (e.g., clean-state validation after artifact removal), giving STAIR an oracle-like hint that the baselines do not receive. The w/o-E ablation shows that the 9.5% improvement over CC-DeepSeek V4 Pro disappears without this channel, so the claimed gain is not evidence of generally better state maintenance or stage-specialized planning; it is evidence that labels distilled from the scoring function improve scores.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents STAIR, an end-to-end agentic planning framework for incident response that combines Graph-as-State (GAS), a stage-specialized multi-agent planner, an Experience Database, and an Execution Harness with bounded tool execution. The authors evaluate the framework on 100 Docker-based cyber ranges with a 70/30 train/test split, reporting an Overall Defense Score (ODS) of 0.9409 against 0.8589 for the strongest baseline (CC-DeepSeek V4 Pro), a 9.5% relative improvement. Additional experiments address robustness across incident complexity tiers, cumulative ablations of the three main components, and transfer of experience to held-out variants.","tokens_in":16197,"tokens_out":3720,"duration_ms":37046,"significance":"If the evaluation concern below is resolved, the work is practically significant: it demonstrates that a stateful, stage-aware architecture with validated experience reuse improves closed-loop incident response over open-ended LLM agents, text planners, and RL-style baselines on a nontrivial executable benchmark. The paper's strengths include a clearly described system design, a held-out test split, cumulative ablations that show coherent degradation, and a separate analysis of experience transfer. The code is linked, which supports reproducibility, although the benchmark itself is not released. The main empirical claim, however, depends on the Experience Database component, whose effect labels are generated from the same incident-specific evaluation profiles used for final scoring; the ablations show this component is responsible for the entire headline gain, so the claimed advantage is not yet established as independent of the scoring function.","major_comments":[{"comment":"The Experience Database is constructed from effect labels y_i produced by the Execution Harness using the same incident-specific evaluation profiles that later compute ODS. The paper states that the profile is not exposed to the planner or written into GAS, but it is used to label experiences, and retrieval returns stage-compatible records ranked by state similarity. For held-out variants of training families, the retrieved positive experiences therefore encode what the hidden scoring profile rewards. Table III shows that removing only the Experience Database reduces ODS from 0.9409 to 0.8250, below the strongest baseline (0.8589), so the entire headline improvement is attributable to this channel. To support the claimed 9.5% improvement, please either build the Experience Database from profiles independent of the test scoring profiles, hold out entire families from both training and retrieval, or otherwise demonstrate that retrieved experiences do not encode the test reward function; the current ablation does not separate 'useful historical experience' from 'oracle access to the scoring function.'","section":"§V(d), §VI-A, Eq. (4), Table III"},{"comment":"All metrics are macro-averaged point estimates from a single run over 30 test ranges, with no variance, confidence intervals, or significance tests. The central 9.5% margin and the ablations in Table III should be accompanied by per-range distributions and paired statistical tests (e.g., bootstrap confidence intervals or a Wilcoxon signed-rank test across ranges). This is especially important because several differences are driven by a small number of cases: for example, AttackSup 0.9333 versus 1.0000 corresponds to one range out of 30.","section":"Table II and Table III"},{"comment":"The claim that experience reuse transfers to held-out incidents is weakened by the composition of the test set: 14 of the 20 held-out cases are seen-family variants, while only 6 come from previously unseen families. For seen-family variants, training experiences from the same family are retrieved under similar GAS states and can carry family-specific information about the scoring profile. RQ4 reports a 14.7% ODS gain on seen-family variants versus only 8.2% on unseen families; please report the seen/unseen split with confidence intervals and treat the unseen-family result as the primary evidence for generalization, since it is the only part of the transfer claim not confounded by profile-specific experience.","section":"§VI-A and §VI-F"}],"minor_comments":[{"comment":"The manuscript mixes notations such as '§4.1' in the figure caption and '§IV-A' in the text; please unify the cross-reference style throughout.","section":"Cross-references"},{"comment":"In the last row, '7.400' appears with inconsistent spacing; also, TTR is only meaningful as a comparison metric when response lengths are commensurable, so please clarify whether the shorter TTR in the ablated settings reflects truncated failures rather than efficiency.","section":"Table III"},{"comment":"The figure legend uses 'CC+DeepSeek' and 'CC+Opus 4.7' while Table II uses 'CC-DeepSeek V4 Pro' and 'CC-Opus 4.7'; please align the names across the paper.","section":"Figure 4"},{"comment":"The code repository is given as an anonymous link only; please add a formal availability statement with version, license, and expected runtime/configuration requirements, and state whether the benchmark ranges will be released.","section":"Reference [30] and artifact statement"}],"recommendation":"major_revision","confidential_remarks":"The central issue is whether the Experience Database provides a fair transferable memory or a privileged view of the scoring function. This is addressable within the manuscript's scope: the authors could rerun the key comparison with experiences built from independent evaluation profiles, or at minimum report the unseen-family transfer results as the main evidence and temper the headline claim. If the authors can show that the 9.5% gain persists without profile-derived experience labels, the paper would be a solid systems contribution; without that, the current evaluation overstates the generality of the result. I also recommend asking for statistical reporting and benchmark release as part of the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a real systems contribution: STAIR combines graph-based incident state, stage-specialized agents, and effect-validated experience retrieval into one working loop, evaluated on 100 Docker cyber ranges with a held-out test split. The architecture is clean, the ablations are cumulative and coherent, and the train/test discipline is better than most work in this area. Second, the headline 9.5% improvement over the best LLM agent baseline is not robust to the paper's own ablation: removing only the Experience Database drops ODS from 0.9409 to 0.8250, below CC-DeepSeek V4 Pro at 0.8589. So the entire headline advantage is carried by component E.\n\nThat is not automatically a flaw. An experience database built from validated past episodes is a legitimate design. But here the effect labels in Eq. (4) are generated by the Execution Harness using the same incident-specific evaluation profiles that later compute ODS. For held-out variants of training families, the retrieved records therefore encode what the hidden scoring profile rewards. Baselines get no equivalent memory. The paper says the profile is not exposed to the planner as text, which is true, but it is exposed indirectly through the experience labels. The stress-test note is mostly right: this gives STAIR a structural information advantage, not just better state tracking or planning.\n\nWhat the paper does well: the held-out split, the complexity grouping, the component attribution, and the concrete worked example in Fig. 5 are all credible. The authors are also honest in the Discussion about limitations. The main gaps are the lack of error bars or significance tests on a 30-range test set, the absence of a static-playbook or human baseline despite the paper motivating against playbooks, and the benchmark not being released, which makes replication hard.\n\nNet assessment: the framework is worth taking seriously, but the evaluation does not yet cleanly separate \"better planning\" from \"better access to the scoring function.\" Send it to peer review, but reviewers should push for either a fairer experience baseline, independent validation signals (e.g., human-annotated effect labels), or a release of the benchmark. If the E advantage survives that, the 9.5% claim becomes solid.\n\nFor the reading group: yes, bring it, because the methodology discussion is exactly what we need to have.","headline":"Solid systems integration with a genuinely held-out benchmark, but the headline 9.5% gain is carried entirely by the experience database, whose effect labels come from the same evaluation profiles used to compute the final score; that structural advantage needs a harder look before the number is taken at face value.","tokens_in":16765,"tokens_out":2960,"would_cite":true,"duration_ms":32241,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A graph-state agent framework routes incident response and lifts defense scores to 0.94.","keywords":["incident response","agentic planning","Graph-as-State","LLM agents","experience reuse","cyber range","recovery stage","SOAR"],"falsifier":"Re-run the 30 held-out ranges twice: once with the experience database built from effect labels generated by one rubric, and once with labels generated by a second, independently designed rubric, keeping the final score computed from the original hidden profiles. If the ODS advantage over the no-experience variant largely disappears or changes sign across the two label rubrics, the reported gain depends on information leaked from the scoring profiles rather than on transferable response knowledge.","tokens_in":15779,"feed_emoji":"🛡️","tokens_out":4186,"duration_ms":34738,"temperature":0.7,"pith_summary":"This paper tries to show that automated incident response gets substantially better when planning is anchored in a live, evolving picture of the incident rather than in open-ended text or tool traces. It presents STAIR, a framework that keeps the incident as a graph, decides which of six recovery stages to advance, hands planning to a stage-specialized agent, and executes actions through a harness that records their real effects. On 100 Docker-based cyber ranges, STAIR reports a normalized overall defense score of 0.94, against 0.86 for the strongest baseline, a relative improvement of 9.5%. If this holds, it would mean long-horizon response failures that plague LLM agents, such as state loss, stage mixing, and weak experience reuse, are addressable by structural workflow design rather than by a bigger model.","feed_headline":"Graph-state agents lift incident response defense score to 0.94","feed_subtitle":"STAIR routes each step through stage-specialized agents and validated experience, beating the strongest baseline by 9.5%.","key_machinery":"The load-bearing mechanism is the coupling of three objects: Graph-as-State (GAS), the Stage-Specialized Agent Planner, and an effect-validated Experience Database. GAS is an attributed graph holding machines, artifacts, defender nodes, attack edges, and response edges, so the incident's past and present live in one updatable structure rather than in a prompt or a trace. The Stage Router turns that graph into a dispatch signal $(s_t, \\rho_t)$, selecting a stage and recovery focus, and the corresponding agent plans with stage-scoped tools and memory. The Execution Harness closes the loop: it runs the action in the live environment, writes feedback into GAS, and labels the action effective, high-risk, or ineffective using an incident-specific profile, which is exactly the signal that builds reusable experience for later states.","core_discovery":"STAIR's central claim is that the decision substrate matters more than model power: response planning should be a stateful, stage-aware, experience-supported loop. The framework represents the incident at each step as Graph-as-State (GAS), a graph whose nodes are machines, artifacts, and defenders and whose edges are attack and response behaviors, updated continuously from alert data and execution feedback. A Stage Router reads GAS and picks one of six recovery stages, containment, assessment, preservation, eviction, hardening, restoration, and dispatches a stage-specialized agent whose tools, memory, and experience skill are scoped to that stage. After each action, the Execution Harness validates the effect against an incident-specific profile and writes the tuple of pre-action state, action, effect label, and feedback into an Experience Database, so future planning retrieves proven actions and warnings. In evaluation on 100 Docker-based ranges (70 training, 30 held-out), STAIR reaches an overall defense score of 0.9409 versus 0.8589 for the best baseline, and the ablations show each component contributes: removing experience drops ODS to 0.8250, removing graph state further to 0.6889, and removing stage planning to 0.6031.","pith_inferences":["An implicit but testable consequence is that the framework's value depends on the independence of the effect-labeling profiles from the final scoring profiles; if they share signal, part of the reported gain is by construction.","The authors' prototype assumes a fixed six-stage recovery lifecycle and a closed set of 33 response capabilities; extending it to adaptive stage definitions or ad-hoc tools would test how much of the benefit is structural versus task-specific.","The same graph-state/stage-dispatch/experience-validation loop could generalize to other long-horizon operational workflows, such as vulnerability remediation or cloud outage recovery, where state, stage, and validated actions are the natural units."],"forward_implications":["If the central claim is right, incident-response agents should be organized as stateful workflows rather than raw tool users; open-ended agents will keep losing track of attack paths in long runs.","A persistent graph state plus stage-scoped tools and memory should keep recovery actions aligned with the current objective, avoiding the stage-objective mixing seen in baselines.","Effect-validated experience reuse should transfer across incident variants, including previously unseen families, without replaying exact trajectories.","The component ablations imply that the 9.5% gain is a joint effect of graph state, stage planning, and experience, not a single trick, so removing any one component measurably degrades response."],"supporting_citations":[{"why":"Supplies the lightweight LLM planning approach that the LLM Planner baseline is built on.","marker":"[11]"},{"why":"Provides the reward-driven SecLoop baseline that STAIR compares against.","marker":"[26]"},{"why":"Defines the Claude Code agentic baseline used to test open-ended tool interaction.","marker":"[43]"},{"why":"Grounds the benchmark in Docker-based executable cyber ranges.","marker":"[45]"},{"why":"Provides the containerlab infrastructure used to build the cyber-range topologies.","marker":"[46]"},{"why":"Supplies the reasoning-and-acting interaction pattern that the agentic baselines build on.","marker":"[22]"},{"why":"Motivates the stage-specialized multi-agent design through LLM multi-agent workflow literature.","marker":"[37]"}],"fun_headline_variants":["STAIR's graph-state agents lift incident defense to 0.94","Stateful, stage-aware agents beat static playbooks by 9.5%","Graph-as-state planning gives incident response a 9.5% edge","STAIR: experience-reusing agents route incident response to 0.94"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the effect labels used to build the experience database are an independent measure of response quality and not just the same evaluation profiles used to compute the final score; if they are the same signal, the system is partly being scored on what it was trained to see.","fun_headline_variants_meta":{"raw":{"variants":["STAIR's graph-state agents lift incident defense to 0.94","Stateful, stage-aware agents beat static playbooks by 9.5%","Graph-as-state planning gives incident response a 9.5% edge","STAIR: experience-reusing agents route incident response to 0.94"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1428,"prompt_tokens":982,"completion_tokens":446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":598,"tokens_out":446,"duration_ms":4289,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:42:00.811030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 30 held-out ranges twice: once with the experience database built from effect labels generated by one rubric, and once with labels generated by a second, independently designed rubric, keeping the final score computed from the original hidden profiles. If the ODS advantage over the no-experience variant largely disappears or changes sign across the two label rubrics, the reported gain depends on information leaked from the scoring profiles rather than on transferable response knowledge.","supporting_citations":[{"cited_title":"Advancing llm-based security automation with customized group relative policy optimization for zero-touch networks,","cited_arxiv_id":null,"evidence_quote":"Provides the reward-driven SecLoop baseline that STAIR compares against."},{"cited_title":"Claude code docs,","cited_arxiv_id":null,"evidence_quote":"Defines the Claude Code agentic baseline used to test open-ended tool interaction."},{"cited_title":"What is docker?","cited_arxiv_id":null,"evidence_quote":"Grounds the benchmark in Docker-based executable cyber ranges."},{"cited_title":"Containerlab,","cited_arxiv_id":null,"evidence_quote":"Provides the containerlab infrastructure used to build the cyber-range topologies."},{"cited_title":"Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,","cited_arxiv_id":null,"evidence_quote":"Motivates the stage-specialized multi-agent design through LLM multi-agent workflow literature."}],"review_version":1}