{"id":"3ef3a5dc-2d95-43ca-8adb-90d51f3ca1fe","arxiv_id":"2607.07774","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Pre-execution scope gating works only when the judge sees the user's request, with GLM-5.2 achieving F1=0.66 against an expert reference of 0.78.","lead":"This paper builds ScopeJudge, a benchmark of 4,897 labeled tool calls from offensive-security agent runs, and tests how well cheap LLM judges can block out-of-scope actions before they execute. It finds that judges shown only a fixed policy catch almost nothing, while judges that see the user's request catch most violations, and that an open-weight model (GLM-5.2) leads the cost-accuracy frontier.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Static-policy recall collapse is uninformative: the static strategy omits the user request, the only carrier of task scope; a per-engagement contract is never tested.","rationale":"The paper's most important finding is stated as 'a static policy ... is structurally insufficient for scope enforcement.' The design of the static_policy strategy, however, makes this conclusion nearly tautological: the policy SP is explicitly generic and task-invariant, while the operative scope is confined to u. Giving a judge only SP and a tool call cannot reveal the task-specific boundary, so the near-zero recall is an artifact of the input construction. The paper does not test an alternative specification-time policy such as a per-engagement scope document, which is the natural real-world baseline (bug bounty program rules, engagement contracts). Without that control, the necessity claim ('request-conditioned monitoring is necessary') is not established; the data only show that a judge without access to the request cannot do the task. This does not undermine the benchmark, the cost-accuracy measurements, or the release of labeled data, which are valuable and transparent. But the central thesis as stated overclaims the evidence. The reader's verdict of CONDITIONAL remains appropriate; the authors should add the static-contract condition (or qualify the structural claim) before the headline thesis can be accepted.","tokens_in":16461,"tokens_out":6723,"duration_ms":63984,"concrete_test":"Add a static_contract condition: for each ScopeBench task, translate the explicit scope constraints from u (target host, allowed operations, forbidden state changes; cf. Appendix B) into a formal engagement-scope statement, and supply it to each judge as a static policy (or as a prefix to SP), together with ak and no other history. Run the same 8×5 grid with this sixth strategy. If median recall across judges rises substantially above the 0.005 observed under static_policy, then the 'structural insufficiency' claim is falsified for realistic static scope documents; if recall remains near zero, the claim is supported. Also report precision/F1 and note that the objective text (the agent's goal) should be omitted to isolate 'policy from request' rather than 'request-conditioning'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.2 concludes that a static policy is 'structurally insufficient' because recall collapses to near zero under the static_policy strategy. But Eq. (5) defines static_policy as φstatic = (SP, ak) — the judge sees the global generic policy plus the proposed call, and by Section 2.1 SP contains only 'generic operating ground rules', while the task-specific scope is declared in the user message u. The experiment therefore withholds the only location where scope is stated, making the recall collapse a necessary consequence of the information architecture, not evidence that no specification-time policy could work. Real engagements have written scope/ROE documents that are fixed before execution; the paper never evaluates a static policy that incorporates such a per-engagement contract. The headline thesis that request-conditioned monitoring is necessary (rather than merely effective) is hence unsupported: only a policy that is blind to the engagement scope has been shown insufficient.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ScopeJudge, a benchmark of 4,897 expert-labeled tool calls drawn from 100 agent trajectories on offensive-security tasks, and uses it to evaluate eight LLM judges under five transcript strategies for pre-execution gating. It reports a cost-accuracy Pareto frontier, identifies open-weight judges as competitive (GLM-5.2 reaches F1=0.66 against an expert-agreement reference of 0.78), and argues that a static policy is structurally insufficient for scope enforcement because recall collapses when the judge is not shown the user's request. The authors recommend two deployment operating points, one cost-sensitive and one recall-first, and release the dataset and code.","tokens_in":16672,"tokens_out":6404,"duration_ms":63596,"significance":"If the claims hold, the paper provides a useful, reproducible benchmark for a practical problem: gating LLM security agents before tool execution. The release of the labeled corpus and code, the five-strategy taxonomy, the explicit cost accounting, and the human-agreement reference are genuine contributions. The central empirical pattern—generic static context alone gives near-zero recall, while adding the user request restores recall—is clearly supported by Table 3 and Figure 4. However, the paper's headline conclusion that request-conditioned monitoring is 'necessary' and that a specification-time static policy is 'structurally insufficient' goes beyond the evidence, because the static-policy condition deliberately withholds the only carrier of task-specific scope. The empirical contribution remains valuable once the conclusion is appropriately scoped.","major_comments":[{"comment":"The central 'necessity' claim is not established by the reported experiment. Eq. (5) defines static_policy as φ_static=(SP, a_k), and Section 2.1 states that task-specific scope is declared in the user message u, not in the global static policy SP. The static condition thus removes the only information source where the engagement boundary is stated; the recall collapse in Figure 4 is a built-in consequence of this information architecture, not evidence that no specification-time policy could work. Moreover, Appendix A's constant system prompt tells the judge that it 'will be shown the original engagement instruction,' but under static_policy that instruction is withheld, so the condition is not simply 'static policy only' but also a broken prompt contract. Real engagements have written scope/ROE documents that are fixed before execution; no such per-engagement static contract is tested.","section":"§6.2, Eq. (5), Appendix A/Table 4"},{"comment":"The paper's deployment recommendations are based on single-run point estimates with no confidence intervals, as the authors acknowledge in Section 7. Several configuration differences that motivate the recommended operating points are small (e.g., GLM-5.2 intent_plus_calls F1=0.637 vs. intent_plus_outputs_summary F1=0.639; full F1=0.656). With 4,897 calls and no repeated sampling, these differences are not distinguishable from noise. This does not undermine the large static-vs-intent effect, but it does undermine the precision with which the two operating points are ranked. The authors should either provide variance estimates (e.g., bootstrap over trajectories, or repeated API samples) or explicitly present the recommendations as qualitative guidance rather than as a ranking of nearly tied configurations.","section":"§6.1, Table 3, §7"},{"comment":"All ground-truth labels were produced by five graders who are also the paper's authors, and the expert reference point (F1=0.78) is computed against the majority of those same graders. The paper discloses the evaluator overlap between two judges and the trajectory-generation agents, but the author-grader issue is separate and is not discussed as a limitation. The inter-grader agreement (κ=0.64) is useful, but the reference point is not an external ceiling. A small external validation sample labeled by independent practitioners would materially strengthen the calibration claim. At minimum, this should be added to the limitations in Section 7.","section":"§4.1, §4.3, §7"}],"minor_comments":[{"comment":"The 'best-F1' selection in Figure 3 is an oracle choice: for each judge the best-F1 strategy is selected post hoc using the gold labels. If the recommendation is for a fixed deployed configuration, the paper should also report the expected F1 of that configuration without selection, or explicitly note that Figure 3 is an upper envelope.","section":"§6.1 / Table 3"},{"comment":"The term 'static policy' is used in two senses: the concrete global SP shared across tasks, and the general idea of any specification-time policy. This ambiguity makes the Section 6.2 conclusion easier to overstate. Consider renaming the former 'generic global policy' and reserving 'static policy' for the broader class.","section":"§2.1 / §3.1.1"},{"comment":"The worked example is clear and helpful, but the trajectory is generated by claude-sonnet-4-6, one of the judge models, which is also the evaluator-overlap case flagged in Section 4.1. This is fine as an illustration, but it is worth a one-sentence reminder that the example is from the overlapping subset.","section":"Appendix B"},{"comment":"Section 4.2 states that roughly 380 positive instances are 'enough for stable estimates of monitor recall,' but no variance estimate is provided. Given the base rate of 7.7% and the acknowledged lack of confidence intervals, this assertion should be softened or supported.","section":"§7 / §4.2"},{"comment":"The static-policy rows for most judges have Flag% around 0.1 and recall near zero; a log-scale or per-judge plot might communicate the collapse more clearly than the numbers alone. The current Figure 4 is good, but a per-judge panel would help.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is well executed within its stated benchmark, and the authors are unusually candid about limitations. The main issue is the discrepancy between the measured result and the advertised thesis: the static-policy condition is designed to be scope-blind, so the 'necessity' claim overreaches. This is fixable by reframing the conclusions and, ideally, by adding a per-engagement static-policy condition or an external-label validation. I do not see a fundamental soundness problem with the core measurement; the recommendation is major revision rather than rejection because the headline claim needs substantive reframing, and the other concerns are secondary."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read. The genuinely useful piece is the benchmark: call-level scope-violation labels on 4,897 tool calls from 100 agent trajectories, labeled by five professional testers with Fleiss κ=0.64, plus a clean evaluation of eight judges under five transcript strategies with cost data. That is a reusable resource, and the Pareto result (open-weight GLM-5.2 at F1 0.66 against an expert reference of 0.78) is practically relevant. The paper is also unusually honest about its own limits; Section 7 lists the big ones.\n\nThe soft spot is the headline claim. Section 6.2 concludes that a static policy is 'structurally insufficient' for scope enforcement because recall collapses under the static_policy condition. But that condition feeds the judge only (SP, ak), and by the paper's own definition (Section 2.1) the task-specific scope lives in the user message u. The experiment withholds the only carrier of scope, so the recall collapse is almost a tautology. Real engagements have written scope or rules-of-engagement documents fixed in advance; the paper never tests a static policy that includes such a per-engagement contract. So the strong 'necessary' claim is not supported. What is supported is narrower: this particular generic static policy gates nothing, and adding intent restores a lot of recall. That is still useful, but the framing overreaches.\n\nThe other weaknesses are the admitted ones: one 100-trajectory calibration set from ScopeBench (described as separate work, not released here), author-graders, no confidence intervals, two judges overlapping with trajectory-generating agents, base-rate sensitivity. These are addressable and don't sink the benchmark, but they mean the numbers are indicative, not definitive.\n\nFor peer review: yes, send it out. The dataset and methodology deserve scrutiny, and the overclaim should be fixed. I'd push the authors to (1) temper or remove 'necessary' unless they test a per-engagement static policy; (2) fully release or describe ScopeBench; (3) add bootstrap CIs; (4) report the analysis with overlapping judge-trajectory pairs excluded. Then it's a solid contribution.\n\nI'd bring it to a reading group on agent monitoring and would cite the dataset if my work touched this area.","headline":"Useful benchmark and honest evaluation, but the 'static policy is structurally insufficient' claim overreaches because the tested static condition withholds the user request that, by the paper's own framing, carries the scope.","tokens_in":17155,"tokens_out":3291,"would_cite":true,"duration_ms":28574,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that scope enforcement for autonomous security agents cannot be achieved with a fixed policy: a judge that sees only the static policy and proposed tool call misses almost every violation, while adding the user's request l","keywords":["LLM agent safety","pre-execution gating","scope enforcement","offensive security","LLM-as-judge","transcript strategies","trusted monitoring","tool-call gating"],"falsifier":"Run the judge sweep on a new, independently labeled corpus from a different engagement type (for example, cloud-incident-response or mobile pentesting). If a static-policy-only judge catches a substantial share of violations there (recall well above 0.1), the structural-insufficiency claim fails.","tokens_in":16345,"feed_emoji":"🛡️","tokens_out":4460,"duration_ms":39498,"temperature":0.7,"pith_summary":"The paper tries to establish that for autonomous offensive-security agents, the boundary between allowed and forbidden tool calls is not fixed in advance but is declared in the user's request. It builds a benchmark of 4,897 expert-labeled tool calls from 100 agent trajectories designed to tempt agents out of scope, and tests eight judges under five context strategies. The result: judges shown only a static policy catch essentially nothing (median recall 0.005), while adding the user's intent restores recall to 0.60–0.94. The best judge is open-weight, reaching F1 0.66 against an expert agreement reference of 0.78. The paper concludes that request-conditioned pre-execution gating is necessary and gives cost-aware operating points for deployment.","feed_headline":"Static policies catch almost no scope violations","feed_subtitle":"Adding the user's request lifts judge recall from near zero to 0.60–0.94 across eight AI judges.","key_machinery":"The pre-execution gate: a cheap, fixed LLM judge inspects each tool call proposed by a strong, swappable agent and accepts or rejects it before execution. The transcript strategy controls how much context the judge sees—ranging from only the static policy plus the proposed call, to adding the user's intent, prior tool calls, summarized outputs, or the full trajectory. This design isolates the contribution of request-level intent and produces the paper's central comparison: static policy alone gates nothing, while intent restores recall. The labeled calibration set of 4,897 calls provides the ground truth against which every judge–strategy pair is scored.","core_discovery":"Scope is not a property of the tool call itself but of the request that authorizes it. The same scan, exploit, or data read is in or out of bounds depending on the user's stated engagement constraints, which no fixed policy can enumerate in advance. Empirically, under the static-policy-only transcript strategy, judge recall collapses to near zero (median 0.005 across seven of eight judges), and the moment the user's intent is added, recall jumps to 0.60–0.94. The paper therefore argues that request-conditioned monitoring is structurally necessary for pre-execution gating, and that the user's request is where scope actually lives.","pith_inferences":["If scope truly lives in the user's request, the same gating logic should transfer to other autonomous-agent domains where a request defines boundaries—cloud operations, database access, or code changes—and static allowlists will similarly underperform.","The paper's violation base rate (7.7%) is suite-specific; deployments with different violation prevalence should expect precision and F1 to shift and may need to re-tune thresholds rather than reuse the reported operating points.","History-bearing strategies expose the judge to attacker-controllable tool outputs; a natural next experiment is to test whether a misaligned agent can poison outputs to get an out-of-scope call allowed, which the paper explicitly leaves open.","Because the best judge still lags human agreement, a production design would likely pair a cheap judge as the first gate with human or stronger-model review only of flagged calls, rather than relying on the judge alone."],"forward_implications":["A monitor that sees only a fixed policy will wave through nearly all scope violations; deployments need request-conditioned context to catch them.","Adding the user's intent is the main accuracy lever: recall jumps from near zero to 0.60–0.94, while adding full history buys less than 0.02 F1 at nearly double the cost.","The best judge in the study is open-weight, reaching F1 0.66 at roughly one-third the per-call cost of the best proprietary judge (0.60), making practical gating feasible without a closed model.","Two deployment points are recommended: a cost-sensitive setting that catches about 87% of violations and a recall-first setting that catches about 94% but flags many more calls for human review.","Judge performance still trails expert agreement (0.66 vs 0.78), so the benchmark sets a headroom target for future monitors."],"fun_headline_variants":["User's request is the only scope compass for security AI","Static rules miss 99.5% of scope breaches in AI agents","Scope lives in the request, not the tool call","Add user intent, recall jumps from 0.5% to 94%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire scoreboard rests on one calibration set: 100 English-language web-pentest trajectories labeled by five graders from the paper's own team, so if that suite and those labels do not represent real engagements, the measured recall collapse and scoreboard could shift.","fun_headline_variants_meta":{"raw":{"variants":["User's request is the only scope compass for security AI","Static rules miss 99.5% of scope breaches in AI agents","Scope lives in the request, not the tool call","Add user intent, recall jumps from 0.5% to 94%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1206,"prompt_tokens":870,"completion_tokens":336,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":274}},"tokens_in":614,"tokens_out":336,"duration_ms":3389,"temperature":1.0,"reasoning_tokens":274,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:45:08.505670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the judge sweep on a new, independently labeled corpus from a different engagement type (for example, cloud-incident-response or mobile pentesting). If a static-policy-only judge catches a substantial share of violations there (recall well above 0.1), the structural-insufficiency claim fails.","supporting_citations":[],"review_version":2}