{"id":"b5d04495-69c5-43b7-80d8-1e04bf4a248e","arxiv_id":"2608.02670","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Under the strictest enforced policy, all 12 coding-agent bundles lost success and gained cost, with success losses from 7.1 to 18.3 percentage points and cost inflation from 16.0% to 167.3%.","lead":"This paper tests 12 coding agents on 89 command-line tasks under three levels of enforced security policy, from fully open to locked-down enterprise-style restrictions. It finds that locked-down policies lower success and raise cost for every model, but by very different amounts, so the best model depends on the policy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Solvability witnesses are the load-bearing point: if the study-authored witnesses for protein-assembly, dna-assembly, or configure-git-webserver merely satisfy verifier checks rather than performing the task, the measured non-uniform policy penalties are distorted.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the validity of the study-authored solvability witnesses. My reading of the full text reinforces that this is the point on which the strongest claim depends. The paper is transparent and carefully engineered: native Linux enforcement, pre-flight probes, a released plugin, and an honest limitations section all speak for the quality of the measurement apparatus. The universal direction of the effect (success falls and cost rises under NIST-derived high) is likely robust, but the specific non-uniform magnitudes and the policy-dependence conclusion are driven by the affected-task subset, whose only evidence of solvability is the author-written witness pool. Because Appendix C independently documents that a verifier in this same benchmark can be fooled by a memorized constant, the witness pool cannot be taken on faith. An independent audit of the three most contrived witnesses is necessary before the headline success-loss and cost-inflation numbers can be treated as policy effects rather than benchmark artifacts. This does not change the reader's verdict: CONDITIONAL remains the right disposition, with the conditions being witness validation and confidence intervals on the headline deltas.","tokens_in":27091,"tokens_out":7769,"duration_ms":76606,"concrete_test":"Independently audit the 32 witnesses by replaying each from a clean NIST-derived-high sandbox and then stress the three flagged shortcuts: (i) for protein-assembly, remove the hardcoded SNAP/FLAG constants and require them to be computed from the retrieved PDB structures; (ii) for dna-assembly, replace the canonical-offset slicing with a dynamic primer-generation step and confirm the verifier's own primer3 accepts the products; (iii) for configure-git-webserver, replace the custom paramiko server with a real sshd (or verify that the custom server correctly handles arbitrary git commands, not only canned requests).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that hardening degrades all bundles non-uniformly and makes model choice policy-dependent rests on separating model failures from policy-foreclosed tasks. That separation is carried by the 32 study-authored solvability witnesses in Appendix B, Table 7. This is genuinely load-bearing because on the 50 unaffected tasks the hardening effect is small: success moves from 73.8% to 71.8%, and the cost multiplier is 1.14x under NIST-derived high. The large, discriminating penalties therefore come almost entirely from the 32 affected tasks. Several of those witnesses are visibly contrived: protein-assembly hardcodes the expected SNAP/FLAG values, dna-assembly builds primers by coreutils slicing at canonical offsets, and configure-git-webserver authenticates via a custom paramiko SSH server with no corresponding UNIX account. If these are verifier-satisfying artifacts rather than genuine task solutions, then the affected tasks are effectively policy-foreclosed or benchmark-corrupted, and the claimed policy penalties conflate forced failures with model adaptation. Appendix C shows the same failure mode is real in this benchmark: count-dataset-tokens passes by submitting an expected constant without performing the task. No independent validation of the witnesses is provided, so the witness pool is the least secure assumption underpinning the paper's strongest conclusion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Boundary-Bench, a hardening plugin that layers operating-system-enforced security policies onto Terminal-Bench 2.1 and evaluates 12 frozen model-harness bundles under three nested policy levels: control, non-root, and NIST-derived high. The central empirical claim is that hardening is non-uniform: under the strictest policy every bundle's point estimate of success decreases and mean cost per task increases, with success losses up to 18.3 percentage points and cost inflation up to 167.3%, so that model choice is policy-dependent. The paper also decomposes policy-induced failures into timeouts, wrong solutions, and early stops; attributes cost inflation to workaround construction; measures blocked actions at the enforcement boundary; and audits task solvability under the strictest policy, including authoring 32 policy-compliant solvability witnesses and additively repairing five over-specified verifiers. Full per-bundle tables, infrastructure provenance, provider-side intervention documentation, and code are released.","tokens_in":27367,"tokens_out":12796,"duration_ms":106281,"significance":"If the central result holds, the paper makes a valuable contribution: it introduces the security policy level as an explicit independent variable for coding-agent evaluation, a dimension missing from standard leaderboards. The strengths are concrete: native Linux enforcement with pre-flight probes, transparent verifier-audit methodology, explicit documentation of provider-side safety interventions and of a benchmark-integrity artifact (Appendix C), and full per-bundle result tables. The claim that reference-solution compatibility anticipates where a policy will cost success and money is a falsifiable and practically useful prediction. The main risk is the validity of the 32 study-authored solvability witnesses, which carry the separation between model failures and policy-foreclosed tasks; because the paper itself shows the same verifier-award-without-computation failure mode in count-dataset-tokens, this risk is concrete rather than hypothetical.","major_comments":[{"comment":"The 32 study-authored solvability witnesses in Table 7 are load-bearing: the paper's own decomposition shows that the large policy penalties are concentrated on the 32 affected tasks (83.5% of additional failures; cost multiplier 2.59x versus 1.14x on unaffected tasks), so the validity of these witnesses determines whether the measured success losses are model failures or policy-foreclosed tasks. Several entries are suspect: protein-assembly hardcodes the SNAP/FLAG values; dna-assembly and dna-insert build primers by pure-coreutils slicing at canonical offsets with no primer-design computation; and configure-git-webserver passes via a custom paramiko SSH server that authenticates a login with no corresponding UNIX account. Appendix C demonstrates that the same verifier-award-without-computation failure mode is real in this benchmark (count-dataset-tokens), so the burden is on the authors to show that each witness performs the substantive task work rather than merely reaching the official passing state. Please provide verifier-level evidence or an independent audit for each of the 32 witnesses, and rerun the affected/unaffected analysis after removing any witness that cannot be defended.","section":"Appendix B, Table 7; Appendix C"},{"comment":"The central claim that under NIST-derived high every bundle's point estimate worsens on both axes is universal, but the supporting numbers are three-trial point estimates with no intervals on costs. In Table 13, for example, MiniMax M3 moves from $18.98 to $19.87 and GPT-5.6 Luna from $12.54 to $13.63 on the 82-task pool, both within plausible run-to-run variability. The paper disclaims ranking and notes the absence of intervals, but the universal direction claim is the paper's headline; please add bootstrap or other confidence intervals for the per-bundle deltas, and a sign test or a sensitivity analysis that excludes the smallest changes, to show that the pattern is not driven by noise.","section":"Results: Restriction Sensitivity; Tables 12-13"},{"comment":"The predictive claim that reference-solution compatibility anticipates where a policy will cost both money and success is partly operationalized by the authors' own witness-authoring effort: a task is affected exactly when the shipped reference solution fails and the authors can write a policy-compliant substitute. If the witness pool is contaminated, the affected/unaffected distinction is not an independent task property and the prediction is circular. Please report how the task-level verifier checks the substantive product for each affected task, and ideally show that the same predictive split holds when the affected set is defined by a property independent of the witness-authoring process, for example whether any canonical toolchain step is blocked by a specific policy axis.","section":"Cost Inflation under Policy; Figure 4"}],"minor_comments":[{"comment":"The trial protocol says invalid attempts are rerun until the cell holds three valid trials, so it is unclear how Table 3 can report a residual 0.4% of runs per condition as provider- and verifier-side errors excluded; please reconcile.","section":"Experimental Setup; Table 3"},{"comment":"The sentence 'removing a roughly common five to seven points from every measured penalty' is not supported by Tables 12 and 13; the per-bundle differences between the 89-task and 82-task success drops range from about 4.5 to 7.5 points.","section":"Benchmark Artifacts under Policy"},{"comment":"The dotted line labeled 'robustness Pareto frontier' is not defined in the caption or the surrounding text; please define the dominance relation used.","section":"Figure 2"},{"comment":"The caption does not state the policy level; make explicit that the frontier is for NIST-derived high on the 82-task pool.","section":"Table 14"},{"comment":"The 97% workaround-construction attribution is based on the authors' reading of 200 trajectories; please state whether this coding was done with a protocol and whether any inter-rater reliability check was performed.","section":"Results: Cost Inflation under Policy"},{"comment":"The relationship between the 6,194 verified-complete trials and the 9,612 total trials is only given in the appendix; consider stating in the main text that blocked-action counts are lower bounds due to trace truncation.","section":"Appendix G"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the witness pool is the weakest link in the paper. If the authors can validate the 32 witnesses, or exclude the indefensible ones and rerun the affected/unaffected analysis, the central pattern may still hold, but the current manuscript does not yet establish it. I recommend major revision, with the witness validation as the primary condition."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: this paper introduces policy severity as an independent variable in coding-agent evaluation, with a graded NIST-mapped ladder and a solvability audit. That is new, and it matters. The central direction — hardening degrades all twelve bundles non-uniformly on both success and cost — is well supported by the disclosed tables, and the pattern is not something a single fixed-configuration study like CaMeL or Progent can see.\n\nThe paper is also unusually honest. It ships code, publishes full per-bundle tables in the appendix, documents provider-side safety interventions and their incidence, and explicitly flags count-dataset-tokens as a benchmark-integrity artifact where the verifier awarded success for submitting an expected constant. The affected/unaffected task split is a good analytic device, and the finding that reference-solution compatibility predicts where policy costs both money and success is both useful and credibly established.\n\nThe soft spots are real but concentrated. First, the headline deltas (18.3pp success loss, 167.3% cost inflation) are three-trial point estimates with no confidence intervals. The authors themselves say the spread is descriptive, not a ranking, but then the abstract repeats the point estimates without that caveat. Second, the solvability witnesses are load-bearing. On the 50 unaffected tasks hardening costs only about 2pp success and a 1.14x cost multiplier; the large, discriminating penalties come almost entirely from the 32 affected tasks. Several of those witnesses look contrived: protein-assembly hardcodes SNAP/FLAG values, dna-assembly slices primers with coreutils at canonical offsets, and configure-git-webserver authenticates through a custom SSH server with no UNIX account. The paper demonstrates in count-dataset-tokens that this benchmark rewards exactly this kind of verifier-satisfying artifact, yet it does not independently validate its own witnesses against that failure mode. If some witnesses are artifacts rather than genuine solutions, the measured policy penalties conflate forced failures with model adaptation.\n\nThese concerns do not sink the paper. The direction of the effect is robust, the audit is transparent, and the limitations section is candid about the single benchmark and three-trial design. But the magnitudes should be presented as illustrative until either more trials or independent witness validation is done.\n\nWho this is for: researchers working on coding-agent evaluation, model selection under constrained environments, and the operational cost of security policy. Yes, it deserves a serious referee — the contribution is solid and the flaws are fixable. My recommendation: send it to review, and push hard for confidence intervals on the headline deltas and a sensitivity analysis that removes or honestly re-classifies the most contrived witnesses.","headline":"A genuinely new benchmark contribution with a transparent audit, but the headline magnitudes rest on three-trial point estimates and author-written solvability witnesses that need independent validation before the numbers are taken as precise.","tokens_in":27875,"tokens_out":1733,"would_cite":true,"duration_ms":17930,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under an enforced enterprise-style policy, every tested coding agent lands on a worse success–cost point, and the best choice depends on the policy.","keywords":["coding agents","security policy","hardened environments","success-cost trade-off","Terminal-Bench","solvability audit","NIST SP 800-53","Pareto frontier"],"falsifier":"Run the 32 witnessed tasks and the seven blocked-by-design tasks from a clean Terminal-Bench 2.1 image under NIST-derived high with the repaired verifiers, without access to the paper's witness trajectories, and have independent implementers attempt each task; then inspect whether each witness passes the verifier by performing the task's intended computation rather than by matching a hardcoded constant or a non-standard route (protein-assembly's SNAP/FLAG values, dna-assembly's coreutils slicing at canonical offsets, or configure-git-webserver's custom SSH server with no UNIX account). If a substantial share of the blocked-by-design tasks turn out to be solvable, or if the witnesses do not reproduce on fresh sandboxes, the measured policy penalties and the model-versus-policy attribution fail.","tokens_in":26903,"feed_emoji":"🔒","tokens_out":8622,"duration_ms":64183,"temperature":0.7,"pith_summary":"This paper's central claim is that an enforced security policy is a first-order evaluation axis for coding agents: when agents must run under scoped credentials, restricted egress, and read-only filesystems, their success rates and per-task costs change in ways that permissive-sandbox leaderboards cannot predict. Evaluating twelve frozen model–harness bundles on Terminal-Bench 2.1 under three nested policies, the authors find that under the strictest NIST-derived level every bundle loses success (up to 18.3 percentage points) and pays more per task (up to 167.3% cost inflation), but bundles pay in different currencies: the model that best preserves success absorbs the largest cost increase, while the largest success loss comes with a comparatively modest cost rise. The paper introduces Boundary-Bench, a hardening plugin that enforces policy through native operating-system mechanisms, and pairs it with a solvability audit that separates model failures from tasks the policy forecloses. If the central claim is right, model selection has to be done under the deployment's own policy, and benchmark reporting should treat the policy level as a variable rather than a fixed permissive default.","feed_headline":"Hardening coding agents costs up to 18 points of success, 167% in cost","feed_subtitle":"Every tested bundle loses success and gains cost under strict policy, so model choice becomes policy-dependent.","key_machinery":"The load-bearing mechanism is Boundary-Bench, a hardening plugin that layers a configurable, operating-system-enforced policy onto Terminal-Bench 2.1 and measures each bundle's success–cost shift relative to its own control. Policies are built from three axes—network egress (N), filesystem scope (F), and privilege (P)—and the paper evaluates three nested levels: control (root, open egress, writable filesystem), non-root (privilege drop only), and NIST-derived high (a fixed 205-domain egress allowlist, read-only system trees outside a small writable set, and a no-new-privileges lockdown), each enforced by native Linux mechanisms such as a default-deny proxy, read-only bind mounts, and an unprivileged user with an empty capability set. The argument is carried by the per-bundle change metrics $\\Delta SR_m(\\ell)$ and $\\Delta C_m(\\ell)$ of Eq. (1) plotted as a success–cost Pareto frontier, plus the solvability audit that replays reference solutions under the strict policy, authors policy-compliant witnesses where needed, and additively repairs over-specified verifiers so that valid non-root solutions are credited. This audit is what lets the paper attribute failures to models rather than to tasks the policy forecloses.","core_discovery":"The paper's own claim is that hardening does not uniformly degrade coding agents: under NIST-derived high, all twelve bundles shift to strictly worse success–cost operating points, but by amounts and in currencies that differ across models, so the ranking of models is policy-dependent. Success losses reach 18.3 percentage points and cost inflation reaches 167.3 percent, and the two axes disagree—the bundle that gives up the least success (Grok 4.5, −7.1 points) pays the largest cost inflation (+167.3 percent), while the bundle that loses the most success (Claude Sonnet 5, −18.3 points) does so at a modest +21.4 percent cost rise. The additional failures are timeouts and completed-but-wrong solutions rather than early stops, and on the two deep-sampled tasks the median per-run cost roughly triples even where the success rate holds. To ground the comparison, the paper establishes solvability witnesses for 82 of the 89 tasks under the strict policy (50 via the unchanged reference solution, 32 via authored policy-compliant witnesses), identifies seven tasks blocked by design, and repairs five over-specified verifiers after showing their original checks rejected valid non-root solutions. The paper explicitly treats all numbers as describing a spread across bundles, not a ranking.","pith_inferences":["The 97% attribution of added cost to workaround construction suggests a concrete design target: agents that detect a denial early and plan a policy-compliant route in one pass could recover most of the efficiency loss, which is a testable engineering claim beyond what the paper measures.","Because the seven blocked-by-design tasks are tied to the specific 205-domain allowlist and read-only mount set, the measured penalties would shift if enterprise policies admit more package mirrors or writable system paths; the paper's policy axis makes such sensitivity explicit rather than hidden.","A policy-conditional leaderboard would reorder models relative to today's unconstrained rankings, and a natural extension is to check whether the policy-conditional ordering is stable across benchmark families or task pools beyond Terminal-Bench.","The cost inflation that appears even on tasks where success holds (median per-run cost roughly tripling on both deep-sampled tasks) implies that organizations should price policy not just in lost tasks but in inference spend on successful work."],"forward_implications":["Unconstrained leaderboards misrepresent deployed performance: a model that looks best in a permissive sandbox can be the wrong choice under a specific enterprise policy, so model selection should be repeated under the deployment's own policy.","Deployment budgets must price hardening: because failed runs under policy are timeouts and wrong solutions rather than early stops, wall-clock and token budgets need headroom, and matched passing runs already cost 13% more wall-clock time, 14% more tool calls, and 26% more tokens.","Reference-solution compatibility predicts where policy will cost money and success before any agent runs, giving benchmark authors a cheap way to flag tasks that will inflate measured penalties.","Benchmarks should report a policy axis as standard practice, since the paper shows that a common restricted environment restores discriminative headroom that permissive leaderboards have lost."],"supporting_citations":[{"why":"Supplies Terminal-Bench 2.1, the benchmark whose 89 tasks and per-task environments form the evaluation substrate.","marker":"Merrill et al. 2026"},{"why":"Supplies NIST SP 800-53 control families that motivate the three hardening axes and the 'NIST-derived high' level.","marker":"Joint Task Force Interagency Working Group 2020"},{"why":"Supplies the success–cost Pareto frontier framing the paper uses to summarize each bundle's operating point under each policy.","marker":"Kapoor et al. 2025"},{"why":"Provides a single fixed-configuration policy-enforcement result whose success drop and token overhead this paper extends to a graded policy ladder.","marker":"Debenedetti et al. 2025"},{"why":"Provides the contrast result that tool-call privilege policies can largely maintain utility, against which the paper's non-uniform degradation is measured.","marker":"Shi et al. 2025"},{"why":"Quantifies the 'verifier tax' of a block-and-revise safety gate, the prior work most directly measuring enforcement's token cost that this study generalizes.","marker":"Sah et al. 2026"},{"why":"Documents verifier over-specification in code-generation benchmarks, the failure mode the paper audits and repairs before comparing policies.","marker":"Sharifloo et al. 2025"},{"why":"Documents mid-trajectory tool failures and bans destabilizing agents, which motivates treating policy denials as a structured runtime perturbation.","marker":"Xiong et al. 2025"}],"fun_headline_variants":["Policy locks cost agents up to 18-pt success, 167% cost — model choice matters","Coding agents under policy: success down 18pts, cost up 167%, rankings shift","Hardening coding agents: up to 18-pt success loss, 167% cost jump — uneven","Stricter policy, costlier agents: max 18-pt success drop, 167% cost, not uniform","Policy-restricted coding agents: 18-pt success loss, 167% cost spike, model-dependent"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 32 study-authored solvability witnesses are genuine proofs that their tasks remain solvable under NIST-derived high, and that a task with no witness really is foreclosed by the policy; if those witnesses are contrived—for example, hardcoded expected outputs, canonical-offset file slicing, or custom servers that bypass the task's intended architecture—then the paper's separation of model failures from policy-foreclosed tasks, and every reported success penalty, is distorted.","fun_headline_variants_meta":{"raw":{"variants":["Policy locks cost agents up to 18-pt success, 167% cost — model choice matters","Coding agents under policy: success down 18pts, cost up 167%, rankings shift","Hardening coding agents: up to 18-pt success loss, 167% cost jump — uneven","Stricter policy, costlier agents: max 18-pt success drop, 167% cost, not uniform","Policy-restricted coding agents: 18-pt success loss, 167% cost spike, model-dependent"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000487,"raw_usage":{"total_tokens":2446,"prompt_tokens":1040,"completion_tokens":1406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":1275}},"tokens_in":656,"tokens_out":1406,"duration_ms":9844,"temperature":1.0,"reasoning_tokens":1275,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:09:16.260154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 32 witnessed tasks and the seven blocked-by-design tasks from a clean Terminal-Bench 2.1 image under NIST-derived high with the repaired verifiers, without access to the paper's witness trajectories, and have independent implementers attempt each task; then inspect whether each witness passes the verifier by performing the task's intended computation rather than by matching a hardcoded constant or a non-standard route (protein-assembly's SNAP/FLAG values, dna-assembly's coreutils slicing at canonical offsets, or configure-git-webserver's custom SSH server with no UNIX account). If a substantial share of the blocked-by-design tasks turn out to be solvable, or if the witnesses do not reproduce on fresh sandboxes, the measured policy penalties and the model-versus-policy attribution fail.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies NIST SP 800-53 control families that motivate the three hardening axes and the 'NIST-derived high' level."},{"cited_title":"S.; Nadgir, N.; and Narayanan, A","cited_arxiv_id":null,"evidence_quote":"Supplies the success–cost Pareto frontier framing the paper uses to summarize each bundle's operating point under each policy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a single fixed-configuration policy-enforcement result whose success drop and token overhead this paper extends to a graded policy ladder."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the contrast result that tool-call privilege policies can largely maintain utility, against which the paper's non-uniform degradation is measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Quantifies the 'verifier tax' of a block-and-revise safety gate, the prior work most directly measuring enforcement's token cost that this study generalizes."}],"review_version":2}