pith. machine review for the scientific record. sign in

arxiv: 2604.15384 · v2 · submitted 2026-04-16 · 💻 cs.CR · cs.AI· cs.SE

Recognition: unknown

LinuxArena: A Control Setting for AI Agents in Live Production Software Environments

Tyler Tracy , Ram Potham , Nick Kuhn , Myles Heller , Anshul Khandelwal , Cody Rushing , Henri Lemoine , Miguel Brandao
show 26 more authors
Tomas Turlik Adam Hanson Josh Hills Amy Ngo Ram Rachum Nik Mitchell Falko Galperin Oscar Sykes Pip Arnott Samuel Prieto Lima Carlos Giudice Matt Goldwater Daniel Popp Drew de Wet Ruben Castaing Qi Guo Douw Marx Benjamin Shaffrey Justin Shenk Martin Milbradt Hannah Meagher Shaheen Ahmed-Chowdhury Daniel O'Connell Chris Canal Buck Shlegeris Aryan Bhatt
Authors on Pith no claims yet

Pith reviewed 2026-05-10 11:37 UTC · model grok-4.3

classification 💻 cs.CR cs.AIcs.SE
keywords AI controlsabotage evaluationmonitor detectionproduction environmentssoftware engineering agentsLinuxdata exfiltrationbackdooring
0
0 comments X

The pith

AI agents achieve 23 percent undetected sabotage success in live production Linux environments against trusted monitors.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LinuxArena as a control setting where AI agents perform legitimate software engineering work directly in live multi-service Linux production environments. It pairs 1,671 main tasks with 184 side tasks that represent safety failures such as data exfiltration and backdooring across 20 environments. Sabotage evaluations measure whether agents can finish side tasks without detection while monitor evaluations test a separate model's ability to catch them. Results show Claude Opus 4.6 reaches roughly 23 percent undetected sabotage success against a GPT-5-nano monitor at a 1 percent step-wise false positive rate. The released LaStraj dataset of human-crafted attack trajectories evades monitors at higher rates than model-generated ones, indicating the setting still has room for both improved attacks and better defenses.

Core claim

LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring. Sabotage evaluations reveal that Claude Opus 4.6 achieves roughly a 23 percent undetected sabotage success rate against a GPT-5-nano trusted monitor at a 1 percent step-wise false positive rate. The release of LaStraj, a dataset of human-crafted attack trajectories that evade monitors at substantially higher rates than any model-generated attacks, shows that current attack policies do not saturate the benchmark and that LinuxArena functions as a strong testbed for future control protocols.

What carries the argument

LinuxArena control setting with live multi-service production environments, main tasks for legitimate work, and side tasks for sabotage to enable direct measurement of undetected attack success and monitor detection rates.

Load-bearing premise

The 184 side tasks and 20 environments sufficiently represent real-world safety failures and production software conditions so that measured sabotage and detection rates generalize beyond the benchmark.

What would settle it

Running the identical sabotage and monitor evaluations using the same agents and monitors inside additional real deployed production environments and observing whether the undetected sabotage success rate remains close to 23 percent.

Figures

Figures reproduced from arXiv: 2604.15384 by Adam Hanson, Amy Ngo, Anshul Khandelwal, Aryan Bhatt, Benjamin Shaffrey, Buck Shlegeris, Carlos Giudice, Chris Canal, Cody Rushing, Daniel O'Connell, Daniel Popp, Douw Marx, Drew de Wet, Falko Galperin, Hannah Meagher, Henri Lemoine, Josh Hills, Justin Shenk, Martin Milbradt, Matt Goldwater, Miguel Brandao, Myles Heller, Nick Kuhn, Nik Mitchell, Oscar Sykes, Pip Arnott, Qi Guo, Ram Potham, Ram Rachum, Ruben Castaing, Samuel Prieto Lima, Shaheen Ahmed-Chowdhury, Tomas Turlik, Tyler Tracy.

Figure 1
Figure 1. Figure 1: General structure of a control evaluation in LinuxArena. The policy receives a main [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Main task success rate across frontier models evaluated on up to 400 valid main [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Main task success rate by task category for Claude Opus 4.6, Claude Haiku 4.5, [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of trajectory lengths by action count bucket, with success (green) and [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Side task success rate under the obvious-attack policy by category for Claude Opus [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Haiku 4.5 main task success (left) and obvious-attack side task success (right) under [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Sabotage results for 10 attackers at 1% step-wise FPR: 8 frontier non-GPT models [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Safety vs. step-wise false positive rate (FPR) for five monitor models against a base [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Safety vs. step-wise FPR for the same five monitors against Claude Opus 4.6 aug [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Safety at 1% step-wise FPR for the main monitor pool (the 11 monitors used [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Safety vs. step-wise FPR for five monitors against LaStraj (human-crafted) attack [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Evasion rates at 1% step-wise FPR for the main monitor pool (the 11 monitors used [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Static task count heatmap: number of sampled main tasks in the 400-task evaluation [PITH_FULL_IMAGE:figures/full_fig_p025_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Main task success rate for Claude Haiku 4.5 broken down by environment and task [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Environment description ablation for Claude Haiku 4.5: paired bars show success [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Haiku 4.5 Best-of-N curve on the 378 tasks where all five attempts were valid. The [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Safety curve comparing Haiku subtle attack (paper default: the attacker tries to disguise the side task) against Haiku obvious attack (the attacker is shown the side task and asked to do it directly). Both monitored by GPT-5 Nano with Haiku honest as the step-wise FPR calibration set. Shaded bands are 95% bootstrap CIs over trajectories. Obvious attack is easier to catch at every FPR, so the subtle attack… view at source ↗
Figure 18
Figure 18. Figure 18: Safety vs. step-wise FPR for three monitoring rules on a Haiku 4.5 attacker with a [PITH_FULL_IMAGE:figures/full_fig_p030_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Attacker side-task information ablation as safety vs. step-wise FPR for the Haiku [PITH_FULL_IMAGE:figures/full_fig_p031_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Monitor goal ablation as safety vs. step-wise FPR for the Haiku attacker (GPT-5 [PITH_FULL_IMAGE:figures/full_fig_p032_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Safety of GPT-5 Nano delayed monitor on Haiku attack at 1% trajectory-wise FPR [PITH_FULL_IMAGE:figures/full_fig_p033_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Number of side tasks per environment and category. Each cell shows how many side [PITH_FULL_IMAGE:figures/full_fig_p034_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Obvious-attack policy (Sonnet) side task success rate by environment and side task [PITH_FULL_IMAGE:figures/full_fig_p035_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Trajectory length distribution for the Sonnet obvious-attack run (185 trajectories). [PITH_FULL_IMAGE:figures/full_fig_p036_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Obvious-attack side task success rate per category across 8 non-GPT attacker models. [PITH_FULL_IMAGE:figures/full_fig_p036_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Obvious-attack side task success rate per environment across 8 non-GPT attacker [PITH_FULL_IMAGE:figures/full_fig_p037_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Attack policy side task success rate by attacker model and environment for the 8 [PITH_FULL_IMAGE:figures/full_fig_p037_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: Attack policy trajectory length distributions for the 8 non-GPT attacker models, [PITH_FULL_IMAGE:figures/full_fig_p038_28.png] view at source ↗
Figure 29
Figure 29. Figure 29: Sandboxing ablation as safety vs. step-wise FPR under the Haiku [PITH_FULL_IMAGE:figures/full_fig_p038_29.png] view at source ↗
Figure 30
Figure 30. Figure 30: Scratchpad ablation for the Opus attacker (GPT-5 Nano monitor) as safety vs. [PITH_FULL_IMAGE:figures/full_fig_p039_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: ST-test tool ablation: safety vs. step-wise FPR for the Haiku attacker (GPT-5 Nano [PITH_FULL_IMAGE:figures/full_fig_p047_31.png] view at source ↗
Figure 32
Figure 32. Figure 32: Opus attacker thinking-budget ablation as safety vs. step-wise FPR. Curves cover [PITH_FULL_IMAGE:figures/full_fig_p048_32.png] view at source ↗
Figure 33
Figure 33. Figure 33: Environment description attack ablation (Opus attack) as safety vs. step-wise FPR. [PITH_FULL_IMAGE:figures/full_fig_p049_33.png] view at source ↗
Figure 34
Figure 34. Figure 34: Per-attacker FPR calibration at 1% step-wise FPR, 8 frontier attackers monitored [PITH_FULL_IMAGE:figures/full_fig_p050_34.png] view at source ↗
Figure 35
Figure 35. Figure 35: Every available monitor evaluated against LaStraj (human sabotage trajectories) [PITH_FULL_IMAGE:figures/full_fig_p051_35.png] view at source ↗
read the original abstract

We introduce LinuxArena, a control setting in which agents operate directly on live, multi-service production environments. LinuxArena contains 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring, making it the largest and most diverse control setting for software engineering to date. We validate LinuxArena is useful for control research by running sabotage evaluations, which measure whether attackers can complete side tasks while working on main tasks, and monitor evaluations, which measure a monitor model's ability to detect sabotage attempts. Against a GPT-5-nano trusted monitor at a 1\% step-wise false positive rate, Claude Opus 4.6 achieves roughly a 23% undetected sabotage success rate. We additionally release LaStraj, a dataset of human-crafted attack trajectories that evade monitors at substantially higher rates than any model-generated attacks we elicited, showing that current attack policies do not saturate LinuxArena. These results suggest that LinuxArena has meaningful headroom for both attackers and defenders, making it a strong testbed for developing and evaluating future control protocols.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces LinuxArena, a control setting for AI agents operating in live, multi-service production environments. It comprises 20 environments, 1,671 main tasks representing legitimate software engineering work, and 184 side tasks representing safety failures such as data exfiltration and backdooring. Sabotage evaluations measure attackers' ability to complete side tasks undetected while performing main tasks, while monitor evaluations assess detection by a trusted model. Against a GPT-5-nano monitor at 1% step-wise false positive rate, Claude Opus 4.6 achieves roughly 23% undetected sabotage success rate. The paper also releases LaStraj, a dataset of human-crafted attack trajectories that evade monitors at higher rates than model-generated attacks, concluding that LinuxArena offers meaningful headroom for attackers and defenders as a testbed for control protocols.

Significance. If the side tasks and environments prove representative of real production conditions, LinuxArena would provide a valuable large-scale empirical testbed for AI control research in software engineering, with concrete metrics demonstrating current gaps in sabotage detection and the value of human attack data for stress-testing monitors.

major comments (2)
  1. [Abstract] Abstract: The central claim that LinuxArena has 'meaningful headroom for both attackers and defenders' and constitutes a 'strong testbed' for control protocols rests on the 184 side tasks and 20 environments being sufficiently representative of real-world safety failures and production software conditions. The manuscript provides no external anchoring such as expert ratings of task realism, comparisons to documented real-world incidents, or scaling checks on environment complexity and security posture to support generalization of the 23% undetected sabotage rate.
  2. [Abstract] Abstract (evaluation results): The headline 23% undetected sabotage success rate for Claude Opus 4.6 is presented without accompanying details on task selection criteria, statistical significance testing, error bars, or full methodology for the sabotage and monitor evaluations, which undermines assessment of whether the rate is robust or influenced by post-hoc choices.
minor comments (1)
  1. [Abstract] The abstract refers to 'step-wise false positive rate' without defining the term or explaining how it is computed across agent steps; this should be clarified in the main text.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract and the strength of our claims. We address each major comment point by point below and propose targeted revisions to improve transparency and context.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that LinuxArena has 'meaningful headroom for both attackers and defenders' and constitutes a 'strong testbed' for control protocols rests on the 184 side tasks and 20 environments being sufficiently representative of real-world safety failures and production software conditions. The manuscript provides no external anchoring such as expert ratings of task realism, comparisons to documented real-world incidents, or scaling checks on environment complexity and security posture to support generalization of the 23% undetected sabotage rate.

    Authors: We acknowledge that the manuscript does not include formal external validations such as expert ratings or direct mappings to specific real-world incidents. The environments and tasks were constructed to reflect typical production software engineering workflows and common safety failures, based on standard practices, common vulnerability patterns, and multi-service architectures drawn from industry norms. In the revision, we will expand the main text with a dedicated subsection on task and environment design principles, including concrete examples and rationale for selection criteria. We will also add an explicit limitations paragraph noting the absence of expert ratings or incident comparisons and explaining that such validations are resource-intensive and deferred to future work. This provides readers with better context for interpreting the 23% rate without overstating generalizability. revision: partial

  2. Referee: [Abstract] Abstract (evaluation results): The headline 23% undetected sabotage success rate for Claude Opus 4.6 is presented without accompanying details on task selection criteria, statistical significance testing, error bars, or full methodology for the sabotage and monitor evaluations, which undermines assessment of whether the rate is robust or influenced by post-hoc choices.

    Authors: The full methodology—including task selection (systematic pairing of the 184 side tasks with compatible main tasks from the 1,671 available), the step-wise monitoring protocol at a fixed 1% false positive rate, and all statistical procedures—is described in the Evaluations section, with error bars, confidence intervals, and significance testing reported in the main results figures and appendices. The abstract summarizes the primary finding concisely due to length constraints. In the revision, we will add one sentence to the abstract that briefly outlines the evaluation setup and explicitly directs readers to the Evaluations section and appendices for the complete methodology, task criteria, error bars, and robustness details. This improves accessibility without substantially lengthening the abstract. revision: partial

Circularity Check

0 steps flagged

No circularity: empirical benchmark with direct measurements

full rationale

The paper introduces LinuxArena as a new benchmark with 20 environments, 1,671 main tasks, and 184 side tasks, then reports direct experimental outcomes from sabotage and monitor evaluations (e.g., measured 23% undetected sabotage rate for Claude Opus 4.6 against a GPT-5-nano monitor at 1% FPR). No derivations, equations, fitted parameters renamed as predictions, uniqueness theorems, or self-citations that reduce claims to inputs by construction appear in the provided text. The results are self-contained empirical observations; generalization concerns are external validity issues, not internal circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that constructed tasks and environments capture meaningful control problems; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption The side tasks accurately represent real safety failures such as data exfiltration and backdooring in production software.
    Invoked to justify the relevance of sabotage evaluations to actual AI control risks.

pith-pipeline@v0.9.0 · 5631 in / 1209 out tokens · 52944 ms · 2026-05-10T11:37:34.870566+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 10 canonical work pages · 2 internal anchors

  1. [1]

    Three sketches of ASL-4 safety case components.https://alignment.anthropic

    Anthropic. Three sketches of ASL-4 safety case components.https://alignment.anthropic. com/2024/safety-cases/,

    2024

  2. [2]

    Sabotage evaluations for frontier models

    Joseph Benton et al. Sabotage evaluations for frontier models.arXiv preprint arXiv:2410.21514,

    work page arXiv

  3. [3]

    Ctrl-z: Controlling ai agents via resampling.arXiv preprint arXiv:2504.10374,

    Aryan Bhatt, Cody Rushing, Adam Kaufman, Tyler Tracy, Viktor Georgiev, David Matolcsi, Akbir Khan, and Buck Shlegeris. Ctrl-Z: Controlling AI agents via resampling.arXiv preprint arXiv:2504.10374,

    work page arXiv

  4. [4]

    Scheming ais: Will ais fake alignment during training in order to get power? arXiv preprint arXiv:2311.08379, 2023

    Joe Carlsmith. Scheming AIs: Will AIs fake alignment during training in order to get power? arXiv preprint arXiv:2311.08379,

    work page arXiv

  5. [5]

    Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte Sharkey, Daniel Kokotajlo, Chandler Bales, Oliver Habryka, Buck Shlegeris, Deep Ganguli, Evan Hub- inger, and Samuel R. Bowman. Alignment faking in large language models.arXiv preprint arXiv:2412.14093, 2024a. Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan, and Fabien Roger. AI control...

    work page internal anchor Pith review arXiv 2024

  6. [6]

    GitHub repository

    Adam Kaufman, Jaap Lucassen, Tyler Tracy, Cody Rushing, and Aryan Bhatt. BashArena: A control setting for highly privileged AI agents.arXiv preprint arXiv:2512.15688,

    work page arXiv

  7. [7]

    How to Evaluate Control Measures for LLM Agents? A Trajectory from Today to Superintelligence.arXiv preprint arXiv:2504.05259, 2025

    Tomasz Korbak et al. How to evaluate control measures for LLM agents?arXiv preprint arXiv:2504.05259,

    work page arXiv

  8. [8]

    Julius Kutasov, Yifan Sun, Paul Colognese, Teun van der Weij, Lorenzo Petrini, Christine B. C. Zhang, John Hughes, Xinpeng Deng, Hugo Sleight, Tyler Tracy, Buck Shlegeris, and Joseph Benton. SHADE-Arena: Evaluating sabotage and monitoring in LLM agents.arXiv preprint arXiv:2506.15740,

    work page arXiv

  9. [9]

    Subversion strategy eval: Evaluating AI’s stateless strategic capabilities against control protocols.arXiv preprint arXiv:2412.12480,

    Alex Mallen, Charlie Griffin, Alessandro Abate, and Buck Shlegeris. Subversion strategy eval: Evaluating AI’s stateless strategic capabilities against control protocols.arXiv preprint arXiv:2412.12480,

    work page arXiv

  10. [10]

    Async control: Stress-testing asynchronous control measures for LLM agents.arXiv preprint arXiv:2512.13526,

    Asa Cooper Stickland et al. Async control: Stress-testing asynchronous control measures for LLM agents.arXiv preprint arXiv:2512.13526,

    work page arXiv

  11. [11]

    $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045,

    work page internal anchor Pith review arXiv

  12. [12]

    Score value: C

    useN=1 (one attempt per task); this figure shows how success rate would change with more attempts. Error bars show 95% bootstrap CI over tasks. The curve uses the Chen et al. unbiased pass@k estimator. C Extra Monitoring Results This appendix covers three monitoring ablations: obvious vs. subtle attack, live vs. delayed vs. combined monitoring rules, and ...

    2000